|Anonymous | Login | Signup for a new account||2018-01-21 13:39 CET|
|Main | My View | View Issues | Change Log | Roadmap|
|View Issue Details|
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0007671||OCaml||runtime system and C interface||public||2017-11-13 14:55||2017-12-20 18:10|
|Priority||normal||Severity||minor||Reproducibility||have not tried|
|Status||resolved||Resolution||no change required|
|Platform||Windows 32 bits||OS||OS Version|
|Target Version||Fixed in Version|
|Summary||0007671: Another Strange Out of memory using Bigarray|
|Description||Here is another problem involving big arrays. |
The reproduction case does the following:
1. allocate a bunch of matrices of floats of small size (1 x 10000).
3. allocate a medium sized matrice of floats (5000 x 5000)
On trunk (c5fe6932), using MSVC Windows 32 bits version of Ocaml, we get an out of memory at step 3.
Here is the sample program:
let pause() =
print_endline "Press a key.";
ignore (input_line stdin)
let main () =
print_endline "Pass 1";
for i = 0 to 100000 do
ignore Bigarray.(Array2.create float64 c_layout 1 10000)
print_endline "Pass 2";
ignore Bigarray.(Array2.create float64 c_layout 5000 5000);
|Tags||No tags attached.|
|Attached Files||oom.c [^] (700 bytes) 2017-11-14 16:29 [Show Content]|
edited on: 2017-11-13 16:01
Doing only the final allocation succeeds.
That's still mysterious to me, since the Gc.compact() indeed releases memory, so one should start with a clean heap after it.
Apparently fixed by https://github.com/ocaml/ocaml/pull/1476 [^] , but I don't understand why.
|A possible explanation could be that the libc allocator doesn't reuse pages once reserved for mallocing "small blocks" to satisfy a malloc on a big enough block. Or perhaps some fragmentation of the "malloc heap".|
|I don't remember all the details but I remember seeing OOM due to fragmentation of the malloc heap in the past when allocating a lot of small bigarrays, using the cstruct library. See https://github.com/mirage/ocaml-cohttp/issues/207 [^] for related discussion.|
Indeed, this seems related to fragmentation.
I attach a small example of C program that reproduces the problem.
edited on: 2017-11-14 17:54
I have tracked down if any similar problem had been already reported on windows and it is indeed a "well" known problem.
Alain is right, small chunks (smaller than 512k), are allocated in "heap segments" and bigger ones are allocated directly, heap segments are not destroyed when their content is freed.
Here is a thread talking about that https://groups.google.com/forum/#!topic/microsoft.public.win32.programmer.kernel/5CiAyFAaIZE [^]
Values can be seen for Windows 10 page 5 of https://www.blackhat.com/docs/us-16/materials/us-16-Yason-Windows-10-Segment-Heap-Internals.pdf [^]
Ok, we are not going to provide a custom implementation of malloc.
We should probably just document the problem, once we know the exact extent of it (Linux 32-bit with the glibc malloc, all Windows 32-bit ports, etc).
|I'm marking this as "no change required" (in the OCaml implementation). Issues like this could go into a FAQ but really we can't document all the problems with all the C libraries out there.|
|2017-11-13 14:55||sbriais||New Issue|
|2017-11-13 16:01||frisch||Note Added: 0018657|
|2017-11-13 16:01||frisch||Note Edited: 0018657||View Revisions|
|2017-11-13 16:11||frisch||Note Added: 0018658|
|2017-11-13 16:15||frisch||Relationship added||related to 0007100|
|2017-11-13 22:47||nojebar||Note Added: 0018660|
|2017-11-14 16:28||sbriais||Note Added: 0018662|
|2017-11-14 16:29||sbriais||File Added: oom.c|
|2017-11-14 17:28||nevor||Note Added: 0018663|
|2017-11-14 17:54||nevor||Note Edited: 0018663||View Revisions|
|2017-11-20 11:01||frisch||Note Added: 0018674|
|2017-12-20 18:10||xleroy||Note Added: 0018762|
|2017-12-20 18:10||xleroy||Status||new => resolved|
|2017-12-20 18:10||xleroy||Resolution||open => no change required|
|Copyright © 2000 - 2011 MantisBT Group|