New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Another Strange Out of memory using Bigarray #7671
Comments
Comment author: @alainfrisch Doing only the final allocation succeeds. That's still mysterious to me, since the Gc.compact() indeed releases memory, so one should start with a clean heap after it. Apparently fixed by #1476 , but I don't understand why. |
Comment author: @alainfrisch A possible explanation could be that the libc allocator doesn't reuse pages once reserved for mallocing "small blocks" to satisfy a malloc on a big enough block. Or perhaps some fragmentation of the "malloc heap". |
Comment author: @nojb I don't remember all the details but I remember seeing OOM due to fragmentation of the malloc heap in the past when allocating a lot of small bigarrays, using the cstruct library. See mirage/ocaml-cohttp#207 for related discussion. |
Comment author: @sbriais Indeed, this seems related to fragmentation. |
Comment author: nevor I have tracked down if any similar problem had been already reported on windows and it is indeed a "well" known problem. Alain is right, small chunks (smaller than 512k), are allocated in "heap segments" and bigger ones are allocated directly, heap segments are not destroyed when their content is freed. Here is a thread talking about that https://groups.google.com/forum/#!topic/microsoft.public.win32.programmer.kernel/5CiAyFAaIZE Values can be seen for Windows 10 page 5 of https://www.blackhat.com/docs/us-16/materials/us-16-Yason-Windows-10-Segment-Heap-Internals.pdf |
Comment author: @alainfrisch Ok, we are not going to provide a custom implementation of malloc. We should probably just document the problem, once we know the exact extent of it (Linux 32-bit with the glibc malloc, all Windows 32-bit ports, etc). |
Comment author: @xavierleroy I'm marking this as "no change required" (in the OCaml implementation). Issues like this could go into a FAQ but really we can't document all the problems with all the C libraries out there. |
Original bug ID: 7671
Reporter: @sbriais
Status: resolved (set by @xavierleroy on 2017-12-20T17:10:42Z)
Resolution: not a bug
Priority: normal
Severity: minor
Platform: Windows 32 bits
Category: runtime system and C interface
Related to: #7100
Bug description
Here is another problem involving big arrays.
The reproduction case does the following:
On trunk (c5fe693), using MSVC Windows 32 bits version of Ocaml, we get an out of memory at step 3.
Here is the sample program:
let pause() =
print_endline "Press a key.";
ignore (input_line stdin)
let main () =
print_endline "Pass 1";
for i = 0 to 100000 do
ignore Bigarray.(Array2.create float64 c_layout 1 10000)
done;
pause();
Gc.compact();
pause();
print_endline "Pass 2";
ignore Bigarray.(Array2.create float64 c_layout 5000 5000);
pause()
File attachments
The text was updated successfully, but these errors were encountered: