Comment author: @alainfrisch

Doing only the final allocation succeeds.

That's still mysterious to me, since the Gc.compact() indeed releases memory, so one should start with a clean heap after it.

Apparently fixed by #1476 , but I don't understand why.

Comment author: @alainfrisch

A possible explanation could be that the libc allocator doesn't reuse pages once reserved for mallocing "small blocks" to satisfy a malloc on a big enough block. Or perhaps some fragmentation of the "malloc heap".

Comment author: @nojb

I don't remember all the details but I remember seeing OOM due to fragmentation of the malloc heap in the past when allocating a lot of small bigarrays, using the cstruct library. See mirage/ocaml-cohttp#207 for related discussion.

Comment author: @sbriais

Indeed, this seems related to fragmentation.
I attach a small example of C program that reproduces the problem.

Comment author: nevor

I have tracked down if any similar problem had been already reported on windows and it is indeed a "well" known problem.

Alain is right, small chunks (smaller than 512k), are allocated in "heap segments" and bigger ones are allocated directly, heap segments are not destroyed when their content is freed.

Here is a thread talking about that https://groups.google.com/forum/#!topic/microsoft.public.win32.programmer.kernel/5CiAyFAaIZE

Values can be seen for Windows 10 page 5 of https://www.blackhat.com/docs/us-16/materials/us-16-Yason-Windows-10-Segment-Heap-Internals.pdf

Comment author: @alainfrisch

Ok, we are not going to provide a custom implementation of malloc.

We should probably just document the problem, once we know the exact extent of it (Linux 32-bit with the glibc malloc, all Windows 32-bit ports, etc).

Comment author: @xavierleroy

I'm marking this as "no change required" (in the OCaml implementation). Issues like this could go into a FAQ but really we can't document all the problems with all the C libraries out there.