Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0007671OCamlruntime system and C interfacepublic2017-11-13 14:552017-11-14 17:28
Reportersbriais 
Assigned To 
PrioritynormalSeverityminorReproducibilityhave not tried
StatusnewResolutionopen 
PlatformWindows 32 bitsOSOS Version
Product Version 
Target VersionFixed in Version 
Summary0007671: Another Strange Out of memory using Bigarray
DescriptionHere is another problem involving big arrays.

The reproduction case does the following:
1. allocate a bunch of matrices of floats of small size (1 x 10000).
2. Gc.compact()
3. allocate a medium sized matrice of floats (5000 x 5000)

On trunk (c5fe6932), using MSVC Windows 32 bits version of Ocaml, we get an out of memory at step 3.

Here is the sample program:

let pause() =
  print_endline "Press a key.";
  ignore (input_line stdin)

let main () =
  print_endline "Pass 1";
  for i = 0 to 100000 do
    ignore Bigarray.(Array2.create float64 c_layout 1 10000)
  done;
  pause();
  Gc.compact();
  pause();
  print_endline "Pass 2";
  ignore Bigarray.(Array2.create float64 c_layout 5000 5000);
  pause()
TagsNo tags attached.
Attached Filesc file icon oom.c [^] (700 bytes) 2017-11-14 16:29 [Show Content]

- Relationships
related to 0007100assigneddoligez Bigarray's caml_ba_alloc doesn't try GC if malloc fails 

-  Notes
(0018657)
frisch (developer)
2017-11-13 16:01
edited on: 2017-11-13 16:01

Doing only the final allocation succeeds.

That's still mysterious to me, since the Gc.compact() indeed releases memory, so one should start with a clean heap after it.

Apparently fixed by https://github.com/ocaml/ocaml/pull/1476 [^] , but I don't understand why.

(0018658)
frisch (developer)
2017-11-13 16:11

A possible explanation could be that the libc allocator doesn't reuse pages once reserved for mallocing "small blocks" to satisfy a malloc on a big enough block. Or perhaps some fragmentation of the "malloc heap".
(0018660)
nojebar (reporter)
2017-11-13 22:47

I don't remember all the details but I remember seeing OOM due to fragmentation of the malloc heap in the past when allocating a lot of small bigarrays, using the cstruct library. See https://github.com/mirage/ocaml-cohttp/issues/207 [^] for related discussion.
(0018662)
sbriais (reporter)
2017-11-14 16:28

Indeed, this seems related to fragmentation.
I attach a small example of C program that reproduces the problem.
(0018663)
nevor (reporter)
2017-11-14 17:28
edited on: 2017-11-14 17:54

I have tracked down if any similar problem had been already reported on windows and it is indeed a "well" known problem.

Alain is right, small chunks (smaller than 512k), are allocated in "heap segments" and bigger ones are allocated directly, heap segments are not destroyed when their content is freed.

Here is a thread talking about that https://groups.google.com/forum/#!topic/microsoft.public.win32.programmer.kernel/5CiAyFAaIZE [^]

Values can be seen for Windows 10 page 5 of https://www.blackhat.com/docs/us-16/materials/us-16-Yason-Windows-10-Segment-Heap-Internals.pdf [^]


- Issue History
Date Modified Username Field Change
2017-11-13 14:55 sbriais New Issue
2017-11-13 16:01 frisch Note Added: 0018657
2017-11-13 16:01 frisch Note Edited: 0018657 View Revisions
2017-11-13 16:11 frisch Note Added: 0018658
2017-11-13 16:15 frisch Relationship added related to 0007100
2017-11-13 22:47 nojebar Note Added: 0018660
2017-11-14 16:28 sbriais Note Added: 0018662
2017-11-14 16:29 sbriais File Added: oom.c
2017-11-14 17:28 nevor Note Added: 0018663
2017-11-14 17:54 nevor Note Edited: 0018663 View Revisions


Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker