|Anonymous | Login | Signup for a new account||2013-12-05 21:28 CET|
|Main | My View | View Issues | Change Log | Roadmap|
|View Issue Details|
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0005813||OCaml||OCaml runtime system||public||2012-11-06 17:36||2013-06-07 16:10|
|Target Version||4.01.0+dev||Fixed in Version||4.01.0+dev|
|Summary||0005813: Out of memory when using the Marshal module|
repeatedly unmarshalling large data structures with the Marshal module causes the process to run out of memory even in cases where constant memory usage is expected. This happens on the 32-bit version of OCaml on Windows.
let () =
let data = Array.make_matrix 10 (1024*1024) 0 in
let data = Marshal.to_string data  in
for i = 0 to 100 do
let content = Marshal.from_string data 0 in
This loop dies at iteration 45 with "Fatal error: exception Out_of_memory".
What we think happens (after looking at the RTS code) is that the OCaml heap is extended each time "Marshal.from_string" is called. Eventually this causes the process to run out of memory and finally a call to C's "malloc" in the RTS fails.
Inserting a Gc.full_major in the loop fixes the problem, which is a work-around for now. However we consider this behaviour to be a bug in the Marshal module/RTS.
Shouldn't the RTS do a garbage collection when running out of memory or before unmarshalling large data structures?
|Tags||No tags attached.|
|(Note: this is with the MSVC port, we did not try the mingw one.)|
edited on: 2013-01-04 15:14
Reproduced with 4.01.0+dev on Mac OS X.
Note that the bug doesn't occur in bytecode. It also doesn't occur if you replace your print_int/print_newline with a printf. This is probably because in native code the loop doesn't allocate in the minor heap, so it doesn't check for the "urgent GC" condition.
The solution is probably to have Marshal.from_string check that condition before extending the heap.
|We've seen the error in real code, and I doubt that that code does not allocate at all in the minor heap. Anyway, it sounds a good idea to check the "urgent GC" condition. (We will then test this fix on our code to see if it addresses our problem.)|
Tentative fix on SVN trunk, commit r13755. Will be in 4.01. We just call caml_check_urgent_gc at the end of the unmarshaling functions. (The comment in intern_alloc() explains why it cannot be done just after caml_alloc_shr, as is usually done.) With this fix, the repro case runs in constant heap space.
|2012-11-06 17:36||waern||New Issue|
|2012-11-06 17:42||frisch||Note Added: 0008432|
|2013-01-04 15:13||doligez||Note Added: 0008692|
|2013-01-04 15:13||doligez||Status||new => acknowledged|
|2013-01-04 15:13||doligez||Target Version||=> 4.01.0+dev|
|2013-01-04 15:14||doligez||Note Edited: 0008692||View Revisions|
|2013-01-05 11:21||frisch||Note Added: 0008697|
|2013-06-07 16:10||xleroy||Note Added: 0009433|
|2013-06-07 16:10||xleroy||Status||acknowledged => resolved|
|2013-06-07 16:10||xleroy||Resolution||open => fixed|
|2013-06-07 16:10||xleroy||Fixed in Version||=> 4.01.0+dev|
|Copyright © 2000 - 2011 MantisBT Group|