Mantis Bug Tracker
Main | My View | View Issues | Change Log | Roadmap
  

View Issue Details [ Jump to Notes ] [ Issue History ] [ Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0005813OCamlOCaml runtime systempublic2012-11-06 17:362013-06-07 16:10
Reporterwaern 
Assigned To 
PrioritynormalSeveritymajorReproducibilityalways
StatusresolvedResolutionfixed 
PlatformOSWindowsOS Version7
Product Version4.00.1 
Target Version4.01.0+devFixed in Version4.01.0+dev 
Summary0005813: Out of memory when using the Marshal module
DescriptionHi,

repeatedly unmarshalling large data structures with the Marshal module causes the process to run out of memory even in cases where constant memory usage is expected. This happens on the 32-bit version of OCaml on Windows.

Example code:

let () =
  let data = Array.make_matrix 10 (1024*1024) 0 in
  let data = Marshal.to_string data [] in
  for i = 0 to 100 do
    print_int i;
    print_newline ();
    let content = Marshal.from_string data 0 in
    ()
  done;

This loop dies at iteration 45 with "Fatal error: exception Out_of_memory".

What we think happens (after looking at the RTS code) is that the OCaml heap is extended each time "Marshal.from_string" is called. Eventually this causes the process to run out of memory and finally a call to C's "malloc" in the RTS fails.

Inserting a Gc.full_major in the loop fixes the problem, which is a work-around for now. However we consider this behaviour to be a bug in the Marshal module/RTS.

Shouldn't the RTS do a garbage collection when running out of memory or before unmarshalling large data structures?
TagsNo tags attached.
Attached Files

- Relationships

-  Notes
(0008432)
frisch (developer)
2012-11-06 17:42

 (Note: this is with the MSVC port, we did not try the mingw one.)
(0008692)
doligez (manager)
2013-01-04 15:13
edited on: 2013-01-04 15:14

 Reproduced with 4.01.0+dev on Mac OS X.

Note that the bug doesn't occur in bytecode. It also doesn't occur if you replace your print_int/print_newline with a printf. This is probably because in native code the loop doesn't allocate in the minor heap, so it doesn't check for the "urgent GC" condition.

The solution is probably to have Marshal.from_string check that condition before extending the heap.
(0008697)
frisch (developer)
2013-01-05 11:21

 We've seen the error in real code, and I doubt that that code does not allocate at all in the minor heap. Anyway, it sounds a good idea to check the "urgent GC" condition. (We will then test this fix on our code to see if it addresses our problem.)
(0009433)
xleroy (administrator)
2013-06-07 16:10

 Tentative fix on SVN trunk, commit r13755. Will be in 4.01. We just call caml_check_urgent_gc at the end of the unmarshaling functions. (The comment in intern_alloc() explains why it cannot be done just after caml_alloc_shr, as is usually done.) With this fix, the repro case runs in constant heap space.

- Issue History
Date Modified Username Field Change
2012-11-06 17:36 waern New Issue
2012-11-06 17:42 frisch Note Added: 0008432
2013-01-04 15:13 doligez Note Added: 0008692
2013-01-04 15:13 doligez Status new => acknowledged
2013-01-04 15:13 doligez Target Version => 4.01.0+dev
2013-01-04 15:14 doligez Note Edited: 0008692 View Revisions
2013-01-05 11:21 frisch Note Added: 0008697
2013-06-07 16:10 xleroy Note Added: 0009433
2013-06-07 16:10 xleroy Status acknowledged => resolved
2013-06-07 16:10 xleroy Resolution open => fixed
2013-06-07 16:10 xleroy Fixed in Version => 4.01.0+dev

Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker