0005813: Out of memory when using the Marshal module

View Issue Details [ Jump to Notes ]

[ Issue History ] [ Print ]

Project

Category

View Status

Date Submitted

Last Update

0005813

OCaml

OCaml runtime system

public

2012-11-06 17:36

2013-01-05 11:21

Reporter

waern

Assigned To

Priority

normal

Severity

major

Reproducibility

always

Status

acknowledged

Resolution

open

Platform

Windows

OS Version

Product Version

4.00.1

Target Version

4.01.0+dev

Fixed in Version

Summary

0005813: Out of memory when using the Marshal module

Description

Hi,

repeatedly unmarshalling large data structures with the Marshal module causes the process to run out of memory even in cases where constant memory usage is expected. This happens on the 32-bit version of OCaml on Windows.

Example code:

let () =
  let data = Array.make_matrix 10 (1024*1024) 0 in
  let data = Marshal.to_string data [] in
  for i = 0 to 100 do
    print_int i;
    print_newline ();
    let content = Marshal.from_string data 0 in
    ()
  done;

This loop dies at iteration 45 with "Fatal error: exception Out_of_memory".

What we think happens (after looking at the RTS code) is that the OCaml heap is extended each time "Marshal.from_string" is called. Eventually this causes the process to run out of memory and finally a call to C's "malloc" in the RTS fails.

Inserting a Gc.full_major in the loop fixes the problem, which is a work-around for now. However we consider this behaviour to be a bug in the Marshal module/RTS.

Shouldn't the RTS do a garbage collection when running out of memory or before unmarshalling large data structures?

Tags

No tags attached.

Attached Files

Relationships

Notes
(0008432) frisch (developer) 2012-11-06 17:42	(Note: this is with the MSVC port, we did not try the mingw one.)

(0008692) doligez (manager) 2013-01-04 15:13 edited on: 2013-01-04 15:14	Reproduced with 4.01.0+dev on Mac OS X. Note that the bug doesn't occur in bytecode. It also doesn't occur if you replace your print_int/print_newline with a printf. This is probably because in native code the loop doesn't allocate in the minor heap, so it doesn't check for the "urgent GC" condition. The solution is probably to have Marshal.from_string check that condition before extending the heap.

(0008697) frisch (developer) 2013-01-05 11:21	We've seen the error in real code, and I doubt that that code does not allocate at all in the minor heap. Anyway, it sounds a good idea to check the "urgent GC" condition. (We will then test this fix on our code to see if it addresses our problem.)

Issue History
Date Modified	Username	Field	Change
2012-11-06 17:36	waern	New Issue
2012-11-06 17:42	frisch	Note Added: 0008432
2013-01-04 15:13	doligez	Note Added: 0008692
2013-01-04 15:13	doligez	Status	new => acknowledged
2013-01-04 15:13	doligez	Target Version	=> 4.01.0+dev
2013-01-04 15:14	doligez	Note Edited: 0008692	View Revisions
2013-01-05 11:21	frisch	Note Added: 0008697