Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0004616OCamlOCaml generalpublic2008-09-22 12:352014-04-23 17:39
Reportershinwell 
Assigned Todoligez 
PrioritynormalSeveritymajorReproducibilityalways
StatusclosedResolutionsuspended 
PlatformOSOS Version
Product Version3.10.2 
Target Version4.01.1+devFixed in Version 
Summary0004616: Repeated custom block allocations lead to space leaks
DescriptionThis one concerns native code compilation; I haven't checked if the behaviour differs for bytecode.

Allocation of a custom block with a finalizer, or of size > Max_young_wosize, increments a counter (caml_extra_heap_resources) according to the user-provided mem/max ratio. Then, caml_adjust_gc_speed is called, which causes a minor GC (including a major slice collection) to be effected at the next suitable time.

This is at odds with the manual, which as I read it claims something along the lines of a full major collection: "If you allocate many custom blocks with used / max = 1 / N, the GC will then do one full cycle (examining every object in the heap and calling finalization functions on those that are unreachable) every N allocations."

The collection that is performed when the ratio tips over the 1.0 boundary uses a complex heuristic to determine how much to collect (== the argument to caml_major_collection_slice). Suppose that heuristic seriously underestimates the amount that's been allocated recently. Then, the slice collection might fail to collect an amount commensurate with the amount allocated, and we can end up with the counter being reset to 0.0 and yet still have a load of garbage on the heap (with associated allocations on the C heap). If we keep on allocating the custom blocks, then the same happens over again, with memory usage in a sort of feedback cycle of allocations and collections. This can lead to bloated programs with oscillatory memory usage.

One way to easily trigger this behaviour is using a program that allocates and then lets go of a lot of Bigarray values. I have seen one circumstance where an executable making heavy use of Bigarrays was slowly growing up through the gigabytes of memory used. Unfortunately it had to be killed before it was obvious whether the growth was part of a stabilisation cycle; it's not clear whether this problem can lead to completely unbounded usage.

Here is a simple example which exhibits the problem. At first sight this program might appear to not use very much memory at all -- but on my machine it oscillates between 200Mb and 280Mb:

let create () = Bigarray.Array1.create Bigarray.int Bigarray.c_layout (32*1024)

let rec f _ =
  let x = create () in
  let y = create () in
  Bigarray.Array1.blit x y;
  f x

let () = f (create ())

Adding some full major collections dramatically reduces the memory usage. (Similarly, the program that was in the gigabytes sits below 10Mb with some regular forced full major collections.) I wonder if the heuristic needs adjusting somehow to cope with this pattern of allocation better?
TagsNo tags attached.
Attached Files

- Relationships

-  Notes
(0005470)
doligez (administrator)
2010-05-20 16:10

The simple example actually behaves as expected: the bigarray library declares that bigarrays are supposed to use 256M of memory, so oscillating between 200 and 280 is pretty good.

I'm still interested in a smallish example that exhibits the problem, especially if you also test it
with Gc.allocation_policy = 1 to avoid fragmentation.
(0009489)
xleroy (administrator)
2013-06-14 13:51

This PR has been in the "feedback" state for 3 years. I move to close it unless we gather new information very soon.
(0011315)
frisch (developer)
2014-04-23 17:39

As suggested by Xavier.

- Issue History
Date Modified Username Field Change
2008-09-22 12:35 shinwell New Issue
2008-09-23 02:08 doligez Status new => assigned
2008-09-23 02:08 doligez Assigned To => doligez
2010-05-20 16:10 doligez Note Added: 0005470
2010-05-20 16:11 doligez Status assigned => feedback
2012-07-06 16:37 doligez Target Version => 4.01.0+dev
2012-07-31 13:36 doligez Target Version 4.01.0+dev => 4.00.1+dev
2012-09-06 19:22 frisch Target Version 4.00.1+dev => 4.00.2+dev
2013-06-14 13:51 xleroy Note Added: 0009489
2013-07-09 17:01 doligez Target Version 4.00.2+dev => 4.01.0+dev
2013-07-29 15:28 doligez Target Version 4.01.0+dev => 4.01.1+dev
2014-04-23 17:39 frisch Note Added: 0011315
2014-04-23 17:39 frisch Status feedback => closed
2014-04-23 17:39 frisch Resolution open => suspended


Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker