|Anonymous | Login | Signup for a new account||2017-10-23 13:29 CEST|
|Main | My View | View Issues | Change Log | Roadmap|
|View Issue Details|
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0004616||OCaml||~DO NOT USE (was: OCaml general)||public||2008-09-22 12:35||2014-04-23 17:39|
|Target Version||4.01.1+dev||Fixed in Version|
|Summary||0004616: Repeated custom block allocations lead to space leaks|
|Description||This one concerns native code compilation; I haven't checked if the behaviour differs for bytecode.|
Allocation of a custom block with a finalizer, or of size > Max_young_wosize, increments a counter (caml_extra_heap_resources) according to the user-provided mem/max ratio. Then, caml_adjust_gc_speed is called, which causes a minor GC (including a major slice collection) to be effected at the next suitable time.
This is at odds with the manual, which as I read it claims something along the lines of a full major collection: "If you allocate many custom blocks with used / max = 1 / N, the GC will then do one full cycle (examining every object in the heap and calling finalization functions on those that are unreachable) every N allocations."
The collection that is performed when the ratio tips over the 1.0 boundary uses a complex heuristic to determine how much to collect (== the argument to caml_major_collection_slice). Suppose that heuristic seriously underestimates the amount that's been allocated recently. Then, the slice collection might fail to collect an amount commensurate with the amount allocated, and we can end up with the counter being reset to 0.0 and yet still have a load of garbage on the heap (with associated allocations on the C heap). If we keep on allocating the custom blocks, then the same happens over again, with memory usage in a sort of feedback cycle of allocations and collections. This can lead to bloated programs with oscillatory memory usage.
One way to easily trigger this behaviour is using a program that allocates and then lets go of a lot of Bigarray values. I have seen one circumstance where an executable making heavy use of Bigarrays was slowly growing up through the gigabytes of memory used. Unfortunately it had to be killed before it was obvious whether the growth was part of a stabilisation cycle; it's not clear whether this problem can lead to completely unbounded usage.
Here is a simple example which exhibits the problem. At first sight this program might appear to not use very much memory at all -- but on my machine it oscillates between 200Mb and 280Mb:
let create () = Bigarray.Array1.create Bigarray.int Bigarray.c_layout (32*1024)
let rec f _ =
let x = create () in
let y = create () in
Bigarray.Array1.blit x y;
let () = f (create ())
Adding some full major collections dramatically reduces the memory usage. (Similarly, the program that was in the gigabytes sits below 10Mb with some regular forced full major collections.) I wonder if the heuristic needs adjusting somehow to cope with this pattern of allocation better?
|Tags||No tags attached.|
The simple example actually behaves as expected: the bigarray library declares that bigarrays are supposed to use 256M of memory, so oscillating between 200 and 280 is pretty good.
I'm still interested in a smallish example that exhibits the problem, especially if you also test it
with Gc.allocation_policy = 1 to avoid fragmentation.
|This PR has been in the "feedback" state for 3 years. I move to close it unless we gather new information very soon.|
|As suggested by Xavier.|
|2008-09-22 12:35||shinwell||New Issue|
|2008-09-23 02:08||doligez||Status||new => assigned|
|2008-09-23 02:08||doligez||Assigned To||=> doligez|
|2010-05-20 16:10||doligez||Note Added: 0005470|
|2010-05-20 16:11||doligez||Status||assigned => feedback|
|2012-07-06 16:37||doligez||Target Version||=> 4.01.0+dev|
|2012-07-31 13:36||doligez||Target Version||4.01.0+dev => 4.00.1+dev|
|2012-09-06 19:22||frisch||Target Version||4.00.1+dev => 4.00.2+dev|
|2013-06-14 13:51||xleroy||Note Added: 0009489|
|2013-07-09 17:01||doligez||Target Version||4.00.2+dev => 4.01.0+dev|
|2013-07-29 15:28||doligez||Target Version||4.01.0+dev => 4.01.1+dev|
|2014-04-23 17:39||frisch||Note Added: 0011315|
|2014-04-23 17:39||frisch||Status||feedback => closed|
|2014-04-23 17:39||frisch||Resolution||open => suspended|
|2017-02-23 16:36||doligez||Category||OCaml general => -OCaml general|
|2017-03-03 17:55||doligez||Category||-OCaml general => -(deprecated) general|
|2017-03-03 18:01||doligez||Category||-(deprecated) general => ~deprecated (was: OCaml general)|
|2017-03-06 17:04||doligez||Category||~deprecated (was: OCaml general) => ~DO NOT USE (was: OCaml general)|
|Copyright © 2000 - 2011 MantisBT Group|