You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Original bug ID: 4616 Reporter:@mshinwell Assigned to:@damiendoligez Status: closed (set by @alainfrisch on 2014-04-23T15:39:04Z) Resolution: suspended Priority: normal Severity: major Version: 3.10.2 Target version: 4.01.1+dev Category: ~DO NOT USE (was: OCaml general) Monitored by:@ygrek dpowers sweeks till letaris bhurt-aw @dbuenzli@Chris00@mmottl
Bug description
This one concerns native code compilation; I haven't checked if the behaviour differs for bytecode.
Allocation of a custom block with a finalizer, or of size > Max_young_wosize, increments a counter (caml_extra_heap_resources) according to the user-provided mem/max ratio. Then, caml_adjust_gc_speed is called, which causes a minor GC (including a major slice collection) to be effected at the next suitable time.
This is at odds with the manual, which as I read it claims something along the lines of a full major collection: "If you allocate many custom blocks with used / max = 1 / N, the GC will then do one full cycle (examining every object in the heap and calling finalization functions on those that are unreachable) every N allocations."
The collection that is performed when the ratio tips over the 1.0 boundary uses a complex heuristic to determine how much to collect (== the argument to caml_major_collection_slice). Suppose that heuristic seriously underestimates the amount that's been allocated recently. Then, the slice collection might fail to collect an amount commensurate with the amount allocated, and we can end up with the counter being reset to 0.0 and yet still have a load of garbage on the heap (with associated allocations on the C heap). If we keep on allocating the custom blocks, then the same happens over again, with memory usage in a sort of feedback cycle of allocations and collections. This can lead to bloated programs with oscillatory memory usage.
One way to easily trigger this behaviour is using a program that allocates and then lets go of a lot of Bigarray values. I have seen one circumstance where an executable making heavy use of Bigarrays was slowly growing up through the gigabytes of memory used. Unfortunately it had to be killed before it was obvious whether the growth was part of a stabilisation cycle; it's not clear whether this problem can lead to completely unbounded usage.
Here is a simple example which exhibits the problem. At first sight this program might appear to not use very much memory at all -- but on my machine it oscillates between 200Mb and 280Mb:
let create () = Bigarray.Array1.create Bigarray.int Bigarray.c_layout (32*1024)
let rec f _ =
let x = create () in
let y = create () in
Bigarray.Array1.blit x y;
f x
let () = f (create ())
Adding some full major collections dramatically reduces the memory usage. (Similarly, the program that was in the gigabytes sits below 10Mb with some regular forced full major collections.) I wonder if the heuristic needs adjusting somehow to cope with this pattern of allocation better?
The text was updated successfully, but these errors were encountered:
The simple example actually behaves as expected: the bigarray library declares that bigarrays are supposed to use 256M of memory, so oscillating between 200 and 280 is pretty good.
I'm still interested in a smallish example that exhibits the problem, especially if you also test it
with Gc.allocation_policy = 1 to avoid fragmentation.
Original bug ID: 4616
Reporter: @mshinwell
Assigned to: @damiendoligez
Status: closed (set by @alainfrisch on 2014-04-23T15:39:04Z)
Resolution: suspended
Priority: normal
Severity: major
Version: 3.10.2
Target version: 4.01.1+dev
Category: ~DO NOT USE (was: OCaml general)
Monitored by: @ygrek dpowers sweeks till letaris bhurt-aw @dbuenzli @Chris00 @mmottl
Bug description
This one concerns native code compilation; I haven't checked if the behaviour differs for bytecode.
Allocation of a custom block with a finalizer, or of size > Max_young_wosize, increments a counter (caml_extra_heap_resources) according to the user-provided mem/max ratio. Then, caml_adjust_gc_speed is called, which causes a minor GC (including a major slice collection) to be effected at the next suitable time.
This is at odds with the manual, which as I read it claims something along the lines of a full major collection: "If you allocate many custom blocks with used / max = 1 / N, the GC will then do one full cycle (examining every object in the heap and calling finalization functions on those that are unreachable) every N allocations."
The collection that is performed when the ratio tips over the 1.0 boundary uses a complex heuristic to determine how much to collect (== the argument to caml_major_collection_slice). Suppose that heuristic seriously underestimates the amount that's been allocated recently. Then, the slice collection might fail to collect an amount commensurate with the amount allocated, and we can end up with the counter being reset to 0.0 and yet still have a load of garbage on the heap (with associated allocations on the C heap). If we keep on allocating the custom blocks, then the same happens over again, with memory usage in a sort of feedback cycle of allocations and collections. This can lead to bloated programs with oscillatory memory usage.
One way to easily trigger this behaviour is using a program that allocates and then lets go of a lot of Bigarray values. I have seen one circumstance where an executable making heavy use of Bigarrays was slowly growing up through the gigabytes of memory used. Unfortunately it had to be killed before it was obvious whether the growth was part of a stabilisation cycle; it's not clear whether this problem can lead to completely unbounded usage.
Here is a simple example which exhibits the problem. At first sight this program might appear to not use very much memory at all -- but on my machine it oscillates between 200Mb and 280Mb:
let create () = Bigarray.Array1.create Bigarray.int Bigarray.c_layout (32*1024)
let rec f _ =
let x = create () in
let y = create () in
Bigarray.Array1.blit x y;
f x
let () = f (create ())
Adding some full major collections dramatically reduces the memory usage. (Similarly, the program that was in the gigabytes sits below 10Mb with some regular forced full major collections.) I wonder if the heuristic needs adjusting somehow to cope with this pattern of allocation better?
The text was updated successfully, but these errors were encountered: