0004616: Repeated custom block allocations lead to space leaks

View Issue Details [ Jump to Notes ]

[ Issue History ] [ Print ]

Project

Category

View Status

Date Submitted

Last Update

0004616

OCaml

OCaml general

public

2008-09-22 12:35

2012-09-06 19:22

Reporter

shinwell

Assigned To

doligez

Priority

normal

Severity

major

Reproducibility

always

Status

feedback

Resolution

open

Platform

OS Version

Product Version

3.10.2

Target Version

4.00.2+dev

Fixed in Version

Summary

0004616: Repeated custom block allocations lead to space leaks

Description

This one concerns native code compilation; I haven't checked if the behaviour differs for bytecode.

Allocation of a custom block with a finalizer, or of size > Max_young_wosize, increments a counter (caml_extra_heap_resources) according to the user-provided mem/max ratio. Then, caml_adjust_gc_speed is called, which causes a minor GC (including a major slice collection) to be effected at the next suitable time.

This is at odds with the manual, which as I read it claims something along the lines of a full major collection: "If you allocate many custom blocks with used / max = 1 / N, the GC will then do one full cycle (examining every object in the heap and calling finalization functions on those that are unreachable) every N allocations."

The collection that is performed when the ratio tips over the 1.0 boundary uses a complex heuristic to determine how much to collect (== the argument to caml_major_collection_slice). Suppose that heuristic seriously underestimates the amount that's been allocated recently. Then, the slice collection might fail to collect an amount commensurate with the amount allocated, and we can end up with the counter being reset to 0.0 and yet still have a load of garbage on the heap (with associated allocations on the C heap). If we keep on allocating the custom blocks, then the same happens over again, with memory usage in a sort of feedback cycle of allocations and collections. This can lead to bloated programs with oscillatory memory usage.

One way to easily trigger this behaviour is using a program that allocates and then lets go of a lot of Bigarray values. I have seen one circumstance where an executable making heavy use of Bigarrays was slowly growing up through the gigabytes of memory used. Unfortunately it had to be killed before it was obvious whether the growth was part of a stabilisation cycle; it's not clear whether this problem can lead to completely unbounded usage.

Here is a simple example which exhibits the problem. At first sight this program might appear to not use very much memory at all -- but on my machine it oscillates between 200Mb and 280Mb:

let create () = Bigarray.Array1.create Bigarray.int Bigarray.c_layout (32*1024)

let rec f _ =
  let x = create () in
  let y = create () in
  Bigarray.Array1.blit x y;
  f x

let () = f (create ())

Adding some full major collections dramatically reduces the memory usage. (Similarly, the program that was in the gigabytes sits below 10Mb with some regular forced full major collections.) I wonder if the heuristic needs adjusting somehow to cope with this pattern of allocation better?

Tags

No tags attached.

Attached Files

Relationships

Notes
(0005470) doligez (manager) 2010-05-20 16:10	The simple example actually behaves as expected: the bigarray library declares that bigarrays are supposed to use 256M of memory, so oscillating between 200 and 280 is pretty good. I'm still interested in a smallish example that exhibits the problem, especially if you also test it with Gc.allocation_policy = 1 to avoid fragmentation.

Issue History
Date Modified	Username	Field	Change
2008-09-22 12:35	shinwell	New Issue
2008-09-23 02:08	doligez	Status	new => assigned
2008-09-23 02:08	doligez	Assigned To	=> doligez
2010-05-20 16:10	doligez	Note Added: 0005470
2010-05-20 16:11	doligez	Status	assigned => feedback
2012-07-06 16:37	doligez	Target Version	=> 4.01.0+dev
2012-07-31 13:36	doligez	Target Version	4.01.0+dev => 4.00.1+dev
2012-09-06 19:22	frisch	Target Version	4.00.1+dev => 4.00.2+dev