Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gc.compact vs. Gc.max_overhead #3305

Closed
vicuna opened this issue Apr 16, 2002 · 4 comments
Closed

Gc.compact vs. Gc.max_overhead #3305

vicuna opened this issue Apr 16, 2002 · 4 comments
Labels

Comments

@vicuna
Copy link

vicuna commented Apr 16, 2002

Original bug ID: 1111
Reporter: administrator
Status: closed
Resolution: fixed
Priority: normal
Severity: minor
Category: ~DO NOT USE (was: OCaml general)

Bug description

Full_Name: Martin Jambon
Version: 3.04
OS: i686/linux
Submission from: pc-bioinfo1.ibcp.fr (193.51.160.63)

Setting Gc.max_overhead to 50 triggers heap compactions as reported by
Gc.print_stat.

In the program I'm developing, this works fine until a certain point where no
compaction is triggered anymore, resulting in memory increase.

Periodic calls to Gc.compact fix the problem, i.e. reduce the memory amount to a
normal size (here, 10 MB).

The program makes a lot of memory allocations and deallocations.
The program has the following structure:
for each file, do
input data from file
extract some interesting information from the data
done

Example of GC statistics:

Gc.print_stat output after the last automatic heap compaction:
(somewhere in the middle of the main loop, ~15000 files have been read,
representing a total amount of ~500 MB of disk space in the form of Marshal'ed
data)

minor_words: 22845867
promoted_words: 1578695
major_words: 104923528
minor_collections: 213
major_collections: 12697
heap_words: 9793536 (* 10 MB *)
heap_chunks: 34
live_words: 1703034
live_blocks: 360724
free_words: 745339
free_blocks: 8
largest_free: 256000
fragments: 11
compactions: 67

Gc.print_stat output at the end of the program:
(just before the end of the main loop, 35000 files have been read)

minor_words: 22845867
promoted_words: 1930136
major_words: 227605106
minor_collections: 364
major_collections: 53380
heap_words: 193458176 (* 190 MB *)
heap_chunks: 728
live_words: 4020819
live_blocks: 763264
free_words: 44343702
free_blocks: 81982
largest_free: 36226
fragments: 23
compactions: 67

After the documentation found in module Gc (see below), this looks like a bug.
If not, maybe a warning message should be added.

mutable max_overhead : int;
(** Heap compaction is triggered when the estimated amount
   of free memory is more than [max_overhead] percent of the amount
   of live data.  If [max_overhead] is set to 0, heap
   compaction is triggered at the end of each major GC cycle
   (this setting is intended for testing purposes only).
   If [max_overhead >= 1000000], compaction is never triggered.
   Default: 1000000. *)
@vicuna
Copy link
Author

vicuna commented Apr 30, 2002

Comment author: administrator

Setting Gc.max_overhead to 50 triggers heap compactions as reported by
Gc.print_stat.

In the program I'm developing, this works fine until a certain point where no
compaction is triggered anymore, resulting in memory increase.

Periodic calls to Gc.compact fix the problem, i.e. reduce the memory amount to
a
normal size (here, 10 MB).

Yes, there is a problem with the automatic triggering of compaction,
I managed to reproduce it with a very simple program (included below
for reference).

The bad news is, I don't really have a good idea of what's going on,
so I can't tell when the problem will be fixed...

-- Damien


(* fails *)
Gc.set { (Gc.get ()) with Gc.verbose = 0x04; Gc.max_overhead = 57 };;

(* works *)
(Gc.set { (Gc.get ()) with Gc.verbose = 0x04; Gc.max_overhead = 56 };;)

let x = ref "";;
while true do
x := !x ^ (String.create 2048);
done;;

@vicuna
Copy link
Author

vicuna commented Apr 30, 2002

Comment author: administrator

On Tue, 30 Apr 2002, Damien Doligez wrote:

Yes, there is a problem with the automatic triggering of compaction,
I managed to reproduce it with a very simple program (included below
for reference).


(* fails *)
Gc.set { (Gc.get ()) with Gc.verbose = 0x04; Gc.max_overhead = 57 };;

(* works *)
(Gc.set { (Gc.get ()) with Gc.verbose = 0x04; Gc.max_overhead = 56 };;)

let x = ref "";;
while true do
x := !x ^ (String.create 2048);
done;;

Effectivement, c'est troublant.

J'avais un exemple (dans un programme compliqué) où l'insertion de la
séquence (Hashtbl.add tbl key data; Hashtbl.remove tbl key) était
suffisante pour faire apparaître le problème.
[je ne sais pas si ça peut aider, mais sait-on jamais...]

Martin

@vicuna
Copy link
Author

vicuna commented May 28, 2002

Comment author: administrator

In the program I'm developing, this works fine until a certain point where no
compaction is triggered anymore, resulting in memory increase.

Je pense que j'ai reussi a faire un truc qui marche mieux pour declencher
les compactages. Si vous voulez l'essayer, il faut prendre la version
CVS (3.04+12). Ou alors attendre la 3.05. Ou bien je peux faire un patch,
mais c'est du boulot...

-- Damien

@vicuna vicuna closed this as completed May 28, 2002
@vicuna
Copy link
Author

vicuna commented May 28, 2002

Comment author: administrator

At 19:06 28/05/02 +0200, you wrote:

In the program I'm developing, this works fine until a certain point
where no
compaction is triggered anymore, resulting in memory increase.

Je pense que j'ai reussi a faire un truc qui marche mieux pour declencher
les compactages. Si vous voulez l'essayer, il faut prendre la version
CVS (3.04+12). Ou alors attendre la 3.05. Ou bien je peux faire un patch,
mais c'est du boulot...

-- Damien

Merci beaucoup. J'essaie d'essayer ça dès que j'ai un peu de temps et je
vous tiens au courant.

Martin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant