Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0004868OCamlOCaml otherlibspublic2009-09-17 23:342012-06-30 16:26
Reporterelehack 
Assigned Todoligez 
PrioritynormalSeverityfeatureReproducibilityN/A
StatusassignedResolutionopen 
PlatformOSOS Version
Product Version3.11.1 
Target VersionFixed in Version 
Summary0004868: Request: make CAML_BA_MAX_MEMORY runtime-configurable
DescriptionCurrently, Bigarray uses a compiled-in constant CAML_BA_MAX_MEMORY, set to 256MiB, as the maximum size for allocating bigarray custom blocks. While find for low- to medium-memory machines, it causes excessive garbage collector thrashing when using large bigarrays on high-memory machines (e.g. arrays of several gigabytes). I propose the following changes:

 - change CAML_BA_MAX_MEMORY from a constant to a variable
 - provide a function to set this variable to a different value if it would help program performance

I have made this change on my local copy of OCaml, set the maximum memory to 12GiB, and the GC is substantially less active in the same code (which does not do heavy allocations, but primarily manipulates the bigarrays).
TagsNo tags attached.
Attached Files? file icon available_ram.sh [^] (594 bytes) 2012-06-30 16:26

- Relationships

-  Notes
(0007627)
gerd (reporter)
2012-06-30 16:25

Supporting this.

I'm also thinking about auto-configuring this. CAML_BA_MAX_MEMORY could be set to 3/4 of the available RAM, where the amount of RAM is calculated by a number of methods: (1) if there is a ulimit, take this (getrlimit for RLIMIT_DATA); (2) otherwise use an OS-specific method (see the attached script available_ram.sh to get an idea).

Also thinking that mmaped files (and other memory not allocated by Bigarray directly) should not count here at all. There could be another mechansim here, e.g. giving the GC a one-time boost when the user indicates that the file should be unmapped asap (or just auto-boost whenever a dead mapping is found, to get some automatic speed regulation). Rationale: memory-mapped files do not allocate memory, but only address space. You can nowadays map giant files, and it does not make any sense to include this space into OCaml's memory management calculations.

- Issue History
Date Modified Username Field Change
2009-09-17 23:34 elehack New Issue
2011-05-31 16:32 doligez Status new => assigned
2011-05-31 16:32 doligez Assigned To => doligez
2012-06-21 20:16 frisch Category OCaml general => OCaml otherlibs
2012-06-30 16:25 gerd Note Added: 0007627
2012-06-30 16:26 gerd File Added: available_ram.sh


Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker