Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0007198OCamlruntime system and C interfacepublic2016-03-27 17:172017-11-15 17:23
Reportermfp 
Assigned Todoligez 
PrioritynormalSeverityminorReproducibilitysometimes
StatusassignedResolutionopen 
PlatformOSOS Version
Product Version4.02.3 
Target Version4.07.0+devFixed in Version 
Summary0007198: caml_alloc_custom/caml_alloc_final API easily leads to GC performance issues
DescriptionI've found a number of instances of the same pattern: a library allocates custom blocks with caml_alloc_custom/caml_alloc_final and hardcodes "reasonable" used,max parameters, which latter lead to excessive GC work being performed when an application uses more than max/used "handles" at a time (this can happen because the used,max parameters were OK many years ago on computers with less memory, or because the particular usage in the application is not the one anticipated by the lib's author).

Some examples: the Event module in the (sys)threads library, regexp handles in ocaml-pcre, statement and DB handles in ocaml-sqlite3.

I expanded on some possible ways to address this in

http://caml.inria.fr/mantis/view.php?id=7158#c15410 [^]

In short, it'd be nice to have one of the following:

* caml_alloc_custom/final-like functions where the memory footprint of the out-of-heap resource can be given so that the GC can adjust its speed using a mechanism similar to the one for in-heap blocks (this would be directly usable in all stubs where the custom blocks represent chunks of memory and not scarce resources)

* a means to (1) register and (2) list and modify "build-time constants" (or more precisely, variables with build-time defaults). This would allow to tweak parameters without having to patch the upstream libs (after the one-time change to use this new API, that is).

The latter would allow to address the caml_alloc_custom issue on an application basis, and should also be useful for other "build-time constants" where it is hard to provide a value suitable for all application domains. It could be argued that each library could provide such functionality on its own, but the caml_alloc_custom issues I found (and the others likely to lie dormant waiting to be stepped upon) show this doesn't happen in practice. Having an official C + OCaml API for this would lower the barrier of adoption.
TagsNo tags attached.
Attached Files

- Relationships
related to 0007158resolved Event.sync forces a full major GC cycle every 5000 calls at most 
related to 0007100assigneddoligez Bigarray's caml_ba_alloc doesn't try GC if malloc fails 

-  Notes
(0018664)
frisch (developer)
2017-11-15 17:23

It's true that even the 1/1000 ratio used for Pervasives channels can be suboptimal. Since the GC doesn't even close the underlying file descriptor anyway, one cannot even argue that this is to avoid fd leaks. Should we use a (default) value more representative of the actual memory usage for the channel structure itself?

- Issue History
Date Modified Username Field Change
2016-03-27 17:17 mfp New Issue
2016-04-06 13:06 doligez Status new => acknowledged
2016-04-06 13:06 doligez Target Version => 4.03.1+dev
2016-04-06 13:07 doligez Relationship added related to 0007158
2017-02-16 14:01 doligez Target Version 4.03.1+dev => undecided
2017-02-21 17:29 doligez Assigned To => doligez
2017-02-21 17:29 doligez Status acknowledged => assigned
2017-02-21 17:29 doligez Target Version undecided => 4.06.0 +dev/beta1/beta2/rc1
2017-02-23 16:43 doligez Category OCaml runtime system => runtime system
2017-03-03 17:45 doligez Category runtime system => runtime system and C interface
2017-10-05 18:07 doligez Relationship added related to 0007100
2017-10-10 15:39 doligez Target Version 4.06.0 +dev/beta1/beta2/rc1 => 4.07.0+dev
2017-11-15 17:23 frisch Note Added: 0018664


Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker