0004108: Memory-mapping of bigarrays may exhaust address space

View Issue Details [ Jump to Notes ]

[ Issue History ] [ Print ]

Project

Category

View Status

Date Submitted

Last Update

0004108

OCaml

OCaml general

public

2006-09-11 18:35

2012-09-07 12:52

Reporter

mottl

Assigned To

Priority

normal

Severity

major

Reproducibility

always

Status

confirmed

Resolution

open

Platform

OS Version

Product Version

3.09.3

Target Version

4.00.2+dev

Fixed in Version

4.00.0+dev

Summary

0004108: Memory-mapping of bigarrays may exhaust address space

Description

This seems to be a bug in the bigarray implementation: when files are memory-mapped to bigarrays, the GC does not work hard enough deallocating and (thus, more importantly here, unmapping) the files. On 32bit architectures this will exhaust the address space if a program maps/forgets large files very quickly, leading to a [Sys_error "Cannot allocate memory"] exception.

The bug seems to be in alloc_bigarray in otherlibs/bigarray/bigarray_stubs.c. The function tests "data == NULL", which is not the case for memory-mapped files, and "size" will therefore not be set to the size of the mapped array. Because "size" has been set to zero before, the alloc_custom-call will therefore assume that memory-mapped files are "for free", which is not quite true: address space is a scarce resource, too. I thus suggest that the "size"-parameter in the call to alloc_custom be initialized correctly for memory-mapped files, too, to make the GC work hard enough.

Tags

No tags attached.

Attached Files

Relationships

Notes
(0007300) xleroy (administrator) 2012-04-09 12:42	Controlling the rate of the incremental GC is a difficult issue that the current API for custom allocation doesn't fully address. In 4.00 I added "release" functions to the Bigarray module that support immediate, explicit reclamation of resources (memory and file mappings) associated with a bigarray, just like finalization does but at predictable times. Using these functions wisely should resolve the issue described in this PR, even though it takes more work from the programmer than letting the GC do the finalization.

(0007415) xleroy (administrator) 2012-05-06 10:26	There were objections to the "release" functions in Bigarray, which were therefore removed. Reopening this PR.

Issue History
Date Modified	Username	Field	Change
2006-09-11 18:35	mottl	New Issue
2006-09-12 14:29	doligez	Status	new => acknowledged
2012-04-09 12:42	xleroy	Note Added: 0007300
2012-04-09 12:42	xleroy	Status	acknowledged => resolved
2012-04-09 12:42	xleroy	Resolution	open => suspended
2012-04-09 12:42	xleroy	Fixed in Version	=> 4.00.0+dev
2012-05-06 10:26	xleroy	Note Added: 0007415
2012-05-06 10:26	xleroy	Status	resolved => confirmed
2012-05-06 10:26	xleroy	Resolution	suspended => open
2012-07-06 16:41	doligez	Target Version	=> 4.01.0+dev
2012-07-31 13:37	doligez	Target Version	4.01.0+dev => 4.00.1+dev
2012-09-07 12:52	frisch	Target Version	4.00.1+dev => 4.00.2+dev