Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0005466OCaml~DO NOT USE (was: OCaml general)public2012-01-05 10:512013-08-31 12:46
ReporterMartin Jambon 
Assigned To 
PlatformOSLinuxOS Version2.6.34
Product Version3.12.1 
Target VersionFixed in Version 
Summary0005466: map_file fails if requested length is larger than RAM+swap
DescriptionThe map_file function as implemented for the Bigarray module uses systematically the PROT_WRITE flag in its call to mmap (otherlibs/bigarray/mmap_unix.c).

This causes the call to mmap to fail with error "Cannot allocate memory" if the requested length exceeds a certain value and the file is opened in read-only mode (O_RDONLY) and therefore no sharing is possible (MAP_PRIVATE). This limit seems to be slightly under the combined total RAM+swap. I don't know whether there is a good reason for such a limit on writable files, but it should not be a problem with read-only mappings.

Using PROT_READ instead of PROT_READ|PROT_WRITE solves the problem, i.e. allows to map the entirety of a large file into a single array even it is longer than the available memory.

A workaround is to open the file in read-write mode and to enable sharing, but the file permissions may not always allow it.
Steps To Reproduce(*
   Create a file test.dat that is larger than total memory+swap.
   ocamlopt -o testmap unix.cmxa bigarray.cmxa

   $ ./testmap
   Fatal error: exception Sys_error("Cannot allocate memory")
let map_file fname =
  let fd = Unix.openfile fname [Unix.O_RDONLY] 0o600 in
  Bigarray.Array1.map_file fd Bigarray.char Bigarray.c_layout false (-1)

let () = ignore (map_file "test.dat")

(* Workaround *)
let map_file fname =
  let fd = Unix.openfile fname [Unix.O_RDWR] 0o600 in
  Bigarray.Array1.map_file fd Bigarray.char Bigarray.c_layout true (-1)

let () = ignore (map_file "test.dat")
TagsNo tags attached.
Attached Files

- Relationships

-  Notes
gerd (reporter)
2012-01-05 23:41

This looks like a bug in Linux or libc. SUS specifies: "If PROT_WRITE is specified, the application must have opened the file descriptor fildes with write permission unless MAP_PRIVATE is specified in the flags parameter as described below." I guess Linux silently assumes MAP_PRIVATE if the write permission is not given instead of returning EACCES (which is even documented in the Linux man page).
xleroy (administrator)
2012-01-14 19:58

The Bigarray library could support mapping files without PROT_WRITE, either via an optional parameter to the map_file functions, or automatically if the file is opened readonly. The problem I have with this solution is that any assignment to such a bigarray would crash the program on a SEGV. This doesn't bode well for a language that claims to be type-safe.
Martin Jambon (reporter)
2012-01-17 09:54

I can imagine an "unsafe_map_file" function that would allow a call to mmap without PROT_WRITE as you suggest. Such function would be clearly marked as unsafe and possibly hidden from the official documentation.

On top of that, some library could provide read-only access to bigarrays via a read-only (but not immutable) abstract type and the corresponding safe map_file function. It could be an extension of the Bigarray module (Bigarray.Array1.Read_only.t), it could be a new module Bigarray_read_only shipping with the bigarray library, or the task could be left to third-party libraries.
xleroy (administrator)
2012-02-05 11:01

At this point, I don't quite know what is the best way to address this feature wish, nor how far we should go to support read-only file mappings in Bigarray. So, I'm putting this PR in the "suspended" state. But feel free to experiment with some of the approaches mentioned and let us know of your findings.

- Issue History
Date Modified Username Field Change
2012-01-05 10:51 Martin Jambon New Issue
2012-01-05 23:41 gerd Note Added: 0006605
2012-01-14 19:58 xleroy Note Added: 0006681
2012-01-14 19:58 xleroy Severity major => minor
2012-01-14 19:58 xleroy Status new => feedback
2012-01-17 09:54 Martin Jambon Note Added: 0006690
2012-01-17 09:54 Martin Jambon Status feedback => new
2012-02-05 11:01 xleroy Note Added: 0006879
2012-02-05 11:01 xleroy Severity minor => feature
2012-02-05 11:01 xleroy Status new => resolved
2012-02-05 11:01 xleroy Resolution open => suspended
2013-08-31 12:46 xleroy Status resolved => closed
2017-02-23 16:36 doligez Category OCaml general => -OCaml general
2017-03-03 17:55 doligez Category -OCaml general => -(deprecated) general
2017-03-03 18:01 doligez Category -(deprecated) general => ~deprecated (was: OCaml general)
2017-03-06 17:04 doligez Category ~deprecated (was: OCaml general) => ~DO NOT USE (was: OCaml general)

Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker