Original bug ID: 5466
Reporter: @mjambon
Status: closed (set by @xavierleroy on 2013-08-31T10:46:24Z)
Resolution: suspended
Priority: normal
Severity: feature
OS: Linux
OS Version: 2.6.34
Version: 3.12.1
Category: ~DO NOT USE (was: OCaml general)
Monitored by: Camarade_Tux @hcarty

Bug description

The map_file function as implemented for the Bigarray module uses systematically the PROT_WRITE flag in its call to mmap (otherlibs/bigarray/mmap_unix.c).

This causes the call to mmap to fail with error "Cannot allocate memory" if the requested length exceeds a certain value and the file is opened in read-only mode (O_RDONLY) and therefore no sharing is possible (MAP_PRIVATE). This limit seems to be slightly under the combined total RAM+swap. I don't know whether there is a good reason for such a limit on writable files, but it should not be a problem with read-only mappings.

Using PROT_READ instead of PROT_READ|PROT_WRITE solves the problem, i.e. allows to map the entirety of a large file into a single array even it is longer than the available memory.

A workaround is to open the file in read-write mode and to enable sharing, but the file permissions may not always allow it.

Steps to reproduce

(*
Create a file test.dat that is larger than total memory+swap.
ocamlopt -o testmap unix.cmxa bigarray.cmxa testmap.ml

$ ./testmap
Fatal error: exception Sys_error("Cannot allocate memory")
*)
let map_file fname =
let fd = Unix.openfile fname [Unix.O_RDONLY] 0o600 in
Bigarray.Array1.map_file fd Bigarray.char Bigarray.c_layout false (-1)

let () = ignore (map_file "test.dat")

(* Workaround *)
let map_file fname =
let fd = Unix.openfile fname [Unix.O_RDWR] 0o600 in
Bigarray.Array1.map_file fd Bigarray.char Bigarray.c_layout true (-1)

let () = ignore (map_file "test.dat")