Sys.command: cannot allocate memory #6488
Comments
|
Comment author: @damiendoligez Do you have any swap space configured on your machine? If the total amount of memory (including swap) is around twice the OCaml process size, I think this is expected. The Unix way of launching a process is to call fork() first, which duplicates the whole memory (and the open file descriptors and other system resources) of the calling process, then (in the child process) to call exec(), which deallocates most of these resources and launches the new command. So in order to launch a new process, you need enough free virtual memory to duplicate the current process and this could explain what you are observing. |
|
Comment author: dhekir Indeed, this is the cause. I have no swap space configure on my machine. I had never actually seen this behavior in C code, and since I know very little about OCaml's memory management, I assumed it might be related to it. I confess I am a bit disappointed with Linux, I expected it to use some sort of copy-on-write mechanics instead of this "stupid" behavior. I managed to reproduce the issue with a C program, and I get the same output. Sorry for the noise, and thanks for the explanation. |
|
Comment author: @damiendoligez In fact Linux does use copy-on-write and gets good performance for fork(), but you still need to allocate the memory, even if you initialize it only on demand. What you are looking for is memory over-commit, which most distributions of Linux also do by default (maybe only if you configure some swap space). There is also the vfork() system call, but that's rather error-prone and we don't have it in the Unix module. (I don't know whether it is in the POSIX standard.) I'm closing this PR, feel free to open a feature request for vfork. |
Original bug ID: 6488
Reporter: dhekir
Status: closed (set by @damiendoligez on 2014-07-16T14:30:18Z)
Resolution: not a bug
Priority: normal
Severity: minor
Version: 4.00.0
Target version: 4.02.1+dev
Category: ~DO NOT USE (was: OCaml general)
Monitored by: @gasche
Bug description
I'd like to confirm if the following behavior is intentional, or if the Sys.command error message might be misleading in some cases.
In a situation where there is not enough memory available in the system, the following error message appears in situations which I would qualify as highly improbable:
Fatal error: exception Sys_error("./test.sh: Cannot allocate memory")
The "improbable" comes from the fact that the OCaml process is using more than 1.5 GB of RAM, while the test.sh script uses no more than 8 MB, and yet the error message seems to blame the script.
Steps to reproduce
I've used the following OCaml file and Bash script to reproduce this situation. Note that the sizes of the OCaml matrix must be defined so that the OCaml process will not be killed by the Linux OOM.
test.ml:
let _ =
let _ = Array.create_matrix 45000 5000 0 in
Sys.command "./test.sh"
test.sh:
#!/bin/bash
for x in {1..20000};do :;done
Using a script to measure memory usage, I obtain values as high as 1.7 GB of memory usage by the OCaml process, while the test script by itself consumes about 5 MB.
Is this expected behavior? I would expect very little memory overhead from launching the child process, which would mean that almost all of the available memory is being consumed by the OCaml process.
But since the OCaml process itself does not crash, then this would mean that either it is aggressively overallocating memory (and later increasing its effective usage when there is a shortage, but being unable to free this memory for the external process), or that the error message might actually be unrelated to the Sys.command call. In the latter case, then there might be a bug, hence my report.
Additional information
Runnina a Fedora 64-bit Linux with 4 GB of installed RAM.
The text was updated successfully, but these errors were encountered: