Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0006488OCamlOCaml generalpublic2014-07-15 20:252014-07-16 16:30
Reporterdhekir 
Assigned To 
PrioritynormalSeverityminorReproducibilityalways
StatusclosedResolutionno change required 
PlatformOSOS Version
Product Version4.00.0 
Target Version4.02.1+devFixed in Version 
Summary0006488: Sys.command: cannot allocate memory
DescriptionI'd like to confirm if the following behavior is intentional, or if the Sys.command error message might be misleading in some cases.

In a situation where there is not enough memory available in the system, the following error message appears in situations which I would qualify as highly improbable:

Fatal error: exception Sys_error("./test.sh: Cannot allocate memory")

The "improbable" comes from the fact that the OCaml process is using more than 1.5 GB of RAM, while the test.sh script uses no more than 8 MB, and yet the error message seems to blame the script.
Steps To ReproduceI've used the following OCaml file and Bash script to reproduce this situation. Note that the sizes of the OCaml matrix must be defined so that the OCaml process will not be killed by the Linux OOM.

test.ml:
  let _ =
    let _ = Array.create_matrix 45000 5000 0 in
    Sys.command "./test.sh"

test.sh:
  #!/bin/bash
  for x in {1..20000};do :;done

Using a script to measure memory usage, I obtain values as high as 1.7 GB of memory usage by the OCaml process, while the test script by itself consumes about 5 MB.

Is this expected behavior? I would expect very little memory overhead from launching the child process, which would mean that almost all of the available memory is being consumed by the OCaml process.
But since the OCaml process itself does not crash, then this would mean that either it is aggressively overallocating memory (and later increasing its effective usage when there is a shortage, but being unable to free this memory for the external process), or that the error message might actually be unrelated to the Sys.command call. In the latter case, then there might be a bug, hence my report.
Additional InformationRunnina a Fedora 64-bit Linux with 4 GB of installed RAM.
TagsNo tags attached.
Attached Files

- Relationships

-  Notes
(0011832)
doligez (administrator)
2014-07-16 10:13

Do you have any swap space configured on your machine?

If the total amount of memory (including swap) is around twice the OCaml process size, I think this is expected. The Unix way of launching a process is to call fork() first, which duplicates the whole memory (and the open file descriptors and other system resources) of the calling process, then (in the child process) to call exec(), which deallocates most of these resources and launches the new command.

So in order to launch a new process, you need enough free virtual memory to duplicate the current process and this could explain what you are observing.
(0011834)
dhekir (reporter)
2014-07-16 10:43

Indeed, this is the cause.

I have no swap space configure on my machine.

I had never actually seen this behavior in C code, and since I know very little about OCaml's memory management, I assumed it might be related to it.

I confess I am a bit disappointed with Linux, I expected it to use some sort of copy-on-write mechanics instead of this "stupid" behavior.

I managed to reproduce the issue with a C program, and I get the same output.

Sorry for the noise, and thanks for the explanation.
(0011853)
doligez (administrator)
2014-07-16 16:30

In fact Linux does use copy-on-write and gets good performance for fork(), but you still need to allocate the memory, even if you initialize it only on demand.

What you are looking for is memory over-commit, which most distributions of Linux also do by default (maybe only if you configure some swap space).

There is also the vfork() system call, but that's rather error-prone and we don't have it in the Unix module. (I don't know whether it is in the POSIX standard.)

I'm closing this PR, feel free to open a feature request for vfork.

- Issue History
Date Modified Username Field Change
2014-07-15 20:25 dhekir New Issue
2014-07-16 10:13 doligez Note Added: 0011832
2014-07-16 10:14 doligez Status new => feedback
2014-07-16 10:14 doligez Target Version => 4.02.1+dev
2014-07-16 10:43 dhekir Note Added: 0011834
2014-07-16 10:43 dhekir Status feedback => new
2014-07-16 16:30 doligez Note Added: 0011853
2014-07-16 16:30 doligez Status new => closed
2014-07-16 16:30 doligez Resolution open => no change required


Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker