|Anonymous | Login | Signup for a new account||2014-11-26 19:43 CET|
|Main | My View | View Issues | Change Log | Roadmap|
|View Issue Details|
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0006001||OCaml||OCaml internal build/install (Makefiles, configure)||public||2013-04-27 10:45||2013-06-06 03:34|
|Platform||1&1 mutualised server||OS||Linux||OS Version||Debian GNU/Linux|
|Target Version||Fixed in Version|
|Summary||0006001: Excessive memory consumption while compiling the OCaml distribution|
|Description||During make world, on a 1&1 mutualised server (restricted memory)|
Fatal error: exception Out_of_memory
Exit code 2 while executing this command:
../ocamlcomp.sh -c -g -warn-error A -w a -I camlp4/boot -I camlp4 -I stdlib -o camlp4/boot/camlp4boot.cmo camlp4/boot/camlp4boot.ml
make: *** [camlp4out] Error 2
make: Leaving directory `/homepages/30/d456759237/htdocs/ocaml-4.00.1'
make: *** [world] Error 2
|Tags||No tags attached.|
|Attached Files|| camlp4-extend-avoid-long-sequences.diff [^] (2,796 bytes) 2013-06-04 17:26 [Show Content]
camlp4-bootstrap.diff [^] (10,449 bytes) 2013-06-04 17:26 [Show Content]
parser.diff [^] (601 bytes) 2013-06-05 05:50 [Show Content]
This is not a "bug" per se: your particular environment (a mutualized server) does not provide you enough memory to compile Camlp4 (camlp4boot.ml is the bootstrap file that is indeed quite large and takes a good portion of the compilation time and memory for Camlp4).
As far as I know, the time and memory consumption of the OCaml distribution is extremely reasonable when compared to other software projects (LLVM, GHC, Java... not to mention Firefox or LibreOffice). If it doesn't compile on your environment, we'd rather blame it.
You could try to:
- cross-compile Camlp4 (or OCaml altogether): compile it from a computer having a similar development environment, and move the resulting binaries to your server
- compile OCaml witout Camlp4 (-no-camlp4 option during ./configure)
- tweak the compilation settings, hoping to consume less memory (I see that the -g flags is passed, it may be better to compile without it)
Any of these may be better discussed on the mailing-list rather than the bugtracker: I'm sure some people reading the list have done some of these, and they don't necessarily participate on the bugtracker.
|For the record, I've heard several other reports of Camlp4's compilation consuming too much memory, and I now think it may be interesting to look for ways to reduce it. If time allows, I will have a look, but I welcome any previous experience on monitoring the memory consumption of the OCaml compiler, or help in that matter.|
|The bootstrapping model for Camlp4 is that it tries to marshal all files into one single module, resulting in a very large file, camlp4boot.ml, 55163 lines. Is this the cause of the problem?|
|Yes, I believe that this is the file for which compilation consumes so much memory -- if you have insights on *non-invasive* changes to split it up, that would be interesting. However, given that I don't remember hearing such memory problems before 4.00, it might also be the a regression in memory use by the compiler (related to new warnings or I don't know what), which could be worrying as well and worth tracking down.|
It turns out this command takes up to 700M memory, this issue has nothing to do with
camlp4, since camlp4boot.ml is simply plain ocaml file.
ocamlopt.opt -c camlp4boot.ml
steps to reproduce
assume you are in the compiler toplevel
ocaml>cp _build/camlp4/Camlp4_import.ml camlp4/Camlp4_config.ml camlp4/Camlp4_config.mli camlp4/boot/
boot>ocamlopt.opt dynlink.cmxa Camlp4_import.ml Camlp4_config.mli Camlp4_config.ml Camlp4.ml camlp4boot.ml
I tried to test it under different compiler versions, however, ocaml 3.11.2 is broken in my machine, if you have ocaml 3.11.2 installed, it would be very helpful to try to compile with the same source code(It should compile) and see the result.
It's possible to change the build system without breaking the compatibility, but it may invalidate the knowledge other maintainers have about camlp4'boot system, I am not sure whether it's worth, though.
|I think this is probably a bug in the compiler, 50,000 lines of code are not that big in reality|
A memory usage analysis ( /usr/bin/time -v for peak memory usage, http://ygrek.org.ua/p/code/pmpa [^] for finer-grained traces of code that allocates ) indicates that the problem is related to the long sequence of Grammar.extend call in Camlp4OCamlRevisedParser.ml (as a sub-file of camlp4boot.ml), which then provokes a bad behavior in the register allocator (constructing the coloration graph consumes hundreds of megabytes of memory).
As with all the problems where unnaturally long code sequences passed to the OCaml compiler, a possible fix is to change the code generator to break these sequences in smaller pieces. It is exactly what I did in the attached patch, and indeed it seems to solve the memory consumption issue for camlp4boot.ml (remains to check that this is the only issue).
The patch changes the way the EXTEND syntactic sugar works, to avoid making too many Grammar.Extend calls in a sequence.
This is only a preliminary attempt at solving the problem (and the code is in a very rough state). It would be more interesting to see if the compiler itself can be fixed, without a change in readability. That said, it may be interesting to include a simpler hack like this one in time for the next release.
Hongbo (or whoever is interested), could you comment on whether this patch indeed fixes the memory consumption problem?
One difficulty is that one needs to perform a Camlp4 bootstrap cycle for the changes to affect camlp4boot.ml. As bootstrapping Camlp4 is not easy, I have also uploaded a patch that does this camlp4 bootstrap -- to be applied on top of trunk.
|Would something like 0004074 improve the situation?|
gasche: I upload a simplified patch without changing the code generator, now ocamlc.opt and ocamlopt.opt consumes the same amount of memory, around 200M in my machine. But 200M memory may still seem to be a lot?
alain: would you provide a patch against the current trunk?
Alain, I *think* that the problem is orthogonal, as the "register use worst case" is here happening entirely inside one expression, not spread among several structure items.
Hongbo, your patch is indeed much simpler. Mine had the advantage of also fixing the problem for users writing large grammars with EXTEND, but that's a secondary aspect at best because I'm not sure there are actual uses of this.
I think you should push your patch upstream; there is really no reason not to, as it is nicely orthogonal to any solution we could find to the graph-coloring memory use issue.
> Alain, I *think* that the problem is orthogonal, as the "register use worst case" is here happening entirely inside one expression, not spread among several structure items.
I see, thanks.
It would be interesting to see how the linear scan allocator (0005324) would behave on that file.
|Fixed in r13749|
|2013-04-27 10:45||hgouraud||New Issue|
|2013-04-27 11:18||gasche||Note Added: 0009223|
|2013-04-27 11:18||gasche||Status||new => resolved|
|2013-04-27 11:18||gasche||Resolution||open => no change required|
|2013-04-27 11:18||gasche||Assigned To||=> gasche|
|2013-05-28 10:13||gasche||Note Added: 0009361|
|2013-05-28 11:56||gasche||Assigned To||gasche =>|
|2013-05-28 11:56||gasche||Severity||block => minor|
|2013-05-28 11:56||gasche||Reproducibility||always => sometimes|
|2013-05-28 11:56||gasche||Resolution||no change required => suspended|
|2013-05-28 19:48||hongboz||Note Added: 0009364|
|2013-05-28 23:58||gasche||Note Added: 0009365|
|2013-05-29 01:35||hongboz||Note Added: 0009366|
|2013-05-30 09:58||gasche||Summary||Fatal error: exception Out_of_memory => Excessive memory consumption while compiling the OCaml distribution|
|2013-05-30 09:58||gasche||Description Updated||View Revisions|
|2013-05-31 22:25||hongboz||Note Added: 0009373|
|2013-06-04 17:25||gasche||Note Added: 0009401|
|2013-06-04 17:26||gasche||File Added: camlp4-extend-avoid-long-sequences.diff|
|2013-06-04 17:26||gasche||File Added: camlp4-bootstrap.diff|
|2013-06-04 17:27||gasche||Status||resolved => confirmed|
|2013-06-04 21:40||frisch||Note Added: 0009406|
|2013-06-05 05:50||hongboz||File Added: parser.diff|
|2013-06-05 05:53||hongboz||Note Added: 0009407|
|2013-06-05 09:43||gasche||Note Added: 0009408|
|2013-06-05 10:19||frisch||Note Added: 0009409|
|2013-06-06 03:33||hongboz||Note Added: 0009417|
|2013-06-06 03:34||hongboz||Assigned To||=> hongboz|
|2013-06-06 03:34||hongboz||Status||confirmed => closed|
|2013-06-06 03:34||hongboz||Resolution||suspended => fixed|
|Copyright © 2000 - 2011 MantisBT Group|