Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0006001OCamlOCaml internal build/install (Makefiles, configure)public2013-04-27 10:452013-06-06 03:34
Reporterhgouraud 
Assigned Tohongboz 
PrioritynormalSeverityminorReproducibilitysometimes
StatusclosedResolutionfixed 
Platform1&1 mutualised serverOSLinuxOS VersionDebian GNU/Linux
Product Version4.00.1 
Target VersionFixed in Version 
Summary0006001: Excessive memory consumption while compiling the OCaml distribution
DescriptionDuring make world, on a 1&1 mutualised server (restricted memory)

Fatal error: exception Out_of_memory
Exit code 2 while executing this command:
  ../ocamlcomp.sh -c -g -warn-error A -w a -I camlp4/boot -I camlp4 -I stdlib -o camlp4/boot/camlp4boot.cmo camlp4/boot/camlp4boot.ml
make[1]: *** [camlp4out] Error 2
make[1]: Leaving directory `/homepages/30/d456759237/htdocs/ocaml-4.00.1'
make: *** [world] Error 2
TagsNo tags attached.
Attached Filesdiff file icon camlp4-extend-avoid-long-sequences.diff [^] (2,796 bytes) 2013-06-04 17:26 [Show Content]
diff file icon camlp4-bootstrap.diff [^] (10,449 bytes) 2013-06-04 17:26 [Show Content]
diff file icon parser.diff [^] (601 bytes) 2013-06-05 05:50 [Show Content]

- Relationships

-  Notes
(0009223)
gasche (developer)
2013-04-27 11:18

This is not a "bug" per se: your particular environment (a mutualized server) does not provide you enough memory to compile Camlp4 (camlp4boot.ml is the bootstrap file that is indeed quite large and takes a good portion of the compilation time and memory for Camlp4).

As far as I know, the time and memory consumption of the OCaml distribution is extremely reasonable when compared to other software projects (LLVM, GHC, Java... not to mention Firefox or LibreOffice). If it doesn't compile on your environment, we'd rather blame it.

You could try to:
- cross-compile Camlp4 (or OCaml altogether): compile it from a computer having a similar development environment, and move the resulting binaries to your server
- compile OCaml witout Camlp4 (-no-camlp4 option during ./configure)
- tweak the compilation settings, hoping to consume less memory (I see that the -g flags is passed, it may be better to compile without it)

Any of these may be better discussed on the mailing-list rather than the bugtracker: I'm sure some people reading the list have done some of these, and they don't necessarily participate on the bugtracker.
(0009361)
gasche (developer)
2013-05-28 10:13

For the record, I've heard several other reports of Camlp4's compilation consuming too much memory, and I now think it may be interesting to look for ways to reduce it. If time allows, I will have a look, but I welcome any previous experience on monitoring the memory consumption of the OCaml compiler, or help in that matter.
(0009364)
hongboz (developer)
2013-05-28 19:48

The bootstrapping model for Camlp4 is that it tries to marshal all files into one single module, resulting in a very large file, camlp4boot.ml, 55163 lines. Is this the cause of the problem?
(0009365)
gasche (developer)
2013-05-28 23:58

Yes, I believe that this is the file for which compilation consumes so much memory -- if you have insights on *non-invasive* changes to split it up, that would be interesting. However, given that I don't remember hearing such memory problems before 4.00, it might also be the a regression in memory use by the compiler (related to new warnings or I don't know what), which could be worrying as well and worth tracking down.
(0009366)
hongboz (developer)
2013-05-29 01:35

It turns out this command takes up to 700M memory, this issue has nothing to do with
camlp4, since camlp4boot.ml is simply plain ocaml file.
------------------------------
ocamlopt.opt -c camlp4boot.ml

------------------------------
steps to reproduce
------------------------------
assume you are in the compiler toplevel
-----
ocaml>ocamlbuild camlp4/Camlp4_import.ml
ocaml>cp _build/camlp4/Camlp4_import.ml camlp4/Camlp4_config.ml camlp4/Camlp4_config.mli camlp4/boot/
ocaml>cd camlp4/boot/
boot>ocamlopt.opt dynlink.cmxa Camlp4_import.ml Camlp4_config.mli Camlp4_config.ml Camlp4.ml camlp4boot.ml
--------------------------------

I tried to test it under different compiler versions, however, ocaml 3.11.2 is broken in my machine, if you have ocaml 3.11.2 installed, it would be very helpful to try to compile with the same source code(It should compile) and see the result.

Other issues:
   It's possible to change the build system without breaking the compatibility, but it may invalidate the knowledge other maintainers have about camlp4'boot system, I am not sure whether it's worth, though.
(0009373)
hongboz (developer)
2013-05-31 22:25

I think this is probably a bug in the compiler, 50,000 lines of code are not that big in reality
(0009401)
gasche (developer)
2013-06-04 17:25

A memory usage analysis ( /usr/bin/time -v for peak memory usage, http://ygrek.org.ua/p/code/pmpa [^] for finer-grained traces of code that allocates ) indicates that the problem is related to the long sequence of Grammar.extend call in Camlp4OCamlRevisedParser.ml (as a sub-file of camlp4boot.ml), which then provokes a bad behavior in the register allocator (constructing the coloration graph consumes hundreds of megabytes of memory).

As with all the problems where unnaturally long code sequences passed to the OCaml compiler, a possible fix is to change the code generator to break these sequences in smaller pieces. It is exactly what I did in the attached patch, and indeed it seems to solve the memory consumption issue for camlp4boot.ml (remains to check that this is the only issue).

The patch changes the way the EXTEND syntactic sugar works, to avoid making too many Grammar.Extend calls in a sequence.

This is only a preliminary attempt at solving the problem (and the code is in a very rough state). It would be more interesting to see if the compiler itself can be fixed, without a change in readability. That said, it may be interesting to include a simpler hack like this one in time for the next release.

Hongbo (or whoever is interested), could you comment on whether this patch indeed fixes the memory consumption problem?

One difficulty is that one needs to perform a Camlp4 bootstrap cycle for the changes to affect camlp4boot.ml. As bootstrapping Camlp4 is not easy, I have also uploaded a patch that does this camlp4 bootstrap -- to be applied on top of trunk.
(0009406)
frisch (developer)
2013-06-04 21:40

Would something like 0004074 improve the situation?
(0009407)
hongboz (developer)
2013-06-05 05:53

gasche: I upload a simplified patch without changing the code generator, now ocamlc.opt and ocamlopt.opt consumes the same amount of memory, around 200M in my machine. But 200M memory may still seem to be a lot?

alain: would you provide a patch against the current trunk?

Thanks
(0009408)
gasche (developer)
2013-06-05 09:43

Alain, I *think* that the problem is orthogonal, as the "register use worst case" is here happening entirely inside one expression, not spread among several structure items.

Hongbo, your patch is indeed much simpler. Mine had the advantage of also fixing the problem for users writing large grammars with EXTEND, but that's a secondary aspect at best because I'm not sure there are actual uses of this.

I think you should push your patch upstream; there is really no reason not to, as it is nicely orthogonal to any solution we could find to the graph-coloring memory use issue.
(0009409)
frisch (developer)
2013-06-05 10:19

> Alain, I *think* that the problem is orthogonal, as the "register use worst case" is here happening entirely inside one expression, not spread among several structure items.

I see, thanks.

It would be interesting to see how the linear scan allocator (0005324) would behave on that file.
(0009417)
hongboz (developer)
2013-06-06 03:33

Fixed in r13749

- Issue History
Date Modified Username Field Change
2013-04-27 10:45 hgouraud New Issue
2013-04-27 11:18 gasche Note Added: 0009223
2013-04-27 11:18 gasche Status new => resolved
2013-04-27 11:18 gasche Resolution open => no change required
2013-04-27 11:18 gasche Assigned To => gasche
2013-05-28 10:13 gasche Note Added: 0009361
2013-05-28 11:56 gasche Assigned To gasche =>
2013-05-28 11:56 gasche Severity block => minor
2013-05-28 11:56 gasche Reproducibility always => sometimes
2013-05-28 11:56 gasche Resolution no change required => suspended
2013-05-28 19:48 hongboz Note Added: 0009364
2013-05-28 23:58 gasche Note Added: 0009365
2013-05-29 01:35 hongboz Note Added: 0009366
2013-05-30 09:58 gasche Summary Fatal error: exception Out_of_memory => Excessive memory consumption while compiling the OCaml distribution
2013-05-30 09:58 gasche Description Updated View Revisions
2013-05-31 22:25 hongboz Note Added: 0009373
2013-06-04 17:25 gasche Note Added: 0009401
2013-06-04 17:26 gasche File Added: camlp4-extend-avoid-long-sequences.diff
2013-06-04 17:26 gasche File Added: camlp4-bootstrap.diff
2013-06-04 17:27 gasche Status resolved => confirmed
2013-06-04 21:40 frisch Note Added: 0009406
2013-06-05 05:50 hongboz File Added: parser.diff
2013-06-05 05:53 hongboz Note Added: 0009407
2013-06-05 09:43 gasche Note Added: 0009408
2013-06-05 10:19 frisch Note Added: 0009409
2013-06-06 03:33 hongboz Note Added: 0009417
2013-06-06 03:34 hongboz Assigned To => hongboz
2013-06-06 03:34 hongboz Status confirmed => closed
2013-06-06 03:34 hongboz Resolution suspended => fixed


Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker