Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ocamlopt.opt 4.04.0+spacetime crashes while compiling package camlp4.4.04+1 #7435

Closed
vicuna opened this issue Dec 19, 2016 · 8 comments
Closed
Assignees

Comments

@vicuna
Copy link

vicuna commented Dec 19, 2016

Original bug ID: 7435
Reporter: khm
Assigned to: @mshinwell
Status: acknowledged (set by @mshinwell on 2017-03-07T12:51:15Z)
Resolution: open
Priority: normal
Severity: minor
Platform: x86_64
OS: Linux
OS Version: Fedora 25
Version: 4.04.0
Category: back end (clambda to assembly)
Monitored by: nekketsuuu @gasche

Bug description

ocamlbuild -classic-display -no-ocamlfind ./build/camlp4-byte-only.sh
/home/khm/.opam/4.04.0+spacetime/bin/ocamlopt.opt unix.cmxa -I /home/khm/.opam/4.04.0+spacetime/lib/ocamlbuild /home/khm/.opam/4.04.0+spacetime/lib/ocamlbuild/ocamlbuildlib.cmxa myocamlbuild_config.ml myocamlbuild.ml /home/khm/.opam/4.04.0+spacetime/lib/ocamlbuild/ocamlbuild.cmx -o myocamlbuild
/home/khm/.opam/4.04.0+spacetime/bin/ocamldep.opt -modules camlp4/boot/camlp4boot.ml > camlp4/boot/camlp4boot.ml.depends
/home/khm/.opam/4.04.0+spacetime/bin/ocamldep.opt -modules camlp4/boot/Camlp4.ml > camlp4/boot/Camlp4.ml.depends
/home/khm/.opam/4.04.0+spacetime/bin/ocamldep.opt -modules camlp4/config/Camlp4_config.mli > camlp4/config/Camlp4_config.mli.depends
cp /home/khm/.opam/4.04.0+spacetime/lib/ocaml/compiler-libs/warnings.cmi camlp4/import/warnings.cmi
cp /home/khm/.opam/4.04.0+spacetime/lib/ocaml/compiler-libs/location.cmi camlp4/import/location.cmi
cp /home/khm/.opam/4.04.0+spacetime/lib/ocaml/compiler-libs/longident.cmi camlp4/import/longident.cmi
cp /home/khm/.opam/4.04.0+spacetime/lib/ocaml/compiler-libs/asttypes.cmi camlp4/import/asttypes.cmi
cp /home/khm/.opam/4.04.0+spacetime/lib/ocaml/compiler-libs/parsetree.cmi camlp4/import/parsetree.cmi
cp /home/khm/.opam/4.04.0+spacetime/lib/ocaml/compiler-libs/outcometree.cmi camlp4/import/outcometree.cmi
cp /home/khm/.opam/4.04.0+spacetime/lib/ocaml/compiler-libs/oprint.cmi camlp4/import/oprint.cmi
cp /home/khm/.opam/4.04.0+spacetime/lib/ocaml/compiler-libs/toploop.cmi camlp4/import/toploop.cmi
cp /home/khm/.opam/4.04.0+spacetime/lib/ocaml/compiler-libs/topdirs.cmi camlp4/import/topdirs.cmi
/home/khm/.opam/4.04.0+spacetime/bin/ocamlc.opt -c -g -safe-string -w Z -I camlp4/import -I camlp4/config -I camlp4 -o camlp4/config/Camlp4_config.cmi camlp4/config/Camlp4_config.mli
/home/khm/.opam/4.04.0+spacetime/bin/ocamlc.opt -c -g -safe-string -w a -I camlp4/import -I camlp4/config -I camlp4/boot -o camlp4/boot/Camlp4.cmo camlp4/boot/Camlp4.ml
/home/khm/.opam/4.04.0+spacetime/bin/ocamlc.opt -c -g -safe-string -w a -I camlp4/import -I camlp4/config -I camlp4/boot -o camlp4/boot/camlp4boot.cmo camlp4/boot/camlp4boot.ml
/home/khm/.opam/4.04.0+spacetime/bin/ocamldep.opt -modules camlp4/config/Camlp4_config.ml > camlp4/config/Camlp4_config.ml.depends
/home/khm/.opam/4.04.0+spacetime/bin/ocamldep.opt -modules camlp4/config/gen_import.ml > camlp4/config/gen_import.ml.depends
/home/khm/.opam/4.04.0+spacetime/bin/ocamlc.opt -c -g -safe-string -w Z -I +compiler-libs -I camlp4/import -I camlp4/config -I camlp4 -o camlp4/config/gen_import.cmo camlp4/config/gen_import.ml
/home/khm/.opam/4.04.0+spacetime/bin/ocamlc.opt dynlink.cma -g -I +compiler-libs ocamlcommon.cma camlp4/config/gen_import.cmo -o camlp4/config/gen_import.byte
camlp4/config/gen_import.byte /home/khm/.opam/4.04.0+spacetime/lib/ocaml
/home/khm/.opam/4.04.0+spacetime/bin/ocamldep.opt -modules camlp4/config/Camlp4_import.ml > camlp4/config/Camlp4_import.ml.depends
/home/khm/.opam/4.04.0+spacetime/bin/ocamlc.opt -c -g -safe-string -w Z -I camlp4/import -I camlp4/config -I camlp4 -o camlp4/config/Camlp4_import.cmo camlp4/config/Camlp4_import.ml
/home/khm/.opam/4.04.0+spacetime/bin/ocamlopt.opt -c -g -safe-string -w Z -I camlp4/import -I camlp4/config -I camlp4 -o camlp4/config/Camlp4_import.cmx camlp4/config/Camlp4_import.ml
/home/khm/.opam/4.04.0+spacetime/bin/ocamlopt.opt -c -g -safe-string -w Z -I camlp4/import -I camlp4/config -I camlp4 -o camlp4/config/Camlp4_config.cmx camlp4/config/Camlp4_config.ml
/home/khm/.opam/4.04.0+spacetime/bin/ocamlopt.opt -c -g -safe-string -w a -I camlp4/import -I camlp4/config -I camlp4/boot -o camlp4/boot/Camlp4.cmx camlp4/boot/Camlp4.ml
/home/khm/.opam/4.04.0+spacetime/bin/ocamlopt.opt -c -g -safe-string -w a -I camlp4/import -I camlp4/config -I camlp4/boot -o camlp4/boot/camlp4boot.cmx camlp4/boot/camlp4boot.ml

  • /home/khm/.opam/4.04.0+spacetime/bin/ocamlopt.opt -c -g -safe-string -w a -I camlp4/import -I camlp4/config -I camlp4/boot -o camlp4/boot/camlp4boot.cmx camlp4/boot/camlp4boot.ml
    Fatal error: exception Stack overflow
    Raised by primitive operation at file "asmcomp/spill.ml", line 70, characters 17-55
    Called from file "asmcomp/spill.ml", line 157, characters 13-58
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 170, characters 37-57
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 173, characters 8-60
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 173, characters 8-60
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 173, characters 8-60
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 147, characters 32-52
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    [...]
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 173, characters 8-60
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Called from file "asmcomp/spill.ml", line 160, characters 32-51
    Command exited with code 2.
    Makefile:13: recipe for target 'byte' failed

Steps to reproduce

  1. opam switch 4.04.0+spacetime
  2. eval opam config env
  3. opam install "camlp4.4.04+1"
@vicuna
Copy link
Author

vicuna commented Dec 19, 2016

Comment author: @mshinwell

If you type "ulimit -s" in the shell before doing this compilation, what does it say?

If you do "ulimit -s unlimited" before compiling this problem should go away. We should check how badly Spacetime is affecting this (I have a suspicion some of the camlp4 files may be close to exceeding typical stack limits even with Spacetime disabled).

@vicuna
Copy link
Author

vicuna commented Dec 19, 2016

Comment author: khm

  1. OS defaults
    ulimit -s
    8192

  2. With
    ulimit -s unlimited
    compilation succeeds.

  3. Plain ocamlopt (spacetime is not enabled) can successfully compile camlp4 with OS defaults for me.

@vicuna
Copy link
Author

vicuna commented Feb 5, 2017

Comment author: twold

I've ran into essentially the same problem except that using option 2 (ulimit -s unlimited) still fails for me.

# File "camlp4/boot/camlp4boot.ml", line 1:
# Error: I/O error: as -o 'camlp4/boot/camlp4boot.o' '/tmp/camlasmc5cf87.s': Cannot allocate memory

I use 7.38G memory and 8.00G swap. The observed behaviour is that the compilation consumes all of my memory and then also swaps a lot (several gigabytes but less than half of the available swap) before dying with the above error.

@vicuna
Copy link
Author

vicuna commented Apr 28, 2017

Comment author: nekketsuuu

I also failed to run opam install camlp4.4.04+1 with ulimit -s unlimited.

Environment: 4.04.0+spacetime, opam 1.2.2, Ubuntu 16.10, x86_64, MemTotal 16GB (MemFree was 11GB at the moment), SwapTotal 4GB (SwapFree 3.6GB) [according to /proc/meminfo]

The following is the error message.

#=== ERROR while installing camlp4.4.04+1 =====================================#
# opam-version 1.2.2
# os           linux
# command      make all
# path         /home/nek/.opam/4.04.0+spacetime/build/camlp4.4.04+1
# compiler     4.04.0+spacetime
# exit-code    2
# env-file     /home/nek/.opam/4.04.0+spacetime/build/camlp4.4.04+1/camlp4-24633-ad8886.env
# stdout-file  /home/nek/.opam/4.04.0+spacetime/build/camlp4.4.04+1/camlp4-24633-ad8886.out
# stderr-file  /home/nek/.opam/4.04.0+spacetime/build/camlp4.4.04+1/camlp4-24633-ad8886.err
### stdout ###
# [...]
# /home/nek/.opam/4.04.0+spacetime/bin/ocamldep.opt -modules camlp4/config/Camlp4_import.ml > camlp4/config/Camlp4_import.ml.depends
# /home/nek/.opam/4.04.0+spacetime/bin/ocamlc.opt -c -g -safe-string -w Z -I camlp4/import -I camlp4/config -I camlp4 -o camlp4/config/Camlp4_import.cmo camlp4/config/Camlp4_import.ml
# /home/nek/.opam/4.04.0+spacetime/bin/ocamlopt.opt -c -g -safe-string -w Z -I camlp4/import -I camlp4/config -I camlp4 -o camlp4/config/Camlp4_import.cmx camlp4/config/Camlp4_import.ml
# /home/nek/.opam/4.04.0+spacetime/bin/ocamlopt.opt -c -g -safe-string -w Z -I camlp4/import -I camlp4/config -I camlp4 -o camlp4/config/Camlp4_config.cmx camlp4/config/Camlp4_config.ml
# /home/nek/.opam/4.04.0+spacetime/bin/ocamlopt.opt -c -g -safe-string -w a -I camlp4/import -I camlp4/config -I camlp4/boot -o camlp4/boot/Camlp4.cmx camlp4/boot/Camlp4.ml
# /home/nek/.opam/4.04.0+spacetime/bin/ocamlopt.opt -c -g -safe-string -w a -I camlp4/import -I camlp4/config -I camlp4/boot -o camlp4/boot/camlp4boot.cmx camlp4/boot/camlp4boot.ml
# + /home/nek/.opam/4.04.0+spacetime/bin/ocamlopt.opt -c -g -safe-string -w a -I camlp4/import -I camlp4/config -I camlp4/boot -o camlp4/boot/camlp4boot.cmx camlp4/boot/camlp4boot.ml
# Fatal error: exception Stack overflow
# Command exited with code 2.
# Makefile:13: recipe for target 'byte' failed
### stderr ###
# + echo camlp4/Camlp4.cmo camlp4/Camlp4Top.cmo camlp4/camlp4prof.byte camlp4/mkcamlp4.byte camlp4/camlp4.byte camlp4/camlp4fulllib.cma camlp4/camlp4boot.byte camlp4/camlp4boot.cma camlp4/camlp4r.byte camlp4/camlp4r.cma camlp4/camlp4rf.byte camlp4/camlp4rf.cma camlp4/camlp4o.byte camlp4/camlp4o.cma camlp4/camlp4of.byte camlp4/camlp4of.cma camlp4/camlp4oof.byte camlp4/camlp4oof.cma camlp4/camlp4orf.byte camlp4/camlp4orf.cma camlp4/Camlp4Parsers/Camlp4AstLoader.cmo camlp4/Camlp4Parsers/Camlp4DebugParser.cmo camlp4/Camlp4Parsers/Camlp4GrammarParser.cmo camlp4/Camlp4Parsers/Camlp4ListComprehension.cmo camlp4/Camlp4Parsers/Camlp4MacroParser.cmo camlp4/Camlp4Parsers/Camlp4OCamlOriginalQuotationExpander.cmo camlp4/Camlp4Parsers/Camlp4OCamlParser.cmo camlp4/Camlp4Parsers/Camlp4OCamlParserParser.cmo camlp4/Camlp4Parsers/Camlp4OCamlReloadedParser.cmo camlp4/Camlp4Parsers/Camlp4OCamlRevisedParser.cmo camlp4/Camlp4Parsers/Camlp4OCamlRevisedParserParser.cmo camlp4/Camlp4Parsers/Camlp4OCamlRevisedQuotationExpander.cmo camlp4/Camlp4Parsers/Camlp4QuotationCommon.cmo camlp4/Camlp4Parsers/Camlp4QuotationExpander.cmo camlp4/Camlp4Printers/Camlp4AstDumper.cmo camlp4/Camlp4Printers/Camlp4AutoPrinter.cmo camlp4/Camlp4Printers/Camlp4NullDumper.cmo camlp4/Camlp4Printers/Camlp4OCamlAstDumper.cmo camlp4/Camlp4Printers/Camlp4OCamlPrinter.cmo camlp4/Camlp4Printers/Camlp4OCamlRevisedPrinter.cmo camlp4/Camlp4Filters/Camlp4AstLifter.cmo camlp4/Camlp4Filters/Camlp4ExceptionTracer.cmo camlp4/Camlp4Filters/Camlp4FoldGenerator.cmo camlp4/Camlp4Filters/Camlp4LocationStripper.cmo camlp4/Camlp4Filters/Camlp4MapGenerator.cmo camlp4/Camlp4Filters/Camlp4MetaGenerator.cmo camlp4/Camlp4Filters/Camlp4Profiler.cmo camlp4/Camlp4Filters/Camlp4TrashRemover.cmo
# make: *** [byte] Error 10

Finally, I succeeded to install it with MemFree=14GB, SwapFree=4GB and ulimit -s unlimited.

@vicuna
Copy link
Author

vicuna commented Oct 24, 2018

Comment author: rand

This also happens for other packages, like tgls, with 4.06.0+spacetime . Works with ulimit -s unlimited

@bluddy
Copy link
Contributor

bluddy commented Aug 23, 2019

Confirmed this still exists on 4.08.0+spacetime. ulimit works, and memory consumption went up to 10GB as well.

@XVilka
Copy link
Contributor

XVilka commented May 17, 2020

With the official end of Camlp4 it is not so relevant anymore. On the other hand it's still a bug in the compiler, independently from the Camlp4 status. It makes sense to minify the reproducer.

@gasche
Copy link
Member

gasche commented May 17, 2020

It is known that various parts of the backend are not tail-recursive and that we can get stack overflow if fed large enough programs. This is fine as long as the large programs are artificial examples, and we have decided accept the price if the large program is the result of a program generator that would like to use OCaml as a backend. We have a problem if human-written programs hit these limits, and they cannot easily rewritten into a less stack-hungry approach.

We have known for a while that Camlp4 is uncomfortably close to the limit. It passes, but variants of it (when compiled under a small machine, or using Flambda with aggressive optimizations, or spacetime) blow up. This is not ideal, but this is a known issue that no one is actively working on right now. (Working on it would mean finding ways to make the stack consumption of the compiler backend constant-time without hurting readability and maintainability; this should be easier now than it was before thanks to let-operators that can make CPS palatable.)

Finally: Spacetime does not receive a lot of testing, and we are hoping to reduce the need for it by introducing a new memory profiler, Statmemprof, that should be more usable in practice. Hopefully this particular source of issue will be gone when the transition happens -- possibly around OCaml 4.11, which should be released in a few months.

With all that, I think that we can close the issue -- as not something that will result in specific compiler work in the foreseeable future.

@gasche gasche closed this as completed May 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants