0004243: Dependencies are missing in Makefiles

Notes
(0004033) doligez (manager) 2007-04-23 16:41	Not sure how to find the missing dependencies...

(0004036) laurent (reporter) 2007-04-26 13:14	I've added a file trying to sum up and explain how to modify the makefile so it works. To test if it works, one can run make -j. Currently, it fails, and if it works it's a good point but it does not mean there is no dependency missing.

(0004341) doligez (manager) 2007-11-22 11:37	It's a bit more subtle than you think, but I guess I can try to make the Makefile parallelizable. As this is not a vital feature, it won't be available before version 3.11.

(0004342) doligez (manager) 2007-11-22 17:43	After spending a few hours on the makefiles, I am not optimistic, for the following reasons: 1. Your method introduces too many dependencies, which will lead to too much recompilation when developing OCaml. For example everything would depend on coldstart, which would get remade all the time. 2. There are a lot of makefiles to modify. 3. You won't get a big speedup anyway, because most of the time is spent in ocamlbuild, which doesn't build in parallel. 4. As always with concurrent stuff, this is hell to debug and testing doesn't give even a small amount of assurance (unlike sequential stuff).

(0004347) xleroy (administrator) 2007-11-24 18:43	I had a try at parallelizing OCaml's Makefiles a couple of years ago. I believe a simpler way to do this is by using recursive invocations of "make" for the top-level targets, e.g. all: $(MAKE) runtime $(MAKE) ocamlc ... instead of all: runtime ocamlc ... You get a little less parallelism this way, but a lot less trouble also. Damien is right that most of the time is spend ocamlbuild-ing Camlp4. ocamlbuild does support parallel make (the undocumented -j N option), but extracts less parallelism than make, because ocamlbuild discovers the dependency graph on the fly during compilation, rather than knowing it in advance. Also, even assuming GNU make, there is no good way to extract whatever -j option the user gave to "make" and pass it back to "ocamlbuild". All in all, I doubt this issue is worth investing significant effort.

(0004985) pipping (reporter) 2009-06-07 11:37	I've tried to get ocaml 3.10.2 (targets world, opt, opt.opt) to compile with -j (thus, no limit to the number of jobs) as well. Here's what my conclusions are: A lot of dependencies are missing. Some of these include: * everything but the coldstart target needs a working ocamlrun. if you run `make -j world`, you'll have make work on the targets 'coldstart' and 'all' in parallel, since 'all' does not depend on coldstart itself. You'll get an error here because there's no ocamlrun. * similarly, a lot of stuff needs ocamlc but doesn't depend on it. You'll get an error here as well. * opt itself does not depend on ocamlopt, contrary to what the makefile says, but some of its subtargets do. so e.g. libraryopt should but doesn't. you can see this easily e.g. this way: [1] * a lot of stuff needs ocamlmklib but the makefiles don't reflect that. ocamlmklib is a problem as you can see here: [2] (the error that you see there will not become a problem with -j1; with -j3 or even -j it will, though). * the debugger needs libraries from otherlibs/unix Another parellization problem I've come across, is in the asmrun Makefile. I'll quote parts of it here: COBJS=startup.o main.o fail.o roots.o globroots.o signals.o signals_asm.o \ misc.o freelist.o major_gc.o minor_gc.o memory.o alloc.o compare.o ints.o \ floats.o str.o array.o io.o extern.o intern.o hash.o sys.o parsing.o \ gc_ctrl.o terminfo.o md5.o obj.o lexing.o printexc.o callback.o weak.o \ compact.o finalise.o custom.o unix.o backtrace.o ASMOBJS=$(ARCH).o OBJS=$(COBJS) $(ASMOBJS) libasmrun.a: $(OBJS) rm -f libasmrun.a ar rc libasmrun.a $(OBJS) $(RANLIB) libasmrun.a That looks good so far. However, with -j I sometimes get an error by 'ar' that roots.o does not exist. That should be impossible because when ar for libasmrun.a is invoked, everything in $(OBJS), thus $(COBJS) and especially roots.o should have been built. The problem is further down in the makefile: .c.p.o: @ if test -f $.o; then mv $.o $.f.o; else :; fi $(CC) -c $(PFLAGS) $< mv $.o $.p.o @ if test -f $.f.o; then mv $.f.o $.o; else :; fi Thus, roots.o will be created by one job. Another job sees that it's there and starts running the libasmrun.a target. During that, the first job will mv roots.o to roots.p.o and when the second job reaches the 'ar' invocation, roots.o will be gone, yielding an error, as in [3]. [1] pipping@pfirsich ~/ocaml-3.10.2 $ make -n libraryopt cd stdlib; make allopt make[1]: Entering directory `/home/pipping/ocaml-3.10.2/stdlib' make[1]: * No rule to make target `../ocamlopt', needed by `pervasives.cmx'. Stop. make[1]: Leaving directory `/home/pipping/ocaml-3.10.2/stdlib' make: * [libraryopt] Error 2 [2] pipping@pfirsich ~/ocaml-3.10.2/tools $ make -n ocamlmklib ../build/mkmyocamlbuild_config.sh cp ../myocamlbuild_config.ml . ../boot/ocamlrun ../boot/ocamlc -nostdlib -I ../boot -c -warn-error A -I ../utils -I ../parsing -I ../typing -I ../bytecomp -I ../asmcomp -I ../driver myocamlbuild_config.ml echo '(* THIS FILE IS GENERATED FROM ocamlmklib.mlp )' >ocamlmklib.ml sed -e "s\|%%BINDIR%%\|/usr/local/bin\|" \ -e "s\|%%SUPPORTS_SHARED_LIBRARIES%%\|true\|" \ -e "s\|%%MKSHAREDLIB%%\|\|" \ -e "s\|%%BYTECCRPATH%%\|-Wl,-rpath,\|" \ -e "s\|%%NATIVECCRPATH%%\|-Wl,-rpath,\|" \ -e "s\|%%MKSHAREDLIBRPATH%%\|-Wl,-rpath,\|" \ -e "s\|%%RANLIB%%\|ranlib\|" \ ocamlmklib.mlp >> ocamlmklib.ml make: No rule to make target `myocamlbuild_config.cmi', needed by `ocamlmklib.cmo'. Stop. [3] pipping@pfirsich ~/ocaml-3.10.2/asmrun $ make -j [ snip ] rm -f libasmrun.a ar rc libasmrun.a startup.o main.o fail.o roots.o globroots.o signals.o signals_asm.o misc.o freelist.o major_gc.o minor_gc.o memory.o alloc.o compare.o ints.o floats.o str.o array.o io.o extern.o intern.o hash.o sys.o parsing.o gc_ctrl.o terminfo.o md5.o obj.o lexing.o printexc.o callback.o weak.o compact.o finalise.o custom.o unix.o backtrace.o amd64.o ar: startup.o: No such file or directory make: * [libasmrun.a] Error 1

(0005760) doligez (manager) 2011-01-06 15:25	I've just committed in trunk a Makefile fix for the .c.p.o problem.

(0006120) meurer (developer) 2011-09-07 18:47	I've just attached a patch that resolves several obvious issues in the Makefile's w/o introducing too many dependencies. At least world and opt do now work with -j most of the time (there are two issues left, which I'm trying to figure out).

Issue History
Date Modified	Username	Field	Change
2007-03-27 14:30	laurent	New Issue
2007-04-23 16:41	doligez	Note Added: 0004033
2007-04-23 16:41	doligez	Status	new => acknowledged
2007-04-26 13:11	laurent	File Added: howtowritemakefile.pdf
2007-04-26 13:14	laurent	Note Added: 0004036
2007-11-22 11:37	doligez	Note Added: 0004341
2007-11-22 11:37	doligez	Assigned To	=> doligez
2007-11-22 11:37	doligez	Projection	none => minor fix
2007-11-22 11:37	doligez	ETA	none => > 1 month
2007-11-22 17:43	doligez	Note Added: 0004342
2007-11-24 18:43	xleroy	Note Added: 0004347
2009-06-07 11:37	pipping	Note Added: 0004985
2009-06-08 18:13	doligez	Relationship added	has duplicate 0004657
2011-01-06 15:25	doligez	Note Added: 0005760
2011-01-06 15:25	doligez	Category	Incoming => OCaml general
2011-01-06 15:27	doligez	Note Added: 0005761
2011-01-06 15:28	doligez	Note Deleted: 0005761
2011-09-07 18:46	meurer	File Added: ocaml-3.12.1-parallel-make-fixes.diff
2011-09-07 18:47	meurer	Note Added: 0006120
2012-09-06 16:43	doligez	Target Version	=> 4.00.1+dev
2012-09-14 23:56	doligez	Target Version	4.00.1+dev => 4.01.0+dev