|Anonymous | Login | Signup for a new account||2013-05-23 15:15 CEST|
|Main | My View | View Issues | Change Log | Roadmap|
|View Issue Details|
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0004243||OCaml||OCaml general||public||2007-03-27 14:30||2012-09-14 23:56|
|Target Version||4.01.0+dev||Fixed in Version|
|Summary||0004243: Dependencies are missing in Makefiles|
Loads of dependencies are missing in Makefiles (and mainly the main one of ocaml).
It works as long as make executes the different targets sequentially but fails if it tries to do it in parallel. I wanted to add the missing dependencies but I don't know well enough the source of ocaml to do it (without adding to much dependencies).
|Additional Information||In Makefiles, the rule|
target: dep1 dep2
means that to build _target_ it has to build first _dep1_ and _dep2_ and then to execute _command_. Make can choose to build first _dep1_ and then _dep2_ (which it does by default) but it should also be able to build _dep2_ and then _dep1_ or both at the same time (if there is no other rule indicating some dependencies between the two). When giving the option -j to make, it tries to build several targets at the same time (especially useful with multi-processor and multi-core), which fails because of the missing dependencies.
|Tags||No tags attached.|
|Attached Files|| howtowritemakefile.pdf [^] (66,241 bytes) 2007-04-26 13:11|
ocaml-3.12.1-parallel-make-fixes.diff [^] (4,797 bytes) 2011-09-07 18:46 [Show Content]
Not sure how to find the missing dependencies...
|I've added a file trying to sum up and explain how to modify the makefile so it works. To test if it works, one can run make -j. Currently, it fails, and if it works it's a good point but it does not mean there is no dependency missing.|
It's a bit more subtle than you think, but I guess I can try
to make the Makefile parallelizable. As this is not a vital
feature, it won't be available before version 3.11.
After spending a few hours on the makefiles, I am not optimistic,
for the following reasons:
1. Your method introduces too many dependencies, which will lead
to too much recompilation when developing OCaml. For example
everything would depend on coldstart, which would get remade
all the time.
2. There are a lot of makefiles to modify.
3. You won't get a big speedup anyway, because most of the time
is spent in ocamlbuild, which doesn't build in parallel.
4. As always with concurrent stuff, this is hell to debug and
testing doesn't give even a small amount of assurance (unlike
I had a try at parallelizing OCaml's Makefiles a couple of years ago. I believe a simpler way to do this is by using recursive invocations of "make" for the top-level targets, e.g.
all: runtime ocamlc ...
You get a little less parallelism this way, but a lot less trouble also.
Damien is right that most of the time is spend ocamlbuild-ing Camlp4. ocamlbuild does support parallel make (the undocumented -j N option), but extracts less parallelism than make, because ocamlbuild discovers the dependency graph on the fly during compilation, rather than knowing it in advance. Also, even assuming GNU make, there is no good way to extract whatever -j option the user gave to "make" and pass it back to "ocamlbuild".
All in all, I doubt this issue is worth investing significant effort.
I've tried to get ocaml 3.10.2 (targets world, opt, opt.opt) to compile with -j (thus, no limit to the number of jobs) as well. Here's what my conclusions are:
A lot of dependencies are missing. Some of these include:
* everything but the coldstart target needs a working ocamlrun. if you run `make -j world`, you'll have make work on the targets 'coldstart' and 'all' in parallel, since 'all' does not depend on coldstart itself. You'll get an error here because there's no ocamlrun.
* similarly, a lot of stuff needs ocamlc but doesn't depend on it. You'll get an error here as well.
* opt itself does not depend on ocamlopt, contrary to what the makefile says, but some of its subtargets do. so e.g. libraryopt should but doesn't. you can see this easily e.g. this way: 
* a lot of stuff needs ocamlmklib but the makefiles don't reflect that. ocamlmklib is a problem as you can see here:  (the error that you see there will not become a problem with -j1; with -j3 or even -j it will, though).
* the debugger needs libraries from otherlibs/unix
Another parellization problem I've come across, is in the asmrun Makefile. I'll quote parts of it here:
COBJS=startup.o main.o fail.o roots.o globroots.o signals.o signals_asm.o \
misc.o freelist.o major_gc.o minor_gc.o memory.o alloc.o compare.o ints.o \
floats.o str.o array.o io.o extern.o intern.o hash.o sys.o parsing.o \
gc_ctrl.o terminfo.o md5.o obj.o lexing.o printexc.o callback.o weak.o \
compact.o finalise.o custom.o unix.o backtrace.o
rm -f libasmrun.a
ar rc libasmrun.a $(OBJS)
That looks good so far. However, with -j I sometimes get an error by 'ar' that roots.o does not exist. That should be impossible because when ar for libasmrun.a is invoked, everything in $(OBJS), thus $(COBJS) and especially roots.o should have been built. The problem is further down in the makefile:
@ if test -f $*.o; then mv $*.o $*.f.o; else :; fi
$(CC) -c $(PFLAGS) $<
mv $*.o $*.p.o
@ if test -f $*.f.o; then mv $*.f.o $*.o; else :; fi
Thus, roots.o will be created by one job. Another job sees that it's there and starts running the libasmrun.a target. During that, the first job will mv roots.o to roots.p.o and when the second job reaches the 'ar' invocation, roots.o will be gone, yielding an error, as in .
 pipping@pfirsich ~/ocaml-3.10.2 $ make -n libraryopt
cd stdlib; make allopt
make: Entering directory `/home/pipping/ocaml-3.10.2/stdlib'
make: *** No rule to make target `../ocamlopt', needed by `pervasives.cmx'. Stop.
make: Leaving directory `/home/pipping/ocaml-3.10.2/stdlib'
make: *** [libraryopt] Error 2
 pipping@pfirsich ~/ocaml-3.10.2/tools $ make -n ocamlmklib
cp ../myocamlbuild_config.ml .
../boot/ocamlrun ../boot/ocamlc -nostdlib -I ../boot -c -warn-error A -I ../utils -I ../parsing -I ../typing -I ../bytecomp -I ../asmcomp -I ../driver myocamlbuild_config.ml
echo '(* THIS FILE IS GENERATED FROM ocamlmklib.mlp *)' >ocamlmklib.ml
sed -e "s|%%BINDIR%%|/usr/local/bin|" \
-e "s|%%SUPPORTS_SHARED_LIBRARIES%%|true|" \
-e "s|%%MKSHAREDLIB%%||" \
-e "s|%%BYTECCRPATH%%|-Wl,-rpath,|" \
-e "s|%%NATIVECCRPATH%%|-Wl,-rpath,|" \
-e "s|%%MKSHAREDLIBRPATH%%|-Wl,-rpath,|" \
-e "s|%%RANLIB%%|ranlib|" \
ocamlmklib.mlp >> ocamlmklib.ml
make: *** No rule to make target `myocamlbuild_config.cmi', needed by `ocamlmklib.cmo'. Stop.
 pipping@pfirsich ~/ocaml-3.10.2/asmrun $ make -j
[ snip ]
rm -f libasmrun.a
ar rc libasmrun.a startup.o main.o fail.o roots.o globroots.o signals.o signals_asm.o misc.o freelist.o major_gc.o minor_gc.o memory.o alloc.o compare.o ints.o floats.o str.o array.o io.o extern.o intern.o hash.o sys.o parsing.o gc_ctrl.o terminfo.o md5.o obj.o lexing.o printexc.o callback.o weak.o compact.o finalise.o custom.o unix.o backtrace.o amd64.o
ar: startup.o: No such file or directory
make: *** [libasmrun.a] Error 1
|I've just committed in trunk a Makefile fix for the .c.p.o problem.|
|I've just attached a patch that resolves several obvious issues in the Makefile's w/o introducing too many dependencies. At least world and opt do now work with -j most of the time (there are two issues left, which I'm trying to figure out).|
|2007-03-27 14:30||laurent||New Issue|
|2007-04-23 16:41||doligez||Note Added: 0004033|
|2007-04-23 16:41||doligez||Status||new => acknowledged|
|2007-04-26 13:11||laurent||File Added: howtowritemakefile.pdf|
|2007-04-26 13:14||laurent||Note Added: 0004036|
|2007-11-22 11:37||doligez||Note Added: 0004341|
|2007-11-22 11:37||doligez||Assigned To||=> doligez|
|2007-11-22 11:37||doligez||Projection||none => minor fix|
|2007-11-22 11:37||doligez||ETA||none => > 1 month|
|2007-11-22 17:43||doligez||Note Added: 0004342|
|2007-11-24 18:43||xleroy||Note Added: 0004347|
|2009-06-07 11:37||pipping||Note Added: 0004985|
|2009-06-08 18:13||doligez||Relationship added||has duplicate 0004657|
|2011-01-06 15:25||doligez||Note Added: 0005760|
|2011-01-06 15:25||doligez||Category||Incoming => OCaml general|
|2011-01-06 15:27||doligez||Note Added: 0005761|
|2011-01-06 15:28||doligez||Note Deleted: 0005761|
|2011-09-07 18:46||meurer||File Added: ocaml-3.12.1-parallel-make-fixes.diff|
|2011-09-07 18:47||meurer||Note Added: 0006120|
|2012-09-06 16:43||doligez||Target Version||=> 4.00.1+dev|
|2012-09-14 23:56||doligez||Target Version||4.00.1+dev => 4.01.0+dev|
|Copyright © 2000 - 2011 MantisBT Group|