|Anonymous | Login | Signup for a new account||2015-08-28 02:22 CEST|
|Main | My View | View Issues | Change Log | Roadmap|
|View Issue Details|
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0004243||OCaml||OCaml general||public||2007-03-27 14:30||2013-07-24 16:40|
|Target Version||4.02.0+dev||Fixed in Version||4.02.0+dev|
|Summary||0004243: Dependencies are missing in Makefiles|
Loads of dependencies are missing in Makefiles (and mainly the main one of ocaml).
It works as long as make executes the different targets sequentially but fails if it tries to do it in parallel. I wanted to add the missing dependencies but I don't know well enough the source of ocaml to do it (without adding to much dependencies).
|Additional Information||In Makefiles, the rule|
target: dep1 dep2
means that to build _target_ it has to build first _dep1_ and _dep2_ and then to execute _command_. Make can choose to build first _dep1_ and then _dep2_ (which it does by default) but it should also be able to build _dep2_ and then _dep1_ or both at the same time (if there is no other rule indicating some dependencies between the two). When giving the option -j to make, it tries to build several targets at the same time (especially useful with multi-processor and multi-core), which fails because of the missing dependencies.
|Tags||No tags attached.|
|Attached Files|| howtowritemakefile.pdf [^] (66,241 bytes) 2007-04-26 13:11|
ocaml-3.12.1-parallel-make-fixes.diff [^] (4,797 bytes) 2011-09-07 18:46 [Show Content]
0001-Poor-men-parallel-compilation.patch [^] (8,634 bytes) 2013-07-04 15:38 [Show Content]
Poor-men-parallel-compilation_simplified.patch [^] (5,060 bytes) 2013-07-09 18:24 [Show Content]
0001-Fix-parallel-compilation-of-opt-version-of-stdlib.patch [^] (637 bytes) 2013-07-09 23:06 [Show Content]
Not sure how to find the missing dependencies...
|I've added a file trying to sum up and explain how to modify the makefile so it works. To test if it works, one can run make -j. Currently, it fails, and if it works it's a good point but it does not mean there is no dependency missing.|
It's a bit more subtle than you think, but I guess I can try
to make the Makefile parallelizable. As this is not a vital
feature, it won't be available before version 3.11.
After spending a few hours on the makefiles, I am not optimistic,
for the following reasons:
1. Your method introduces too many dependencies, which will lead
to too much recompilation when developing OCaml. For example
everything would depend on coldstart, which would get remade
all the time.
2. There are a lot of makefiles to modify.
3. You won't get a big speedup anyway, because most of the time
is spent in ocamlbuild, which doesn't build in parallel.
4. As always with concurrent stuff, this is hell to debug and
testing doesn't give even a small amount of assurance (unlike
I had a try at parallelizing OCaml's Makefiles a couple of years ago. I believe a simpler way to do this is by using recursive invocations of "make" for the top-level targets, e.g.
all: runtime ocamlc ...
You get a little less parallelism this way, but a lot less trouble also.
Damien is right that most of the time is spend ocamlbuild-ing Camlp4. ocamlbuild does support parallel make (the undocumented -j N option), but extracts less parallelism than make, because ocamlbuild discovers the dependency graph on the fly during compilation, rather than knowing it in advance. Also, even assuming GNU make, there is no good way to extract whatever -j option the user gave to "make" and pass it back to "ocamlbuild".
All in all, I doubt this issue is worth investing significant effort.
I've tried to get ocaml 3.10.2 (targets world, opt, opt.opt) to compile with -j (thus, no limit to the number of jobs) as well. Here's what my conclusions are:
A lot of dependencies are missing. Some of these include:
* everything but the coldstart target needs a working ocamlrun. if you run `make -j world`, you'll have make work on the targets 'coldstart' and 'all' in parallel, since 'all' does not depend on coldstart itself. You'll get an error here because there's no ocamlrun.
* similarly, a lot of stuff needs ocamlc but doesn't depend on it. You'll get an error here as well.
* opt itself does not depend on ocamlopt, contrary to what the makefile says, but some of its subtargets do. so e.g. libraryopt should but doesn't. you can see this easily e.g. this way: 
* a lot of stuff needs ocamlmklib but the makefiles don't reflect that. ocamlmklib is a problem as you can see here:  (the error that you see there will not become a problem with -j1; with -j3 or even -j it will, though).
* the debugger needs libraries from otherlibs/unix
Another parellization problem I've come across, is in the asmrun Makefile. I'll quote parts of it here:
COBJS=startup.o main.o fail.o roots.o globroots.o signals.o signals_asm.o \
misc.o freelist.o major_gc.o minor_gc.o memory.o alloc.o compare.o ints.o \
floats.o str.o array.o io.o extern.o intern.o hash.o sys.o parsing.o \
gc_ctrl.o terminfo.o md5.o obj.o lexing.o printexc.o callback.o weak.o \
compact.o finalise.o custom.o unix.o backtrace.o
rm -f libasmrun.a
ar rc libasmrun.a $(OBJS)
That looks good so far. However, with -j I sometimes get an error by 'ar' that roots.o does not exist. That should be impossible because when ar for libasmrun.a is invoked, everything in $(OBJS), thus $(COBJS) and especially roots.o should have been built. The problem is further down in the makefile:
@ if test -f $*.o; then mv $*.o $*.f.o; else :; fi
$(CC) -c $(PFLAGS) $<
mv $*.o $*.p.o
@ if test -f $*.f.o; then mv $*.f.o $*.o; else :; fi
Thus, roots.o will be created by one job. Another job sees that it's there and starts running the libasmrun.a target. During that, the first job will mv roots.o to roots.p.o and when the second job reaches the 'ar' invocation, roots.o will be gone, yielding an error, as in .
 pipping@pfirsich ~/ocaml-3.10.2 $ make -n libraryopt
cd stdlib; make allopt
make: Entering directory `/home/pipping/ocaml-3.10.2/stdlib'
make: *** No rule to make target `../ocamlopt', needed by `pervasives.cmx'. Stop.
make: Leaving directory `/home/pipping/ocaml-3.10.2/stdlib'
make: *** [libraryopt] Error 2
 pipping@pfirsich ~/ocaml-3.10.2/tools $ make -n ocamlmklib
cp ../myocamlbuild_config.ml .
../boot/ocamlrun ../boot/ocamlc -nostdlib -I ../boot -c -warn-error A -I ../utils -I ../parsing -I ../typing -I ../bytecomp -I ../asmcomp -I ../driver myocamlbuild_config.ml
echo '(* THIS FILE IS GENERATED FROM ocamlmklib.mlp *)' >ocamlmklib.ml
sed -e "s|%%BINDIR%%|/usr/local/bin|" \
-e "s|%%SUPPORTS_SHARED_LIBRARIES%%|true|" \
-e "s|%%MKSHAREDLIB%%||" \
-e "s|%%BYTECCRPATH%%|-Wl,-rpath,|" \
-e "s|%%NATIVECCRPATH%%|-Wl,-rpath,|" \
-e "s|%%MKSHAREDLIBRPATH%%|-Wl,-rpath,|" \
-e "s|%%RANLIB%%|ranlib|" \
ocamlmklib.mlp >> ocamlmklib.ml
make: *** No rule to make target `myocamlbuild_config.cmi', needed by `ocamlmklib.cmo'. Stop.
 pipping@pfirsich ~/ocaml-3.10.2/asmrun $ make -j
[ snip ]
rm -f libasmrun.a
ar rc libasmrun.a startup.o main.o fail.o roots.o globroots.o signals.o signals_asm.o misc.o freelist.o major_gc.o minor_gc.o memory.o alloc.o compare.o ints.o floats.o str.o array.o io.o extern.o intern.o hash.o sys.o parsing.o gc_ctrl.o terminfo.o md5.o obj.o lexing.o printexc.o callback.o weak.o compact.o finalise.o custom.o unix.o backtrace.o amd64.o
ar: startup.o: No such file or directory
make: *** [libasmrun.a] Error 1
|I've just committed in trunk a Makefile fix for the .c.p.o problem.|
|I've just attached a patch that resolves several obvious issues in the Makefile's w/o introducing too many dependencies. At least world and opt do now work with -j most of the time (there are two issues left, which I'm trying to figure out).|
I have just done some benchmark on Xavier proposition. On a dual core, without compiling camlp4, the speed-up is almost x2. When camlp4 is also compiled, the gain falls to 25%.
In the attached patch, the compilation of camlp4 does not use parallelism, but camlp4 is compiled in parallel of 'otherlibs', ocamldoc and others independent tools.
The patch also includes some small fixes on dependencies computation for ocamldoc and labltk.
Here are some timings on a recent Intel processor, hyper-threaded dual-core:
make world 74.08s user 3.98s system 92% cpu 1:24.07 total
make -j4 world 95.23s user 4.15s system 259% cpu 38.360 total
make world.opt 126.31s user 7.42s system 92% cpu 2:23.97 total
make -j4 world.opt 154.46s user 7.70s system 214% cpu 1:15.47 total
make world 163.88s user 6.17s system 106% cpu 2:39.55 total
make -j4 world 173.76s user 6.15s system 161% cpu 1:51.68 total
make opt 47.98s user 2.20s system 93% cpu 53.477 total
make -j4 opt 55.46s user 2.28s system 157% cpu 36.695 total
make world.opt 225.33s user 11.59s system 98% cpu 4:01.13 total
make -j4 world.opt 253.00s user 12.36s system 153% cpu 2:52.55 total
I regularly use the -no-camlp4 configure option when I plan frequent build/test cycles of the compiler sources for some development purpose (this will be more widely applicable as findlib doesn't depend on Camlp4 anymore).
I am therefore fairly interested in solutions even if they don't tackle the Camlp4 issue. Any review of the change by someone that actually understands makefiles would be appreciated.
One more data point - also 2 cores 4 threads. More than 2x without camlp4, 25% full build.
make world 42.54 user 4.30 system 0:50.90 elapsed 92%CPU
make -j4 world 50.28 user 3.42 system 0:23.89 elapsed 224%CPU
make world.opt 81.83 user 8.92 system 1:40.68 elapsed 90%CPU
make -j4 world.opt 93.32 user 7.28 system 0:45.40 elapsed 221%CPU
make world 109.24 user 8.58 system 2:08.68 elapsed 91%CPU
make -j4 world 109.48 user 6.36 system 1:29.23 elapsed 129%CPU
make world.opt 144.94 user 13.65 system 2:53.49 elapsed 91%CPU
make -j4 world.opt 160.50 user 12.53 system 2:06.01 elapsed 137%CPU
|I am also interested in changing the build system of camlp4, it is still valuable even if the patch does not tackle the camlp4 issue|
I have two questions about this latest patch:
+ $(MAKE) otherlibrariesopt ocamltoolsopt
+ $(MAKE) ocamllex.opt ocamltoolsopt.opt ocamldoc.opt \
+ ocamlbuild.native $(CAMLP4OPT)
In the original, ocamllex.opt is placed between otherlibrariesopt and ocamltoolsopt. Is there a reason to move it down?
+ -I generators
This is in ocamldoc. You are adding an include directory. Is it needed to parallelize the makefiles, or is it a fix for an unrelated problem?
> In the original, ocamllex.opt is placed between otherlibrariesopt and ocamltoolsopt. Is there a reason to move it down?
No real reason. I just wanted to allow as many items as possible to compile in parallel of camlp4, and for wrong reasons I temporary thought that 'ocamltoolsopt' and 'ocamltoolsopt.opt' could not be compiled in parallel. Then, I forgot to restore the original order. One can safely do:
+ $(MAKE) otherlibrariesopt
+ $(MAKE) ocamllex.opt ocamltoolsopt ocamltoolsopt.opt ocamldoc.opt \
+ ocamlbuild.native $(CAMLP4OPT)
> This is in ocamldoc. You are adding an include directory. Is it needed to parallelize the makefiles, or is it a fix for an unrelated problem?
Neither. Once again I forgot to remove that one. This was a first tentative fix for ocamldoc parallel compilation. This is not required when we modify the 'all' rule in ocamldoc/Makefile.
I have attached a simplified patch.
There was a small problem remaining for compiling the rule libraryopt:
Compiling profiling and normal version must be sequential.
Also the commited stdlib/.depend in the repository trunk is not up to date: there are no dependencies for .p.cmx rules. A make depend is required
Last two patches (...simplified and ...of-stdlib) applied to trunk, stdlib dependencies remade (rev 13931).
|2007-03-27 14:30||laurent||New Issue|
|2007-04-23 16:41||doligez||Note Added: 0004033|
|2007-04-23 16:41||doligez||Status||new => acknowledged|
|2007-04-26 13:11||laurent||File Added: howtowritemakefile.pdf|
|2007-04-26 13:14||laurent||Note Added: 0004036|
|2007-11-22 11:37||doligez||Note Added: 0004341|
|2007-11-22 11:37||doligez||Assigned To||=> doligez|
|2007-11-22 11:37||doligez||Projection||none => minor fix|
|2007-11-22 11:37||doligez||ETA||none => > 1 month|
|2007-11-22 17:43||doligez||Note Added: 0004342|
|2007-11-24 18:43||xleroy||Note Added: 0004347|
|2009-06-07 11:37||pipping||Note Added: 0004985|
|2009-06-08 18:13||doligez||Relationship added||has duplicate 0004657|
|2011-01-06 15:25||doligez||Note Added: 0005760|
|2011-01-06 15:25||doligez||Category||Incoming => OCaml general|
|2011-01-06 15:27||doligez||Note Added: 0005761|
|2011-01-06 15:28||doligez||Note Deleted: 0005761|
|2011-09-07 18:46||meurer||File Added: ocaml-3.12.1-parallel-make-fixes.diff|
|2011-09-07 18:47||meurer||Note Added: 0006120|
|2012-09-06 16:43||doligez||Target Version||=> 4.00.1+dev|
|2012-09-14 23:56||doligez||Target Version||4.00.1+dev => 4.01.0+dev|
|2013-07-04 15:37||hnrgrgr||Note Added: 0009690|
|2013-07-04 15:38||hnrgrgr||File Added: 0001-Poor-men-parallel-compilation.patch|
|2013-07-04 15:45||gasche||Note Added: 0009691|
|2013-07-05 05:53||ygrek||Note Added: 0009696|
|2013-07-07 16:50||hongboz||Note Added: 0009717|
|2013-07-09 16:41||doligez||Note Added: 0009735|
|2013-07-09 18:23||hnrgrgr||Note Added: 0009737|
|2013-07-09 18:24||hnrgrgr||File Added: Poor-men-parallel-compilation_simplified.patch|
|2013-07-09 23:06||chambart||File Added: 0001-Fix-parallel-compilation-of-opt-version-of-stdlib.patch|
|2013-07-09 23:11||chambart||Note Added: 0009738|
|2013-07-24 16:40||doligez||Note Added: 0009856|
|2013-07-24 16:40||doligez||Status||acknowledged => resolved|
|2013-07-24 16:40||doligez||Resolution||open => fixed|
|2013-07-24 16:40||doligez||Fixed in Version||=> 4.02.0+dev|
|2013-07-24 16:40||doligez||Target Version||4.01.0+dev => 4.02.0+dev|
|Copyright © 2000 - 2011 MantisBT Group|