Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0004243OCamlOCaml generalpublic2007-03-27 14:302013-07-24 16:40
Reporterlaurent 
Assigned Todoligez 
PrioritynormalSeveritytweakReproducibilityalways
StatusresolvedResolutionfixed 
PlatformOSOS Version
Product Version3.09.3 
Target Version4.02.0+devFixed in Version4.02.0+dev 
Summary0004243: Dependencies are missing in Makefiles
DescriptionHello,

Loads of dependencies are missing in Makefiles (and mainly the main one of ocaml).
It works as long as make executes the different targets sequentially but fails if it tries to do it in parallel. I wanted to add the missing dependencies but I don't know well enough the source of ocaml to do it (without adding to much dependencies).

Cheers,
Laurent.
Additional InformationIn Makefiles, the rule
target: dep1 dep2
    command
means that to build _target_ it has to build first _dep1_ and _dep2_ and then to execute _command_. Make can choose to build first _dep1_ and then _dep2_ (which it does by default) but it should also be able to build _dep2_ and then _dep1_ or both at the same time (if there is no other rule indicating some dependencies between the two). When giving the option -j to make, it tries to build several targets at the same time (especially useful with multi-processor and multi-core), which fails because of the missing dependencies.
TagsNo tags attached.
Attached Filespdf file icon howtowritemakefile.pdf [^] (66,241 bytes) 2007-04-26 13:11
diff file icon ocaml-3.12.1-parallel-make-fixes.diff [^] (4,797 bytes) 2011-09-07 18:46 [Show Content]
patch file icon 0001-Poor-men-parallel-compilation.patch [^] (8,634 bytes) 2013-07-04 15:38 [Show Content]
patch file icon Poor-men-parallel-compilation_simplified.patch [^] (5,060 bytes) 2013-07-09 18:24 [Show Content]
patch file icon 0001-Fix-parallel-compilation-of-opt-version-of-stdlib.patch [^] (637 bytes) 2013-07-09 23:06 [Show Content]

- Relationships
has duplicate 0004657resolved Compilation of ocaml fails in labltk with parallel jobs 

-  Notes
(0004033)
doligez (administrator)
2007-04-23 16:41

Not sure how to find the missing dependencies...
(0004036)
laurent (reporter)
2007-04-26 13:14

I've added a file trying to sum up and explain how to modify the makefile so it works. To test if it works, one can run make -j. Currently, it fails, and if it works it's a good point but it does not mean there is no dependency missing.
(0004341)
doligez (administrator)
2007-11-22 11:37

It's a bit more subtle than you think, but I guess I can try
to make the Makefile parallelizable. As this is not a vital
feature, it won't be available before version 3.11.
(0004342)
doligez (administrator)
2007-11-22 17:43

After spending a few hours on the makefiles, I am not optimistic,
for the following reasons:

1. Your method introduces too many dependencies, which will lead
   to too much recompilation when developing OCaml. For example
   everything would depend on coldstart, which would get remade
   all the time.

2. There are a lot of makefiles to modify.

3. You won't get a big speedup anyway, because most of the time
   is spent in ocamlbuild, which doesn't build in parallel.

4. As always with concurrent stuff, this is hell to debug and
   testing doesn't give even a small amount of assurance (unlike
   sequential stuff).
(0004347)
xleroy (administrator)
2007-11-24 18:43

I had a try at parallelizing OCaml's Makefiles a couple of years ago. I believe a simpler way to do this is by using recursive invocations of "make" for the top-level targets, e.g.

all:
   $(MAKE) runtime
   $(MAKE) ocamlc
   ...

instead of

all: runtime ocamlc ...

You get a little less parallelism this way, but a lot less trouble also.

Damien is right that most of the time is spend ocamlbuild-ing Camlp4. ocamlbuild does support parallel make (the undocumented -j N option), but extracts less parallelism than make, because ocamlbuild discovers the dependency graph on the fly during compilation, rather than knowing it in advance. Also, even assuming GNU make, there is no good way to extract whatever -j option the user gave to "make" and pass it back to "ocamlbuild".

All in all, I doubt this issue is worth investing significant effort.
(0004985)
pipping (reporter)
2009-06-07 11:37

I've tried to get ocaml 3.10.2 (targets world, opt, opt.opt) to compile with -j (thus, no limit to the number of jobs) as well. Here's what my conclusions are:

A lot of dependencies are missing. Some of these include:

 * everything but the coldstart target needs a working ocamlrun. if you run `make -j world`, you'll have make work on the targets 'coldstart' and 'all' in parallel, since 'all' does not depend on coldstart itself. You'll get an error here because there's no ocamlrun.
 * similarly, a lot of stuff needs ocamlc but doesn't depend on it. You'll get an error here as well.
 * opt itself does not depend on ocamlopt, contrary to what the makefile says, but some of its subtargets do. so e.g. libraryopt should but doesn't. you can see this easily e.g. this way: [1]
 * a lot of stuff needs ocamlmklib but the makefiles don't reflect that. ocamlmklib is a problem as you can see here: [2] (the error that you see there will not become a problem with -j1; with -j3 or even -j it will, though).
 * the debugger needs libraries from otherlibs/unix

Another parellization problem I've come across, is in the asmrun Makefile. I'll quote parts of it here:

COBJS=startup.o main.o fail.o roots.o globroots.o signals.o signals_asm.o \
  misc.o freelist.o major_gc.o minor_gc.o memory.o alloc.o compare.o ints.o \
  floats.o str.o array.o io.o extern.o intern.o hash.o sys.o parsing.o \
  gc_ctrl.o terminfo.o md5.o obj.o lexing.o printexc.o callback.o weak.o \
  compact.o finalise.o custom.o unix.o backtrace.o

ASMOBJS=$(ARCH).o

OBJS=$(COBJS) $(ASMOBJS)

libasmrun.a: $(OBJS)
        rm -f libasmrun.a
        ar rc libasmrun.a $(OBJS)
        $(RANLIB) libasmrun.a

That looks good so far. However, with -j I sometimes get an error by 'ar' that roots.o does not exist. That should be impossible because when ar for libasmrun.a is invoked, everything in $(OBJS), thus $(COBJS) and especially roots.o should have been built. The problem is further down in the makefile:

.c.p.o:
        @ if test -f $*.o; then mv $*.o $*.f.o; else :; fi
        $(CC) -c $(PFLAGS) $<
        mv $*.o $*.p.o
        @ if test -f $*.f.o; then mv $*.f.o $*.o; else :; fi

Thus, roots.o will be created by one job. Another job sees that it's there and starts running the libasmrun.a target. During that, the first job will mv roots.o to roots.p.o and when the second job reaches the 'ar' invocation, roots.o will be gone, yielding an error, as in [3].

[1] pipping@pfirsich ~/ocaml-3.10.2 $ make -n libraryopt
cd stdlib; make allopt
make[1]: Entering directory `/home/pipping/ocaml-3.10.2/stdlib'
make[1]: *** No rule to make target `../ocamlopt', needed by `pervasives.cmx'. Stop.
make[1]: Leaving directory `/home/pipping/ocaml-3.10.2/stdlib'
make: *** [libraryopt] Error 2

[2] pipping@pfirsich ~/ocaml-3.10.2/tools $ make -n ocamlmklib
../build/mkmyocamlbuild_config.sh
cp ../myocamlbuild_config.ml .
../boot/ocamlrun ../boot/ocamlc -nostdlib -I ../boot -c -warn-error A -I ../utils -I ../parsing -I ../typing -I ../bytecomp -I ../asmcomp -I ../driver myocamlbuild_config.ml
echo '(* THIS FILE IS GENERATED FROM ocamlmklib.mlp *)' >ocamlmklib.ml
sed -e "s|%%BINDIR%%|/usr/local/bin|" \
            -e "s|%%SUPPORTS_SHARED_LIBRARIES%%|true|" \
            -e "s|%%MKSHAREDLIB%%||" \
            -e "s|%%BYTECCRPATH%%|-Wl,-rpath,|" \
            -e "s|%%NATIVECCRPATH%%|-Wl,-rpath,|" \
            -e "s|%%MKSHAREDLIBRPATH%%|-Wl,-rpath,|" \
            -e "s|%%RANLIB%%|ranlib|" \
          ocamlmklib.mlp >> ocamlmklib.ml
make: *** No rule to make target `myocamlbuild_config.cmi', needed by `ocamlmklib.cmo'. Stop.

[3] pipping@pfirsich ~/ocaml-3.10.2/asmrun $ make -j
[ snip ]
rm -f libasmrun.a
ar rc libasmrun.a startup.o main.o fail.o roots.o globroots.o signals.o signals_asm.o misc.o freelist.o major_gc.o minor_gc.o memory.o alloc.o compare.o ints.o floats.o str.o array.o io.o extern.o intern.o hash.o sys.o parsing.o gc_ctrl.o terminfo.o md5.o obj.o lexing.o printexc.o callback.o weak.o compact.o finalise.o custom.o unix.o backtrace.o amd64.o
ar: startup.o: No such file or directory
make: *** [libasmrun.a] Error 1
(0005760)
doligez (administrator)
2011-01-06 15:25

I've just committed in trunk a Makefile fix for the .c.p.o problem.
(0006120)
meurer (developer)
2011-09-07 18:47

I've just attached a patch that resolves several obvious issues in the Makefile's w/o introducing too many dependencies. At least world and opt do now work with -j most of the time (there are two issues left, which I'm trying to figure out).
(0009690)
hnrgrgr (developer)
2013-07-04 15:37

I have just done some benchmark on Xavier proposition. On a dual core, without compiling camlp4, the speed-up is almost x2. When camlp4 is also compiled, the gain falls to 25%.

In the attached patch, the compilation of camlp4 does not use parallelism, but camlp4 is compiled in parallel of 'otherlibs', ocamldoc and others independent tools.

The patch also includes some small fixes on dependencies computation for ocamldoc and labltk.

Here are some timings on a recent Intel processor, hyper-threaded dual-core:

#####

./configure -no-camlp4

make world 74.08s user 3.98s system 92% cpu 1:24.07 total
make -j4 world 95.23s user 4.15s system 259% cpu 38.360 total

make world.opt 126.31s user 7.42s system 92% cpu 2:23.97 total
make -j4 world.opt 154.46s user 7.70s system 214% cpu 1:15.47 total

#####

./configure

make world 163.88s user 6.17s system 106% cpu 2:39.55 total
make -j4 world 173.76s user 6.15s system 161% cpu 1:51.68 total

make opt 47.98s user 2.20s system 93% cpu 53.477 total
make -j4 opt 55.46s user 2.28s system 157% cpu 36.695 total

make world.opt 225.33s user 11.59s system 98% cpu 4:01.13 total
make -j4 world.opt 253.00s user 12.36s system 153% cpu 2:52.55 total
(0009691)
gasche (developer)
2013-07-04 15:45

I regularly use the -no-camlp4 configure option when I plan frequent build/test cycles of the compiler sources for some development purpose (this will be more widely applicable as findlib doesn't depend on Camlp4 anymore).

I am therefore fairly interested in solutions even if they don't tackle the Camlp4 issue. Any review of the change by someone that actually understands makefiles would be appreciated.
(0009696)
ygrek (reporter)
2013-07-05 05:53

One more data point - also 2 cores 4 threads. More than 2x without camlp4, 25% full build.

./configure -no-camlp4

        make world 42.54 user 4.30 system 0:50.90 elapsed 92%CPU
    make -j4 world 50.28 user 3.42 system 0:23.89 elapsed 224%CPU

    make world.opt 81.83 user 8.92 system 1:40.68 elapsed 90%CPU
make -j4 world.opt 93.32 user 7.28 system 0:45.40 elapsed 221%CPU


./configure

        make world 109.24 user 8.58 system 2:08.68 elapsed 91%CPU
    make -j4 world 109.48 user 6.36 system 1:29.23 elapsed 129%CPU

    make world.opt 144.94 user 13.65 system 2:53.49 elapsed 91%CPU
make -j4 world.opt 160.50 user 12.53 system 2:06.01 elapsed 137%CPU
(0009717)
hongboz (developer)
2013-07-07 16:50

I am also interested in changing the build system of camlp4, it is still valuable even if the patch does not tackle the camlp4 issue
(0009735)
doligez (administrator)
2013-07-09 16:41

I have two questions about this latest patch:

+ $(MAKE) otherlibrariesopt ocamltoolsopt
+ $(MAKE) ocamllex.opt ocamltoolsopt.opt ocamldoc.opt \
+ ocamlbuild.native $(CAMLP4OPT)

In the original, ocamllex.opt is placed between otherlibrariesopt and ocamltoolsopt. Is there a reason to move it down?


+ -I generators

This is in ocamldoc. You are adding an include directory. Is it needed to parallelize the makefiles, or is it a fix for an unrelated problem?
(0009737)
hnrgrgr (developer)
2013-07-09 18:23

> In the original, ocamllex.opt is placed between otherlibrariesopt and ocamltoolsopt. Is there a reason to move it down?

No real reason. I just wanted to allow as many items as possible to compile in parallel of camlp4, and for wrong reasons I temporary thought that 'ocamltoolsopt' and 'ocamltoolsopt.opt' could not be compiled in parallel. Then, I forgot to restore the original order. One can safely do:

+ $(MAKE) otherlibrariesopt
+ $(MAKE) ocamllex.opt ocamltoolsopt ocamltoolsopt.opt ocamldoc.opt \
+ ocamlbuild.native $(CAMLP4OPT)

> This is in ocamldoc. You are adding an include directory. Is it needed to parallelize the makefiles, or is it a fix for an unrelated problem?

Neither. Once again I forgot to remove that one. This was a first tentative fix for ocamldoc parallel compilation. This is not required when we modify the 'all' rule in ocamldoc/Makefile.

I have attached a simplified patch.
(0009738)
chambart (developer)
2013-07-09 23:11

There was a small problem remaining for compiling the rule libraryopt:
Compiling profiling and normal version must be sequential.

Also the commited stdlib/.depend in the repository trunk is not up to date: there are no dependencies for .p.cmx rules. A make depend is required
(0009856)
doligez (administrator)
2013-07-24 16:40

Last two patches (...simplified and ...of-stdlib) applied to trunk, stdlib dependencies remade (rev 13931).

- Issue History
Date Modified Username Field Change
2007-03-27 14:30 laurent New Issue
2007-04-23 16:41 doligez Note Added: 0004033
2007-04-23 16:41 doligez Status new => acknowledged
2007-04-26 13:11 laurent File Added: howtowritemakefile.pdf
2007-04-26 13:14 laurent Note Added: 0004036
2007-11-22 11:37 doligez Note Added: 0004341
2007-11-22 11:37 doligez Assigned To => doligez
2007-11-22 11:37 doligez Projection none => minor fix
2007-11-22 11:37 doligez ETA none => > 1 month
2007-11-22 17:43 doligez Note Added: 0004342
2007-11-24 18:43 xleroy Note Added: 0004347
2009-06-07 11:37 pipping Note Added: 0004985
2009-06-08 18:13 doligez Relationship added has duplicate 0004657
2011-01-06 15:25 doligez Note Added: 0005760
2011-01-06 15:25 doligez Category Incoming => OCaml general
2011-01-06 15:27 doligez Note Added: 0005761
2011-01-06 15:28 doligez Note Deleted: 0005761
2011-09-07 18:46 meurer File Added: ocaml-3.12.1-parallel-make-fixes.diff
2011-09-07 18:47 meurer Note Added: 0006120
2012-09-06 16:43 doligez Target Version => 4.00.1+dev
2012-09-14 23:56 doligez Target Version 4.00.1+dev => 4.01.0+dev
2013-07-04 15:37 hnrgrgr Note Added: 0009690
2013-07-04 15:38 hnrgrgr File Added: 0001-Poor-men-parallel-compilation.patch
2013-07-04 15:45 gasche Note Added: 0009691
2013-07-05 05:53 ygrek Note Added: 0009696
2013-07-07 16:50 hongboz Note Added: 0009717
2013-07-09 16:41 doligez Note Added: 0009735
2013-07-09 18:23 hnrgrgr Note Added: 0009737
2013-07-09 18:24 hnrgrgr File Added: Poor-men-parallel-compilation_simplified.patch
2013-07-09 23:06 chambart File Added: 0001-Fix-parallel-compilation-of-opt-version-of-stdlib.patch
2013-07-09 23:11 chambart Note Added: 0009738
2013-07-24 16:40 doligez Note Added: 0009856
2013-07-24 16:40 doligez Status acknowledged => resolved
2013-07-24 16:40 doligez Resolution open => fixed
2013-07-24 16:40 doligez Fixed in Version => 4.02.0+dev
2013-07-24 16:40 doligez Target Version 4.01.0+dev => 4.02.0+dev


Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker