Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dependencies are missing in Makefiles #4243

Closed
vicuna opened this issue Mar 27, 2007 · 16 comments
Closed

Dependencies are missing in Makefiles #4243

vicuna opened this issue Mar 27, 2007 · 16 comments
Assignees
Labels
Milestone

Comments

@vicuna
Copy link

vicuna commented Mar 27, 2007

Original bug ID: 4243
Reporter: laurent
Assigned to: @damiendoligez
Status: closed (set by @xavierleroy on 2015-12-11T18:21:09Z)
Resolution: fixed
Priority: normal
Severity: tweak
Version: 3.09.3
Target version: 4.02.0+dev
Fixed in version: 4.02.0+dev
Category: ~DO NOT USE (was: OCaml general)
Has duplicate: #4657
Monitored by: meurer @ygrek "Julien Signoles" @hcarty

Bug description

Hello,

Loads of dependencies are missing in Makefiles (and mainly the main one of ocaml).
It works as long as make executes the different targets sequentially but fails if it tries to do it in parallel. I wanted to add the missing dependencies but I don't know well enough the source of ocaml to do it (without adding to much dependencies).

Cheers,
Laurent.

Additional information

In Makefiles, the rule
target: dep1 dep2
command
means that to build target it has to build first dep1 and dep2 and then to execute command. Make can choose to build first dep1 and then dep2 (which it does by default) but it should also be able to build dep2 and then dep1 or both at the same time (if there is no other rule indicating some dependencies between the two). When giving the option -j to make, it tries to build several targets at the same time (especially useful with multi-processor and multi-core), which fails because of the missing dependencies.

File attachments

@vicuna
Copy link
Author

vicuna commented Apr 23, 2007

Comment author: @damiendoligez

Not sure how to find the missing dependencies...

@vicuna
Copy link
Author

vicuna commented Apr 26, 2007

Comment author: laurent

I've added a file trying to sum up and explain how to modify the makefile so it works. To test if it works, one can run make -j. Currently, it fails, and if it works it's a good point but it does not mean there is no dependency missing.

@vicuna
Copy link
Author

vicuna commented Nov 22, 2007

Comment author: @damiendoligez

It's a bit more subtle than you think, but I guess I can try
to make the Makefile parallelizable. As this is not a vital
feature, it won't be available before version 3.11.

@vicuna
Copy link
Author

vicuna commented Nov 22, 2007

Comment author: @damiendoligez

After spending a few hours on the makefiles, I am not optimistic,
for the following reasons:

  1. Your method introduces too many dependencies, which will lead
    to too much recompilation when developing OCaml. For example
    everything would depend on coldstart, which would get remade
    all the time.

  2. There are a lot of makefiles to modify.

  3. You won't get a big speedup anyway, because most of the time
    is spent in ocamlbuild, which doesn't build in parallel.

  4. As always with concurrent stuff, this is hell to debug and
    testing doesn't give even a small amount of assurance (unlike
    sequential stuff).

@vicuna
Copy link
Author

vicuna commented Nov 24, 2007

Comment author: @xavierleroy

I had a try at parallelizing OCaml's Makefiles a couple of years ago. I believe a simpler way to do this is by using recursive invocations of "make" for the top-level targets, e.g.

all:
$(MAKE) runtime
$(MAKE) ocamlc
...

instead of

all: runtime ocamlc ...

You get a little less parallelism this way, but a lot less trouble also.

Damien is right that most of the time is spend ocamlbuild-ing Camlp4. ocamlbuild does support parallel make (the undocumented -j N option), but extracts less parallelism than make, because ocamlbuild discovers the dependency graph on the fly during compilation, rather than knowing it in advance. Also, even assuming GNU make, there is no good way to extract whatever -j option the user gave to "make" and pass it back to "ocamlbuild".

All in all, I doubt this issue is worth investing significant effort.

@vicuna
Copy link
Author

vicuna commented Jun 7, 2009

Comment author: pipping

I've tried to get ocaml 3.10.2 (targets world, opt, opt.opt) to compile with -j (thus, no limit to the number of jobs) as well. Here's what my conclusions are:

A lot of dependencies are missing. Some of these include:

  • everything but the coldstart target needs a working ocamlrun. if you run make -j world, you'll have make work on the targets 'coldstart' and 'all' in parallel, since 'all' does not depend on coldstart itself. You'll get an error here because there's no ocamlrun.
  • similarly, a lot of stuff needs ocamlc but doesn't depend on it. You'll get an error here as well.
  • opt itself does not depend on ocamlopt, contrary to what the makefile says, but some of its subtargets do. so e.g. libraryopt should but doesn't. you can see this easily e.g. this way: [1]
  • a lot of stuff needs ocamlmklib but the makefiles don't reflect that. ocamlmklib is a problem as you can see here: [2] (the error that you see there will not become a problem with -j1; with -j3 or even -j it will, though).
  • the debugger needs libraries from otherlibs/unix

Another parellization problem I've come across, is in the asmrun Makefile. I'll quote parts of it here:

COBJS=startup.o main.o fail.o roots.o globroots.o signals.o signals_asm.o
misc.o freelist.o major_gc.o minor_gc.o memory.o alloc.o compare.o ints.o
floats.o str.o array.o io.o extern.o intern.o hash.o sys.o parsing.o
gc_ctrl.o terminfo.o md5.o obj.o lexing.o printexc.o callback.o weak.o
compact.o finalise.o custom.o unix.o backtrace.o

ASMOBJS=$(ARCH).o

OBJS=$(COBJS) $(ASMOBJS)

libasmrun.a: $(OBJS)
rm -f libasmrun.a
ar rc libasmrun.a $(OBJS)
$(RANLIB) libasmrun.a

That looks good so far. However, with -j I sometimes get an error by 'ar' that roots.o does not exist. That should be impossible because when ar for libasmrun.a is invoked, everything in $(OBJS), thus $(COBJS) and especially roots.o should have been built. The problem is further down in the makefile:

.c.p.o:
@ if test -f $.o; then mv $.o $.f.o; else :; fi
$(CC) -c $(PFLAGS) $<
mv $
.o $.p.o
@ if test -f $
.f.o; then mv $.f.o $.o; else :; fi

Thus, roots.o will be created by one job. Another job sees that it's there and starts running the libasmrun.a target. During that, the first job will mv roots.o to roots.p.o and when the second job reaches the 'ar' invocation, roots.o will be gone, yielding an error, as in [3].

[1] pipping@pfirsich ~/ocaml-3.10.2 $ make -n libraryopt
cd stdlib; make allopt
make[1]: Entering directory /home/pipping/ocaml-3.10.2/stdlib' make[1]: *** No rule to make target ../ocamlopt', needed by pervasives.cmx'. Stop. make[1]: Leaving directory /home/pipping/ocaml-3.10.2/stdlib'
make: *** [libraryopt] Error 2

[2] pipping@pfirsich ~/ocaml-3.10.2/tools $ make -n ocamlmklib
../build/mkmyocamlbuild_config.sh
cp ../myocamlbuild_config.ml .
../boot/ocamlrun ../boot/ocamlc -nostdlib -I ../boot -c -warn-error A -I ../utils -I ../parsing -I ../typing -I ../bytecomp -I ../asmcomp -I ../driver myocamlbuild_config.ml
echo '(* THIS FILE IS GENERATED FROM ocamlmklib.mlp *)' >ocamlmklib.ml
sed -e "s|%%BINDIR%%|/usr/local/bin|"
-e "s|%%SUPPORTS_SHARED_LIBRARIES%%|true|"
-e "s|%%MKSHAREDLIB%%||"
-e "s|%%BYTECCRPATH%%|-Wl,-rpath,|"
-e "s|%%NATIVECCRPATH%%|-Wl,-rpath,|"
-e "s|%%MKSHAREDLIBRPATH%%|-Wl,-rpath,|"
-e "s|%%RANLIB%%|ranlib|"
ocamlmklib.mlp >> ocamlmklib.ml
make: *** No rule to make target myocamlbuild_config.cmi', needed by ocamlmklib.cmo'. Stop.

[3] pipping@pfirsich ~/ocaml-3.10.2/asmrun $ make -j
[ snip ]
rm -f libasmrun.a
ar rc libasmrun.a startup.o main.o fail.o roots.o globroots.o signals.o signals_asm.o misc.o freelist.o major_gc.o minor_gc.o memory.o alloc.o compare.o ints.o floats.o str.o array.o io.o extern.o intern.o hash.o sys.o parsing.o gc_ctrl.o terminfo.o md5.o obj.o lexing.o printexc.o callback.o weak.o compact.o finalise.o custom.o unix.o backtrace.o amd64.o
ar: startup.o: No such file or directory
make: *** [libasmrun.a] Error 1

@vicuna
Copy link
Author

vicuna commented Jan 6, 2011

Comment author: @damiendoligez

I've just committed in trunk a Makefile fix for the .c.p.o problem.

@vicuna
Copy link
Author

vicuna commented Sep 7, 2011

Comment author: meurer

I've just attached a patch that resolves several obvious issues in the Makefile's w/o introducing too many dependencies. At least world and opt do now work with -j most of the time (there are two issues left, which I'm trying to figure out).

@vicuna
Copy link
Author

vicuna commented Jul 4, 2013

Comment author: hnrgrgr

I have just done some benchmark on Xavier proposition. On a dual core, without compiling camlp4, the speed-up is almost x2. When camlp4 is also compiled, the gain falls to 25%.

In the attached patch, the compilation of camlp4 does not use parallelism, but camlp4 is compiled in parallel of 'otherlibs', ocamldoc and others independent tools.

The patch also includes some small fixes on dependencies computation for ocamldoc and labltk.

Here are some timings on a recent Intel processor, hyper-threaded dual-core:

./configure -no-camlp4

make world 74.08s user 3.98s system 92% cpu 1:24.07 total
make -j4 world 95.23s user 4.15s system 259% cpu 38.360 total

make world.opt 126.31s user 7.42s system 92% cpu 2:23.97 total
make -j4 world.opt 154.46s user 7.70s system 214% cpu 1:15.47 total

./configure

make world 163.88s user 6.17s system 106% cpu 2:39.55 total
make -j4 world 173.76s user 6.15s system 161% cpu 1:51.68 total

make opt 47.98s user 2.20s system 93% cpu 53.477 total
make -j4 opt 55.46s user 2.28s system 157% cpu 36.695 total

make world.opt 225.33s user 11.59s system 98% cpu 4:01.13 total
make -j4 world.opt 253.00s user 12.36s system 153% cpu 2:52.55 total

@vicuna
Copy link
Author

vicuna commented Jul 4, 2013

Comment author: @gasche

I regularly use the -no-camlp4 configure option when I plan frequent build/test cycles of the compiler sources for some development purpose (this will be more widely applicable as findlib doesn't depend on Camlp4 anymore).

I am therefore fairly interested in solutions even if they don't tackle the Camlp4 issue. Any review of the change by someone that actually understands makefiles would be appreciated.

@vicuna
Copy link
Author

vicuna commented Jul 5, 2013

Comment author: @ygrek

One more data point - also 2 cores 4 threads. More than 2x without camlp4, 25% full build.

./configure -no-camlp4

    make world  42.54 user 4.30 system 0:50.90 elapsed 92%CPU
make -j4 world  50.28 user 3.42 system 0:23.89 elapsed 224%CPU

make world.opt  81.83 user 8.92 system 1:40.68 elapsed 90%CPU

make -j4 world.opt 93.32 user 7.28 system 0:45.40 elapsed 221%CPU

./configure

    make world  109.24 user 8.58 system 2:08.68 elapsed 91%CPU
make -j4 world  109.48 user 6.36 system 1:29.23 elapsed 129%CPU

make world.opt  144.94 user 13.65 system 2:53.49 elapsed 91%CPU

make -j4 world.opt 160.50 user 12.53 system 2:06.01 elapsed 137%CPU

@vicuna
Copy link
Author

vicuna commented Jul 7, 2013

Comment author: @bobzhang

I am also interested in changing the build system of camlp4, it is still valuable even if the patch does not tackle the camlp4 issue

@vicuna
Copy link
Author

vicuna commented Jul 9, 2013

Comment author: @damiendoligez

I have two questions about this latest patch:

  • $(MAKE) otherlibrariesopt ocamltoolsopt
  • $(MAKE) ocamllex.opt ocamltoolsopt.opt ocamldoc.opt \
  •       ocamlbuild.native $(CAMLP4OPT)
    

In the original, ocamllex.opt is placed between otherlibrariesopt and ocamltoolsopt. Is there a reason to move it down?

  • -I generators

This is in ocamldoc. You are adding an include directory. Is it needed to parallelize the makefiles, or is it a fix for an unrelated problem?

@vicuna
Copy link
Author

vicuna commented Jul 9, 2013

Comment author: hnrgrgr

In the original, ocamllex.opt is placed between otherlibrariesopt and ocamltoolsopt. Is there a reason to move it down?

No real reason. I just wanted to allow as many items as possible to compile in parallel of camlp4, and for wrong reasons I temporary thought that 'ocamltoolsopt' and 'ocamltoolsopt.opt' could not be compiled in parallel. Then, I forgot to restore the original order. One can safely do:

  • $(MAKE) otherlibrariesopt
  • $(MAKE) ocamllex.opt ocamltoolsopt ocamltoolsopt.opt ocamldoc.opt \
  • ocamlbuild.native $(CAMLP4OPT)

This is in ocamldoc. You are adding an include directory. Is it needed to parallelize the makefiles, or is it a fix for an unrelated problem?

Neither. Once again I forgot to remove that one. This was a first tentative fix for ocamldoc parallel compilation. This is not required when we modify the 'all' rule in ocamldoc/Makefile.

I have attached a simplified patch.

@vicuna
Copy link
Author

vicuna commented Jul 9, 2013

Comment author: @chambart

There was a small problem remaining for compiling the rule libraryopt:
Compiling profiling and normal version must be sequential.

Also the commited stdlib/.depend in the repository trunk is not up to date: there are no dependencies for .p.cmx rules. A make depend is required

@vicuna
Copy link
Author

vicuna commented Jul 24, 2013

Comment author: @damiendoligez

Last two patches (...simplified and ...of-stdlib) applied to trunk, stdlib dependencies remade (rev 13931).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants