New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dependencies are missing in Makefiles #4243
Comments
Comment author: @damiendoligez Not sure how to find the missing dependencies... |
Comment author: laurent I've added a file trying to sum up and explain how to modify the makefile so it works. To test if it works, one can run make -j. Currently, it fails, and if it works it's a good point but it does not mean there is no dependency missing. |
Comment author: @damiendoligez It's a bit more subtle than you think, but I guess I can try |
Comment author: @damiendoligez After spending a few hours on the makefiles, I am not optimistic,
|
Comment author: @xavierleroy I had a try at parallelizing OCaml's Makefiles a couple of years ago. I believe a simpler way to do this is by using recursive invocations of "make" for the top-level targets, e.g. all: instead of all: runtime ocamlc ... You get a little less parallelism this way, but a lot less trouble also. Damien is right that most of the time is spend ocamlbuild-ing Camlp4. ocamlbuild does support parallel make (the undocumented -j N option), but extracts less parallelism than make, because ocamlbuild discovers the dependency graph on the fly during compilation, rather than knowing it in advance. Also, even assuming GNU make, there is no good way to extract whatever -j option the user gave to "make" and pass it back to "ocamlbuild". All in all, I doubt this issue is worth investing significant effort. |
Comment author: pipping I've tried to get ocaml 3.10.2 (targets world, opt, opt.opt) to compile with -j (thus, no limit to the number of jobs) as well. Here's what my conclusions are: A lot of dependencies are missing. Some of these include:
Another parellization problem I've come across, is in the asmrun Makefile. I'll quote parts of it here: COBJS=startup.o main.o fail.o roots.o globroots.o signals.o signals_asm.o ASMOBJS=$(ARCH).o OBJS=$(COBJS) $(ASMOBJS) libasmrun.a: $(OBJS) That looks good so far. However, with -j I sometimes get an error by 'ar' that roots.o does not exist. That should be impossible because when ar for libasmrun.a is invoked, everything in .c.p.o: Thus, roots.o will be created by one job. Another job sees that it's there and starts running the libasmrun.a target. During that, the first job will mv roots.o to roots.p.o and when the second job reaches the 'ar' invocation, roots.o will be gone, yielding an error, as in [3]. [1] pipping@pfirsich ~/ocaml-3.10.2 $ make -n libraryopt [2] pipping@pfirsich ~/ocaml-3.10.2/tools $ make -n ocamlmklib [3] pipping@pfirsich ~/ocaml-3.10.2/asmrun $ make -j |
Comment author: @damiendoligez I've just committed in trunk a Makefile fix for the .c.p.o problem. |
Comment author: meurer I've just attached a patch that resolves several obvious issues in the Makefile's w/o introducing too many dependencies. At least world and opt do now work with -j most of the time (there are two issues left, which I'm trying to figure out). |
Comment author: hnrgrgr I have just done some benchmark on Xavier proposition. On a dual core, without compiling camlp4, the speed-up is almost x2. When camlp4 is also compiled, the gain falls to 25%. In the attached patch, the compilation of camlp4 does not use parallelism, but camlp4 is compiled in parallel of 'otherlibs', ocamldoc and others independent tools. The patch also includes some small fixes on dependencies computation for ocamldoc and labltk. Here are some timings on a recent Intel processor, hyper-threaded dual-core: ./configure -no-camlp4 make world 74.08s user 3.98s system 92% cpu 1:24.07 total make world.opt 126.31s user 7.42s system 92% cpu 2:23.97 total ./configure make world 163.88s user 6.17s system 106% cpu 2:39.55 total make opt 47.98s user 2.20s system 93% cpu 53.477 total make world.opt 225.33s user 11.59s system 98% cpu 4:01.13 total |
Comment author: @gasche I regularly use the -no-camlp4 configure option when I plan frequent build/test cycles of the compiler sources for some development purpose (this will be more widely applicable as findlib doesn't depend on Camlp4 anymore). I am therefore fairly interested in solutions even if they don't tackle the Camlp4 issue. Any review of the change by someone that actually understands makefiles would be appreciated. |
Comment author: @ygrek One more data point - also 2 cores 4 threads. More than 2x without camlp4, 25% full build. ./configure -no-camlp4
make -j4 world.opt 93.32 user 7.28 system 0:45.40 elapsed 221%CPU ./configure
make -j4 world.opt 160.50 user 12.53 system 2:06.01 elapsed 137%CPU |
Comment author: @bobzhang I am also interested in changing the build system of camlp4, it is still valuable even if the patch does not tackle the camlp4 issue |
Comment author: @damiendoligez I have two questions about this latest patch:
In the original, ocamllex.opt is placed between otherlibrariesopt and ocamltoolsopt. Is there a reason to move it down?
This is in ocamldoc. You are adding an include directory. Is it needed to parallelize the makefiles, or is it a fix for an unrelated problem? |
Comment author: hnrgrgr
No real reason. I just wanted to allow as many items as possible to compile in parallel of camlp4, and for wrong reasons I temporary thought that 'ocamltoolsopt' and 'ocamltoolsopt.opt' could not be compiled in parallel. Then, I forgot to restore the original order. One can safely do:
Neither. Once again I forgot to remove that one. This was a first tentative fix for ocamldoc parallel compilation. This is not required when we modify the 'all' rule in ocamldoc/Makefile. I have attached a simplified patch. |
Comment author: @chambart There was a small problem remaining for compiling the rule libraryopt: Also the commited stdlib/.depend in the repository trunk is not up to date: there are no dependencies for .p.cmx rules. A make depend is required |
Comment author: @damiendoligez Last two patches (...simplified and ...of-stdlib) applied to trunk, stdlib dependencies remade (rev 13931). |
Original bug ID: 4243
Reporter: laurent
Assigned to: @damiendoligez
Status: closed (set by @xavierleroy on 2015-12-11T18:21:09Z)
Resolution: fixed
Priority: normal
Severity: tweak
Version: 3.09.3
Target version: 4.02.0+dev
Fixed in version: 4.02.0+dev
Category: ~DO NOT USE (was: OCaml general)
Has duplicate: #4657
Monitored by: meurer @ygrek "Julien Signoles" @hcarty
Bug description
Hello,
Loads of dependencies are missing in Makefiles (and mainly the main one of ocaml).
It works as long as make executes the different targets sequentially but fails if it tries to do it in parallel. I wanted to add the missing dependencies but I don't know well enough the source of ocaml to do it (without adding to much dependencies).
Cheers,
Laurent.
Additional information
In Makefiles, the rule
target: dep1 dep2
command
means that to build target it has to build first dep1 and dep2 and then to execute command. Make can choose to build first dep1 and then dep2 (which it does by default) but it should also be able to build dep2 and then dep1 or both at the same time (if there is no other rule indicating some dependencies between the two). When giving the option -j to make, it tries to build several targets at the same time (especially useful with multi-processor and multi-core), which fails because of the missing dependencies.
File attachments
The text was updated successfully, but these errors were encountered: