Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0004934OCaml-for ocamlbuild use 17:012016-02-16 01:23
Assigned To 
PlatformOSOS Version
Product Version3.11.1 
Target VersionlaterFixed in Version 
Summary0004934: ocamlbuild takes up to one minute to scan already done parts of Coq sources

I'm investigating the use of ocamlbuild for building Coq. I'd be glad to completely get rid of nasty Makefile stuff, but currently, two major issues prevent a complete switch to ocamlbuild. One is the lack of proper parallel compilation (see a forthcoming bug report). Another is that the delay needed by ocamlbuild to scan already built parts (> 1min here) prohibits using it
for interactive development (edit a file, recompile, and so on). I've already had private discussions on this topic, without clear solutions, so I turn this into a proper bug report, for the records.

Disclaimer: I clearly do not claim my file to be perfect, so
this inefficiency can be caused by the way it is written. Any hints are welcome...

Best regards,
Pierre Letouzey

Instructions to reproduce:

0) you'll need camlp5 installed
1) get a copy of coq development archive
   svn checkout svn:// [^]
2) ./configure -local -opt
3) Launch a first build via ocamlbuild (./build is a small wrapper)
4) Launch ./build again

Here, the first build says:
Finished, 2716 targets (0 cached) in 00:13:10.

While the second says:
Finished, 2716 targets (2716 cached) in 00:01:07.

With old-style make, the second run of make says instantly that nothing is to be done.

NB: I've tried the patch for bug 0004922, unfortunately it doesn't help.
We're not recompiling too many files here, simply taking too much time
scanning cached things.
Attached Filespatch file icon 0001-cache-expand_module.patch [^] (1,010 bytes) 2011-05-17 16:41 [Show Content]
patch file icon 0002-disable-logging-in-Solver.self.patch [^] (2,413 bytes) 2011-05-17 16:42 [Show Content]
patch file icon 0003-specialize-List.union.patch [^] (3,220 bytes) 2011-05-17 16:42 [Show Content]
patch file icon 0004-specialize-hashtbl-for-Resource.Cache.patch [^] (861 bytes) 2011-05-17 16:42 [Show Content]

- Relationships
related to 0005754resolved Not optimal parallel build of dependencies 
related to 0005756closedgasche ocamlbuild spends a lot of time logging 

-  Notes
ygrek (reporter)
2011-05-17 16:43

Here are some observations.
Ocaml_utils.expand_module is pure and called too often (and generates lots of garbage) with same arguments - simple memoisation reduces the number of invocations by the factor of 100 (for coq sources) and cuts running time by 30 sec (out of 2 minutes).
Solver.self calls dprintf which doesn't print anything at default logging level, but still incurs too much overhead, commenting it out cuts another 40 seconds. (One day I hope to have type-safe camlp4 printf which doesn't parse format string at runtime..)
Another offender is generic compare_val called from various places - specializing My_std.List.union and Hashtbl in Resource.Cache casts away 8 more seconds.
Contrary to my expectation using Set(String) instead of list to detect circular deps in Solver.self didn't give speedup..

All in all - 42 seconds vs 2+ minutes. Quick'n'dirty patches attached.

BTW better names for anonymous functions and better debug-symbol info would be greatly appreciated for debugging such issues
ygrek (reporter)
2012-05-22 11:26

Why not apply at least the first patch?
meyer (developer)
2012-09-04 01:53

Ygrek, your patch is being considered for an inclusion - at least the first one, however it looks we have general scalability problems with ocamlbuild, and until we will know what we can do to improve it in general case, I will postpone applying it. I don't believe scanning the tree should take that long. What we can also do is to spawn several processes scanning the tree - but this will be the last thing we we will do, however.
meyer (developer)
2012-09-04 02:04

As for the logging - we could functorize the implementation, and replace the logging functions with a dummy ones if the user requested logging. I'll look into that.
meyer (developer)
2012-09-04 02:12
edited on: 2015-02-26 21:15

I raised 0005756 for this chunk of work.

meyer (developer)
2012-09-04 02:17
edited on: 2015-02-26 21:15

Ygrek, for your patch # 4: I think what we want to do is to get rid of lists all together, and replace them with something better (e.g: Set). Nevertheless it highlights some obvious optimisations we could do.

ygrek (reporter)
2012-09-04 10:33

I agree that there are speed problems at different levels, and not doing unnecessary work is the best optimisation anyway :)
I doubt it is possible to fix logging with functorizing.. Maybe better wrap it oldschool way with : `if loglevel X then ...`
ygrek (reporter)
2015-07-27 10:14

FTR patch #1 was included in 4.01.0
For the other patches see [^]
egallego (reporter)
2016-02-16 00:55

This issue is superseded by [^]
gasche (administrator)
2016-02-16 01:23

I'm closing the ticket to migrate further discussions to the new ocamlbuild bugtracker. Note that ygrek's patches from [^] have been merged upstream.

- Issue History
Date Modified Username Field Change
2009-12-08 17:01 letouzey New Issue
2011-05-02 12:38 doligez Status new => acknowledged
2011-05-17 16:41 ygrek File Added: 0001-cache-expand_module.patch
2011-05-17 16:42 ygrek File Added: 0002-disable-logging-in-Solver.self.patch
2011-05-17 16:42 ygrek File Added: 0003-specialize-List.union.patch
2011-05-17 16:42 ygrek File Added: 0004-specialize-hashtbl-for-Resource.Cache.patch
2011-05-17 16:43 ygrek Note Added: 0005902
2011-06-06 13:02 xclerc Status acknowledged => assigned
2011-06-06 13:02 xclerc Assigned To => xclerc
2012-02-02 15:17 protz Category OCamlbuild => OCamlbuild (the tool)
2012-05-22 11:26 ygrek Note Added: 0007447
2012-07-06 16:33 doligez Target Version => 4.01.0+dev
2012-07-06 16:33 doligez Description Updated View Revisions
2012-07-31 13:36 doligez Target Version 4.01.0+dev => 4.00.1+dev
2012-09-03 02:08 meyer Relationship added related to 0005754
2012-09-04 01:53 meyer Note Added: 0008012
2012-09-04 02:04 meyer Note Added: 0008013
2012-09-04 02:11 meyer Relationship added related to 0005756
2012-09-04 02:12 meyer Note Added: 0008014
2012-09-04 02:17 meyer Note Added: 0008015
2012-09-04 10:33 ygrek Note Added: 0008017
2012-09-06 19:20 frisch Target Version 4.00.1+dev => 4.00.2+dev
2013-01-20 01:01 meyer Assigned To xclerc => meyer
2013-06-16 18:33 gasche Target Version 4.00.2+dev => 4.02.0+dev
2013-06-16 21:25 gasche Severity major => minor
2013-07-12 18:15 doligez Target Version 4.02.0+dev => 4.01.1+dev
2013-09-04 21:06 doligez Tag Attached: patch
2014-01-21 14:01 doligez Assigned To meyer =>
2014-05-25 20:20 doligez Target Version 4.01.1+dev => 4.02.0+dev
2014-08-15 15:47 gasche Target Version 4.02.0+dev => 4.02.1+dev
2014-09-04 00:25 doligez Target Version 4.02.1+dev => undecided
2014-09-26 17:45 doligez Target Version undecided => 4.02.2+dev / +rc1
2015-02-26 21:15 doligez Note Edited: 0008014 View Revisions
2015-02-26 21:15 doligez Note Edited: 0008015 View Revisions
2015-02-26 21:17 doligez Target Version 4.02.2+dev / +rc1 => 4.03.0+dev / +beta1
2015-07-27 10:14 ygrek Note Added: 0014277
2015-12-02 16:05 gasche Target Version 4.03.0+dev / +beta1 => later
2016-02-16 00:55 egallego Note Added: 0015361
2016-02-16 01:23 gasche Note Added: 0015363
2016-02-16 01:23 gasche Status assigned => closed
2016-02-16 01:23 gasche Resolution open => suspended
2017-02-23 16:34 doligez Category OCamlbuild (the tool) => for ocamlbuild use [^]
2017-02-23 16:44 doligez Category for ocamlbuild use [^] => -for ocamlbuild use [^]

Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker