Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0006094OCamlOCamlbuild (the tool)public2013-07-27 03:032013-08-19 16:50
Reportermeyer 
Assigned To 
PriorityhighSeverityfeatureReproducibilityalways
StatusfeedbackResolutionopen 
PlatformOSOS Version
Product Version4.02.0+dev 
Target Version4.02.0+devFixed in Version 
Summary0006094: Reading META files
DescriptionAs proposed by dbuenzli, and already implemented in other build systems, we can easily parse META information and extract neeeded information, possibly about the dependencies.
Additional InformationAgain I am setting the target to the current release, as it would really nice to break the usual convenance that ocamlbuild development is behind a firewall.
TagsNo tags attached.
Attached Files

- Relationships

-  Notes
(0009892)
william (reporter)
2013-07-27 09:16
edited on: 2013-07-27 09:17

while it is good to integrate ocamlfind, implementing rules to read META files is like forking findlib, and this is bad idea I think. If Gerd makes changes to findlib, you would have to
- either run after him and also integrate it;
- or do not update things, and that would do 2 versions of the thing.

A first step would rather be to ask Gerd if he would be willing to maintain findlib whithin ocaml distribution, and see if that works ?

note that there already are some rules to read META files I think in ocamlbuild. Which I think should be removed.

(0009893)
Camarade_Tux (reporter)
2013-07-27 12:35
edited on: 2013-07-27 12:35

I don't like what ocp-build does. Findlib is a ... library and that means we can and should use it for such support.

Relying on findlib causes a chicken-and-egg issue however. Maybe this could be handled through dynlink: if the cmxs is available, load it, if it isn't, fallback to a dumber method (spawning ocamlfind maybe).

I haven't thought about this a lot but I believe we have to try hard to avoid duplication and avoid re-doing findlib.

(0009895)
gerd (reporter)
2013-07-27 13:29

William: I'm not opposed to contributing findlib to the official ocaml tree. But this is a major step, and it will require approval by INRIA, and there is also paperwork to do in order to transfer the intellectual rights. (I'm self-employed, so this is generally possible.) So nothing you can do shortly.

Generally, I think it's a bad idea to have several parsers and interpreters of META files. This may create incompatibilities. I don't think, however, it is possible to load findlib as library into ocamlbuild, because you run then into a big bootstrap problem (which is only cleanly solvable by including findlib into the ocaml tree).

I don't know exactly what the intention of having library information directly in ocamlbuild. If it is just about avoiding the costs of running ocamlfind for every compiler call, we could talk about running ocamlfind in a "pipe mode" where it continuously accepts queries on stdin, and outputs the compiler calls (or other requested information) on stdout.
(0009897)
gasche (developer)
2013-07-27 20:45
edited on: 2013-07-27 20:46

I also don't understand what reading META files is supposed to do for ocamlbuild. Wojciech or Daniel, could you be more explicit on what you would expect ocamlbuild to do with those META file?

(The idea to include findlib in the distribution is overkill and unreasonable. Distribution maintainers want to have less things to maintain, not more. Besides, it is much simpler to add a setting to ocamlbuild to bootstrap without some features than to enter such a heavy process. So "no".)

gerd > something I had been thinking about a few days ago would be a -show-command option added to the ocamlfind commands, to return the command that would be run if files were passed in arguments, instead of actually running the command.

For example,
  ocamlfind ocamlc -package foo -linkpkg -show-tag
would return something along the lines of
  ocamlc -I /usr/lib/ocaml/.../foo bar.cma
if the package "foo" only has an archive "bar.cma" and no other link options

My idea was to use that in ocamlbuild (when -use-ocamlfind is set) to cache ocamlfind calls. Instead of actually calling ocamlfind for all files, we could build a memoized mapping (for the duration of the set of one ocamlbuild run) from the set of arguments passed to ocamlfind to the set of actual parameters using this -show-command, without changing in any way the interface observed by the user.

It looks like the pipe mode may allow for strictly less system calls (but I'm not sure that would necessarily be a performance win if you have more communication instead), but also be a bit more invasive on the findlib side. Do you have a strong preference for either option? We could also consider proposing patches for both...

(0009898)
gasche (developer)
2013-07-27 21:03

I wanted to make an additional point that Adrien Nader just reminded me about: *I* don't have observed any problem with the current way -use-ocamlfind is implemented, and am not personally obsessed with the number of system calls made during an ocamlbuild compilation process.

The reason why this came on the table is that some people mentioned this as a problem of -use-ocamlfind, in particular Anil, speaking about Mirage builds. I am only discussing the issue from hearsay -- I have not done any profiling of ocamlbuild performances yet.

So it's not like the number of system calls is my number one priority right now. It was indirectly mentioned in this bugtracker discussion, and I was happy to see Gerd participate so I threw some ideas about it, but I don't know whether working on that would be worth it right now.

Gerd, I'm happy to get design advice or indications of which kind of patches we could consider sending you if/when we get conclusive data on ocamlfind performance impact, but I would be embarrassed if you invested more time than that on the matter for now.
(0009899)
Camarade_Tux (reporter)
2013-07-27 21:12

I need to add something: there have been complaints about the speed impact of ocamlbuild with ocamlfind but currently it's an issue that is opened in the bug trackers without enough information.

Which free software exhibit this issue besides Mirage? What are the characteristics? Are there camlp4 calls and is so, how much? How many files are there, how are they laid out on disk? How are the inter-files dependencies? Are packs involved? How many libraries are available on the system through ocamlfind? Are some of the libraries that are used not installed (i.e. living in src/lib1 instead of /usr)?

That's to name a few. More details are needed because right now it feels like optimizing for an imaginary workload (I'm not saying that it is, I'm saying that it feels like it is).
(0009900)
gerd (reporter)
2013-07-27 21:35

Here is an easy observation about the speed of ocamlfind:

$ time ocamlfind 2>/dev/null
real 0m0.001s
user 0m0.000s
sys 0m0.000s

This is basically the time for launching the executable. As you see, only 1ms (on Linux).

$ time ocamlfind ocamlc
real 0m0.004s
user 0m0.000s
sys 0m0.004s

Here it is doing a bit more, but not much. The difference is that it runs ocamlc.opt as subprocess. Roughly, 1ms goes on ocamlfind's account, and 3ms on ocamlc.opt's.

$ time ocamlfind ocamlc -package netstring,pxp,lwt,ssl,yojson,zip
real 0m0.012s
user 0m0.008s
sys 0m0.004s

This command looks a number of packages up, parses the META file, resolves dependencies, and calls ocamlc.opt with a lot of -I switches. These are extra 8ms, completely eaten by user code (i.e. it's not the syscalls). This would need more analysis. Maybe caching META data would help here (the suggested pipe mode would also be good for this). However, is 8ms something we should worry about, when the following compiler invocation consumes a multitude of it? Also, maybe some performance tweaks in ocamlfind could also help (not sure which, but if you look close enough you usually find something).
(0009901)
gerd (reporter)
2013-07-27 21:44

Also I was looking into the findlib source code. There was a performance fix in 1.4 (processing of long argument lists), so maybe this plays a role here?

Anyway, we need a concrete case before we can look into it.
(0009902)
meyer (developer)
2013-07-27 23:26

gasche: so dry run indeed would be useful, except is not possible without executing the body of the rule (as you know ocamlbuild discovers dependencies at each invocation of rule). If that was possible it would be much easier to improve paralelisation.

For the question of why we would like to parse META files, I think the biggest benefit is to get per tree package usage, dependencies that normally are specified with "package(foo)". If others do it and it's useful then we might be doing that as well.
(0009909)
meyer (developer)
2013-07-28 01:38
edited on: 2013-07-28 08:38

So possible uses of META files:

* automatic resolution of findlib dependencies
* specification of targets to be build
* installation of the needed files to the findlib repository

Gerd: One of the solutions to mitigate the need of including the findlib in ocamlbuild tree (probably difficult at the moment at least, but worth to invesigating at some point), is to allow ocamlfind to read meta files from the file system, and then query should work as usual, except META is read from an explicit location.

(0009922)
gasche (developer)
2013-07-28 09:01

I feel that you're stepping on OASIS' shoes when going in this direction. Oasis does require metadata that can be used to generate both a META file, and an ocamlbuild scaffolding.

Thinking of use-cases without OASIS:
(1) the first point may be convenient (it allows to not repeat ocamlfind dependencies in both the META file)
(3) the third feels outside the scope of a build system (we may provide feature to specify which files has been built, that is Daniel B├╝nzli's PR#6067, but installing them is not a build system's job)
(2) the second is a bit in between; I agree it could make sense, but I suspect that an install system knows more about targets to be built than the META file -- which misses for example .mli files and documentation.
(0009924)
william (reporter)
2013-07-28 11:12
edited on: 2013-07-28 11:15

I tested the "-use-ocamlfind" option on ocamlbuild 3.12.1, on debian (packaged stable version).

1) I run ocamlbuild with the "-use-ocamlfind" option :
Finished, 406 targets (0 cached) in 00:02:20.
real 2m20.919s /2m21.734s
user 2m10.160s /2m11.076s

2) I run ocamlbuild without the option. The funny thing being that it also works! (yes, ocamlbuild, at least 3.12.1, already supports some ocamlfind features...) :
Finished, 406 targets (0 cached) in 00:02:18.
real 2m18.449s/2m19.035s
user 2m9.760s /2m9.484s

Note that for my project most of the time is spent on actually compiling stuff, so using or not using ocamlfind is no performance issue at all. It contains objects (which takes time to compile) and it was done on an old laptop (core 2 duo proc)

(0009928)
meyer (developer)
2013-07-28 13:23

While I agree that this OASIS job to do certain things, we need to be able to be flexible enough to support this, this way or other way. At the moment I agree with gasche that maybe full support for META might be an overkill, however, with the proposal of the adding user options to the compilation commandline of plugins;

http://caml.inria.fr/mantis/view.php?id=6093 [^]

makes it relatively easy to write a plugin, distributed for instance as ocamlfind package, to do the job of parsing the META files.
(0010200)
doligez (administrator)
2013-08-19 16:50

I'm not sure I understand the remark about setting the target to the current release (my goal is to have zero PR with 4.01.0 as target before we release). I'm setting this to feedback because there is obviously an interesting discussion going on. Move it to "confirmed" when we have consensus on what needs to be done.

- Issue History
Date Modified Username Field Change
2013-07-27 03:03 meyer New Issue
2013-07-27 09:16 william Note Added: 0009892
2013-07-27 09:17 william Note Edited: 0009892 View Revisions
2013-07-27 12:35 Camarade_Tux Note Added: 0009893
2013-07-27 12:35 Camarade_Tux Note Edited: 0009893 View Revisions
2013-07-27 13:29 gerd Note Added: 0009895
2013-07-27 20:45 gasche Note Added: 0009897
2013-07-27 20:46 gasche Note Edited: 0009897 View Revisions
2013-07-27 21:03 gasche Note Added: 0009898
2013-07-27 21:12 Camarade_Tux Note Added: 0009899
2013-07-27 21:35 gerd Note Added: 0009900
2013-07-27 21:44 gerd Note Added: 0009901
2013-07-27 23:26 meyer Note Added: 0009902
2013-07-28 01:38 meyer Note Added: 0009909
2013-07-28 08:38 meyer Note Edited: 0009909 View Revisions
2013-07-28 09:01 gasche Note Added: 0009922
2013-07-28 11:12 william Note Added: 0009924
2013-07-28 11:14 william Note Edited: 0009924 View Revisions
2013-07-28 11:15 william Note Edited: 0009924 View Revisions
2013-07-28 13:23 meyer Note Added: 0009928
2013-08-19 16:50 doligez Note Added: 0010200
2013-08-19 16:50 doligez Status new => feedback
2013-08-19 16:50 doligez Target Version 4.01.0+dev => 4.02.0+dev


Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker