Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reading META files #6094

Closed
vicuna opened this issue Jul 27, 2013 · 15 comments
Closed

Reading META files #6094

vicuna opened this issue Jul 27, 2013 · 15 comments

Comments

@vicuna
Copy link

vicuna commented Jul 27, 2013

Original bug ID: 6094
Reporter: meyer
Status: resolved (set by @damiendoligez on 2017-03-01T15:45:21Z)
Resolution: suspended
Priority: high
Severity: feature
Version: 4.02.0+dev
Target version: undecided
Category: -for ocamlbuild use https://github.com/ocaml/ocamlbuild/issues
Monitored by: meyer @gasche Camarade_Tux kerneis @hcarty gerd

Bug description

As proposed by dbuenzli, and already implemented in other build systems, we can easily parse META information and extract neeeded information, possibly about the dependencies.

Additional information

Again I am setting the target to the current release, as it would really nice to break the usual convenance that ocamlbuild development is behind a firewall.

@vicuna
Copy link
Author

vicuna commented Jul 27, 2013

Comment author: william

while it is good to integrate ocamlfind, implementing rules to read META files is like forking findlib, and this is bad idea I think. If Gerd makes changes to findlib, you would have to

  • either run after him and also integrate it;
  • or do not update things, and that would do 2 versions of the thing.

A first step would rather be to ask Gerd if he would be willing to maintain findlib whithin ocaml distribution, and see if that works ?

note that there already are some rules to read META files I think in ocamlbuild. Which I think should be removed.

@vicuna
Copy link
Author

vicuna commented Jul 27, 2013

Comment author: Camarade_Tux

I don't like what ocp-build does. Findlib is a ... library and that means we can and should use it for such support.

Relying on findlib causes a chicken-and-egg issue however. Maybe this could be handled through dynlink: if the cmxs is available, load it, if it isn't, fallback to a dumber method (spawning ocamlfind maybe).

I haven't thought about this a lot but I believe we have to try hard to avoid duplication and avoid re-doing findlib.

@vicuna
Copy link
Author

vicuna commented Jul 27, 2013

Comment author: gerd

William: I'm not opposed to contributing findlib to the official ocaml tree. But this is a major step, and it will require approval by INRIA, and there is also paperwork to do in order to transfer the intellectual rights. (I'm self-employed, so this is generally possible.) So nothing you can do shortly.

Generally, I think it's a bad idea to have several parsers and interpreters of META files. This may create incompatibilities. I don't think, however, it is possible to load findlib as library into ocamlbuild, because you run then into a big bootstrap problem (which is only cleanly solvable by including findlib into the ocaml tree).

I don't know exactly what the intention of having library information directly in ocamlbuild. If it is just about avoiding the costs of running ocamlfind for every compiler call, we could talk about running ocamlfind in a "pipe mode" where it continuously accepts queries on stdin, and outputs the compiler calls (or other requested information) on stdout.

@vicuna
Copy link
Author

vicuna commented Jul 27, 2013

Comment author: @gasche

I also don't understand what reading META files is supposed to do for ocamlbuild. Wojciech or Daniel, could you be more explicit on what you would expect ocamlbuild to do with those META file?

(The idea to include findlib in the distribution is overkill and unreasonable. Distribution maintainers want to have less things to maintain, not more. Besides, it is much simpler to add a setting to ocamlbuild to bootstrap without some features than to enter such a heavy process. So "no".)

gerd > something I had been thinking about a few days ago would be a -show-command option added to the ocamlfind commands, to return the command that would be run if files were passed in arguments, instead of actually running the command.

For example,
ocamlfind ocamlc -package foo -linkpkg -show-tag
would return something along the lines of
ocamlc -I /usr/lib/ocaml/.../foo bar.cma
if the package "foo" only has an archive "bar.cma" and no other link options

My idea was to use that in ocamlbuild (when -use-ocamlfind is set) to cache ocamlfind calls. Instead of actually calling ocamlfind for all files, we could build a memoized mapping (for the duration of the set of one ocamlbuild run) from the set of arguments passed to ocamlfind to the set of actual parameters using this -show-command, without changing in any way the interface observed by the user.

It looks like the pipe mode may allow for strictly less system calls (but I'm not sure that would necessarily be a performance win if you have more communication instead), but also be a bit more invasive on the findlib side. Do you have a strong preference for either option? We could also consider proposing patches for both...

@vicuna
Copy link
Author

vicuna commented Jul 27, 2013

Comment author: @gasche

I wanted to make an additional point that Adrien Nader just reminded me about: I don't have observed any problem with the current way -use-ocamlfind is implemented, and am not personally obsessed with the number of system calls made during an ocamlbuild compilation process.

The reason why this came on the table is that some people mentioned this as a problem of -use-ocamlfind, in particular Anil, speaking about Mirage builds. I am only discussing the issue from hearsay -- I have not done any profiling of ocamlbuild performances yet.

So it's not like the number of system calls is my number one priority right now. It was indirectly mentioned in this bugtracker discussion, and I was happy to see Gerd participate so I threw some ideas about it, but I don't know whether working on that would be worth it right now.

Gerd, I'm happy to get design advice or indications of which kind of patches we could consider sending you if/when we get conclusive data on ocamlfind performance impact, but I would be embarrassed if you invested more time than that on the matter for now.

@vicuna
Copy link
Author

vicuna commented Jul 27, 2013

Comment author: Camarade_Tux

I need to add something: there have been complaints about the speed impact of ocamlbuild with ocamlfind but currently it's an issue that is opened in the bug trackers without enough information.

Which free software exhibit this issue besides Mirage? What are the characteristics? Are there camlp4 calls and is so, how much? How many files are there, how are they laid out on disk? How are the inter-files dependencies? Are packs involved? How many libraries are available on the system through ocamlfind? Are some of the libraries that are used not installed (i.e. living in src/lib1 instead of /usr)?

That's to name a few. More details are needed because right now it feels like optimizing for an imaginary workload (I'm not saying that it is, I'm saying that it feels like it is).

@vicuna
Copy link
Author

vicuna commented Jul 27, 2013

Comment author: gerd

Here is an easy observation about the speed of ocamlfind:

$ time ocamlfind 2>/dev/null
real 0m0.001s
user 0m0.000s
sys 0m0.000s

This is basically the time for launching the executable. As you see, only 1ms (on Linux).

$ time ocamlfind ocamlc
real 0m0.004s
user 0m0.000s
sys 0m0.004s

Here it is doing a bit more, but not much. The difference is that it runs ocamlc.opt as subprocess. Roughly, 1ms goes on ocamlfind's account, and 3ms on ocamlc.opt's.

$ time ocamlfind ocamlc -package netstring,pxp,lwt,ssl,yojson,zip
real 0m0.012s
user 0m0.008s
sys 0m0.004s

This command looks a number of packages up, parses the META file, resolves dependencies, and calls ocamlc.opt with a lot of -I switches. These are extra 8ms, completely eaten by user code (i.e. it's not the syscalls). This would need more analysis. Maybe caching META data would help here (the suggested pipe mode would also be good for this). However, is 8ms something we should worry about, when the following compiler invocation consumes a multitude of it? Also, maybe some performance tweaks in ocamlfind could also help (not sure which, but if you look close enough you usually find something).

@vicuna
Copy link
Author

vicuna commented Jul 27, 2013

Comment author: gerd

Also I was looking into the findlib source code. There was a performance fix in 1.4 (processing of long argument lists), so maybe this plays a role here?

Anyway, we need a concrete case before we can look into it.

@vicuna
Copy link
Author

vicuna commented Jul 27, 2013

Comment author: meyer

gasche: so dry run indeed would be useful, except is not possible without executing the body of the rule (as you know ocamlbuild discovers dependencies at each invocation of rule). If that was possible it would be much easier to improve paralelisation.

For the question of why we would like to parse META files, I think the biggest benefit is to get per tree package usage, dependencies that normally are specified with "package(foo)". If others do it and it's useful then we might be doing that as well.

@vicuna
Copy link
Author

vicuna commented Jul 27, 2013

Comment author: meyer

So possible uses of META files:

  • automatic resolution of findlib dependencies
  • specification of targets to be build
  • installation of the needed files to the findlib repository

Gerd: One of the solutions to mitigate the need of including the findlib in ocamlbuild tree (probably difficult at the moment at least, but worth to invesigating at some point), is to allow ocamlfind to read meta files from the file system, and then query should work as usual, except META is read from an explicit location.

@vicuna
Copy link
Author

vicuna commented Jul 28, 2013

Comment author: @gasche

I feel that you're stepping on OASIS' shoes when going in this direction. Oasis does require metadata that can be used to generate both a META file, and an ocamlbuild scaffolding.

Thinking of use-cases without OASIS:
(1) the first point may be convenient (it allows to not repeat ocamlfind dependencies in both the META file)
(3) the third feels outside the scope of a build system (we may provide feature to specify which files has been built, that is Daniel Bünzli's #6067, but installing them is not a build system's job)
(2) the second is a bit in between; I agree it could make sense, but I suspect that an install system knows more about targets to be built than the META file -- which misses for example .mli files and documentation.

@vicuna
Copy link
Author

vicuna commented Jul 28, 2013

Comment author: william

I tested the "-use-ocamlfind" option on ocamlbuild 3.12.1, on debian (packaged stable version).

  1. I run ocamlbuild with the "-use-ocamlfind" option :
    Finished, 406 targets (0 cached) in 00:02:20.
    real 2m20.919s /2m21.734s
    user 2m10.160s /2m11.076s

  2. I run ocamlbuild without the option. The funny thing being that it also works! (yes, ocamlbuild, at least 3.12.1, already supports some ocamlfind features...) :
    Finished, 406 targets (0 cached) in 00:02:18.
    real 2m18.449s/2m19.035s
    user 2m9.760s /2m9.484s

Note that for my project most of the time is spent on actually compiling stuff, so using or not using ocamlfind is no performance issue at all. It contains objects (which takes time to compile) and it was done on an old laptop (core 2 duo proc)

@vicuna
Copy link
Author

vicuna commented Jul 28, 2013

Comment author: meyer

While I agree that this OASIS job to do certain things, we need to be able to be flexible enough to support this, this way or other way. At the moment I agree with gasche that maybe full support for META might be an overkill, however, with the proposal of the adding user options to the compilation commandline of plugins;

#6093

makes it relatively easy to write a plugin, distributed for instance as ocamlfind package, to do the job of parsing the META files.

@vicuna
Copy link
Author

vicuna commented Aug 19, 2013

Comment author: @damiendoligez

I'm not sure I understand the remark about setting the target to the current release (my goal is to have zero PR with 4.01.0 as target before we release). I'm setting this to feedback because there is obviously an interesting discussion going on. Move it to "confirmed" when we have consensus on what needs to be done.

@vicuna
Copy link
Author

vicuna commented Mar 1, 2017

Comment author: @damiendoligez

ocamlbuild is now a separate project that lives on GitHub.
PR transferred to ocaml/ocamlbuild#195

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant