Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some dependencies cannot be detected by ocamldep #5624

Closed
vicuna opened this issue May 25, 2012 · 13 comments
Closed

Some dependencies cannot be detected by ocamldep #5624

vicuna opened this issue May 25, 2012 · 13 comments

Comments

@vicuna
Copy link

vicuna commented May 25, 2012

Original bug ID: 5624
Reporter: @alainfrisch
Status: acknowledged (set by @damiendoligez on 2013-06-28T15:27:08Z)
Resolution: open
Priority: normal
Severity: minor
Target version: later
Category: tools (ocaml{lex,yacc,dep,debug,...})
Related to: #7080 #7470
Monitored by: @nojb @ygrek @glondu "Julien Signoles"

Bug description

ocamldep, because it operates purely syntactically on the source files, misses some dependencies need to unroll type abbreviations. Example: c.ml refers to a type B.t, which is defined in b.mli by "type t = A.t"; if c.ml does not otherwise refer to module A, there is no possibility for ocamldep to detect a dependency from c.cmo to a.cmi. Nevertheless, the compiler will indeed need a.cmi when compiling c.cmo.

Usually, this is not problematic, but it can be with non-trivial build rules which copy files around.

Example:

foo/a.mli:  type t = int
foo/b.mli:  type t = A.t
c.ml:       let x : B.t = 2 

and the following Makefile

## Implicit compilation rules
.SUFFIXES: .ml .mli .cmi .cmo
.mli.cmi:
    ocamlc -c -I $(shell dirname $<) $<
.ml.cmo:
    ocamlc -c -I $(shell dirname $<) $<

## Installation rules
a.cmi: foo/a.cmi
    cp $< $@
b.cmi: foo/b.cmi
    cp $< $@

## Dependencies (as understood by ocamldep)
c.cmo: b.cmi
foo/b.cmi: foo/a.cmi

Dependencies have been obtained with ocamldep -modules, and then translated to file dependencies according to the corresponding -I flags used for compilation (this is how ocamldep, and -- I assume -- ocamlbuild work). The inferred dependencies are:

foo/a.mli:
foo/b.mli: A
c.ml: B

Then make c.cmo fails because of a missing dependency from c.cmo to a.cmi.

Several approaches are possible to "fix" this:

  • Prevent the compiler from looking for a file a.cmi unless ocamldep -modules finds a dependency to module A (i.e. run the dependency analysis before type-checking). This is not very nice, since some type abbreviations are interpreted as abstract types, and this can break type-checking in ways which are not straightforward to understand (the fix is to add a dummy reference to A if the abbreviation is needed). This is the approach we follow at LexiFi.

  • Include dependent .cmi. In the example above, b.cmi would contain a copy of a.cmi. Of course, this can lead to very large .cmi files.

  • An optimization is to inline only the required external type abbreviations in the .cmi files (in the example above, b.cmi would be extended to include the fact that A.t = int). Note that this solution can also improve compilation time, because the compiler doesn't need to search and open .cmi files only to expand abbreviations.

@vicuna
Copy link
Author

vicuna commented May 25, 2012

Comment author: @gasche

ocamldep is an heuristic tool based on some assumptions about how OCaml users typically organize their development. I'm not sure it is reasonable to try to evolve it into an omniscient tool capable of dealing with your sophisticated file manipulations. That's what "by hand" compilation scripts (or special-case tools designed hand-in-hand with your quite specific build system) are for: absolute flexibility.

I'm not convinced in particular that doing deep modification to the semantics of .cmi files and/or the compilation process to work around limitations of ocamldep is such a good idea.

I would rather welcome richer ways to inform the compiler of the mapping from Ocaml world compilation unit names to filesystem object files than the current rules allow; and/or potentially a type-checker design facilitating discovery of dependencies and interaction with library providers and build systems (implement what currently is ocamldep as a layer between parsing and type-checking, that resolves such mapping, is able to access existing .cmi files for semantically accurate naming information, and to make asynchronous queries to a build system for the missing .cmi).

@vicuna
Copy link
Author

vicuna commented May 25, 2012

Comment author: @alainfrisch

I don't think it is reasonable to expect large code base to be based on a manual analysis of dependencies across the project: this needs to be automated. At least omake (and maybe ocamlbuild, I don't know) allows using customized rules (compiling individual files with different flags and -I dirs, copying rules, code generation like ocamlyacc and even when the generator tool itself is part of the project, etc) and still supports automatic analysis of dependencies. It really works well, except for the specific issue I raise here.

Only my first proposal changes the "semantics" of the system. The second and third ones should not introduce any bad behavior.

That said, I agree with you. More advanced build tools that communicate directly with the compilers would open new doors.

@vicuna
Copy link
Author

vicuna commented Jun 26, 2012

Comment author: @damiendoligez

I'm setting this PR to "feedback" because a fix is not obvious and it still needs discussion.
Everyone's input is welcome.

@vicuna
Copy link
Author

vicuna commented Jun 12, 2013

Comment author: @alainfrisch

Pushing to target version = "later". This problem has been around for a long time and is not critical.

@vicuna
Copy link
Author

vicuna commented Jun 12, 2013

Comment author: @glondu

In the given example, I wouldn't blame ocamldep, but rather a missing dependency from b.cmi to a.cmi.

Why is the missing dependency from c.cmi to a.cmi considered a bug?

@vicuna
Copy link
Author

vicuna commented Jun 12, 2013

Comment author: @alainfrisch

ocamldep reports correctly that foo/b.mli depends on module A, which is then interpreted as a file dependency "foo/b.cmi: foo/a.cmi". Where do you see a missing dependency from b.cmi to a.cmi?

Why is the missing dependency from c.cmi to a.cmi considered a bug?

If foo/a.cmi changes, c.cmo needs to be rebuild. It is generally assumed that it is fine if ocamldep returns correct dependencies "modulo a transitive closure", because the build system will perform this transitive closure. But the example shows that this is not the case. When foo/a.mli changes, and we ask make to ensure that c.cmo is up to date, it will refresh b.cmi, forcing foo/b.cmi to be refreshed, forcing foo/a.cmi to be refreshed. But when c.ml is finally recompiled, nothing forced a.cmi to be refreshed by copying foo/a.cmi over. This is because c.cmo does not depend on a.cmi, which is precisely our problem here.

@vicuna
Copy link
Author

vicuna commented Jun 12, 2013

Comment author: @glondu

Where do you see a missing dependency from b.cmi to a.cmi?

I meant without the leading "foo/".

If you add "b.cmi: a.cmi" (without leading "foo/") to your Makefile, then everything works as expected:

First time

$ make c.cmo
ocamlc -c -I foo foo/a.mli
ocamlc -c -I foo foo/b.mli
cp foo/a.cmi a.cmi
cp foo/b.cmi b.cmi
ocamlc -c -I . c.ml

Second time

$ touch foo/a.mli
$ make c.cmo
ocamlc -c -I foo foo/a.mli
ocamlc -c -I foo foo/b.mli
cp foo/a.cmi a.cmi
cp foo/b.cmi b.cmi
ocamlc -c -I . c.ml

Just to be sure...

$ make c.cmo
make: « c.cmo » est à jour.

@vicuna
Copy link
Author

vicuna commented Jun 12, 2013

Comment author: @alainfrisch

If you add "b.cmi: a.cmi"

Where would this information come from? Maybe the example with a Makefile is not very good, because the Makefile are usually not very compositional. Imagine an entire project controlled by, say, omake. In the foo/ directory, generic rules apply and deal with the dependency from b.cmi to a.cmi. Then, we somehow inject copy rules from foo/ to another directory (say, the root of the project, in the example), with proper dependencies (say, from b.cmi to foo/b.cmi). I don't know how to tell omake, in a modular way, that it should also copy the dependency from foo/b.cmi to foo/a.cmi in order to produce a dependency from b.cmi to a.cmi. And anyway, this would not be conceptually very clean: b.cmi does not depend on a.cmi, since b.cmi is obtained directly from foo/b.cmi; concretely, if I remove a.cmi at the root, there is no need to copy it back in order to bring b.cmi up to date. So forcing a dependency from b.cmi to a.cmi at the root is really a hack to work-around the problem.

For me, the notion of dependency is rather simple: if the compilation command required to produce c.cmo needs to read a.cmi, then c.cmo depends on a.cmi. The fact is that ocamldep has no way to "predict" that.

@github-actions
Copy link

This issue has been open one year with no activity. Consequently, it is being marked with the "stale" label. What this means is that the issue will be automatically closed in 30 days unless more comments are added or the "stale" label is removed. Comments that provide new information on the issue are especially welcome: is it still reproducible? did it appear in other contexts? how critical is it? etc.

@github-actions github-actions bot added the Stale label May 15, 2020
@nojb nojb removed the Stale label May 15, 2020
@yawaramin
Copy link
Contributor

@nojb given that dune exists, is ocamldep supported any more? It's not mentioned as part of OCaml Platform.

@nojb
Copy link
Contributor

nojb commented Jan 10, 2021

@nojb given that dune exists, is ocamldep supported any more? It's not mentioned as part of OCaml Platform.

ocamldep is very much supported! dune depends on it to compute dependencies and it is pretty much the only game in town. (There is also https://github.com/Octachron/codept, but am not sure what is the status of it and its future prospects ... cc @Octachron.)

I guess the reason it is not mentioned by the platform docs is that it is supposed to be used behind the scenes by other tools (eg dune) and maybe not directly invoked by the casual user.

@github-actions
Copy link

This issue has been open one year with no activity. Consequently, it is being marked with the "stale" label. What this means is that the issue will be automatically closed in 30 days unless more comments are added or the "stale" label is removed. Comments that provide new information on the issue are especially welcome: is it still reproducible? did it appear in other contexts? how critical is it? etc.

@Octachron
Copy link
Member

Concerning codept, it works on current OCaml version, and it is able to unroll dependencies generated by type abbreviations. Thus this issue is theoretically solved; even if in practice, there is no (few?) users of codept.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants