Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0005624OCamlOCaml typingpublic2012-05-25 15:122013-06-28 17:27
Reporterfrisch 
Assigned To 
PrioritynormalSeverityminorReproducibilityhave not tried
StatusacknowledgedResolutionopen 
PlatformOSOS Version
Product Version 
Target VersionlaterFixed in Version 
Summary0005624: Some dependencies cannot be detected by ocamldep
Descriptionocamldep, because it operates purely syntactically on the source files, misses some dependencies need to unroll type abbreviations. Example: c.ml refers to a type B.t, which is defined in b.mli by "type t = A.t"; if c.ml does not otherwise refer to module A, there is no possibility for ocamldep to detect a dependency from c.cmo to a.cmi. Nevertheless, the compiler will indeed need a.cmi when compiling c.cmo.

Usually, this is not problematic, but it can be with non-trivial build rules which copy files around.

Example:

foo/a.mli: type t = int
foo/b.mli: type t = A.t
c.ml: let x : B.t = 2

and the following Makefile


===================================================================
## Implicit compilation rules
.SUFFIXES: .ml .mli .cmi .cmo
.mli.cmi:
    ocamlc -c -I $(shell dirname $<) $<
.ml.cmo:
    ocamlc -c -I $(shell dirname $<) $<

## Installation rules
a.cmi: foo/a.cmi
    cp $< $@
b.cmi: foo/b.cmi
    cp $< $@

## Dependencies (as understood by ocamldep)
c.cmo: b.cmi
foo/b.cmi: foo/a.cmi
===================================================================


Dependencies have been obtained with "ocamldep -modules", and then translated to file dependencies according to the corresponding -I flags used for compilation (this is how ocamldep, and -- I assume -- ocamlbuild work). The inferred dependencies are:

foo/a.mli:
foo/b.mli: A
c.ml: B

Then "make c.cmo" fails because of a missing dependency from c.cmo to a.cmi.



Several approaches are possible to "fix" this:

 - Prevent the compiler from looking for a file a.cmi unless "ocamldep -modules" finds a dependency to module A (i.e. run the dependency analysis before type-checking). This is not very nice, since some type abbreviations are interpreted as abstract types, and this can break type-checking in ways which are not straightforward to understand (the fix is to add a dummy reference to A if the abbreviation is needed). This is the approach we follow at LexiFi.

 - Include dependent .cmi. In the example above, b.cmi would contain a copy of a.cmi. Of course, this can lead to very large .cmi files.

 - An optimization is to inline only the required external type abbreviations in the .cmi files (in the example above, b.cmi would be extended to include the fact that "A.t = int"). Note that this solution can also improve compilation time, because the compiler doesn't need to search and open .cmi files only to expand abbreviations.

TagsNo tags attached.
Attached Files

- Relationships

-  Notes
(0007464)
gasche (developer)
2012-05-25 17:58

ocamldep is an heuristic tool based on some assumptions about how OCaml users typically organize their development. I'm not sure it is reasonable to try to evolve it into an omniscient tool capable of dealing with your sophisticated file manipulations. That's what "by hand" compilation scripts (or special-case tools designed hand-in-hand with your quite specific build system) are for: absolute flexibility.

I'm not convinced in particular that doing deep modification to the semantics of .cmi files and/or the compilation process to work around limitations of ocamldep is such a good idea.

I would rather welcome richer ways to inform the compiler of the mapping from Ocaml world compilation unit names to filesystem object files than the current rules allow; and/or potentially a type-checker design facilitating discovery of dependencies and interaction with library providers and build systems (implement what currently is ocamldep as a layer between parsing and type-checking, that resolves such mapping, is able to access existing .cmi files for semantically accurate naming information, and to make asynchronous queries to a build system for the missing .cmi).
(0007465)
frisch (developer)
2012-05-25 18:10

I don't think it is reasonable to expect large code base to be based on a manual analysis of dependencies across the project: this needs to be automated. At least omake (and maybe ocamlbuild, I don't know) allows using customized rules (compiling individual files with different flags and -I dirs, copying rules, code generation like ocamlyacc and even when the generator tool itself is part of the project, etc) and still supports automatic analysis of dependencies. It really works well, except for the specific issue I raise here.

Only my first proposal changes the "semantics" of the system. The second and third ones should not introduce any bad behavior.

That said, I agree with you. More advanced build tools that communicate directly with the compilers would open new doors.
(0007612)
doligez (administrator)
2012-06-26 18:12

I'm setting this PR to "feedback" because a fix is not obvious and it still needs discussion.
Everyone's input is welcome.
(0009464)
frisch (developer)
2013-06-12 17:16

Pushing to target version = "later". This problem has been around for a long time and is not critical.
(0009467)
glondu (reporter)
2013-06-12 17:53

In the given example, I wouldn't blame ocamldep, but rather a missing dependency from b.cmi to a.cmi.

Why is the missing dependency from c.cmi to a.cmi considered a bug?
(0009469)
frisch (developer)
2013-06-12 18:21

ocamldep reports correctly that foo/b.mli depends on module A, which is then interpreted as a file dependency "foo/b.cmi: foo/a.cmi". Where do you see a missing dependency from b.cmi to a.cmi?

> Why is the missing dependency from c.cmi to a.cmi considered a bug?

If foo/a.cmi changes, c.cmo needs to be rebuild. It is generally assumed that it is fine if ocamldep returns correct dependencies "modulo a transitive closure", because the build system will perform this transitive closure. But the example shows that this is not the case. When foo/a.mli changes, and we ask make to ensure that c.cmo is up to date, it will refresh b.cmi, forcing foo/b.cmi to be refreshed, forcing foo/a.cmi to be refreshed. But when c.ml is finally recompiled, nothing forced a.cmi to be refreshed by copying foo/a.cmi over. This is because c.cmo does not depend on a.cmi, which is precisely our problem here.
(0009471)
glondu (reporter)
2013-06-12 20:15

> Where do you see a missing dependency from b.cmi to a.cmi?

I meant without the leading "foo/".

If you add "b.cmi: a.cmi" (without leading "foo/") to your Makefile, then everything works as expected:

# First time
$ make c.cmo
ocamlc -c -I foo foo/a.mli
ocamlc -c -I foo foo/b.mli
cp foo/a.cmi a.cmi
cp foo/b.cmi b.cmi
ocamlc -c -I . c.ml
# Second time
$ touch foo/a.mli
$ make c.cmo
ocamlc -c -I foo foo/a.mli
ocamlc -c -I foo foo/b.mli
cp foo/a.cmi a.cmi
cp foo/b.cmi b.cmi
ocamlc -c -I . c.ml
# Just to be sure...
$ make c.cmo
make: « c.cmo » est à jour.
(0009472)
frisch (developer)
2013-06-12 21:08

> If you add "b.cmi: a.cmi"

Where would this information come from? Maybe the example with a Makefile is not very good, because the Makefile are usually not very compositional. Imagine an entire project controlled by, say, omake. In the foo/ directory, generic rules apply and deal with the dependency from b.cmi to a.cmi. Then, we somehow inject copy rules from foo/ to another directory (say, the root of the project, in the example), with proper dependencies (say, from b.cmi to foo/b.cmi). I don't know how to tell omake, in a modular way, that it should also copy the dependency from foo/b.cmi to foo/a.cmi in order to produce a dependency from b.cmi to a.cmi. And anyway, this would not be conceptually very clean: b.cmi does not depend on a.cmi, since b.cmi is obtained directly from foo/b.cmi; concretely, if I remove a.cmi at the root, there is no need to copy it back in order to bring b.cmi up to date. So forcing a dependency from b.cmi to a.cmi at the root is really a hack to work-around the problem.

For me, the notion of dependency is rather simple: if the compilation command required to produce c.cmo needs to read a.cmi, then c.cmo depends on a.cmi. The fact is that ocamldep has no way to "predict" that.

- Issue History
Date Modified Username Field Change
2012-05-25 15:12 frisch New Issue
2012-05-25 17:58 gasche Note Added: 0007464
2012-05-25 18:10 frisch Note Added: 0007465
2012-06-26 18:12 doligez Note Added: 0007612
2012-06-26 18:12 doligez Status new => feedback
2012-07-09 14:54 doligez Target Version => 4.01.0+dev
2012-07-31 13:36 doligez Target Version 4.01.0+dev => 4.00.1+dev
2012-09-21 14:26 doligez Target Version 4.00.1+dev => 4.01.0+dev
2013-06-12 17:16 frisch Note Added: 0009464
2013-06-12 17:16 frisch Status feedback => new
2013-06-12 17:16 frisch Status new => feedback
2013-06-12 17:16 frisch Target Version 4.01.0+dev => later
2013-06-12 17:53 glondu Note Added: 0009467
2013-06-12 18:21 frisch Note Added: 0009469
2013-06-12 18:21 frisch Status feedback => new
2013-06-12 20:15 glondu Note Added: 0009471
2013-06-12 21:08 frisch Note Added: 0009472
2013-06-28 17:27 doligez Status new => acknowledged


Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker