Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0005283OCamlOCaml generalpublic2011-06-06 18:192013-10-08 15:43
Reporterlefessan 
Assigned Toxleroy 
PrioritynormalSeverityfeatureReproducibilityhave not tried
StatusfeedbackResolutionopen 
PlatformallOSallOS Versionall
Product Version3.12.0 
Target VersionFixed in Version 
Summary0005283: patch to pack functors
DescriptionThis patch aims at building multi-units functors, i.e. a functor whose body is defined in several compilation units.

An example is provided in test-run.tar.gz.

Each compilation unit in the functor should be compiled with "-functor x.mli", where x.mli (or x.cmi) defines the interface of the functor argument (several -functor arguments can be provided).
To generate a functor, the option "-pack-functor FunctorName" should be provided. It will generate a module (as defined by "-o NameOfModule.cmo/.cmx"), containing the functor FunctorName, with one argument (corresponding to the -functor used for all its arguments), and all the provided modules as body.

As an independant feature, it allows to pack interfaces:
ocamlc -pack -o abc.cmi a.cmi b.cmi c.cmi
an option that was not implemented before, while still useful.

The patch is applied against 3.12.0, but should work with minimal changes against next versions.
Steps To ReproduceExample of usage:

        $(OCAMLC) -c arguments/x.mli
        $(OCAMLC) -c -functor arguments/x.mli arguments/y.mli
        $(OCAMLC) -c -functor arguments/y.mli -functor arguments/x.mli a.mli
        $(OCAMLC) -c -functor arguments/y.mli -for-pack Lib.Make_a -functor arguments/x.mli a.ml
        $(OCAMLC) -I arguments -functor arguments/x.mli -for-pack Lib -pack-functor Make -o make_a.$(CMO) a.$(CMO)
        $(OCAMLC) -c -functor arguments/x.mli -for-pack Lib b.ml
        $(OCAMLC) -I arguments -pack-functor Make -o lib.$(CMO) make_a.$(CMO) b.$(CMO)
Tagspatch
Attached Filespatch file icon lib-as-functor-3.12.0.2011-06-06.patch [^] (99,641 bytes) 2011-06-06 18:19 [Show Content]
gz file icon test-run.tar.gz [^] (1,400 bytes) 2011-06-06 18:19

- Relationships

-  Notes
(0005984)
lefessan (developer)
2011-06-06 18:24

I should add it was a request of Jane Street, and it builds on Alain's idea to extend the pack mechanism for functors, from this caml-list thread:

http://groups.google.com/group/fa.caml/browse_thread/thread/86d82441e7ce7e9a/246f18fe94ee8b06?hl=en&ie=UTF-8&q=yminsky+functorize+multiple+files&pli=1 [^]
(0005985)
guesdon (manager)
2011-06-06 18:31

I'm afraid that programs will become harder to understandif part of the code moves to the Makefile.
And it will become a nightmare to develop tools (I'm thinking about ocamldoc here, for example).
(0006007)
yminsky (reporter)
2011-06-14 00:09
edited on: 2011-06-14 00:12

Fabrice mentioned that Xavier was interested in seeing some examples. We certainly have them. Sadly, the description is a little abstract, but here goes:

One of our core applications is a big framework organized around a central functor where the business logic is contained. In that core application, we have a somewhat awkward mix of polymorphism and functor application, to avoid the need to functorize the entire library.

That core application worked out OK because the polymorphism that needed to be extended out was fairly small. But then we built a user-interface system for connecting to instances of this core application. The polymorphism in the UI was out of control, so that we ended up with types that had 8 free type variables. It was horribly confusing, and the whole thing would have gone away if we could have functorized the whole thing around a single module.

In the end, we used first-class packaged modules to dynamically choose the module in question based on an environment variable. That got rid of the confusing type parameters. But it's an ugly solution, and it also required us to restructure our code considerably to make this approach work.

Basically, we had weeks of rewriting of an application to deal with the lack of this feature, and we ended up with an uglier solution than we would have liked.

Sadly, I can point to no examples of this in our publically released code. That said, this thread from 9 years ago on the caml list came up in my work on the SKS keyserver, and hits on very much the same issue:

http://caml.inria.fr/pub/ml-archives/caml-list/2002/03/67cc0f6f75a9bd7ced9f03f724a92bdf.en.html [^]

Here's the source for that:

http://code.google.com/p/sks-keyserver/ [^]

(0006595)
chambart (developer)
2012-01-05 11:20

This can also be usefull for ocsigen.

In ocsigen a site is registered by calling the service creation functions in the right context. This is achieved by dynlinking the cma file of the site at the right time during the server initialisation.

The problem is that it is difficult to do static linking of the whole server with the site: There is no right order of evaluation of the different modules, the initialisation must occur in the middle of a function.

It is possible to circumvent that by doing the whole site initialisation inside a single function, but that adds the restriction that every services must be declared in the same file.

With this functor pack option, we could transform every site into a functor, and registration whould be applying the functor to its context.
(0006771)
xleroy (administrator)
2012-01-23 12:01

This proposal was discussed among the dev team, so let me record the pros and cons.

- On the positive side: this is a natural extension of the "-pack" mechanism that can indeed be useful to functorize code a posteriori without the burden of functorizing by hand each compilation unit.

- On the negative side: several important users of OCaml (e.g. Jane Street), as well as many of the dev team members, are unhappy about "-pack" (inefficient linking, lack of tool support, etc) and have been pushing for other approaches to namespace management that would subsume "-pack". Several namespace designs were floated on the caml-devel and caml-consortium mailing lists. My big concern here is that if we add packed functors (this patch) on top of packed modules (the -pack option), we get something that is very very hard to subsume via a namespace mechanism. In turn, this would, in my opinion, delay or even prohibit the integration of a different namespace mechanism.

So, here is my question to the supporters of this proposal: if you have to choose between packed functors (this proposal) or a future namespace mechanism, which one do you choose, knowing that you won't get both? Which one is more important to you?

To finish, Alain Frisch also remarked that for small to medium-sized collections of compilation units, one can always use "cat" and a bit of shell scripting to produce the packed functor as an OCaml source file, then compile it as usual.

(0006772)
frisch (developer)
2012-01-23 12:26

We don't have a real need either for the current -pack or for a new namespace mechanism. Being able to split the implementation of a functor's body over several files would be useful to us.

Another approach to this problem could be an ad hoc (independent of -pack) solution where the compiler is instructed to compile one functor by taking the source code in several files; this would be cleaner than the "cat" solution.
(0006777)
hcarty (reporter)
2012-01-23 14:57

I would much rather have a namespace mechanism.
(0006784)
glondu (reporter)
2012-01-23 16:35

Generating big files with a mechanism external to the compiler looks fine to me. A namespace mechanism looks more useful.
(0006789)
yminsky (reporter)
2012-01-24 02:45

A few thoughts:

A cat based solution seems decidedly second class. Among things that won't work well:

a) resolution of error messages in the tools to the proper files. Maybe there's some way to make this work nicely, but cat isn't it.
b) It also plays poorly with other parts of the building process. What if you want to run different syntax extensions on different parts of the code thus combined?
c) We have some non-trivial codebases that we'd want to be able to use this with. Since this kills separate compilation, such codebases would become quite inconvenient to build.

All of which is to say: the "cat" implementation seems pretty painful. Maybe there's some intermediate solution that doesn't use -pack, as Alain suggests.

I agree that namespaces are more important, though, since some workaround for library-wide functors that addresses at least point (a) above is probably possible.
(0006802)
frisch (developer)
2012-01-25 20:20

Addressing a) is easy with a shell script (or within omake directly); it's just about adding a #line-directive before each included file.

For b), I've no real solution. Do you really use incompatible syntax extensions for various modules intended to be part of the same functor's body?

For c), are you concerned with the complexity of the build system or with slower builds?
(0008548)
hongboz (developer)
2012-11-30 18:04

For b) it can be solved in the future, each file can describe which syntax it used, self-explainable, and we could provide a include which respects this

- Issue History
Date Modified Username Field Change
2011-06-06 18:19 lefessan New Issue
2011-06-06 18:19 lefessan Status new => assigned
2011-06-06 18:19 lefessan Assigned To => xleroy
2011-06-06 18:19 lefessan File Added: lib-as-functor-3.12.0.2011-06-06.patch
2011-06-06 18:19 lefessan File Added: test-run.tar.gz
2011-06-06 18:24 lefessan Note Added: 0005984
2011-06-06 18:31 guesdon Note Added: 0005985
2011-06-14 00:09 yminsky Note Added: 0006007
2011-06-14 00:12 yminsky Note Edited: 0006007
2012-01-05 11:20 chambart Note Added: 0006595
2012-01-23 12:01 xleroy Note Added: 0006771
2012-01-23 12:01 xleroy Status assigned => feedback
2012-01-23 12:26 frisch Note Added: 0006772
2012-01-23 14:57 hcarty Note Added: 0006777
2012-01-23 16:35 glondu Note Added: 0006784
2012-01-24 02:45 yminsky Note Added: 0006789
2012-01-25 20:20 frisch Note Added: 0006802
2012-11-30 18:04 hongboz Note Added: 0008548
2013-10-08 15:43 doligez Tag Attached: patch
2013-10-08 15:43 doligez Target Version 3.12.0 =>


Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker