Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0006402OCamlOCamlbuild (the tool)public2014-05-07 11:052014-10-18 12:11
Reporterhhugo 
Assigned To 
PrioritynormalSeverityfeatureReproducibilityN/A
StatusacknowledgedResolutionopen 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version 
Summary0006402: move ocamlbuild to its own repository.
Description - so releases are not necessary mapped to OCaml's one
 - so improvement can benefit build with other OCaml version
TagsNo tags attached.
Attached Files

- Relationships

-  Notes
(0011379)
gildor (developer)
2014-05-07 11:31

I think this will be an overall improvement:
- hopefully benefit from external users contribution
- allow to fix rules outside of OCaml release cycle

+1
(0011381)
jpdeplaix (reporter)
2014-05-07 14:01

I also think it would be a good thing to do.

Some times ago, this idea of spliting the tools which are in the distribution was raised and I wasn't sure whenever it was a good idea or not for ocamlbuild. My main con, was that I think that one reason why ocamlbuild is used is that it's distributed with ocaml.

But now, after thinking of it a little bit more, I think this con can be erased by « massive » contributions and a shorter release cycle.

I tried to do the split last week but I didn't had the time to finish it.
(0011383)
hhugo (reporter)
2014-05-07 15:26

https://github.com/hhugo/ocamlbuild [^]
(0011384)
gildor (developer)
2014-05-07 15:45

It should be in https://github.com/ocaml/ocamlbuild... [^]

Maybe contact Anil or Fabrice Lefessan about that.
(0011385)
gildor (developer)
2014-05-07 15:45

Also, if you get approval for the move to github/ocaml/, I am willing to spend five minute to transfer it.
(0011386)
hhugo (reporter)
2014-05-07 16:34

I can already change the github repo owner to the ocaml organisation
(0011387)
gildor (developer)
2014-05-07 16:48

hhugo, but there will be some conflict if you do so! Please contact Anil/Fabrice before.
(0011389)
avsm (reporter)
2014-05-07 17:59

Glad to see you working on this! It would be best to leave it as a private fork until there is agreement from the development team that this is a direction we want a future release of OCaml to go in.

OCaml 4.02 is in feature freeze right now in preparation for branching for release, so it would probably be best to bring this up in a week or so after that branch has been created, and trunk is 4.03dev.
(0011394)
hhugo (reporter)
2014-05-07 21:37

Thanks for the details.

I've just synced the repo with the latest trunk (with "safe-string" reverted to build on 4.01). gildor, jpdeplaix, you now both have access to this repo if you wish to contribute
(0011395)
gildor (developer)
2014-05-07 21:51

Draft of TODO goals:
https://github.com/hhugo/ocamlbuild/wiki [^]

If you want I can setup a forge project for mailing list/tarballs/homepage...
(0011397)
avsm (reporter)
2014-05-07 23:30

Please don't do that until there's consensus from the core team, as otherwise it'll be forking confusion as things get out of sync.
(0011398)
gildor (developer)
2014-05-08 00:32

ACK, I don't play against the core team. I suppose hhugo will agree that if the core team is opposed to this decision we will just withdraw the ocamlbuild git repository and that will be the end of the story.
(0011399)
berke.durak (reporter)
2014-05-08 05:07

Not sure about this; I think it's a good thing that a proper build tool ships with Ocaml itself. If you move it to its own repo, people will inevitably want to use this or that external module, and you can't ship that with the compiler.

I seem to recall that I had to write Ocamlbuild's glob module in pure Ocaml to avoid a dependency on Str. And Nicolas wrote a My_unix module so that Ocamlbuild would work even with "degraded" Unix environments.

On the other hand it is true that the plugin/rule/tag system is difficult to use.

Why not fork Ocaml, work on Ocamlbuild, and use pull requests on Github?
(0011400)
gildor (developer)
2014-05-08 09:08

I think ocamlbuild is a quite separate project and easily splitable. Given the fact that you don't use ocamlbuild inside OCaml sources that a good candidate to be moved away.

If you have a different project, you'll be able to follow a different release cycle (shorter hopefully).

E.g. fix the .cmxs rule and release rather than wait for all other patches on all other topics to be merged.

Given the fact that camlp4 and tk lib has been moved to their own project, I didn't think it was a problem.

On the topic of using external module and rewriting code like "glob" or "my_unix", I don't share your opinion... I would use Str and Unix and if mandatory I would enforce thing like depending on findlib. I don't have enough time to redo everything.
(0011402)
gasche (developer)
2014-05-08 11:18
edited on: 2014-05-08 11:23

The main problem is that there are currently extremely few people remotely motivated to contribute to ocamlbuild's development. I count two, jpdeplaix and myself. That remains a problem whether or not ocamlbuild is split.

It's very good to be able to make releases independently of OCaml's release schedule. But on how many occasion would it have made sense, between the 4.01 and the 4.02 release? Answer: zero times. Nobody gave a damn about helping with ocamlbuild development (except jpdeplaix above) since the 4.01 release, so there was nothing to release.

Will having ocamlbuild in its own github repository encourage people to contribute? Well:
- ocaml already has a github repository, and we even monitor pull requests now, where would the added convenience be?
- camlp4 has been split into a separate github repository just before 4.01. How does being split as its own github work for camlp4? Besides Jérémie Dimino's maintenance work, there have been precisely four external contributions, three of them coming from regular contributors to the OCaml distribution itself. I can't see a wave of new contributors here.

(I also have a draft of improved OCamlbuild documentation, publicly announced 8 months ago, with a github repository. So far, besides one batch of typo fixes, no substantial contribution.)

I feel neutral about splitting ocamlbuild as a separate project: I see no reason why that would make any difference -- besides making "which build system to use?" discussions even funnier.

(0011403)
gildor (developer)
2014-05-08 11:55
edited on: 2014-05-08 11:56

What about taking it the other way: is keeping ocamlbuild inside OCaml sources providing a significant benefit to the core team?

Actually, I would be motivated to work on it. Esp. fixing .cmxs incremental build which is a pain point for OASIS. I don't contribute because I always postpone it given the time frame of releases. Now you may argue that I can do it and wait but it will not account for the thousands other urgent matters that I have to process. Lets call it lack of "immediate reward", something which our brain is pretty accustomed to.

Anyway, all these are just wild guesses and I suppose the best way to see if there will be an improvement in contributors is to try.

On the topic of "which build system to use?", I think this is the usual flamewar. I stopped reading caml-list 2 years ago because of this kind of discussion. Having "ocamlbuild" as an external project will not prevent 10 new "ultimate, parallel, fast, so much better and written by me" build systems to be created in 2014.

(0011404)
gasche (developer)
2014-05-08 12:19

Regardless of whether ocamlbuild is split, I'm certainly interested in trying to make OASIS's life easier. However, I have very little feedback about what your needs about OASIS would be; the little I get being in the form of occasional contributions from Jacques-Pascal (jpdeplaix), drawn out of any original OASIS context. Is your "cmxs incremental build" issue reported anywhere?
(0011405)
gildor (developer)
2014-05-08 12:41

cmxs issue is on my personal todo list and this is related to http://caml.inria.fr/mantis/view.php?id=5653 [^]

The bug is marked as solved, but this is a pain point and I cannot figure out an easy way to fix it.

Basically, in certain case you always build .cmxs, even if nothing has been updated. If you are interested I can find an example project that shows this behavior.

Anyway this is OT for the current bug ;-)
(0011406)
avsm (reporter)
2014-05-08 12:46

> It's very good to be able to make releases independently of OCaml's release schedule.
> But on how many occasion would it have made sense, between the 4.01 and the
> 4.02 release? Answer: zero times.

That's circular: the reason that I (for example) haven't contributed ocamlbuild bugfixes
back in the past is that I can't wait a year for the fix to show up in a release to unblock
my build. If ocamlbuild were easier to recompile separately (e.g. via an opam pin),
then working on it is a viable option. Otherwise, a Makefile or an alternative build
tool is the only way to get on with real work.

Given that ocamlbuild isn't used by the compiler itself, I think splitting it out after
this release would at least give it a fighting chance to become more usable.
(0011407)
garrigue (manager)
2014-05-08 15:17

Now that camlp4 is no longer in the distribution, the is no real point in keeping ocamlbuild there.
Moreover, ocamlbuild includes support for findlib, which is not part of the distribution.
So yes, I think there are very good reasons to split it, and let users improve it.
(0011417)
Camarade_Tux (reporter)
2014-05-10 19:42

I'm a bit parted on this issue.

It's been more than one year that I've stated I would prefer to have camlp4, (labltk,) ocamlbuild, ocamldoc separated from other bits of the compiler.
Separated or at least that it's possible to configure, build and install them on their own. So overall, I'm in favor of that.

That said, I'm worried about how the split would be done. Part of these worries is a general feeling that doing a proper split for ocamlbuild is more difficult than for camlp4 or labltk.
Camlp4 was something that could be disabled for quite some time and there was camlp5 to prove such a tool could live outside of the compiler. Labltk was an optional component for an even longer time and there was again something (lablgtk) to prove it could live nicely outside.
Now, ocamlbuild, most libraries and programs rely on it and the situation regarding build systems is weird enough that it justifies taking extra time and thinking _before_ moving it.

For starters, I believe the split of ocamlbuild should be decided before 4.02 and that a notice should be put in the release notes and in ./configure if it is to be split for 4.03.

I don't have strong justifictions for my worries; it's mainly a feeling but a feeling of someone who's dived very deep into build systems, including OCaml's.

One thing that definitely has to be changed is with the various "config" files: myocamlbuild_config.ml and config/Makefile. They hold the result of ./configure in two different formats. The information is valuable and has to be made available in a proper format to subsequent tools. I consider this a blocking issue before ocamlbuild can be split.

I'll try to come up with the various other issues in the coming weeks but with the preparation of the new version of http://win-builds.org [^] I've clearly fallen behind the cross-compilation work.
(0011442)
jpdeplaix (reporter)
2014-05-13 16:47

Now that the 4.02 has been froze, I guess we have to hope to have this for the 4.03 ? :'(
(0011445)
protz (manager)
2014-05-13 17:45
edited on: 2014-05-13 17:46

> One thing that definitely has to be changed is with the various "config" files:
myocamlbuild_config.ml and config/Makefile. They hold the result of ./configure
in two different formats. The information is valuable and has to be made
available in a proper format to subsequent tools. I consider this a blocking
issue before ocamlbuild can be split.

This comment above is the biggest concern imho and has not been answered yet. Those of you (gildor, jpdeplaix) who want to kick ocamlbuild out, what is your opinion on this particular point?

Thanks,

~ jonathan

(0011446)
gildor (developer)
2014-05-13 18:05

Can't we extract this information from 'ocamlc -config' ?
(0011447)
gildor (developer)
2014-05-13 18:12

I just had a quick look at myocamlbuild_config.ml and I think the best is to register what will be missing in 'ocamlc -config' (which means that it will help not only ocamlbuild but other build tools BTW) but we can already be in a pretty good shape with just what we have in ocamlc.

The other good point on relying on the output of 'ocamlc -config' is that we can use ocamlbuild with older version of ocaml...

Note that I already use this kind of information in OASIS.
(0011448)
Camarade_Tux (reporter)
2014-05-13 18:37
edited on: 2014-05-13 18:39

I hadn't thought about this earlier on but separating ocamlbuild from this data might be quite nice for the cross-compilation support.

There is no reason to have such values hardcoded into a build system and being able to point ocamlbuild to a different configuration file at runtime both through environment variables and commandl-line switches would be a plus imho.

And now I notice how much that could overlap with what ocamlfind is able to do. Maybe the data could be read by ocamlfind and then re-used by other tools?

PS: currently I cross-compile oasis/ocamlbuild-based build systems by pointing ocamlfind to specific configuration files that are setup for cross-compilation through environment variables; this is very nice but I'd hate to add another environment variable for a config file that duplicates some of the information.

edit: the process to create this file is: configure runs, creates config/Makefile at some point later on this file is transformed into myocamlbuild_config.ml; a solution that would unify both into a nicer format be nice to have (thinking about it, it is also possible to have configure write both files at the same time painlessly so this might not be a huge concern).

(0011463)
gasche (developer)
2014-05-14 17:16
edited on: 2014-05-14 17:16

It seems there is a consensus on the fact that the move is a good thing.

I think we should wait for the 4.02 release in any case.

(0011464)
gildor (developer)
2014-05-14 17:34

Here might be a proposal:
Step 1. make https://github.com/hhugo/ocamlbuild [^] builds on its own with latest version of ocaml
Step 2. create a PR for https://github.com/ocaml/ocaml [^] to strip ocamlbuild from it and check this works too and provide more information if needed
Step 3. discuss and try to get approval from INRIA core team to do the split officialy
Step 4. move https://github.com/hhugo/ocamlbuild [^] to https://github.com/ocaml/ocamlbuild, [^] integrate PR from Step 2 into the official OCaml repository and do the blog post/announce of the official split (if we get approval from Step 3)

This will help to have real proof of concept that it is doable. At this point, I think most of the question can be answered by a few weeks of coding and checking everything work.

hhugo, jpdeplaix do you agree with this plan ?
(0011469)
jpdeplaix (reporter)
2014-05-14 23:59

I've already committed something for removing ocamlbuild from trunk if you want: https://github.com/jpdeplaix/ocaml/tree/remove-ocamlbuild [^]

hhugo, I don't know if you already did that. Maybe you have something a bit different.

Anyway, I agree with the plan.
An other possible discussion would be: Do we want to have the separated ocamlbuild compatible with older ocaml versions (older than 4.03) or do we follow the same line as camlp4 does ?
(0011471)
Camarade_Tux (reporter)
2014-05-15 07:58

> Do we want to have the separated ocamlbuild compatible with older ocaml versions (older than 4.03) or do we follow the same line as camlp4 does ?

I'd say yes, at least for 4.02. It will make the transition much smoother and will make initiali development much easier: packages would use the version in the compiler tree for now and in a few months can switch to the external one by adding the package and configuring the compiler sources without ocamlbuild (unless I'm mistaken, that bit is in 4.02).
(0011479)
jpdeplaix (reporter)
2014-05-15 11:00

By « older version » I meant all versions up to 3.12.1, because the with the external ocamlbuild, the api would be the same and it would be pretty convenient for packages that uses ocamlbuild to have a good backward compatibility.
(0011484)
gildor (developer)
2014-05-15 12:36

The problem with "older version" is that they will be shipped with their own version of ocamlbuild and this will conflict with installing the split version of ocamlbuild.

Maybe we should take advantage of this fact and just consider >= 4.02 (I would like to start with 4.02 so that during dev we may rely on the latest stable version rather than the trunk of OCaml).
(0011490)
gildor (developer)
2014-05-16 10:45

I have updated this page with the task to be done:
https://github.com/hhugo/ocamlbuild/wiki [^]

I think we should coordinate through the github wiki to know what we will be working on.
(0011491)
avsm (reporter)
2014-05-16 11:05

Not to tell anyone how to spend their time, but I would encourage spending some time dealing with the 4.02 breaking changes before starting yet another feature breakout in the middle of a release cycle.

For instance, it woud be very useful for the 4.02 branch right now identify packages that do not compile with camlp4 correctly (see http://github.com/avsm/opam-bulk-logs [^]), and fix upstream packages (such as OASIS) that warn due to the new Bytes module. Once all this is stable, it would be much easier to use 4.02 as a base on which to break out ocamlbuild and deal with all the subsequent fallout.
(0011982)
jpdeplaix (reporter)
2014-08-06 17:53

What's the status of this ?
(0011984)
gasche (developer)
2014-08-06 18:00

I asked at the last developer meeting mid-July, but unfortunately we didn't come to a definitive conclusion.
(0012090)
jpdeplaix (reporter)
2014-09-04 13:18

When is the next developer meeting ?
(0012338)
gasche (developer)
2014-10-09 16:05

What is the current opinion on this? Is it worth bringing up the question again?

As said above, I asked the dev team, before the 4.02 release, about the idea of splitting off ocamlbuild, but there was no clear consensus. I'm now considering bringing that question to the table again, but I haven't heard for anyone willing to work on ocamlbuild matters since -- while hhugo, jpdeplaix and gildor previously expressed interest in helping with ocamlbuild's maintenance if it was moved of the repository.

I still have no strong opinion about the move, but currently I'm doing 100% of the maintenance work going in ocamlbuild. Among the people that would support a move out of the repository, are there still some that would commit some time to maintenance work? Is there any particular reason why these people wouldn't consider contributing for now? For example, a couple of bugs have been fixed between 4.02.0 and 4.02.1, and some other may still be fixable.
(0012340)
jpdeplaix (reporter)
2014-10-09 23:40

I will not talk as them but I think the main reason is that the bugs or the features they can implement is delayed to the next ocaml release. This is too complicated if you want to support several version of the compiler and have this feature/bug fixed. Plus, you have to wait a considerable amount of time.
(0012393)
Camarade_Tux (reporter)
2014-10-18 12:11

I'd like opinions on an intermediate proposal/step: separate the build ocamlbuild from the build of the compiler, i.e. the compiler would have to be built and installed before ocamlbuild can be built.

The only drawback I can think of involves testing ocamlbuild when doing changes in the compiler but thanks to gasche's DESTDIR support in the compiler it should only be a matter of "make install DESTDIR=some/dir" plus setting PATH and OCAMLPATH. A couple additional steps but simple ones.

That work would be required if ocamlbuild is moved to its own repository anyway so no work lost in any case and it would simplify support for cross-compilation (both build-time and because it would ensure a single ocamlbuild executable isn't tied to the compiler used to build it) while being fairly simple to implement.

Can anyone think of drawbacks to this?

- Issue History
Date Modified Username Field Change
2014-05-07 11:05 hhugo New Issue
2014-05-07 11:31 gildor Note Added: 0011379
2014-05-07 14:01 jpdeplaix Note Added: 0011381
2014-05-07 15:26 hhugo Note Added: 0011383
2014-05-07 15:45 gildor Note Added: 0011384
2014-05-07 15:45 gildor Note Added: 0011385
2014-05-07 16:34 hhugo Note Added: 0011386
2014-05-07 16:48 gildor Note Added: 0011387
2014-05-07 17:59 avsm Note Added: 0011389
2014-05-07 21:37 hhugo Note Added: 0011394
2014-05-07 21:51 gildor Note Added: 0011395
2014-05-07 23:30 avsm Note Added: 0011397
2014-05-08 00:32 gildor Note Added: 0011398
2014-05-08 05:07 berke.durak Note Added: 0011399
2014-05-08 09:08 gildor Note Added: 0011400
2014-05-08 11:18 gasche Note Added: 0011402
2014-05-08 11:23 gasche Note Edited: 0011402 View Revisions
2014-05-08 11:55 gildor Note Added: 0011403
2014-05-08 11:56 gildor Note Edited: 0011403 View Revisions
2014-05-08 12:19 gasche Note Added: 0011404
2014-05-08 12:19 gasche Severity minor => feature
2014-05-08 12:19 gasche Status new => acknowledged
2014-05-08 12:41 gildor Note Added: 0011405
2014-05-08 12:46 avsm Note Added: 0011406
2014-05-08 15:17 garrigue Note Added: 0011407
2014-05-10 19:42 Camarade_Tux Note Added: 0011417
2014-05-13 16:47 jpdeplaix Note Added: 0011442
2014-05-13 17:45 protz Note Added: 0011445
2014-05-13 17:46 protz Note Edited: 0011445 View Revisions
2014-05-13 18:05 gildor Note Added: 0011446
2014-05-13 18:12 gildor Note Added: 0011447
2014-05-13 18:37 Camarade_Tux Note Added: 0011448
2014-05-13 18:39 Camarade_Tux Note Edited: 0011448 View Revisions
2014-05-14 17:16 gasche Note Added: 0011463
2014-05-14 17:16 gasche Note Edited: 0011463 View Revisions
2014-05-14 17:34 gildor Note Added: 0011464
2014-05-14 23:59 jpdeplaix Note Added: 0011469
2014-05-15 07:58 Camarade_Tux Note Added: 0011471
2014-05-15 11:00 jpdeplaix Note Added: 0011479
2014-05-15 12:36 gildor Note Added: 0011484
2014-05-16 10:45 gildor Note Added: 0011490
2014-05-16 11:05 avsm Note Added: 0011491
2014-08-06 17:53 jpdeplaix Note Added: 0011982
2014-08-06 18:00 gasche Note Added: 0011984
2014-09-04 13:18 jpdeplaix Note Added: 0012090
2014-10-09 16:05 gasche Note Added: 0012338
2014-10-09 23:40 jpdeplaix Note Added: 0012340
2014-10-18 12:11 Camarade_Tux Note Added: 0012393


Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker