Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

move ocamlbuild to its own repository. #6402

Closed
vicuna opened this issue May 7, 2014 · 41 comments
Closed

move ocamlbuild to its own repository. #6402

vicuna opened this issue May 7, 2014 · 41 comments

Comments

@vicuna
Copy link

vicuna commented May 7, 2014

Original bug ID: 6402
Reporter: @hhugo
Assigned to: @gasche
Status: closed (set by @xavierleroy on 2017-09-24T15:31:50Z)
Resolution: fixed
Priority: normal
Severity: feature
Fixed in version: 4.03.0+dev / +beta1
Category: -for ocamlbuild use https://github.com/ocaml/ocamlbuild/issues
Monitored by: @gasche @ygrek @jmeber @hcarty @dbuenzli gildor

Bug description

  • so releases are not necessary mapped to OCaml's one
  • so improvement can benefit build with other OCaml version
@vicuna
Copy link
Author

vicuna commented May 7, 2014

Comment author: gildor

I think this will be an overall improvement:

  • hopefully benefit from external users contribution
  • allow to fix rules outside of OCaml release cycle

+1

@vicuna
Copy link
Author

vicuna commented May 7, 2014

Comment author: jpdeplaix

I also think it would be a good thing to do.

Some times ago, this idea of spliting the tools which are in the distribution was raised and I wasn't sure whenever it was a good idea or not for ocamlbuild. My main con, was that I think that one reason why ocamlbuild is used is that it's distributed with ocaml.

But now, after thinking of it a little bit more, I think this con can be erased by « massive » contributions and a shorter release cycle.

I tried to do the split last week but I didn't had the time to finish it.

@vicuna
Copy link
Author

vicuna commented May 7, 2014

Comment author: @hhugo

https://github.com/hhugo/ocamlbuild

@vicuna
Copy link
Author

vicuna commented May 7, 2014

Comment author: gildor

It should be in https://github.com/ocaml/ocamlbuild...

Maybe contact Anil or Fabrice Lefessan about that.

@vicuna
Copy link
Author

vicuna commented May 7, 2014

Comment author: gildor

Also, if you get approval for the move to github/ocaml/, I am willing to spend five minute to transfer it.

@vicuna
Copy link
Author

vicuna commented May 7, 2014

Comment author: @hhugo

I can already change the github repo owner to the ocaml organisation

@vicuna
Copy link
Author

vicuna commented May 7, 2014

Comment author: gildor

hhugo, but there will be some conflict if you do so! Please contact Anil/Fabrice before.

@vicuna
Copy link
Author

vicuna commented May 7, 2014

Comment author: @avsm

Glad to see you working on this! It would be best to leave it as a private fork until there is agreement from the development team that this is a direction we want a future release of OCaml to go in.

OCaml 4.02 is in feature freeze right now in preparation for branching for release, so it would probably be best to bring this up in a week or so after that branch has been created, and trunk is 4.03dev.

@vicuna
Copy link
Author

vicuna commented May 7, 2014

Comment author: @hhugo

Thanks for the details.

I've just synced the repo with the latest trunk (with "safe-string" reverted to build on 4.01). gildor, jpdeplaix, you now both have access to this repo if you wish to contribute

@vicuna
Copy link
Author

vicuna commented May 7, 2014

Comment author: gildor

Draft of TODO goals:
https://github.com/hhugo/ocamlbuild/wiki

If you want I can setup a forge project for mailing list/tarballs/homepage...

@vicuna
Copy link
Author

vicuna commented May 7, 2014

Comment author: @avsm

Please don't do that until there's consensus from the core team, as otherwise it'll be forking confusion as things get out of sync.

@vicuna
Copy link
Author

vicuna commented May 7, 2014

Comment author: gildor

ACK, I don't play against the core team. I suppose hhugo will agree that if the core team is opposed to this decision we will just withdraw the ocamlbuild git repository and that will be the end of the story.

@vicuna
Copy link
Author

vicuna commented May 8, 2014

Comment author: berke.durak

Not sure about this; I think it's a good thing that a proper build tool ships with Ocaml itself. If you move it to its own repo, people will inevitably want to use this or that external module, and you can't ship that with the compiler.

I seem to recall that I had to write Ocamlbuild's glob module in pure Ocaml to avoid a dependency on Str. And Nicolas wrote a My_unix module so that Ocamlbuild would work even with "degraded" Unix environments.

On the other hand it is true that the plugin/rule/tag system is difficult to use.

Why not fork Ocaml, work on Ocamlbuild, and use pull requests on Github?

@vicuna
Copy link
Author

vicuna commented May 8, 2014

Comment author: gildor

I think ocamlbuild is a quite separate project and easily splitable. Given the fact that you don't use ocamlbuild inside OCaml sources that a good candidate to be moved away.

If you have a different project, you'll be able to follow a different release cycle (shorter hopefully).

E.g. fix the .cmxs rule and release rather than wait for all other patches on all other topics to be merged.

Given the fact that camlp4 and tk lib has been moved to their own project, I didn't think it was a problem.

On the topic of using external module and rewriting code like "glob" or "my_unix", I don't share your opinion... I would use Str and Unix and if mandatory I would enforce thing like depending on findlib. I don't have enough time to redo everything.

@vicuna
Copy link
Author

vicuna commented May 8, 2014

Comment author: @gasche

The main problem is that there are currently extremely few people remotely motivated to contribute to ocamlbuild's development. I count two, jpdeplaix and myself. That remains a problem whether or not ocamlbuild is split.

It's very good to be able to make releases independently of OCaml's release schedule. But on how many occasion would it have made sense, between the 4.01 and the 4.02 release? Answer: zero times. Nobody gave a damn about helping with ocamlbuild development (except jpdeplaix above) since the 4.01 release, so there was nothing to release.

Will having ocamlbuild in its own github repository encourage people to contribute? Well:

  • ocaml already has a github repository, and we even monitor pull requests now, where would the added convenience be?
  • camlp4 has been split into a separate github repository just before 4.01. How does being split as its own github work for camlp4? Besides Jérémie Dimino's maintenance work, there have been precisely four external contributions, three of them coming from regular contributors to the OCaml distribution itself. I can't see a wave of new contributors here.

(I also have a draft of improved OCamlbuild documentation, publicly announced 8 months ago, with a github repository. So far, besides one batch of typo fixes, no substantial contribution.)

I feel neutral about splitting ocamlbuild as a separate project: I see no reason why that would make any difference -- besides making "which build system to use?" discussions even funnier.

@vicuna
Copy link
Author

vicuna commented May 8, 2014

Comment author: gildor

What about taking it the other way: is keeping ocamlbuild inside OCaml sources providing a significant benefit to the core team?

Actually, I would be motivated to work on it. Esp. fixing .cmxs incremental build which is a pain point for OASIS. I don't contribute because I always postpone it given the time frame of releases. Now you may argue that I can do it and wait but it will not account for the thousands other urgent matters that I have to process. Lets call it lack of "immediate reward", something which our brain is pretty accustomed to.

Anyway, all these are just wild guesses and I suppose the best way to see if there will be an improvement in contributors is to try.

On the topic of "which build system to use?", I think this is the usual flamewar. I stopped reading caml-list 2 years ago because of this kind of discussion. Having "ocamlbuild" as an external project will not prevent 10 new "ultimate, parallel, fast, so much better and written by me" build systems to be created in 2014.

@vicuna
Copy link
Author

vicuna commented May 8, 2014

Comment author: @gasche

Regardless of whether ocamlbuild is split, I'm certainly interested in trying to make OASIS's life easier. However, I have very little feedback about what your needs about OASIS would be; the little I get being in the form of occasional contributions from Jacques-Pascal (jpdeplaix), drawn out of any original OASIS context. Is your "cmxs incremental build" issue reported anywhere?

@vicuna
Copy link
Author

vicuna commented May 8, 2014

Comment author: gildor

cmxs issue is on my personal todo list and this is related to #5653

The bug is marked as solved, but this is a pain point and I cannot figure out an easy way to fix it.

Basically, in certain case you always build .cmxs, even if nothing has been updated. If you are interested I can find an example project that shows this behavior.

Anyway this is OT for the current bug ;-)

@vicuna
Copy link
Author

vicuna commented May 8, 2014

Comment author: @avsm

It's very good to be able to make releases independently of OCaml's release schedule.
But on how many occasion would it have made sense, between the 4.01 and the
4.02 release? Answer: zero times.

That's circular: the reason that I (for example) haven't contributed ocamlbuild bugfixes
back in the past is that I can't wait a year for the fix to show up in a release to unblock
my build. If ocamlbuild were easier to recompile separately (e.g. via an opam pin),
then working on it is a viable option. Otherwise, a Makefile or an alternative build
tool is the only way to get on with real work.

Given that ocamlbuild isn't used by the compiler itself, I think splitting it out after
this release would at least give it a fighting chance to become more usable.

@vicuna
Copy link
Author

vicuna commented May 8, 2014

Comment author: @garrigue

Now that camlp4 is no longer in the distribution, the is no real point in keeping ocamlbuild there.
Moreover, ocamlbuild includes support for findlib, which is not part of the distribution.
So yes, I think there are very good reasons to split it, and let users improve it.

@vicuna
Copy link
Author

vicuna commented May 10, 2014

Comment author: Camarade_Tux

I'm a bit parted on this issue.

It's been more than one year that I've stated I would prefer to have camlp4, (labltk,) ocamlbuild, ocamldoc separated from other bits of the compiler.
Separated or at least that it's possible to configure, build and install them on their own. So overall, I'm in favor of that.

That said, I'm worried about how the split would be done. Part of these worries is a general feeling that doing a proper split for ocamlbuild is more difficult than for camlp4 or labltk.
Camlp4 was something that could be disabled for quite some time and there was camlp5 to prove such a tool could live outside of the compiler. Labltk was an optional component for an even longer time and there was again something (lablgtk) to prove it could live nicely outside.
Now, ocamlbuild, most libraries and programs rely on it and the situation regarding build systems is weird enough that it justifies taking extra time and thinking before moving it.

For starters, I believe the split of ocamlbuild should be decided before 4.02 and that a notice should be put in the release notes and in ./configure if it is to be split for 4.03.

I don't have strong justifictions for my worries; it's mainly a feeling but a feeling of someone who's dived very deep into build systems, including OCaml's.

One thing that definitely has to be changed is with the various "config" files: myocamlbuild_config.ml and config/Makefile. They hold the result of ./configure in two different formats. The information is valuable and has to be made available in a proper format to subsequent tools. I consider this a blocking issue before ocamlbuild can be split.

I'll try to come up with the various other issues in the coming weeks but with the preparation of the new version of http://win-builds.org I've clearly fallen behind the cross-compilation work.

@vicuna
Copy link
Author

vicuna commented May 13, 2014

Comment author: jpdeplaix

Now that the 4.02 has been froze, I guess we have to hope to have this for the 4.03 ? :'(

@vicuna
Copy link
Author

vicuna commented May 13, 2014

Comment author: @protz

One thing that definitely has to be changed is with the various "config" files:
myocamlbuild_config.ml and config/Makefile. They hold the result of ./configure
in two different formats. The information is valuable and has to be made
available in a proper format to subsequent tools. I consider this a blocking
issue before ocamlbuild can be split.

This comment above is the biggest concern imho and has not been answered yet. Those of you (gildor, jpdeplaix) who want to kick ocamlbuild out, what is your opinion on this particular point?

Thanks,

~ jonathan

@vicuna
Copy link
Author

vicuna commented May 13, 2014

Comment author: gildor

Can't we extract this information from 'ocamlc -config' ?

@vicuna
Copy link
Author

vicuna commented May 13, 2014

Comment author: gildor

I just had a quick look at myocamlbuild_config.ml and I think the best is to register what will be missing in 'ocamlc -config' (which means that it will help not only ocamlbuild but other build tools BTW) but we can already be in a pretty good shape with just what we have in ocamlc.

The other good point on relying on the output of 'ocamlc -config' is that we can use ocamlbuild with older version of ocaml...

Note that I already use this kind of information in OASIS.

@vicuna
Copy link
Author

vicuna commented May 13, 2014

Comment author: Camarade_Tux

I hadn't thought about this earlier on but separating ocamlbuild from this data might be quite nice for the cross-compilation support.

There is no reason to have such values hardcoded into a build system and being able to point ocamlbuild to a different configuration file at runtime both through environment variables and commandl-line switches would be a plus imho.

And now I notice how much that could overlap with what ocamlfind is able to do. Maybe the data could be read by ocamlfind and then re-used by other tools?

PS: currently I cross-compile oasis/ocamlbuild-based build systems by pointing ocamlfind to specific configuration files that are setup for cross-compilation through environment variables; this is very nice but I'd hate to add another environment variable for a config file that duplicates some of the information.

edit: the process to create this file is: configure runs, creates config/Makefile at some point later on this file is transformed into myocamlbuild_config.ml; a solution that would unify both into a nicer format be nice to have (thinking about it, it is also possible to have configure write both files at the same time painlessly so this might not be a huge concern).

@vicuna
Copy link
Author

vicuna commented May 14, 2014

Comment author: @gasche

It seems there is a consensus on the fact that the move is a good thing.

I think we should wait for the 4.02 release in any case.

@vicuna
Copy link
Author

vicuna commented May 14, 2014

Comment author: gildor

Here might be a proposal:
Step 1. make https://github.com/hhugo/ocamlbuild builds on its own with latest version of ocaml
Step 2. create a PR for https://github.com/ocaml/ocaml to strip ocamlbuild from it and check this works too and provide more information if needed
Step 3. discuss and try to get approval from INRIA core team to do the split officialy
Step 4. move https://github.com/hhugo/ocamlbuild to https://github.com/ocaml/ocamlbuild, integrate PR from Step 2 into the official OCaml repository and do the blog post/announce of the official split (if we get approval from Step 3)

This will help to have real proof of concept that it is doable. At this point, I think most of the question can be answered by a few weeks of coding and checking everything work.

hhugo, jpdeplaix do you agree with this plan ?

@vicuna
Copy link
Author

vicuna commented May 14, 2014

Comment author: jpdeplaix

I've already committed something for removing ocamlbuild from trunk if you want: https://github.com/jpdeplaix/ocaml/tree/remove-ocamlbuild

hhugo, I don't know if you already did that. Maybe you have something a bit different.

Anyway, I agree with the plan.
An other possible discussion would be: Do we want to have the separated ocamlbuild compatible with older ocaml versions (older than 4.03) or do we follow the same line as camlp4 does ?

@vicuna
Copy link
Author

vicuna commented May 15, 2014

Comment author: Camarade_Tux

Do we want to have the separated ocamlbuild compatible with older ocaml versions (older than 4.03) or do we follow the same line as camlp4 does ?

I'd say yes, at least for 4.02. It will make the transition much smoother and will make initiali development much easier: packages would use the version in the compiler tree for now and in a few months can switch to the external one by adding the package and configuring the compiler sources without ocamlbuild (unless I'm mistaken, that bit is in 4.02).

@vicuna
Copy link
Author

vicuna commented May 15, 2014

Comment author: jpdeplaix

By « older version » I meant all versions up to 3.12.1, because the with the external ocamlbuild, the api would be the same and it would be pretty convenient for packages that uses ocamlbuild to have a good backward compatibility.

@vicuna
Copy link
Author

vicuna commented May 15, 2014

Comment author: gildor

The problem with "older version" is that they will be shipped with their own version of ocamlbuild and this will conflict with installing the split version of ocamlbuild.

Maybe we should take advantage of this fact and just consider >= 4.02 (I would like to start with 4.02 so that during dev we may rely on the latest stable version rather than the trunk of OCaml).

@vicuna
Copy link
Author

vicuna commented May 16, 2014

Comment author: gildor

I have updated this page with the task to be done:
https://github.com/hhugo/ocamlbuild/wiki

I think we should coordinate through the github wiki to know what we will be working on.

@vicuna
Copy link
Author

vicuna commented May 16, 2014

Comment author: @avsm

Not to tell anyone how to spend their time, but I would encourage spending some time dealing with the 4.02 breaking changes before starting yet another feature breakout in the middle of a release cycle.

For instance, it woud be very useful for the 4.02 branch right now identify packages that do not compile with camlp4 correctly (see http://github.com/avsm/opam-bulk-logs), and fix upstream packages (such as OASIS) that warn due to the new Bytes module. Once all this is stable, it would be much easier to use 4.02 as a base on which to break out ocamlbuild and deal with all the subsequent fallout.

@vicuna
Copy link
Author

vicuna commented Aug 6, 2014

Comment author: jpdeplaix

What's the status of this ?

@vicuna
Copy link
Author

vicuna commented Aug 6, 2014

Comment author: @gasche

I asked at the last developer meeting mid-July, but unfortunately we didn't come to a definitive conclusion.

@vicuna
Copy link
Author

vicuna commented Sep 4, 2014

Comment author: jpdeplaix

When is the next developer meeting ?

@vicuna
Copy link
Author

vicuna commented Oct 9, 2014

Comment author: @gasche

What is the current opinion on this? Is it worth bringing up the question again?

As said above, I asked the dev team, before the 4.02 release, about the idea of splitting off ocamlbuild, but there was no clear consensus. I'm now considering bringing that question to the table again, but I haven't heard for anyone willing to work on ocamlbuild matters since -- while hhugo, jpdeplaix and gildor previously expressed interest in helping with ocamlbuild's maintenance if it was moved of the repository.

I still have no strong opinion about the move, but currently I'm doing 100% of the maintenance work going in ocamlbuild. Among the people that would support a move out of the repository, are there still some that would commit some time to maintenance work? Is there any particular reason why these people wouldn't consider contributing for now? For example, a couple of bugs have been fixed between 4.02.0 and 4.02.1, and some other may still be fixable.

@vicuna
Copy link
Author

vicuna commented Oct 9, 2014

Comment author: jpdeplaix

I will not talk as them but I think the main reason is that the bugs or the features they can implement is delayed to the next ocaml release. This is too complicated if you want to support several version of the compiler and have this feature/bug fixed. Plus, you have to wait a considerable amount of time.

@vicuna
Copy link
Author

vicuna commented Oct 18, 2014

Comment author: Camarade_Tux

I'd like opinions on an intermediate proposal/step: separate the build ocamlbuild from the build of the compiler, i.e. the compiler would have to be built and installed before ocamlbuild can be built.

The only drawback I can think of involves testing ocamlbuild when doing changes in the compiler but thanks to gasche's DESTDIR support in the compiler it should only be a matter of "make install DESTDIR=some/dir" plus setting PATH and OCAMLPATH. A couple additional steps but simple ones.

That work would be required if ocamlbuild is moved to its own repository anyway so no work lost in any case and it would simplify support for cross-compilation (both build-time and because it would ensure a single ocamlbuild executable isn't tied to the compiler used to build it) while being fairly simple to implement.

Can anyone think of drawbacks to this?

@vicuna
Copy link
Author

vicuna commented Feb 9, 2016

Comment author: @gasche

The split was done in trunk (future 4.03):

#443
https://github.com/ocaml/ocamlbuild/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants