Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

relocatable ocaml compiler #5950

Closed
vicuna opened this issue Mar 14, 2013 · 14 comments
Closed

relocatable ocaml compiler #5950

vicuna opened this issue Mar 14, 2013 · 14 comments

Comments

@vicuna
Copy link

vicuna commented Mar 14, 2013

Original bug ID: 5950
Reporter: kanig
Status: resolved (set by @xavierleroy on 2013-03-24T16:01:02Z)
Resolution: suspended
Priority: normal
Severity: feature
Category: ~DO NOT USE (was: OCaml general)
Monitored by: @glondu @hcarty

Bug description

Dear OCaml Team,

I wonder whether you considered making the ocaml compiler relocatable? Basically, this avoids hardcoding the "prefix", or any other paths that depend on the installation location, into the binaries. In consequence, the ocaml install can be moved around after compilation and installation. That would greatly simplify binary distribution.

A common technique is to compute the prefix from the path to the executable, as e.g. returned by Sys.executable_name. I actually tried to modify utils/config.ml accordingly, but it seems it's not enough to make the compiler relocatable.

What do you think?

Steps to reproduce

./configure --prefix=/opt
make world opt opt.opt
make install
mv /opt /opt2
<compiler does not work anymore ...>

File attachments

@vicuna
Copy link
Author

vicuna commented Mar 14, 2013

Comment author: @alainfrisch

FWIW, we do something along the line you proposer for LexiFi's version of OCaml, which we redistribute together with our application to our clients. In native code, this works well (and we do it only for native code). For bytecode (except in -custom mode), the tricky part is that the "header" (either a #! directive under Unix or a small executable) needs a way to locate ocamlrun which has no reason to be in a fixed position relative to the bytecode executable.

@vicuna
Copy link
Author

vicuna commented Mar 14, 2013

Comment author: kanig

Thanks for your feedback.

Your patch for the native code sounds interesting. Can you share it? We do not actually distribute OCaml ourselves, but even the setup for nightly tests would be simplified with that.

I do not understand your remark concerning bytecode, but that's probably because I don't know much about the OCaml compiler. Is the path to the ocamlrun executable actually hardcoded in the result of bytecode compilation? Indeed, that would make it very hard to obtain something relocatable.

That makes me think that I should retry my own attempt, because I only tried bytecode, not native ... For our purposes, native would be sufficient as well.

@vicuna
Copy link
Author

vicuna commented Mar 24, 2013

Comment author: @xavierleroy

You can achieve some degree of relocation by setting the OCAMLLIB and CAML_LD_LIBRARY_PATH environment variables. This is what the Windows binary distribution of OCaml does.

One problem, mentioned by Alain, is for bytecode executables generated by ocamlc, more precisely: which ocamlrun is used to execute them? Under Unix, ocamlc hard-wires #!/some/path/to/ocamlrun. Under Windows, ocamlrun is searched in the PATH when the bytecode executable is started. None of these solutions is good.

Another problem is that the OCaml compilers can't assume that Sys.argv.(0) is the correct full path to their own executable. It is guaranteed under Linux because we use /proc/self/exe, a nonstandard Linux extension to the Unix standards. On other systems, we use portable heuristics that may fail.

Finally, the general trend recently is to make it easier to install and compile OCaml (via OPAM), rather than binary distribution.

Unless you can make a really convincing use case for relocating a binary distribution of OCaml on a non-Windows OS, I move to close this PR.

@vicuna
Copy link
Author

vicuna commented Mar 24, 2013

Comment author: kanig

Thanks for your input. I have a few questions, although I am not sure if it the best place to discuss these issues.

Why the difference in behavior to locate ocamlrun, on Windows and Linux? It seems that searching the PATH is slightly better, because it actually allows relocating ocamlrun. But of course there is the danger of confusing two installed compilers.

Yes, looking at Sys.argv.(0) is not enough, in the general case you need to search in the PATH variable to find your own executable. I know that GCC does such trickery, and I assumed that Sys.executable_name also did. Am I wrong?

Why do you want me to provide a use case for non-Windows OS only, do you feel that the problem is solved on Windows?

Use cases for binary distribution are: shipping the OCaml compiler with another product (see Alain's comment), having a common OCaml install copied to different build systems (instead of having to compile it on each of them). There is no difference here between Windows and Linux.

But as you said, it is not a blocking issue, there are (at least) two workarounds: use the environment variables you mentioned, or compile from source.

@vicuna
Copy link
Author

vicuna commented Mar 24, 2013

Comment author: @xavierleroy

A few answers and then I "resolve/suspend" this PR.

Why the difference in behavior to locate ocamlrun, on Windows and Linux?

Unix has the #! mechanism, which we really want to use for bytecode executables as it is compact and efficient, but it requires an absolute path: #!/path/to/ocamlrun. In contrast, Windows lacks any equivalent of #!, so we make do with a small C executable for which it was easier to search ocamlrun in PATH.

It seems that searching the PATH is slightly better

Not in general, because you risk selecting the wrong ocamlrun to run your bytecode executable, if you have several versions of OCaml installed in different places and select one by playing on PATH (a common scenario - I do that all the time).

Yes, looking at Sys.argv.(0) is not enough, in the general case you need to search in the PATH variable to find your own executable.

The OCaml runtime does this, obviously. But it is still not reliable, as nothing prevents the shell or the program launching your OCaml compiler to pass the wrong argv[0], e.g.

execl("/dir/not/in/path/ocamlc", "ocamlc", /* other args */, NULL);

Why do you want me to provide a use case for non-Windows OS only, do you feel that the problem is solved on Windows?

Not 100%, but Windows is the only platform I know where it is customary to 1- install precompiled binaries, and 2- install them in any directory of the user's choice. The Unix tradition, as embodied in modern package managers, is either to compile from sources before installing, or install precompiled binaries at known locations (/usr/bin, etc).

@vicuna vicuna closed this as completed Mar 24, 2013
@vicuna
Copy link
Author

vicuna commented Mar 25, 2013

Comment author: @alainfrisch

FWIW, we need to ship a binary Linux version of the toolchain (bytecode + native code) to a customer, and they need to be able to install it where they want. The current situation is fine for us, though. The tools we distribute are only the .opt versions (ocamlc.opt, ocamlopt.opt, ocamldoc.opt, ocamldep.opt, etc), and for the bytecode programs compiled by the customer, we advise them to use to use "-custom" or to call "ocamlrun" explicitly.

@vicuna
Copy link
Author

vicuna commented Aug 13, 2013

Comment author: nycbrit

The "unix tradition" of fixed install paths causes insurmountable problems for users who don't have full control of their file system & can't rebuild the binary for logistical reasons.
Can you get ocamlc to look in the same (canonical) directory for ocamlrun?

@vicuna
Copy link
Author

vicuna commented Jul 2, 2015

Comment author: @samoht

Finally, the general trend recently is to make it easier to install and compile OCaml (via OPAM), rather than binary distribution.

even with OPAM, having a relocatable compiler would be very helpful to avoid recompiling the same compiler on different switches. Could this be a configure-time option (which will be disable on systems where /proc/self/exe is not available?

@vicuna
Copy link
Author

vicuna commented Jul 2, 2015

Comment author: @alainfrisch

For the compilers, it's indeed quite easy to support relocation (using a relative path starting from the compiler/toplevel to locate the library directory). The difficult part is for bytecode user programs (not compiled with -custom) and this is likely to be required to build many packages. These programs somehow need a way to locate some compatible ocamlrun. Either this is searched in the PATH (as for Windows today; and perhaps using "#!/usr/bin/env ocamlrun" under Unix), but this means that the PATH must be altered, which is not very nice. Or we continue hard-coding the path to the desired ocamlrun in the bytecode program (and I think we should be uniform between Windows and Unix, i.e. the "prefix program" under Windows should extract the path to ocamlrun from the bytecode program), but then the same program would not work if installed on a machine where the toolchain is at a different location.

All in all, I believe that the proper way to address that would be to deprecate the notion of calling the bytecode program (if not compiled with -custom) directly, and encourage users to do instead "ocamlrun myprog". This way, they can choose whether to rely on the PATH or on a specific toolchain, and we remove the expectation that the bytecode program can find ocamlrun on its own (which is incompatible with the idea of relocation).

@vicuna
Copy link
Author

vicuna commented Mar 2, 2019

Comment author: cullmann

Hi ;=)

I just stumbled up on this bug during my search how to relocate a compiled OCaml for Linux/macOS.

If you use OCaml inside a CI, it is really beneficial if one can compile it once and just reuse it again. This is what we do e.g. for llvm/clang but isn't possible for non-Windows OCaml. The CI can't use the same prefix twice, even with containers that is hard without ugly mount tricks that then lead to other issues for people wanting to get the same compile env. outside of the container.

Would it be hard to support it at least optionally? I understood from this report that more or less the biggest hassle is the #!... in the interpreter binaries. Would it be ok to have the Windows like way at least optionally?

At the moment we need to compile OCaml for each CI run, which takes ~10 minutes in addition to the other stuff.

@vicuna
Copy link
Author

vicuna commented Mar 3, 2019

Comment author: cullmann

The attached diff shows the changes needed to get a relocatable OCaml for e.g. Linux & macOS (if you set afterwards PATH/OCAMLLIB/CAML_LD_LIBRARY_PATH properly).

I tried that here with our CI for Astree & CompCert, seems to work nicely.

Would such a change be ok to contribute via a configure flag?
One could flag Windows as "always relocatable", that would even avoid the opsys check in ocamlmklib and replace it with the "is relocatable" check to be introduced.

@cemerick
Copy link

Just to echo the motivating use case mentioned in #5950 (comment):

At the moment we need to compile OCaml for each CI run, which takes ~10 minutes in addition to the other stuff.

I was bitten by the un-relocatability of switches in a CI context (to be fair, after being warned), and am now employing a different workaround to avoid the costs of local switch creation:

https://discuss.ocaml.org/t/basic-gitlab-ci-configuration/3327/12?u=cemerick

tl;dr: if your CI platform provides caching of temporaries across builds, then you can use that to mostly avoid the cost associated with repeatably working with clean switches.

@christoph-cullmann
Copy link
Contributor

At AbsInt we use these two patches to have some relocatable OCaml that is good enough to compile our products.

patch1.txt
patch2.txt

@gasche
Copy link
Member

gasche commented Nov 14, 2019

At the last developer meeting, @dra27 has proposed a plan that makes compilers relocatable, and is also designed for a more graceful integration of several different versions of the compiler/runtime co-existing in the path. (Basically the idea is to hash the "runtime configuration", and to use this hash as a uniquely-identifying suffix for the ocamlrun binary filepath, and other location-specific information.)

There is a reasonable consensus with going forward with this plan, so I hope that we will see progress on compiler relocatability in, say, one or two years.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants