Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wish: Support for cross-compilation #4303

Open
vicuna opened this issue May 25, 2007 · 17 comments
Open

Wish: Support for cross-compilation #4303

vicuna opened this issue May 25, 2007 · 17 comments

Comments

@vicuna
Copy link

vicuna commented May 25, 2007

Original bug ID: 4303
Reporter: n8gray
Status: acknowledged (set by @xavierleroy on 2017-02-20T09:56:21Z)
Resolution: open
Priority: normal
Severity: feature
Version: 3.10.0
Category: platform support (windows, cross-compilation, etc)
Related to: #6266 #6613
Monitored by: @ygrek @glondu Snark Camarade_Tux jm @hcarty "Richard Jones" @dra27 @avsm

Bug description

It would be nice if OCaml supported cross-compilation. There are a number of interesting platforms in existence today that are powerful enough to support an ocaml runtime environment but are not well suited for hosting the compiler itself, in particular mobile platforms (PocketPCs, Linux phones, Nintendo DS, PSP, etc). Cross-compilation is also attractive for Mac OS X users since it would allow us to build universal binaries of OCaml programs. There are various one-off patches floating around the net to get specific versions of OCaml to cross-compile to specific platforms, but it would be better to be able to build a single compiler supporting multiple architectures.

@vicuna
Copy link
Author

vicuna commented Jun 23, 2007

Comment author: karl

Second this wish, I want ocaml for ARM platform.

@vicuna
Copy link
Author

vicuna commented Nov 10, 2007

Comment author: @xavierleroy

On the CVS head version, the configure script was modified to allow specifying a cross C compiler and a cross assembler to be used. This removes one hurdle in building a Caml cross-compiler. I agree there should be a "HOWTO" explaining the steps to take. The major issue is the autoconfiguration stuff: either you manage to run it on the target device and copy the results to the build platform, or you have to write the configuration files by hand.

What you will never have, however, is a single OCaml compiler executable that can generate code for several target architectures.

@vicuna
Copy link
Author

vicuna commented Nov 15, 2008

Comment author: Richard Jones

We are building a Windows cross-compiler environment
as part of the Fedora MinGW project. You can find our
test builds here:

http://hg.et.redhat.com/misc/fedora-mingw--devel/
(Click 'manifest' then one of the ocaml subdirectories)

@vicuna
Copy link
Author

vicuna commented Apr 1, 2009

Comment author: @xavierleroy

Xavier Clerc is making progress on this issue, see:

http://brion.inria.fr/gallium/index.php/CrossCompiler

@vicuna
Copy link
Author

vicuna commented Dec 15, 2012

Comment author: Camarade_Tux

I've noticed two iesues with this patch so far:

  • doesn't really integrate with the build system
  • seems unable to compile C (i.e. "ocamlc -c foo.c" will not work)

@vicuna
Copy link
Author

vicuna commented Feb 20, 2017

Comment author: @xavierleroy

Un-assigning from xclerc

@vicuna
Copy link
Author

vicuna commented Mar 3, 2017

Comment author: @damiendoligez

related to:

#6266
#940 #940
#766 #766
#620 #620
#183 #183

@github-actions
Copy link

This issue has been open one year with no activity. Consequently, it is being marked with the "stale" label. What this means is that the issue will be automatically closed in 30 days unless more comments are added or the "stale" label is removed. Comments that provide new information on the issue are especially welcome: is it still reproducible? did it appear in other contexts? how critical is it? etc.

@github-actions github-actions bot added the Stale label May 18, 2020
@nojb
Copy link
Contributor

nojb commented May 18, 2020

Still relevant.

@github-actions
Copy link

This issue has been open one year with no activity. Consequently, it is being marked with the "stale" label. What this means is that the issue will be automatically closed in 30 days unless more comments are added or the "stale" label is removed. Comments that provide new information on the issue are especially welcome: is it still reproducible? did it appear in other contexts? how critical is it? etc.

@github-actions github-actions bot added the Stale label Aug 10, 2022
@shindere shindere self-assigned this Aug 11, 2022
@shindere shindere removed the Stale label Aug 11, 2022
@shindere
Copy link
Contributor

shindere commented Oct 11, 2022 via email

@nojb
Copy link
Contributor

nojb commented Oct 11, 2022

Sorry for the naïve question, but is there a roadmap that is being followed here? Right now, I get the feeling that these preliminary steps (single Makefile, out-of-tree compilation, etc) are nice-to-have but personally I don't have a clear picture of how everything is supposed to come together in the end.

In my opinion, having a clear HOWTO describing the roadmap would be most useful: are these intermediate steps (single Makefile, out-of-tree compilation) logically necessary for cross-compilation? What comes after? etc. Also, having such a roadmap would allow splitting the work into smaller pieces and allow other contributors to participate as well.

Judging by the fact that there are patches that achieve different forms of cross-compilation maintained by the community, it seems that some of these intermediate steps are not absolutely necessary. Can we take inspiration from these projects? If not, why not? What is missing from these projects? How do they work? etc. Having a document with all this information would already be a huge step forward.

Just my 2c.

@dra27
Copy link
Member

dra27 commented Oct 11, 2022

It's a very reasonable 2¢, @nojb!

The main thing with a lot of the work towards this is that there are a lot of strands of work which pull in the same general direction. I often use the analogy of a ship-in-a-bottle, where at some point hopefully soon all the strings get gently pulled and we magically have a cross-compiler in the bottle - but at the moment it looks like a weird set of unrelated changes 🙂

There is a very limited roadmap from a few years ago in the developer-meetings repo which could be surfaced into a public RFC, yes. Based on my own experiments (February and June 2020), neither single Makefile nor out-of-tree builds are necessary for cross-compilation, but the result (certainly with in-tree builds) is considerably more complicated and involves a lot of filename mangling and dependency generation mangling.

For the existing solutions (I apologise in advance to their maintainers if my assessment is now out-of-date) - in particular, the opam-cross mechanisms build cross-compilers effectively by "tricking" our build system into using an existing host compiler (and opam-cross only has slower bytecode versions of the tools - i.e. there's no ocamlopt-compiled cross-compiler, unless that's changed). Both existing solutions (opam-cross and I think there's a version based on esy, but I'm not sure of the package names) rely on external systems to build the cross compilers (i.e. opam, esy, etc.). That said - and here's one of the strange unrelated strands - there's a possible benefit which @stedolan is investigating where being able to start-up the compiler's build using a pre-existing - and possibly older - compiler gives a boost to the build.

For the two things @shindere's actively working on now, I think it's worth differentiating "on the path to cross-compilation" from "worthwhile in their own right". For the unified Makefile work, again my own experiments trying to speed up the build of the compiler for opam users find multiple-minute savings just by tweaking the order in which we build things in the existing build infrastructure. However, those changes are hard to reason about, because the author is deciding that things can be safely built before other things, etc., rather than allowing the build system to know that via its dependency graph. The hunch is that with a unified Makefile, these performance boosts will be found and be easier to reason about. They certainly exist, because the Dune build of the compiler is much faster (not just the upstreamed one, but also from the flambda2 team's experience). Part of that work factors out build instructions., recipes, flags and so forth to one place - all of which need tweaking to be building cross-compilers.

For the out-of-tree builds, there are a few non-cross-compilation benefits. The bootstrap cycle becomes slightly more clear if it uses separate build trees for its phases (at present, it does a partialclean between each phase, saving the required artefacts from each stage in boot/ as it goes). We already have a partial out-of-tree build in that we maintain a partial compiler in boot/ and a full compiler in the root - both of which require myriad flags to execute and both of which could be made to execute "naturally" if combined with both my relocatable work and an out-of-tree ("installation") layout.

My own development preference for changes like this, as you might have noticed in the past, is huge numbers of commits, completely demonstrating the feature (vis-à-vis both "relocatable" and "no scripting" 🙂) but that comes with with two problems. Firstly, the work is terrifying to behold for reviewers (!!), and secondly, especially where the build system is concerned, it's exceptionally painful to rebase. This is why @shindere I think quite rightly prefers to be working on earlier stages as one project - it avoids the nasty rebases, but at the cost of not having a prototype in-tree cross-compiler to demonstrate the long-term benefit of the change.

@nojb
Copy link
Contributor

nojb commented Oct 11, 2022

Thanks for the explanations @dra27!

@jonahbeckford
Copy link

I think we can lift most of the material we'd need for a roadmap from what I wrote in https://diskuv.gitlab.io/diskuv-ocaml/doc/CompilingInDepth.html. That document was distilled into the dkml-base-compiler package in the opam repository, which supports cross-compiling macOS x86_64 <-> macOS arm64 and the various Android ABI cross-compiles (although I haven't done a public announcement).

Simplifying the cross-compiling procedure

The cross-compiling procedure is in https://diskuv.gitlab.io/diskuv-ocaml/doc/CompilingInDepth.html#changing-ocaml-to-do-cross-compilation

  1. Just look at how often the procedure says "recompile" and "regenerate". This OCaml issue thread already mentioned
    the need to stop interleaving build artifacts so let's be explicit:
    • Roadmap: Stop interleaving build artifacts. A solution could be to provide "staged" build directories, where each build directory is focused on one phase (or stage) of the compilation.
  2. This OCaml issue thread also mentioned simplifying down to a single Makefile. I didn't talk about that, but in dkml-base-compiler I had to write a bridge from CMake (the build system that Android uses) into ./configure. So for completeness I'll repeat it here:
    • Roadmap: Provide a single Makefile. The Makefile must be able to set the Target Machine configuration that is currently in utils/config.ml and runtime/sys.c

Removing Limitations

Each "limitation" can be a roadmap item. See the Limitations at https://diskuv.gitlab.io/diskuv-ocaml/doc/CompilingInDepth.html#limitations:

  1. if your Target Machine is a 32-bit system, make sure you use a 32-bit Host Machine compiler. That equalizes [word_size]

    becomes

    • Roadmap: Solve the dual-purposes of word_size. A possible solution is to separate word_size into host_word_size and target_word_size
  2. When ocamlopt is linking object files into an executable on Windows, it uses an executable called flexlink.exe that expects Windows .obj (COFF) object files for linking. However Linux uses ELF object files and macOS uses Mach-O object files, so a Windows Host Machine cannot support a non-Windows Target Machine.
    becomes

    • Roadmap: Solve the COFF dependency for flexlink.exe. A possible solution is to make flexlink.exe work with ELF and Mach-O object files
  3. When ocamlopt is linking object files into an executable on Windows, the Host/Target Machine compiler must match (ie. MSVC or MinGW) and the Host/Target Machine word size (ie. 32 or 64) must match because flexlink.exe bundles a word size + compiler named object file flexdll_msvc.obj, flexdll_mingw64.obj, etc. into the final executable.
    becomes

    • Roadmap: Solve the flexdll Windows ABI dependency.
  4. if your Target Machine is a Unix system, make sure you run the Host Machine cross-compiler on a Unix system. That equalizes ostype_unix

    and

    if your Target Machine is a Windows system, make sure you run the Host Machine cross-compiler on a Windows system. That equalizes ostype_windows

    becomes

    • Roadmap: Solve the ostype_unix and ostype_win32 and ostype_cygwin seperation.

@ChenQi1989
Copy link

@dra27 Sorry to bother. I've been looking at the configure and Makefile for some time. I found it's impossible to generate a cross-compiler (runs on linux x86-64 and generate codes for linux aarch64). The build process involves running 'ocamlrun', which is also expected to run on target. Am I missing something?

@shindere
Copy link
Contributor

shindere commented Oct 18, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants