MantisBT - OCaml
View Issue Details
0005049OCaml~DO NOT USE (was: OCaml general)public2010-05-18 10:292013-08-31 12:46
glondu 
frisch 
normalfeaturealways
closedno change required 
ARM
 
 
0005049: natdynlink is broken on arm
Hello,

While rebuilding all Debian packages (armel port) with ocamlopt enabled (using trunk r10396), ssreflect failed to build. The full build log is available at [1].

The following command crashes with a segmentation fault:

  /usr/bin/coqc -dump-glob theories/ssreflect.glob -q -R src Ssreflect -R theories Ssreflect theories/ssreflect

It uses (nat)dynlink. The bytecode counterpart (by adding "-byte") doesn't segfault. Generating a Coq toplevel with ssreflect statically linked in (with "coqmktop -opt -o ssrcoq src/ssreflect.cmx"), and then using it (by adding "-image ./ssrcoq" to the command above) doesn't segfault either.

Running ocsigen in native code on arm also segfaults:
> root@daribow:/tmp# ocsigen.opt -V
> -- Dependencies of ocsigen.ext.staticmod: ocsigen.commandline, camlp4, ocsigen.ext.staticmod
> -- Needed: /usr/lib/ocaml/ocsigen/parsecommandline.cmxs, /usr/lib/ocaml/ocsigen/staticmod.cmxs
> Loading extension /usr/lib/ocaml/ocsigen/parsecommandline.cmxs
> Loading extension /usr/lib/ocaml/ocsigen/staticmod.cmxs
> Segmentation fault

These errors don't happen on amd64.


Cheers,

--
Stéphane
[1] http://ocaml.debian.net/debian/ocaml3120dev23r10396/failures/ssreflect_1.2%2Bdfsg-4%2B3.12.0%2Bdev23%2B10396%2B1_armel.build [^]
No tags attached.
Issue History
2010-05-18 10:29glonduNew Issue
2010-05-25 11:01frischNote Added: 0005498
2010-05-25 11:26glonduNote Added: 0005499
2010-05-25 11:38frischNote Added: 0005500
2010-05-25 13:36xleroyNote Added: 0005502
2010-05-25 13:36xleroyStatusnew => feedback
2010-05-25 17:49glonduNote Added: 0005514
2010-05-25 19:36Julien SignolesNote Added: 0005515
2010-05-26 00:26glonduNote Added: 0005516
2010-05-26 08:08frischNote Added: 0005519
2010-05-26 09:50Julien SignolesNote Added: 0005520
2010-05-28 12:19frischNote Added: 0005526
2010-05-28 12:20frischSeverityblock => feature
2010-05-28 12:20frischStatusfeedback => acknowledged
2010-05-28 12:20frischPlatform => ARM
2010-05-28 15:20mehdiNote Added: 0005530
2010-05-28 17:12frischNote Added: 0005531
2010-05-28 17:13frischNote Added: 0005532
2010-05-28 17:13frischStatusacknowledged => resolved
2010-05-28 17:13frischResolutionopen => no change required
2010-05-28 17:13frischAssigned To => frisch
2012-02-04 11:19meurerNote Added: 0006878
2013-08-31 12:46xleroyStatusresolved => closed
2017-02-23 16:36doligezCategoryOCaml general => -OCaml general
2017-03-03 17:55doligezCategory-OCaml general => -(deprecated) general
2017-03-03 18:01doligezCategory-(deprecated) general => ~deprecated (was: OCaml general)
2017-03-06 17:04doligezCategory~deprecated (was: OCaml general) => ~DO NOT USE (was: OCaml general)

Notes
(0005498)
frisch   
2010-05-25 11:01   
Natdynlink is indeed supported only a limited number of platforms. We simply plan to declare ARM not to be in this list (and, accordingly, it will not be installed).
(0005499)
glondu   
2010-05-25 11:26   
What criterion determines whether natdynlink is supported on a given architecture or not? So far, all architectures that support native code generation supported by Debian support natdynlink as well, so I assumed Linux and FreeBSD as a whole did. Could you summarize the technical difficulties? (al least for future reference)

FWIW, armel is currently the third most popular architecture in Debian [1] (after i386 and amd64). I (and I guess many other people) consider it more valuable that e.g. sparc.

[1] http://popcon.debian.org/ [^]
(0005500)
frisch   
2010-05-25 11:38   
It turns out that natdynlink is (was) compiled on all platforms that support dynamic linking, but it was never supported on all these platforms.

The problem is that there are also some constraints on the code generated by ocamlopt in order to have a working natdynlink.

I don't think anyone on the caml development team is going to spend energy trying to make natdynlink work for arm, but if you can identify the problem and submit a patch, it will most likely be considered for inclusion.


Would it make things simpler for you if a version of natdynlink were always installed, even when not functional (trying to use it would raise a clean runtime exception)? As opposed to, say, not install dynlink.cmxa at all.
(0005502)
xleroy   
2010-05-25 13:36   
For the record, here are the conditions to be met by a platform for natdynlink to work:

- Either non-PIC code can be put in shared libraries. Then, the ocamlopt code generator need no changes. This is the case for x86-32 bits under Linux, at least. (Recent versions of MacOS X broke this. Under Windows, this works thanks to FlexDLL magic.)

- Or the ocamlopt code generator was changed specifically to generate PIC code. This is the case for x86-64 bits, where the changes and the performance hit were minimal.

- Or the ocamlopt code generator just happens to generate PIC code. I don't think this is the case of any of the other code gens, but Alpha might come very close.

A quick look at ARM code generated by gcc -fPIC suggests that significant work is needed to make ocamlopt produce PIC code on this platform, and the run-time overhead of PIC code is large. So, don't expect a working natdynlink on ARM any time soon.

> FWIW, armel is currently the third most popular architecture in Debian [1] (after i386 and amd64).

... at 1% of the installed base...

> I (and I guess many other people) consider it more valuable that e.g. sparc.

I certainly agree with you on that; witness my efforts to switch ARM to EABI. That doesn't mean that making natdynlinked-ssreflect work on ARM is a priority.
(0005514)
glondu   
2010-05-25 17:49   
frish wrote:
> Would it make things simpler for you if a version of natdynlink were always installed, even when not functional (trying to use it would raise a clean runtime exception)? As opposed to, say, not install dynlink.cmxa at all.

This should probably be asked to a wider audience...

My personal opinion as a programmer would be not to install dynlink.cmxa... as in the OCaml spirit: prefer compile-time errors (such as inexistent dynlink.cmxa) rather than runtime errors. Either way, software relying on dynlink will have to deal with situations where there is ocamlopt, but no natdynlink (the possibilities I see are providing a static counterpart or falling back to bytecode, the latter being the simplest one).

From a maintainer PoV, installing dynlink.cmxa and failing at run-time could have worked if dynlink was never used during the build process... but this approach seems fragile and not very satisfactory.

xleroy wrote:
> the run-time overhead of PIC code is large. So, don't expect a working natdynlink on ARM any time soon.

Does that mean that a patch for this would be rejected, as opposed to Alain's invitation to provide such a patch? I don't say that I would write such a patch myself, but other people might have interest...

> ... at 1% of the installed base...

I didn't hide it :-)

> witness my efforts to switch ARM to EABI

Of course I do, and thank you for that.
(0005515)
Julien Signoles   
2010-05-25 19:36   
=====
Glondu wrote:
frish wrote:
> Would it make things simpler for you if a version of natdynlink were always installed, even when not functional (trying to use it would raise a clean runtime exception)? As opposed to, say, not install dynlink.cmxa at all.

This should probably be asked to a wider audience...
=====

I agree with Stéphane: it is better to have no dynlink.cmxa that a broken one.

Currently the 3 situations already occur anyway:
- no dynlink.cmxa (architectures that do not support natdynlink)
- a broken dynlink.cmxa generating exceptions at runtime (some versions of Mac OS X at least)
- a working dynlink.cmxa

Thus we have to deal with all of these (that is what Frama-C did for instance).
(0005516)
glondu   
2010-05-26 00:26   
I forgot this one:

> - Or the ocamlopt code generator just happens to generate PIC code. I don't think this is the case of any of the other code gens, but Alpha might come very close.

As far as Debian is concerned, ocamlopt is not installed on alpha, but it is on powerpc and sparc and at least ssreflect happens to work there.
(0005519)
frisch   
2010-05-26 08:08   
> Either way, software relying on dynlink will have to deal with situations where there is ocamlopt, but no natdynlink (the possibilities I see are providing a static counterpart or falling back to bytecode, the latter being the simplest one).

I believe it is not uncommon to have applications that depends on dynlink only when the user explicitly ask to load some kind of addins, but are still fully functional without dynlink. Having the natdynlink module always available with a way to check whether it works would simplify the code organization and build system for these programs. It is always possible to check the configuration statically if needed (there will be a NATDYNLINK variable in the OCaml config file). There are still cases where dynlink.cmxa is not installed, but this could easily be addressed.

I'm still not 100% sure about the right solution.
(0005520)
Julien Signoles   
2010-05-26 09:50   
Anyway if you want to be compatible with old versions of ocaml (< 3.11) as well as many architectures, you have to handle the case where dynlink.cmxa is not installed. Furthermore you have to handle the incompatibility of Dynlink interface.

Thus, from a developer point of view, I believe that the simplest way to handle all the possible behaviors is to write an unique front-end to Dynlink and to statically generate the implementation according to the configuration. That a work done by any reasonable build system.

But it is still better to have no dynlink.cmxa that a broken one since the error is detected earlier (as Stéphane already said). Moreover, for a build system, it is easier to detect a missing file that a buggy one whatever is the ocaml config file/version.

Above all, I believe the most important think is to be consistent: either you provide dynlink.cmxa on **all** architectures with an easy and documented way to check whether it is working, either you don't provide it on architectures without an usable natdynlink.
(0005526)
frisch   
2010-05-28 12:19   
Ok, you convinced me. dynlink.cmxa (and related files) are no longer installed when native dynlink is not supported.

The list of platforms where natdynlink is supported is given in the configure script. Currently, the list is quite conservative, but it should be extended with feedback from users who can test natdynlink on other platforms:

  case "$host" in
    *-*-cygwin*) natdynlink=true;;
    i[3456]86-*-linux*) natdynlink=true;;
    x86_64-*-linux*) natdynlink=true;;
  esac
(0005530)
mehdi   
2010-05-28 15:20   
Maybe I'm not too late here… But, I think that providing dynlink.cmxa on *all* architectures is a better approach for the following reasons:

- It allows to have the same code to compile on any architecture. Otherwise, if we do use natdynlink in some application, we have to modify the code before compiling depending on natdynlink-less native architectures.

- "unix.cmxa" is another example where you have a completely broken module on some architectures (e.g. Windows). Nonetheless, it's installed on Windows but fails gracefully with an exception.

It would be really nice to have a natdynlink module even when dynlink is broken and raise an exception when broken (and even add a "val has_dynlink : bool" (like the test "is_native") to exploit constant inlining…).

> ... at 1% of the installed base...
Keep in mind that on this kind of architecture, popcon is usually disabled (s390 is also very under estimated).
(0005531)
frisch   
2010-05-28 17:12   
I'm afraid you're too late, I've already reverted the changes that implemented a Dynlink.is_supported flag, and I don't feel to re-revert it, unless there is some overwhelming popular demand.

(As a side node, I wouldn't say Unix is completely broken on Windows: a lot of its functions work pretty-well.)
(0005532)
frisch   
2010-05-28 17:13   
Natdynlink is indeed supported only on some specific platforms, which does not include ARM. Third party packages should be prepared to deal with other platforms and degrade gracefully.
(0006878)
meurer   
2012-02-04 11:19   
Natdynlink should now work for ARM starting with revision 12124 (trunk).