Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Patch from PR6733 does not work on OS X #6927

Closed
vicuna opened this issue Jul 12, 2015 · 13 comments
Closed

Patch from PR6733 does not work on OS X #6927

vicuna opened this issue Jul 12, 2015 · 13 comments

Comments

@vicuna
Copy link

vicuna commented Jul 12, 2015

Original bug ID: 6927
Reporter: @whitequark
Assigned to: @whitequark
Status: resolved (set by @damiendoligez on 2017-02-24T12:07:38Z)
Resolution: suspended
Priority: normal
Severity: minor
OS: OS X
Version: 4.02.2
Target version: later
Category: -for ocamlbuild use https://github.com/ocaml/ocamlbuild/issues
Tags: patch
Related to: #6733
Monitored by: @gasche

Bug description

Amusingly, this is a case of doing more work than necessary and failing at it, because the linker invocation already includes the -shared/-bundle flag, added by the compiler driver (ocamlc/ocamlopt). On Linux the second instance of -shared, added by the flag ["ocaml"; "link"; "output_shared"] built-in rule, is ignored, but on OS X it conflicts with -bundle.

A patch is attached. I verified that it works on Linux and OS X. It probably also works on Windows, since now output_so includes no platform details not used in the compiler elsewhere, but I'm not sure. Flexlink might or might not be a problem for the 3rd party code that wishes to load the resulting DLLs.

Additional information

The target is a bundle, but it is called x.{byte,native}.so. This is actually common practice, and there is no generally agreed upon file extension for bundles (although .bundle is often used), but ocamlbuild rules mention .dylib, which does not actually happen. A second patch is attached to alleviate that.

File attachments

@vicuna
Copy link
Author

vicuna commented Jul 12, 2015

Comment author: @whitequark

Note that this leaves the output_shared tag used but not defined. This was an oversight at first, but then I realized that it would be convenient for myocamlbuild.ml authors dealing with some kind of weird platform, so I think it should stay.

@vicuna
Copy link
Author

vicuna commented Jul 12, 2015

Comment author: @whitequark

Anyone reading this and using 4.02.2 can use the following myocamlbuild rules to override the broken behavior:

open Ocamlbuild_plugin
open Ocamlbuild_pack

dispatch (function
Before_rules ->
let native_link_gen linker =
Ocaml_compiler.link_gen "cmx" "cmxa"
!Options.ext_lib [!Options.ext_obj; "cmi"] linker
in
let native_output_obj x = native_link_gen Ocaml_compiler.ocamlopt_link_prog
(fun tags -> tags++"ocaml"++"link"++"native"++"output_obj") x
in
rule "ocaml: cmx & o -> native.(so|dll)"
~prod:("%.native"-.-(!Options.ext_dll))
~deps:["%.cmx"; "%.o"]
(native_output_obj "%.cmx" ("%.native"-.-(!Options.ext_dll)));
| _ -> ())

@vicuna
Copy link
Author

vicuna commented Jul 19, 2015

Comment author: @whitequark

The issue is actually more complex than this, because the OS X bundles are not equivalent to Linux shared libraries and Windows dll's: you cannot link them directly and have to use dlopen() and friends.

The differences between bundles and dylibs are rather subtle. At 10.4 and before, you could not dlopen() a dylib--you could only link to it, in fact there was no dlopen()--and you could use the corresponding NS* API to load and unload a bundle. After 10.4, bundles and dylibs are almost entirely equivalent; you can dlopen() and dlclose() both. The only differences are that 1) you cannot directly link a dylib and 2) rpath is not handled in the same way. (I have grepped ld64, XNU and dyld from Apple's opensource repository to independently confirm this information.)

Now, let's look at OCaml. OCaml creates dynamic libraries in three different ways:

  • when building bytecode stubs (dllX.so);
  • when building a .cmxs plugin;
  • when building a .native.so or even .byte.so.

Currently, all of these create bundles: the former two because plugins and stubs match the semantics of the bundles as originally created (i.e. at 10.4 and before), the latter because ocamlbuild reuses the configuration options. As a side effect, the .native.so can not be linked against, which is a severe problem for the people trying to link OCaml code into a larger binary portably.

I propose to always build dylibs instead. At 10.5 and later, none of the behavior that OCaml depends on will change (the rpath handling change is irrelevant because neither OCaml-built plugins nor OCaml-built .{byte,native}.so files import any libraries except system ones). Thus, both plugins and .{byte,native}.so files could be both dynamically loaded and linked against. Admittedly, linking against plugins would not be very useful, though who knows.

@vicuna
Copy link
Author

vicuna commented Jul 19, 2015

Comment author: @whitequark

OK, there is actually one more problem on OS X. That is, without the -all_load flag (mostly equivalent to -Wl,--whole-archive) flag, the generated library does not include the symbols from libasmrun.a(startup.o), notably caml_startup. On the other hand, with the -all_load flag, not all _caml_curry functions are generated that are required for the stdlib.

The reason this happens is that OCaml's behavior of lazily generating curry functions depends on the absence of -Wl,--whole-archive flag, because it determines which curry functions to include on the level of an imported .cmx, corresponding to a .o inside a .a.

I'm actually not completely sure why the startup.o file is included on Linux. It might be some kind of platform-specific symbol dependency.

Either way, one way to solve it would be to pass -force_load /path/to/libasmrun.a when linking a .so with -output-complete-obj on OS X, so that all the archive members are included. I'm not sure if there are other ways.

@vicuna
Copy link
Author

vicuna commented Jul 19, 2015

Comment author: @whitequark

Let me know what the maintainers think about this and I can produce patches.

@vicuna
Copy link
Author

vicuna commented Jul 19, 2015

Comment author: @gasche

I asked (I don't know anything about OSX myself, sorry).

@vicuna
Copy link
Author

vicuna commented Jul 20, 2015

Comment author: @damiendoligez

This startup.o problem may be related to #5693, see #5693#c7797 .

@vicuna
Copy link
Author

vicuna commented Jul 20, 2015

Comment author: @whitequark

Yes, this seems to be the same issue.

@vicuna
Copy link
Author

vicuna commented Jul 20, 2015

Comment author: @garrigue

Answering 14211:
I'm a bit surprised that we are still using bundles under OSX.
Since OCaml 3.11, we have switched from using bundle specific code in the runtime to using generic dlopen, so there should be no difficulty in switching to dylib's, isn't it?
Of course the change of file name could confuse the installer for external libraries.
If we can make the transition smooth, this seems to be the way to go.

@vicuna
Copy link
Author

vicuna commented Jul 20, 2015

Comment author: @whitequark

There are no problems with building the dylibs. I have tried building OCaml like that and it works. However, I anticipate problems with tools, since none of them know about the extension.

Specifically:

So on one hand, using .so for .dylib is kind of user-hostile. A .so is not a .dylib and there are significant semantic differences between the two, most importantly with the XCode GUI tools, which would be the most likely way a .dylib will be processed by an end user on OS X. On another one, this change would probably break a substantial part of the OCaml ecosystem on OS X, which is already under-tested, until at least one major release.

@vicuna
Copy link
Author

vicuna commented Jul 20, 2015

Comment author: @whitequark

Oh, and this is compounded by the fact that the stdlib does not allow to determine whether you're running on Darwin or not. I'm actually not sure what would be a decent way to do this from OCaml, at all.

@vicuna
Copy link
Author

vicuna commented Feb 24, 2017

Comment author: @damiendoligez

Related to #988

@vicuna
Copy link
Author

vicuna commented Feb 24, 2017

Comment author: @damiendoligez

ocamlbuild is now a separate project that lives on GitHub.
PR transferred to ocaml/ocamlbuild#149

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant