New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Registering threads created from C #6764
Comments
Comment author: @mshinwell I think it is the case that you only need to use [caml_c_thread_register] from a particular thread if: (a) the OCaml code is compiled to use threads; and If the OCaml code isn't compiled to use threads then you obviously also need to ensure that your C program does not attempt to execute OCaml code in a re-entrant or parallel manner. I presume you have access to the third-party application's code, since you can't directly call an arbitrary OCaml function from a C program, owing to the difference in calling conventions. As such, I presume there is either a modification to the third-party program or some intervening veneer; in either case, you could call the thread registration function before the OCaml callback invocation, I suspect. In case it is useful: you can call [caml_c_thread_register] more than once from the same thread without trouble, and I think there's little performance penalty to doing so. |
Comment author: braibant Just to clarify: I do not have access to the third party application. I am building a shared library that implements a given interface. The app loads the dynamic library and perform lookup for known symbols. There is indeed a shim around the OCaml code, implemented using inverted stubs from Ctypes. I will hack the ctypes generated code to add the correct calls to the thread registration function before the OCaml callback invocation. (Btwm I indeed noticed that the [caml_c_thread_register] function can be called more than once, but was a bit afraid of the runtime cost. Since the need for that function was not obvious the way I understood the documentation, I argued against adding these calls everywhere.) |
Comment author: @mshinwell Yeah, that solution sounds correct. I wouldn't worry about the runtime cost of [caml_c_thread_register], at least in the first instance. |
Comment author: braibant Ok, I think I need to increase the severity of this issue: I have patched ctypes to wrap calls to [register] and [unregister] around the calls to OCaml callbacks. The code that is generated is similar to the following gist. https://gist.github.com/braibant/827b3b2158f94f461847 Running the third party application with this code yields a segfault. The backtrace is the following one
The content of th is as follows
I think this issue is somewhat similar to the one described in #4702 Namely:
|
Comment author: @mshinwell Ack |
Comment author: @mshinwell I think the problem is that you need to call [caml_thread_initialize] before [caml_c_thread_register]. It may be worth trying to fix the runtime so this requirement can be removed. For posterity, I managed to reproduce the bug using the following nasty little test case. (Revised version here now that actually works...) ==> build.sh <== set -x -u -e OCAML=/path/to/ocaml/installation gcc -fPIC -g -c -I$INCLUDE shared_lib_c.c ==> main.c <== static volatile int num_threads = 0; static void* thread_body(void* lib) int main(int argc, char* argv[]) lib = dlopen("shared_lib.so", RTLD_NOW); fn = dlsym(lib, "call_ocaml_init"); printf("starting thread creation loop\n"); return 0; ==> shared_lib_c.c <== void call_ocaml_init(char** argv) value call_ocaml_in_shared_lib(void) ==> shared_lib.ml <== let ocaml_in_shared_lib x = let () = Build and run (on Linux, after editing the path to your OCaml installation in build.sh) with: |
Comment author: braibant I can reprodue a segfault with your example but I cannot reproduce a working behavior: when building and running your example, num active threads keeps increasing, and I never see a call to the ocaml_in_shared_lib function. Are you seeing a different behavior? |
Comment author: braibant I copied your code here: https://github.com/braibant/ocaml-pr6764 |
Comment author: @mshinwell Sorry, I've fixed the test case properly now. |
Comment author: @mshinwell As a follow-up: maybe it should be ensured by the compiler that the [Thread] module's initializer is always run whenever -thread is specified. Or something. Also: you can't just call [caml_thread_initialize] from C directly; it needs to be called from an OCaml context, I think. It's safest just to make sure the [Thread] module is initialized. |
Comment author: braibant Another follow up: if I remove the call to Callback.register, I get another segfault. It is not clear from the manual that this call to Callback.register is mandatory. Unfortunately, the way Ctypes uses callback, they are not always registered, and it's hard to register them all... |
Comment author: @mshinwell I think you must always use [Callback.register] if you are going to invoke the function from C, otherwise [caml_named_value] will return a null pointer, dereferencing of which will cause a segfault. |
Comment author: braibant Ok, I mixed up things and my comment was not founded. Just to be crystal clear: one need to use [Callback.register] if and only if the callback is called from C using the [caml_named_value], right? |
Comment author: @mshinwell I believe that is correct, yes. |
Comment author: @mshinwell Does your program now work? |
Comment author: braibant I am still looking for the cause of a segfault, but it looks like it's coming from ctypes. I will keep you informed (and probably write a summary of this whole experiment) |
Comment author: @mshinwell In case it helps: in the case of a delayed segfault due to heap corruption, you can sometimes narrow it down by calling [Gc.compact] in an attempt to trigger it earlier. (For example, you could do this before releasing the runtime lock in each callback.) |
Comment author: braibant I am still working on a segfault in our dll. I have nailed down an issue in the code we call when the shared object is unloaded: the OCaml documentation is a bit scarce about what is required to cleanup the runtime. Any idea about how to do that in the multithreaded context of your above example? (edit to link a related github pull request #71 ) |
Comment author: @mshinwell Not completely sure, but if you're unloading the runtime anyway as part of the DLL, I'm not sure why it matters. Can you give any more information as to the location of the segfault? |
Comment author: braibant I have updated the code here https://github.com/braibant/ocaml-pr6764 to illustrate the issue. |
Comment author: @mshinwell I'm going to close this issue, since it appears #6776 (killing the ticker thread) is the only fix required. |
Original bug ID: 6764
Reporter: braibant
Assigned to: @mshinwell
Status: closed (set by @mshinwell on 2015-05-06T15:31:20Z)
Resolution: not a bug
Priority: normal
Severity: text
Version: 4.02.1
Target version: 4.02.2+dev / +rc1
Category: documentation
Related to: #4702 #6776
Monitored by: @gasche @yallop
Bug description
Looking at section 19.10.1 "Registering threads created from C" in the manual, I am slightly puzzled by the documentation. I am not quite sure if that means what it means in the context of an OCaml library that's called from a third party C program using dynamic linking.
Obviously, there is no way to register all the threads that this third party application is going to create. Does that mean that it is not possible to use this kind of mix of OCaml/C? This does not seem to be the case reading the next paragraph in the manual, so I am a bit lost here.
The text was updated successfully, but these errors were encountered: