Comment author: administrator

Dear Gerd,

I recently got a bug report for one of my libraries (equeue) that
did not work in a multi-threaded program. Actually, the program
wasn't multi-threaded, but compiled with -thread and threads.cma
(because of a another library), so the multi-threading machinery was
initialized. My library is an enhanced version of "system",
i.e. does fork + exec, and sometimes SIGCHILD signals are emitted. I
never thought it worked in an mt program because of other reasons
(problems with fork, no access to the thread-specific signal
mask). However, it turns out that the problems are much more
fundamental, and can hang the O'Caml runtime at any time (although
this is very unlikely if the program doesn't use signals for
application purposes).

Thank you for this interesting bug report and for the repro case,
and thanks Samuel for the proposed fix (which however exhibits another
symmetrical race condition).

Damien Doligez and I partially fixed the bug in the CVS trunk (should be in
the 3.09 release).

The fix is partial in that it assumes that the POSIX thread function
pthread_mutex_trylock() is async-signal safe, which is not guaranteed
by the POSIX thread spec. For example, the fix works under Linux
(both the old LinuxThreads library and the new NPTL library implement
trylock() in a way that is async-signal safe), but the deadlock is
still there under MacOSX.

Given the way OCaml processes signals and the limitations of POSIX
threads, it is fundamentally impossible to implement things so that
threads and signals work well together. So, the current fix is a best
effort, and improves over the previous implementation, but there are
still no guarantees that threads and signal combine well.

Another observation: In a signal handler, POSIX forbids to call any pthread
function. The O'Caml runtime does, however

Yes, and it calls many other functions that should not be called from
signal handlers according to POSIX. Basically, the OCaml runtime
system makes the assumption that if the main program is blocked on a
syscall, it is safe to do arbitrary computations in a signal handler.
That assumption has proved effective in practice for "true" system
calls, but as the MacOSX example demonstrates, is not always true for
POSIX thread primitives. Again, if we were to do things by the specs,
I'm afraid there would be no signal handling at all in OCaml...

Best wishes,

• Xavier

Comment author: administrator

Am Freitag, den 29.07.2005, 15:48 +0200 schrieb Xavier Leroy:

The fix is partial in that it assumes that the POSIX thread function
pthread_mutex_trylock() is async-signal safe, which is not guaranteed
by the POSIX thread spec. For example, the fix works under Linux
(both the old LinuxThreads library and the new NPTL library implement
trylock() in a way that is async-signal safe), but the deadlock is
still there under MacOSX.

Given the way OCaml processes signals and the limitations of POSIX
threads, it is fundamentally impossible to implement things so that
threads and signals work well together. So, the current fix is a best
effort, and improves over the previous implementation, but there are
still no guarantees that threads and signal combine well.

Another observation: In a signal handler, POSIX forbids to call any pthread
function. The O'Caml runtime does, however

Yes, and it calls many other functions that should not be called from
signal handlers according to POSIX. Basically, the OCaml runtime
system makes the assumption that if the main program is blocked on a
syscall, it is safe to do arbitrary computations in a signal handler.
That assumption has proved effective in practice for "true" system
calls, but as the MacOSX example demonstrates, is not always true for
POSIX thread primitives. Again, if we were to do things by the specs,
I'm afraid there would be no signal handling at all in OCaml...

I think one of the problems is that OCaml supports asynchronous signals,
i.e. in a blocking section the signal handler is immediately executed.
Imagine what would happen when such signals would be simply deferred
until the blocking section is left. The implementation would be much
simpler, and I think it could be done conforming to the specs. The
visible difference to the current implementation depends on the type of
the blocking section. If it is a direct syscall, nothing will change,
because the syscall immediately returns with EINTR (most blocking
sections are of this type; I don't know whether Windows behaves the
same). If it is a C routine like name service lookup, there will be a
small, but limited delay until the signal handler is executed. The
problematic case is an external event loop like labltk. However, I think
this case can be handled when the event loop allows it to inject custom
events, i.e. signals are mapped to events. Of course, all these
libraries need to be changed (but there aren't that many libraries of
this type).

Interestingly, the multi-threading tick signal isn't affected at all,
because it needs not to wake up blocking sections. (Actually, I am
wondering why a real OS-level signal is used. In principle, this signal
only interrupts the execution of OCaml code from time to time to emulate
time slices, and this can be done without OS-level signals. One never
needs to interrupt blocked threads - or am I missing something?)

I think we need a reliable coexistence of threads and signals for many
real-world applications. This is more important than mimicking the C
behaviour of signals as closely as possible (OCaml does not support AIO
anyway, and for almost other applications small delays in signal
handling are acceptable). Even now, if there was a switch to turn off
asynchronous signals in favour of absolute reliability, I would do this
for some applications (e.g. networking applications).

So my suggestion is to provide such a switch in 3.09, and allow people
to experiment whether they can accept the new semantics of signals.

Gerd

Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany
gerd@gerd-stolpmann.de http://www.gerd-stolpmann.de
Telefon: 06151/153855 Telefax: 06151/997714

Comment author: administrator

Dear Gerd,

Imagine what would happen when such signals would be simply deferred
until the blocking section is left. The implementation would be much
simpler, and I think it could be done conforming to the specs. The
visible difference to the current implementation depends on the type of
the blocking section. If it is a direct syscall, nothing will change,
because the syscall immediately returns with EINTR (most blocking
sections are of this type; I don't know whether Windows behaves the
same).

Interesting suggestion. Actually, my first attempt at signal
handling in Caml Light circa 1990 was along these lines, but did not
work because we were working on BSD systems (SunOS 3 and 4) at this
time, and in BSD many system calls restart on a signal rather than
returning EINTR. The situation is likely to be different now. I'll
have to check whether POSIX and the Single Unix Specification
guarantee the EINTR behavior by default.

Note that we could get the best of both worlds (?) as follows: use
synchronous signal processing as in 3.08 for single-threaded code, and
use your deferred strategy for multi-threaded code, therefore avoiding
the problematic pthread_mutex_trylock call in the signal handler. The
new signal handling code Damien and I just put in 3.09 makes it easy
to switch to the deferred strategy when the threading library
initializes.

There are no problems with Windows: the only signal-like mechanism is
ctrl-C keyboard interrupts, and it is already implemented along the
lines of your deferred strategy, i.e. just record the interrupt but
don't act over it immediately. The reason is that Windows runs the
ctrl-C handler function in a new thread, so immediate callback to Caml
code just doesn't work.

Interestingly, the multi-threading tick signal isn't affected at all,
because it needs not to wake up blocking sections. (Actually, I am
wondering why a real OS-level signal is used. In principle, this signal
only interrupts the execution of OCaml code from time to time to emulate
time slices, and this can be done without OS-level signals. One never
needs to interrupt blocked threads - or am I missing something?)

No, you're correct. Actually, the systhreads implementation (over
POSIX/Win32 threads) does not use an OS periodic signal a la setitimer():
there is a separate thread that loops over a "sleep; post a pending signal"
sequence.

I think we need a reliable coexistence of threads and signals for many
real-world applications.

I still believe you're taking serious risks by mixing signals and
threads: this is the darkest corner of the POSIX thread spec. But,
yes, it would be best to make Caml itself handle the mixture reliably,
so that all the bugs you're going to run into are those of your code
and the external libraries you call, but not ours :-)

Best wishes,

• Xavier

Comment author: administrator

see also #3680. Fixed in 3.09 by XL and DD, 2005-07-29