For anyone who is interested, we've resolved this issue.  We were
mistaken to think this was a GC issue at all.  The long delays simply
resulted from some odd decisions by the Linux scheduler.  We resolved
them by changing SCHED_RR scheduling for the threads in question.

Yaron

On 8/31/05, Yaron Minsky <yminsky@gmail.com> wrote:
> I've been running into some very odd behavior with the GC and
> Mutex.lock.  We have an application where a bunch of threads are all
> pushing updates through a synchronized queue, so all of the threads are
> contending for a global lock.  Under some rare circumstances, we've seen
> very odd GC delays coming up, and we're at a bit of a loss to explain
> them.
> 
> The basic situation is that when the system comes under unusually high
> GC load, we will sometimes see one of the threads take a long time (on
> the order of half a second) when it tries to acquire the lock. We
> instrumented the acquisition of the lock as follows:
> 
> let pre_stat = Gc.quick_stat () in
> Mutex.lock mutex
> let post_stat = Gc.quick_stat () in
> 
> The thread that gets delays shows hundreds or thousands of minor
> collections and 5-10 major collections between the calls to
> Gc.quick_stat ().  What's also weird is that it's always the same thread
> that gets blocked, and other threads are not blocked during this period.
> They seem to be able to acquire and release the lock without problems.
> And the thread that does get delayed isn't even the one causing the
> unusual load (although the delayed thread does do a big chunk of
> allocation that is immediately unreachable right before it gets blocked).
> 
> I can't come up with a reasonable explanation for how this might happen,
> and I'm hoping someone with a deeper understanding of the GC than I
> might be able to help.
> 
> Yaron
>