English version
Accueil     À propos     Téléchargement     Ressources     Contactez-nous    

Ce site est rarement mis à jour. Pour les informations les plus récentes, rendez-vous sur le nouveau site OCaml à l'adresse ocaml.org.

Browse thread
Rewriting the Digest module causes linking errors
[ Home ] [ Index: by date | by threads ]
[ Search: ]

[ Message by date: previous | next ] [ Message in thread: previous | next ] [ Thread: previous | next ]
Date: 2010-03-18 (10:58)
From: Goswin von Brederlow <goswin-v-b@w...>
Subject: Re: Random segfaults / out of memory
Goswin von Brederlow <goswin-v-b@web.de> writes:

> Goswin von Brederlow <goswin-v-b@web.de> writes:
>>> On Wed, Mar 17, 2010 at 09:27:30AM +0100, Goswin von Brederlow wrote:
>>>> I want to rewrite the Digest module to expose a more lowlevel interface
>>>> to the md5 digest and add support to digest Bigarrays. I've patched the
>>>> respective files involved and it all looks alright but when I try to
>>>> build ocaml I get the following error:
> Ok, so I managed to bootstrap the compiler properly and build debian
> packages with my new Digest interface. But something is still wrong as I
> randomly get segfaults or
> <No room for growing heap
> Fatal error: out of memory.
> The more threads I use to compute digests in parallel the more likely
> the error becomes. But that might just be an issue with more allocations
> hapening and not a race condition between threads.

I finaly tracked down the issue with the help of Erkki Seppala.

First problem:

I had the function declared as "noalloc" but used CAMLparam2() in
it. That seems to cause random segfaults. I don't understand why but if
I remove the "noalloc" then it works.

Second problem:

When I remove the CAMLparam2() the finalizer is called too early:

CAMLprim value md5_update_bigarray(value context, value vb)
  //CAMLparam2(context, vb);
  struct helper *helper = (struct helper*)Data_custom_val(context);
  struct MD5Context *ctx = helper->ctx;
  fprintf(stderr, "update_bigarray: helper = %p, ctx = %p\n", helper, ctx);
  struct caml_ba_array * b = Caml_ba_array_val(vb);
  unsigned char *data = b->data;
  uintnat len = caml_ba_byte_size(b);
  caml_MD5Update(ctx, data, len);
  return Val_unit;

  let rec loop () =
    Mutex.lock mutex;
    if !num = 0
    then Mutex.unlock mutex
    else begin
      decr num;
      Mutex.unlock mutex;
      let context = context () in
      let () = update_bigarray context buf
	loop ()
    loop ()

This sometimes results in the following code flow:

 context ()				<- allocates memory
 update_bigarray context buf
 GC runs
 context is finalized			<- frees memory
   caml_MD5Update(ctx, data, len);	<- writes to ctx which is freeed

Looks like ocamlopt really is so smart that is sees that context is
never used after the call to update_bigarray and removes it from the
root set before calling update_bigarray. It assumes the update_bigarray
will hold all its arguments alive itself, which is a valid assumption.

This is a tricky situation. The md5_update_bigarray() on its own is a
"noalloc" function. But due to the caml_enter_blocking_section() another
thread can alloc and trigger a GC run in parallel. So I guess that makes
the function actually not "noalloc".

Well, problem solved, lesson learned. :)