Version française
Home     About     Download     Resources     Contact us    

This site is updated infrequently. For up-to-date information, please visit the new OCaml website at

Browse thread
CamlMPI: sending marshalled objects
[ Home ] [ Index: by date | by threads ]
[ Search: ]

[ Message by date: previous | next ] [ Message in thread: previous | next ] [ Thread: previous | next ]
Date: -- (:)
From: Thomas Fischbacher <tf@f...>
Subject: CamlMPI: sending marshalled objects

Hello everybody,

there is an old Caml/MPI interface, written by Xavier himself in 1998.
This provides a pair of C functions caml_mpi_send/caml_mpi_receive,
which may look a bit funny considering present day ML-C-interface coding
conventions, but GC-wise is actually valid code as far as I can judge.
For reference, I added them at the end of this email.

As far as I can see, output_value_to_malloc() actually should be able to
serialize virtually any ML object (which does not contain alloc_custom()
blocks and similar stuff) to a buffer. Sending this over the net also
should not be much of a problem, regardless of the data being sent. If,
on the other hand, I manually serialized to a string and tried to send
that, I would be bound by the 16 MB maximal string length limitation on
32-bit architectures.

So much the theory. But what I actually find in reality is that -- even
using caml_mpi_send/caml_mpi_receive rather than intermediate strings --
my code crashes when I try to pass around serialized ML data over MPI
which is larger than 16 MB. I am not yet 100% sure that this really is
the problem, but at present, I have every reason to believe so.

So, (1) what is going on here, and (2) how can I repair it?
Is there a quick fix?

value caml_mpi_send(value data, value flags,
                     value dest, value tag, value vcomm)
   MPI_Comm comm = Comm_val(vcomm);
   char * buffer;
   long len;

   Begin_root(vcomm)             /* prevent deallocation of communicator */
     output_value_to_malloc(data, flags, &buffer, &len);
     /* This also allocates the buffer */
     MPI_Send(buffer, len, MPI_BYTE, Int_val(dest), Int_val(tag), comm);
   return Int_val(len);

value caml_mpi_receive(value vlen, value source, value tag, value vcomm)
   MPI_Comm comm = Comm_val(vcomm);
   mlsize_t len = Long_val(vlen);
   char * buffer;
   MPI_Status status;
   value res;

   Begin_root(vcomm)             /* prevent deallocation of communicator */
     buffer = stat_alloc(len);
     MPI_Recv(buffer, len, MPI_BYTE,
              Int_val(source), Int_val(tag), comm, &status);
     res = input_value_from_malloc(buffer, 0);
     /* This also deallocates the buffer */
   return res;

best regards,
Thomas Fischbacher