Hello,

I need to implement (meta) heuristic algorithms that
uses parallelism in order to (attempt to) solve a (hard)
machine learning problem that is inherently exponential.
The aim is to take maximum advantage of the multi-core
processors I have access to.

To that effect I have revisited many of the lively
discussions in threads related to concurrency, parallelism
and shared memory in this mailing list. I however still
have many doubts, some of which are very basic.

My initial objective is to make a very simple tests that
launches k jobs. Each of these jobs must access
a common data set that is read-only. Each of the k threads
in turn generates its own data. The data generated by the k
jobs are then placed in a queue for further processing.

The process continues by launching (or reusing) k/2 jobs.
Each job consumes two elements from the queue that where
previously generated (the common data set must still be
available). The process repeats itself until k=1. Note
that the queued data is not small nor can I determine
a fixed maximum size for it.

I have opted to use "kernel-level threads" that allow use
of the (multi-core) processors but still allow "easy"
access to shared memory".

I have done a cursory look at:
- Ocaml.Threads
- Ocaml.Unix (LinuxThreads)
- coThreads
- Ocamlnet2/3 (netshm, netcamlbox)
(An eThreads library exists in the forge but I did not examine this)

My first concern is to take advantage of the multi-cores so:

1. The thread library is not the answer
    Chapter 24 - "The threads library is implemented by time-sharing on 
a
    single processor. It will not take advantage of multi-processor
    machines." [1]

2. LinuxThreads seems to be what I need
    "The main strength of this approach is that it can take full
     advantage of multiprocessors." [2]


Issue 1

In the manual [3] I see only references to function for the creation
and  use of processes. I see no calls that allow me to simply generate
and assign a function (job) to a thread (such as val create : ('a -> 'b)
  -> 'a -> t in the Thread module). The unix library where LinuxThreads
is now integrated shows the same API. Am I missing something or
is their no way to launch "threaded functions" from the Unix module?
Naturally I assume that threads and processes are not the same thing.

Issue 2

If I cannot launch kernel-threads to allow for easy memory sharing, what
other options do I have besides netshm? The data I must share is defined
by a recursive variant and is not simple numerical data.

I would appreciate any comments.

TIA,
Hugo F.


[1] http://caml.inria.fr/pub/docs/manual-ocaml/manual038.html
[2] http://pauillac.inria.fr/~xleroy/linuxthreads/
[3] http://caml.inria.fr/pub/docs/manual-ocaml/libref/ThreadUnix.html
[4] http://caml.inria.fr/pub/docs/manual-ocaml/manual035.html