Interfacing C with Caml Light

This chapter describes how user-defined primitives, written in C, can be added to the Caml Light runtime system and called from Caml Light code.

Overview and compilation information

Declaring primitives

User primitives are declared in a module interface (a .mli file), in the same way as a regular ML value, except that the declaration is followed by the = sign, the function arity (number of arguments), and the name of the corresponding C function. For instance, here is how the input primitive is declared in the interface for the standard library module io:

        value input : in_channel -> string -> int -> int -> int
                    = 4 "input"

Values thus declared primitive in a module interface must not be implemented in the module implementation (the .ml file). They can be used inside the module implementation.

Implementing primitives

User primitives with arity n <= 5 are implemented by C functions that take n arguments of type value, and return a result of type value. The type value is the type of the representations for Caml Light values. It encodes objects of several base types (integers, floating-point numbers, strings, ...), as well as Caml Light data structures. The type value and the associated conversion functions and macros are described in details below. For instance, here is the declaration for the C function implementing the input primitive:

        value input(channel, buffer, offset, length)
                value channel, buffer, offset, length;
        {
         ...
        }

When the primitive function is applied in a Caml Light program, the C function is called with the values of the expressions to which the primitive is applied as arguments. The value returned by the function is passed back to the Caml Light program as the result of the function application.

User primitives with arity greater than 5 are implemented by C functions that receive two arguments: a pointer to an array of Caml Light values (the values for the arguments), and an integer which is the number of arguments provided:

        value prim_with_lots_of_args(argv, argn)
                value * argv;
                int argn;
        {
          ... argv[0] ...;              /* The first argument */
          ... argv[6] ...;              /* The seventh argument */
        }

Implementing a user primitive is actually two separate tasks: on the one hand, decoding the arguments to extract C values from the given Caml Light values, and encoding the return value as a Caml Light value; on the other hand, actually computing the result from the arguments. Except for very simple primitives, it is often preferable to have two distinct C functions to implement these two tasks. The first function actually implements the primitive, taking native C values as arguments and returning a native C value. The second function, often called the ``stub code'', is a simple wrapper around the first function that converts its arguments from Caml Light values to C values, call the first function, and convert the returned C value to Caml Light value. For instance, here is the stub code for the input primitive:

        value input(channel, buffer, offset, length)
                value channel, buffer, offset, length;
        {
          return Val_long(getblock((struct channel *) channel,
                                   &Byte(buffer, Long_val(offset)),
                                   Long_val(length)));
        }

        long getblock(channel, p, n)
             struct channel * channel;
             char * p;
             long n;
        {
          ...
        }

To write C code that operates on Caml Light values, the following include files are provided:

Include file	Provides
mlvalues.h	definition of the value type, and conversion macros
alloc.h	allocation functions (to create structured Caml Light objects)
memory.h	miscellaneous memory-related functions (for in-place modification of structures, etc).

These files reside in the Caml Light standard library directory (usually /usr/local/lib/caml-light).

Linking C code with Caml Light code

The Caml Light runtime system comprises three main parts: the bytecode interpreter, the memory manager, and a set of C functions that implement the primitive operations. Some bytecode instructions are provided to call these C functions, designated by their offset in a table of functions (the table of primitives).

In the default mode, the Caml Light linker produces bytecode for the standard runtime system, with a standard set of primitives. References to primitives that are not in this standard set result in the ``unavailable C primitive'' error.

In the ``custom runtime'' mode, the Caml Light linker scans the bytecode object files (.zo files) and determines the set of required primitives. Then, it builds a suitable runtime system, by calling the native code linker with:

• the table of the required primitives
• a library that provides the bytecode interpreter, the memory manager, and the standard primitives
• libraries and object code files (.o files) mentioned on the command line for the Caml Light linker, that provide implementations for the user's primitives.

To link in ``custom runtime'' mode, execute the camlc command with:

• the -custom option
• the names of the desired Caml Light object files (.zo files)
• the names of the C object files and libraries (.o and .a files) that implement the required primitives. (Libraries can also be specified with the usual -l syntax.)

The value type

All Caml Light objects are represented by the C type value, defined in the include file mlvalues.h, along with macros to manipulate values of that type. An object of type value is either:

• an unboxed integer
• a pointer to a block inside the heap (such as the blocks allocated through one of the alloc_* functions below)
• a pointer to an object outside the heap (e.g., a pointer to a block allocated by malloc, or to a C variable).

Integer values

Integer values encode 31-bit signed integers. They are unboxed (unallocated).

Blocks

Blocks in the heap are garbage-collected, and therefore have strict structure constraints. Each block includes a header containing the size of the block (in words), and the tag of the block. The tag governs how the contents of the blocks are structured. A tag lower than No_scan_tag indicates a structured block, containing well-formed values, which is recursively traversed by the garbage collector. A tag greater than or equal to No_scan_tag indicates a raw block, whose contents are not scanned by the garbage collector. For the benefits of ad-hoc polymorphic primitives such as equality and structured input-output, structured and raw blocks are further classified according to their tags as follows:

Tag	Contents of the block
0 to No_scan_tag-1	A structured block (an array of Caml Light objects). Each field is a value.
Closure_tag	A closure representing a functional value. The first word is a pointer to a piece of bytecode, the second word is a value containing the environment.
String_tag	A character string.
Double_tag	A double-precision floating-point number.
Abstract_tag	A block representing an abstract datatype.
Final_tag	A block representing an abstract datatype with a ``finalization'' function, to be called when the block is deallocated.

Pointers to outside the heap

Any pointer to outside the heap can be safely cast to and from the type value. This includes pointers returned by malloc, and pointers to C variables obtained with the & operator.

Representation of Caml Light data types

This section describes how Caml Light data types are encoded in the value type.

Atomic types

Caml type	Encoding
int	Unboxed integer values.
char	Unboxed integer values (ASCII code).
float	Blocks with tag Double_tag.
string	Blocks with tag String_tag.

Product types

Tuples and arrays are represented by pointers to blocks, with tag 0.

Records are also represented by zero-tagged blocks. The ordering of labels in the record type declaration determines the layout of the record fields: the value associated to the label declared first is stored in field 0 of the block, the value associated to the label declared next goes in field 1, and so on.

Concrete types

Constructed terms are represented by blocks whose tag encode the constructor. The constructors for a given concrete type are numbered from 0 to the number of constructors minus one, following the order in which they appear in the concrete type declaration. Constant constructors are represented by zero-sized blocks (atoms), tagged with the constructor number. Non-constant constructors declared with a n-tuple as argument are represented by a block of size n, tagged with the constructor number; the n fields contain the components of its tuple argument. Other non-constant constructors are represented by a block of size 1, tagged with the constructor number; the field 0 contains the value of the constructor argument. Example:

Constructed term	Representation
()	Size = 0, tag = 0
false	Size = 0, tag = 0
true	Size = 0, tag = 1
[]	Size = 0, tag = 0
h::t	Size = 2, tag = 1, first field = h, second field = t

Operations on values

Kind tests

• Is_int(v) is true if value v is an immediate integer, false otherwise
• Is_block(v) is true if value v is a pointer to a block, and false if it is an immediate integer.

Operations on integers

• Val_long(l) returns the value encoding the long int l
• Long_val(v) returns the long int encoded in value v
• Val_int(i) returns the value encoding the int i
• Int_val(v) returns the int encoded in value v

Accessing blocks

• Wosize_val(v) returns the size of value v, in words, excluding the header.
• Tag_val(v) returns the tag of value v.
• Field(v,n) returns the value contained in the n-th field of the structured block v. Fields are numbered from 0 to Wosize_val(v)-1.
• Code_val(v) returns the code part of the closure v.
• Env_val(v) returns the environment part of the closure v.
• string_length(v) returns the length (number of characters) of the string v.
• Byte(v,n) returns the n-th character of the string v, with type char. Characters are numbered from 0 to string_length(v)-1.
• Byte_u(v,n) returns the n-th character of the string v, with type unsigned char. Characters are numbered from 0 to string_length(v)-1.
• String_val(v) returns a pointer to the first byte of the string v, with type char *. This pointer is a valid C string: there is a null character after the last character in the string. However, Caml Light strings can contain embedded null characters, that will confuse the usual C functions over strings.
• Double_val(v) returns the floating-point number contained in value v, with type double.

Allocating blocks

From the standpoint of the allocation functions, blocks are divided according to their size as zero-sized blocks, small blocks (with size less than or equal to Max_young_wosize), and large blocks (with size greater than to Max_young_wosize). The constant Max_young_wosize is declared in the include file mlvalues.h. It is guaranteed to be at least 64 (words), so that any block with constant size less than or equal to 64 can be assumed to be small. For blocks whose size is computed at run-time, the size must be compared against Max_young_wosize to determine the correct allocation procedure.

• Atom(t) returns an ``atom'' (zero-sized block) with tag t. Zero-sized blocks are preallocated outside of the heap. It is incorrect to try and allocate a zero-sized block using the functions below. For instance, Atom(0) represents (), false and []; Atom(1) represents true. (As a convenience, mlvalues.h defines the macros Val_unit, Val_false and Val_true.)
• alloc(n,t) returns a fresh small block of size n <= Max_young_wosize words, with tag t. If this block is a structured block (i.e. if t < No_scan_tag), then the fields of the block (initially containing garbage) must be initialized with legal values (using direct assignment to the fields of the block) before the next allocation.
• alloc_tuple(n) returns a fresh small block of size n <= Max_young_wosize words, with tag 0. The fields of this block must be filled with legal values before the next allocation or modification.
• alloc_shr(n,t) returns a fresh block of size n, with tag t. The size of the block can be greater than Max_young_wosize. (It can also be smaller, but in this case it is more efficient to call alloc instead of alloc_shr.) If this block is a structured block (i.e. if t < No_scan_tag), then the fields of the block (initially containing garbage) must be initialized with legal values (using the initialize function described below) before the next allocation.
• alloc_string(n) returns a string value of length n characters. The string initially contains garbage.
• copy_string(s) returns a string value containing a copy of the null-terminated C string s (a char *).
• copy_double(d) returns a floating-point value initialized with the double d.
• alloc_array(f,a) allocates an array of values, calling function f over each element of the input array a to transform it into a value. The array a is an array of pointers terminated by the null pointer. The function f receives each pointer as argument, and returns a value. The zero-tagged block returned by alloc_array(f,a) is filled with the values returned by the successive calls to f.
• copy_string_array(p) allocates an array of strings, copied from the pointer to a string array p (a char **).

Raising exceptions

C functions cannot raise arbitrary exceptions. However, two functions are provided to raise two standard exceptions:

• failwith(s), where s is a null-terminated C string (with type char *), raises exception Failure with argument s.
• invalid_argument(s), where s is a null-terminated C string (with type char *), raises exception Invalid_argument with argument s.

Living in harmony with the garbage collector

Unused blocks in the heap are automatically reclaimed by the garbage collector. This requires some cooperation from C code that manipulates heap-allocated blocks.

Rule:

After a structured block (a block with tag less than No_scan_tag) is allocated, all fields of this block must be filled with well-formed values before the next allocation operation. If the block has been allocated with alloc or alloc_tuple, filling is performed by direct assignment to the fields of the block:

        Field(v, n) = vn;

If the block has been allocated with alloc_shr, filling is performed through the initialize function:

        initialize(&Field(v, n), vn);

The next allocation can trigger a garbage collection. The garbage collector assumes that all structured blocks contain well-formed values. Newly created blocks contain random data, which generally do not represent well-formed values.

If you really need to allocate before the fields can receive their final value, first initialize with a constant value (e.g. Val_long(0)), then allocate, then modify the fields with the correct value (see rule 3).

Rule:

Local variables containing values must be registered with the garbage collector (using the Push_roots and Pop_roots macros), if they are to survive a call to an allocation function.

Registration is performed with the Push_roots and Pop_roots macros. Push_roots(r,n) declares an array r of n values and registers them with the garbage collector. The values contained in r[0] to r[n-1] are treated like roots by the garbage collector. A root value has the following properties: if it points to a heap-allocated block, this block (and its contents) will not be reclaimed; moreover, if this block is relocated by the garbage collector, the root value is updated to point to the new location for the block. Push_roots(r,n) must occur in a C block exactly between the last local variable declaration and the first statement in the block. To un-register the roots, Pop_roots() must be called before the C block containing Push_roots(r,n) is exited. (Roots are automatically un-registered if a Caml exception is raised.)

Rule:

Direct assignment to a field of a block, as in

        Field(v, n) = w;

is safe only if v is a block newly allocated by alloc or alloc_tuple; that is, if no allocation took place between the allocation of v and the assignment to the field. In all other cases, never assign directly. If the block has just been allocated by alloc_shr, use initialize to assign a value to a field for the first time:

        initialize(&Field(v, n), w);

Otherwise, you are updating a field that previously contained a well-formed value; then, call the modify function:

        modify(&Field(v, n), w);

To illustrate the rules above, here is a C function that builds and returns a list containing the two integers given as parameters:

value alloc_list_int(i1, i2)
        int i1, i2;
{
  value result;
  Push_roots(r, 1);
  r[0] = alloc(2, 1);                     /* Allocate a cons cell */
  Field(r[0], 0) = Val_int(i2);           /* car = the integer i2 */
  Field(r[0], 1) = Atom(0);               /* cdr = the empty list [] */
  result = alloc(2, 1);                   /* Allocate the other cons cell */
  Field(result, 0) = Val_int(i1);         /* car = the integer i1 */
  Field(result, 1) = r[0];                /* cdr = the first cons cell */
  Pop_roots();
  return result;
}

In the example above, the list is built bottom-up. Here is an alternate way, that proceeds top-down. It is less efficient, but illustrates the use of modify.

value alloc_list_int(i1, i2)
        int i1, i2;
{
  value tail;
  Push_roots(r, 1);
  r[0] = alloc(2, 1);                     /* Allocate a cons cell */
  Field(r[0], 0) = Val_int(i1);           /* car = the integer i1 */
  Field(r[0], 1) = Val_int(0);            /* A dummy value
  tail = alloc(2, 1);                     /* Allocate the other cons cell */
  Field(tail, 0) = Val_int(i2);           /* car = the integer i2 */
  Field(tail, 1) = Atom(0);               /* cdr = the empty list [] */
  modify(&Field(r[0], 1), tail);          /* cdr of the result = tail */
  Pop_roots();
  return r[0];
}

A complete example

This section outlines how the functions from the Unix curses library can be made available to Caml Light programs. First of all, here is the interface curses.mli that declares the curses primitives and data types:

type window;;                   (* The type "window" remains abstract *)
value initscr: unit -> window = 1 "curses_initscr"
  and endwin: unit -> unit = 1 "curses_endwin"
  and refresh: unit -> unit = 1 "curses_refresh"
  and wrefresh : window -> unit = 1 "curses_wrefresh"
  and newwin: int -> int -> int -> int -> window = 4 "curses_newwin"
  and mvwin: window -> int -> int -> unit = 3 "curses_mvwin"
  and addch: char -> unit = 1 "curses_addch"
  and mvwaddch: window -> int -> int -> char -> unit = 4 "curses_mvwaddch"
  and addstr: string -> unit = 1 "curses_addstr"
  and mvwaddstr: window -> int -> int -> string -> unit = 4 "curses_mvwaddstr"
;; (* lots more omitted *)

        camlc -c curses.mli

To implement these functions, we just have to provide the stub code; the core functions are already implemented in the curses library. The stub code file, curses.o, looks like:

#include <curses.h>
#include <mlvalues.h>

value curses_initscr(unit)
        value unit;
{
  return (value) initscr();     /* OK to coerce directly from WINDOW * to value
                                   since that's a block created by malloc() */
}

value curses_wrefresh(win)
        value win;
{
  wrefresh((WINDOW *) win);
  return Val_unit;
}

value curses_newwin(nlines, ncols, x0, y0)
        value nlines, ncols, x0, y0;
{
  return (value) newwin(Int_val(nlines), Int_val(ncols),
                        Int_val(x0), Int_val(y0));
}

value curses_addch(c)
        value c;
{
  addch(Int_val(c));            /* Characters are encoded like integers */
  return Val_unit;
}

value curses_addstr(s)
        value s;
{
  addstr(String_val(s));
  return Val_unit;
}

/* This goes on for pages. */

The file curses.c can be compiled with:

        cc -c -I/usr/local/lib/caml-light curses.c

        camlc -c curses.c

Now, here is a sample Caml Light program test.ml that uses the curses module:

#open "curses";;
let main_window = initscr () in
let small_window = newwin 10 5 20 10 in
  mvwaddstr main_window 10 2 "Hello";
  mvwaddstr small_window 4 3 "world";
  refresh();
  for i = 1 to 100000 do () done;
  endwin()
;;

        camlc -c test.ml

        camlc -custom -o test test.zo curses.o -lcurses