Original bug ID: 5008
Reporter: Antoine Mine
Assigned to: @xavierleroy
Status: closed (set by @xavierleroy on 2011-05-29T10:19:59Z)
Resolution: fixed
Severity: crash
Version: 3.11.2
Fixed in version: 3.12.0+dev
Category: ~DO NOT USE (was: OCaml general)
Monitored by: @alainfrisch

Bug description

Hi,

1. Symptoms

On ocaml-3.11.2 (also verified on ocaml-3.11.1) compiled with Visual Studio 9.0 (i.e., 2008) and Windows SDK 6.0A on Windows 7 in 64-bit mode.

We define a C function with "float" calling convention:

double ml_single_precision_add_float(double a, double b)
{
return (float) (a + b);
}

and call:

external add : t->t->t = "ml_single_precision_add" "ml_single_precision_add_float" "float"

let x = add 1. 2. in ...

within a large OCaml program whose source I cannot disclose, unfortunately.

Then, non-deterministically (approx. 2 out of 10 runs), x will get a value which is not 3. (checked with Printf.printf), although the C function gets the right arguments and computes the right result (checked with printf).
I was not able to construct a small program exhibiting the same behavior, unfortunately.

2. Tentative explanation

I am not sure I understand fully what happens, but here is a guess.
After add returns successfully its result in XMM0, caml_call_gc is called.
It saves XMM0 in [rsp] and calls caml_garbage_collection, a C function without any argument.
On Win64, I was led to believe that a function must reserve 32 bytes on its stack, even a void function, as the space might be used by the callee as a scratchpad.
I guess some C code in caml_garbage_collection may smash the saved XMM0 in [rsp].

3. Tentative fix

In amd64nt.asm, caml_call_gc, change:

    call    caml_garbage_collection

into:

sub     rsp, 32
    call    caml_garbage_collection
add     rsp, 32

On my program, this seems to fix the issue.
Again, I do not claim to fully understand this.

I've also heard that the stack should be kept aligned on 16-bytes.
Here, rsp is decreased by an odd number of qwords.
I don't know whether this might cause a problem.

Best regards,

Antoine