Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0005008OCamlOCaml generalpublic2010-03-24 16:522011-05-29 12:19
ReporterAntoine Mine 
Assigned Toxleroy 
PrioritynoneSeveritycrashReproducibilityrandom
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version3.11.2 
Target VersionFixed in Version3.12.0+dev 
Summary0005008: Stack smash(?) on Win64 causes C "float" function to return random results
DescriptionHi,

1) Symptoms

On ocaml-3.11.2 (also verified on ocaml-3.11.1) compiled with Visual Studio 9.0 (i.e., 2008) and Windows SDK 6.0A on Windows 7 in 64-bit mode.

We define a C function with "float" calling convention:

double ml_single_precision_add_float(double a, double b)
{
  return (float) (a + b);
}

and call:

external add : t->t->t = "ml_single_precision_add" "ml_single_precision_add_float" "float"

let x = add 1. 2. in ...

within a large OCaml program whose source I cannot disclose, unfortunately.

Then, non-deterministically (approx. 2 out of 10 runs), x will get a value which is not 3. (checked with Printf.printf), although the C function gets the right arguments and computes the right result (checked with printf).
I was not able to construct a small program exhibiting the same behavior, unfortunately.

2) Tentative explanation

I am not sure I understand fully what happens, but here is a guess.
After add returns successfully its result in XMM0, caml_call_gc is called.
It saves XMM0 in [rsp] and calls caml_garbage_collection, a C function without any argument.
On Win64, I was led to believe that a function must reserve 32 bytes on its stack, even a void function, as the space might be used by the callee as a scratchpad.
I guess some C code in caml_garbage_collection may smash the saved XMM0 in [rsp].

3) Tentative fix

In amd64nt.asm, caml_call_gc, change:

        call caml_garbage_collection

into:

    sub rsp, 32
        call caml_garbage_collection
    add rsp, 32

On my program, this seems to fix the issue.
Again, I do not claim to fully understand this.

I've also heard that the stack should be kept aligned on 16-bytes.
Here, rsp is decreased by an odd number of qwords.
I don't know whether this might cause a problem.

Best regards,
-
Antoine
TagsNo tags attached.
Attached Files

- Relationships

-  Notes
(0005273)
xleroy (administrator)
2010-03-28 10:09

Hello Antoine,

Thanks for the detective work. You are correct that the bottom 32 bytes of stack are scratch space in the Win64 calling conventions. ocamlopt-generated code as well as other functions in amd64nt.asm carefully reserve this space, but caml_call_gc did not. I've applied your fix; it will go in release 3.12.0.

- Issue History
Date Modified Username Field Change
2010-03-24 16:52 Antoine Mine New Issue
2010-03-28 10:09 xleroy Note Added: 0005273
2010-03-28 10:09 xleroy Assigned To => xleroy
2010-03-28 10:09 xleroy Priority normal => none
2010-03-28 10:09 xleroy Status new => resolved
2010-03-28 10:09 xleroy Resolution open => fixed
2010-03-28 10:09 xleroy Fixed in Version => 3.12.0+dev
2011-05-29 12:19 xleroy Status resolved => closed


Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker