0005008: Stack smash(?) on Win64 causes C "float" function to return random results

View Issue Details [ Jump to Notes ]

[ Issue History ] [ Print ]

Project

Category

View Status

Date Submitted

Last Update

0005008

OCaml

OCaml general

public

2010-03-24 16:52

2011-05-29 12:19

Reporter

Antoine Mine

Assigned To

xleroy

Priority

none

Severity

crash

Reproducibility

random

Status

closed

Resolution

fixed

Platform

OS Version

Product Version

3.11.2

Target Version

Fixed in Version

3.12.0+dev

Summary

0005008: Stack smash(?) on Win64 causes C "float" function to return random results

Description

Hi,

1) Symptoms

On ocaml-3.11.2 (also verified on ocaml-3.11.1) compiled with Visual Studio 9.0 (i.e., 2008) and Windows SDK 6.0A on Windows 7 in 64-bit mode.

We define a C function with "float" calling convention:

double ml_single_precision_add_float(double a, double b)
{
  return (float) (a + b);
}

and call:

external add : t->t->t = "ml_single_precision_add" "ml_single_precision_add_float" "float"

let x = add 1. 2. in ...

within a large OCaml program whose source I cannot disclose, unfortunately.

Then, non-deterministically (approx. 2 out of 10 runs), x will get a value which is not 3. (checked with Printf.printf), although the C function gets the right arguments and computes the right result (checked with printf).
I was not able to construct a small program exhibiting the same behavior, unfortunately.

2) Tentative explanation

I am not sure I understand fully what happens, but here is a guess.
After add returns successfully its result in XMM0, caml_call_gc is called.
It saves XMM0 in [rsp] and calls caml_garbage_collection, a C function without any argument.
On Win64, I was led to believe that a function must reserve 32 bytes on its stack, even a void function, as the space might be used by the callee as a scratchpad.
I guess some C code in caml_garbage_collection may smash the saved XMM0 in [rsp].

3) Tentative fix

In amd64nt.asm, caml_call_gc, change:

        call caml_garbage_collection

into:

    sub rsp, 32
        call caml_garbage_collection
    add rsp, 32

On my program, this seems to fix the issue.
Again, I do not claim to fully understand this.

I've also heard that the stack should be kept aligned on 16-bytes.
Here, rsp is decreased by an odd number of qwords.
I don't know whether this might cause a problem.

Best regards,
-
Antoine

Tags

No tags attached.

Attached Files

Relationships

Notes
(0005273) xleroy (administrator) 2010-03-28 10:09	Hello Antoine, Thanks for the detective work. You are correct that the bottom 32 bytes of stack are scratch space in the Win64 calling conventions. ocamlopt-generated code as well as other functions in amd64nt.asm carefully reserve this space, but caml_call_gc did not. I've applied your fix; it will go in release 3.12.0.

Issue History
Date Modified	Username	Field	Change
2010-03-24 16:52	Antoine Mine	New Issue
2010-03-28 10:09	xleroy	Note Added: 0005273
2010-03-28 10:09	xleroy	Assigned To	=> xleroy
2010-03-28 10:09	xleroy	Priority	normal => none
2010-03-28 10:09	xleroy	Status	new => resolved
2010-03-28 10:09	xleroy	Resolution	open => fixed
2010-03-28 10:09	xleroy	Fixed in Version	=> 3.12.0+dev
2011-05-29 12:19	xleroy	Status	resolved => closed