Comment author: @mlasson

Hello,

I was able to reproduce this issue on linux 32bits (full disclosure : on ubuntu 14.04 with ocaml 4.01 inside a virtual machine executed on a 64 bits host).

Also, I tried to reproduce the performance issue outside of caml with the following piece of assembly code :

.extern cos
.section .data
pi: .double 3.14
.section .text
.global slowcos
.global fastcos

slowcos:
pushl pi+4
pushl pi
call cos
addl $8, %esp
ret

fastcos:
fldl pi
subl $8, %esp
fstpl 0(%esp)
call cos
addl $8, %esp
ret

These two functions compute "cos(3.14)" one is copying "3.14" on the stack using two pushes (the slow one) and the other using one float load and one float pop on the float stack (the fast one).

I have a no idea why this matters that much during the computation of cos.

I join to this note a file "cos.asm" and a file "test.c". You can compile it on 64bits system with (drop --32 and -m32 to compile it on a 32bits system):
as --32 cos.asm -o cos.o
gcc -m32 -O0 cos.o test.c -lm -o test

For reference, I've also replaced the calls to cos with a call to an identity function. Here is the output on my computer (for 40 millions iterations):

1.62459s
slowcos = -0.999999
1.06661s
fastcos = -0.999999
0.254804s
fastpi = 3.14
0.265336s
slowpi = 3.14

Comment author: @alainfrisch

A discussion on this topic:

http://stackoverflow.com/questions/32445429/why-one-of-these-functions-runs-almost-two-times-slower-than-the-other/32455660

Comment author: @mlasson

I'm quite convinced by the explanation provided by EOF, one should not push double on the stack by cutting them in half.

The following patch replaces in "i386/emit.mlp" the lines that push double on the stack by x87 instructions.

It relies on the assumption that there is always one free slot on the float stack (is it an invariant maintained by the compiler ?).

It solves the problem, but I hope it does not create others.

Index: asmcomp/i386/emit.mlp

--- asmcomp/i386/emit.mlp
+++ asmcomp/i386/emit.mlp
@@ -744,9 +744,10 @@
stack_offset := !stack_offset + 8
| {loc = Stack sl; typ = Float} ->
let ofs = slot_offset sl 1 in

•        I.push (mem32 DWORD (ofs + 4) RSP);
  

•        I.push (mem32 DWORD (ofs + 4) RSP);
  

•        I.fld (mem32 REAL8 ofs RSP);
  

•        I.sub (int 8) esp;
         cfi_adjust_cfa_offset 8;
  

•        I.fstp (mem32 REAL8 0 RSP);
         stack_offset := !stack_offset + 8
     | _ ->
         I.push (reg r);
  

Comment author: @mlasson

I have two more remarks to add.

1. it seems that gcc is almost always using the pair fldl/fstpl to push doubles on the stack.

For instance here is the assembly produces by "gcc -S" to comple the function "double f() { double x = id(3.14), y = id(x); return x + y; }".
f:
.cfi_startproc
pushl %ebp
.cfi_def_cfa_offset 8
.cfi_offset 5, -8
movl %esp, %ebp
.cfi_def_cfa_register 5

    subl    $24, %esp
    fldl    .LC1
    fstpl   (%esp)
    call    id
    fstpl   -16(%ebp)
    fldl    -16(%ebp)
    fstpl   (%esp)
    call    id
    fstpl   -8(%ebp)
    fldl    -16(%ebp)
    faddl   -8(%ebp)
    leave

2. To confirm that this phenomenon is not specific to special math-functions (like cos and exp), the assembly code below (warp in test.c) ouputs :

0.365847s
fastpi = 28.26
1.24235s
slowpi = 28.26

It computes 3.14 + 3.14 + 3.14 + 3.14 + 3.14 + 3.14 + 3.14 + 3.14 + 3.14.
Both version uses the same instructions, but one first copies using "movs" then copies with "fstpl" while the other do the "fstpl" first and the mov after. Thus the first one is more "cache-friendly".

.section .data
pi: .double 3.14
.section .text
.global slowpi
.global fastpi

fastpi:
fldl pi
subl $8, %esp

movl pi, %eax
movl %eax, 0(%esp)
movl pi+4, %eax
movl %eax, 4(%esp)

fstpl 0(%esp)

fldl 0(%esp)

faddl 0(%esp)
faddl 0(%esp)
faddl 0(%esp)
faddl 0(%esp)
faddl 0(%esp)
faddl 0(%esp)
faddl 0(%esp)
faddl 0(%esp)

addl $8, %esp
ret

slowpi:
fldl pi
subl $8, %esp

fstpl 0(%esp)

movl pi, %eax
movl %eax, 0(%esp)
movl pi+4, %eax
movl %eax, 4(%esp)

fldl 0(%esp)

faddl 0(%esp)
faddl 0(%esp)
faddl 0(%esp)
faddl 0(%esp)
faddl 0(%esp)
faddl 0(%esp)
faddl 0(%esp)
faddl 0(%esp)

addl $8, %esp
ret

Comment author: @xavierleroy

Thanks for the detective work.

The two-pushl idiom was what GCC used to generate a while ago. It made sense at that time because fldl/fstpl was reputed to be slow, owing perhaps to the 64 bits -> 80 bits -> 64 bits conversions involved. Also, store forwarding was probably less aggressive on the x86 processors of that time.

I see that today GCC uses the fldl/fstpl sequence, so I'm fine with using it as well in the x86-32 back-end of OCaml.

Comment author: @alainfrisch

Thanks Xavier! Do you think Marc's patch (in a note below) is valid, i.e. that one can safely assume the existence of a free slot on the x87 pseudo stack when the move happens?

Comment author: @xavierleroy

No need to worry about FP stack overflow. The x87 stack is used only during the evaluation of pure subexpressions, and is empty otherwise, in particular when we push the arguments to a C function call. Besides, owing to the use of Ershov's algorithm in ocamlopt, you'd need a really huge FP expression to make use of all 8 slots of the FP stack. So, you can go ahead and merge Marc's patch.

Comment author: @alainfrisch

Thanks Marc and Xavier! Suggested fix committed to trunk, rev 16414.