Woo hoo.  Language advocacy with benchmarks again.

Feel free to replace this whole post with a comment about "lies, damned 
lies, and cross language benchmarks".  It amounts to the same thing.

On Fri, 3 Jan 2003, Xavier Leroy wrote:

> > Is it normal that my ocaml program is only 2 times faster than the java 
> > counterpart ?(using the same method and complied into native. jdk is 1.4.1
> 
> You know, many compiler researchers would kill their whole families to
> get speedups by a factor of 2 :-)
> 
> James Gosling gave a talk at INRIA recently where he repeated the
> party line that JDK 1.4 runs as fast, or even faster, than C++.

Quibble #1: *what* C++?  Most of the time, when I see C++ benchmarked,
what's really being benchmarked is C compiled with a C++ compiler, or at
most C with classes.  My experience with C++ tells me that if you actually
use the features of C++- RTTI, templates, STL, exceptions, operator
overloading, etc- the code you produce is often much *slower* than Java.  
With a language as feature rich/bloated as C++, which subset of the
language you use makes a huge difference in your performance.  Ocaml has
the same problem in a lot of ways.

Quibble #2: define "equivelent program".

> So, by transitivity, you're implying that OCaml is twice as fast as C++.
> Yippee! 
> 
> More seriously: Java is nowhere as fast as a good C++ compiler (see
> e.g. http://www.coyotegulch.com/reviews/almabench.html for an
> independent, cross-language benchmark in numerical computing),

I note the coyote gulch benchmark shows IBM's Java to be more-or-less on
par with GCC 3.2.  I note, btw, that GCC 3.2 is signifigantly better at
optimization than GCC 2.9x, producing code about 10% faster on average
IIRC according to the GCC maintainers themselves.  Which tells me that
IBM's Java *is* better than GCC 2.9x.  Which is still the most commonly
used compiler on Linux systems.  Ditto for Windows.  My own experience and
tests show me that MS VC++ 6.0 is no better than, and in many cases worse
than, GCC 2.9x for optimization.

> but it's not that slow either.  A factor of 2 slower than ocamlopt
> sounds broadly reasonable, especially if the program doesn't stress
> the GC too much.  Bagley's shootout (http://www.bagley.org/~doug/shootout/)
> seems to suggest a larger factor (JDK 1.3 slightly slower than OCaml
> bytecode), but his figures may be lowered by Java's slow start-up times.

Startup costs dominate in Bagley's shootout.  Look at matrix 
multiplication- the fastest tests (C, C++, and Ocaml) are running in 
70-110 milliseconds.  Most timers are accurate only to ~10 milliseconds, 
which means the time for the C program to run could be anything from 
600 millisecond to 800 milliseconds, for an error of +/-14.3%.

Java has huge start up costs.  First off, you have the JIT.  Then, there 
is a time delay before hotspot kicks in an actually starts optimizing the 
code to any signifigant extent.  Notice that the pro-Java benchmarks run 
the code to be benchmarked a few thousands or tens of thousands of times 
before starting the timer, so that the hotspot optimizer has already been 
over the code a couple of times.  Or at least once, to bypass JIT time.   
Is this a legitimate tactic?  Lies, damned lies, and cross-language 
benchmarks.

Note that I can also claim, with a straight face, that Ocaml is 5x 
*slower* than Java.  Take a look at Bagley's shootout on matrix 
multiplication, comparing byte-code interpreted Java with byte-code 
intepreted Ocaml.  Which is a much more apples to apples comparison.

Then there is the question of *future* performance of the languages.  In 
the pro-Java camp, I direct your attention to HP's Dynamo project:
http://www.arstechnica.com/reviews/1q00/dynamo/dynamo-1.html
http://www.hpl.hp.com/cambridge/projects/Dynamo/
which showed that a virtual PA-RISC emulator could run the code up to 20%
faster than running the same code native.  In the pro-Ocaml camp, Caml's
innate ease of reasoning about code open up, I think, a much larger array
of potiental optimizations for the compiler.

Of course, Java, Ocaml, and C++ all pale in comparison to the performance 
of hand-tuned assembly language.  Ergo, anyone who is using performance of 
the generated code as the primary reason for picking a language should, by 
all logic, be coding in assembly language.

Note that I, personally, think that performance should be the last reason 
used to pick a language.  Things like correctness of the code, available 
libraries and environments, and existing talents and skills of the 
workforce, should instead take precedence.

Brian


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners