]> Woo hoo. Language advocacy with benchmarks again. Feel free to replace this whole post with a comment about "lies, damned lies, and cross language benchmarks". It amounts to the same thing. On Fri, 3 Jan 2003, Xavier Leroy wrote: > > Is it normal that my ocaml program is only 2 times faster than the java > > counterpart ?(using the same method and complied into native. jdk is 1.4.1 > > You know, many compiler researchers would kill their whole families to > get speedups by a factor of 2 :-) > > James Gosling gave a talk at INRIA recently where he repeated the > party line that JDK 1.4 runs as fast, or even faster, than C++. Quibble #1: *what* C++? Most of the time, when I see C++ benchmarked, what's really being benchmarked is C compiled with a C++ compiler, or at most C with classes. My experience with C++ tells me that if you actually use the features of C++- RTTI, templates, STL, exceptions, operator overloading, etc- the code you produce is often much *slower* than Java. With a language as feature rich/bloated as C++, which subset of the language you use makes a huge difference in your performance. Ocaml has the same problem in a lot of ways. Quibble #2: define "equivelent program". > So, by transitivity, you're implying that OCaml is twice as fast as C++. > Yippee! > > More seriously: Java is nowhere as fast as a good C++ compiler (see > e.g. http://www.coyotegulch.com/reviews/almabench.html for an > independent, cross-language benchmark in numerical computing), I note the coyote gulch benchmark shows IBM's Java to be more-or-less on par with GCC 3.2. I note, btw, that GCC 3.2 is signifigantly better at optimization than GCC 2.9x, producing code about 10% faster on average IIRC according to the GCC maintainers themselves. Which tells me that IBM's Java *is* better than GCC 2.9x. Which is still the most commonly used compiler on Linux systems. Ditto for Windows. My own experience and tests show me that MS VC++ 6.0 is no better than, and in many cases worse than, GCC 2.9x for optimization. > but it's not that slow either. A factor of 2 slower than ocamlopt > sounds broadly reasonable, especially if the program doesn't stress > the GC too much. Bagley's shootout (http://www.bagley.org/~doug/shootout/) > seems to suggest a larger factor (JDK 1.3 slightly slower than OCaml > bytecode), but his figures may be lowered by Java's slow start-up times. Startup costs dominate in Bagley's shootout. Look at matrix multiplication- the fastest tests (C, C++, and Ocaml) are running in 70-110 milliseconds. Most timers are accurate only to ~10 milliseconds, which means the time for the C program to run could be anything from 600 millisecond to 800 milliseconds, for an error of +/-14.3%. Java has huge start up costs. First off, you have the JIT. Then, there is a time delay before hotspot kicks in an actually starts optimizing the code to any signifigant extent. Notice that the pro-Java benchmarks run the code to be benchmarked a few thousands or tens of thousands of times before starting the timer, so that the hotspot optimizer has already been over the code a couple of times. Or at least once, to bypass JIT time. Is this a legitimate tactic? Lies, damned lies, and cross-language benchmarks. Note that I can also claim, with a straight face, that Ocaml is 5x *slower* than Java. Take a look at Bagley's shootout on matrix multiplication, comparing byte-code interpreted Java with byte-code intepreted Ocaml. Which is a much more apples to apples comparison. Then there is the question of *future* performance of the languages. In the pro-Java camp, I direct your attention to HP's Dynamo project: http://www.arstechnica.com/reviews/1q00/dynamo/dynamo-1.html http://www.hpl.hp.com/cambridge/projects/Dynamo/ which showed that a virtual PA-RISC emulator could run the code up to 20% faster than running the same code native. In the pro-Ocaml camp, Caml's innate ease of reasoning about code open up, I think, a much larger array of potiental optimizations for the compiler. Of course, Java, Ocaml, and C++ all pale in comparison to the performance of hand-tuned assembly language. Ergo, anyone who is using performance of the generated code as the primary reason for picking a language should, by all logic, be coding in assembly language. Note that I, personally, think that performance should be the last reason used to pick a language. Things like correctness of the code, available libraries and environments, and existing talents and skills of the workforce, should instead take precedence. Brian ------------------- To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/ Beginner's list: http://groups.yahoo.com/group/ocaml_beginners