On Fri, 2007-02-09 at 11:32 +0100,
ls-ocaml-developer-2006@m-e-leypold.de wrote:
> ls-ocaml-developer-2006@m-e-leypold.de writes:
> 
> > optimizations. I'd feel better if the code is benchmarked in a way
> > that the result of the multiplication is output to a file and to
> > subtract the constant contribution of that to the run time that the
> > time is measured for various problem sizes (number of matrices). 
> 
> Correction: ... the constant contribution of writing the result to a
> file should be subtracted from the run time by measuring with various
> problem sizes.
> 
> Hope that is understandable now.

There is no need for that IMHO. The right way to benchmark simple
things like matrix operations AND simultaneously validate them
is to evaluate known laws of arithmetic, such as the associative
law, on various samples.

This doesn't require any I/O, nor a 'known correct' program
to generate the comparison data.

Generating the samples is harder .. I once used things like
Fibonacci to generate input for integer calculator tests.
You really want to run these benchmarks for at least 5 minutes
and randomly to eliminate cache effects.

If you can generate increasing sized problems based on a single
linear integer parameter, my Python test harness can handle
randomising the tests, run multiple processes, 
and also draw a nice plot of the results, such as these:

http://felix.sourceforge.net/speed/en_flx_perf_0005.html
http://felix.sourceforge.net/speed/en_flx_perf_0012.html

full sources etc are here to read:

http://felix.sourceforge.net/speed/en_flx_perf_top.html

or you can grab from Felix svn archive (the test harness
is independent of Felix). Or I can email to you (saves
building Felix just to get the test harness ..)

Ocamlopt scores quite well thank you! This data is from
two AMD64 machines running Ubuntu Linux and is the result
of HOURS of run time.

I don't know if it is cache, random context switches, or what,
but on my boxes variations up to 20% on tests under 5 seconds are
normal. Note the measurements are real time. Other measurements
are suspect .. you really SHOULD count VM paging for example.


-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net