On Sat, 2007-04-28 at 01:12 +0200, Oliver Bandel wrote:

> So, I then checked my mboxlib and saw that it is quite slow,
> compared to what I expected ( expect! I did not tried it
> on my development machine because I have nomutt installed there)
> and even if native-code smuch faster, it's nevertheless slow...
> ...so I thought I have to redesign my scanner-stage.
> (I use Str-module and ocamnllex mixed together; maybe
>  using a plain selfwritten  OCaml-scanner might be better here).

Ocamllex generates very fast scanner: it is using
a very high-tech tagged deterministic finite state automaton
with a driver written in C (so no boxing etc processing
text buffers). I doubt you can hand code anything as
fast as Ocamllex in C, let alone in Ocaml.

You should check the size (number of states) of the generated
lexer. It will run faster with small number of states where
the matrix fits easily in the cache.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net