Hi,

> if the lexer cannot decide it on the tokens seen, a packrat
> parser (like Aurochs) may be a better choice, since in a PEG
> there is no seperate lexer, it's all one grammar, so you don't
> have this problem.

But does Aurochs also handle UTF8 streams?

In the meantime I've implemented the parser using Ulex/Menhir
with the "dummy action" trick I mentioned before.  It allowed
me to simplify the tokenizer tremendously, though it's still
present:

https://forge.ocamlcore.org/plugins/scmsvn/viewcvs.php/trunk/lambdoc/src/lib/lambdoc_read_lambtex/?root=lambdoc

Cheers,
Dario Teixeira