I am interested in building a dynamically extensible lexer.. can
anyone recommend any libraries or tools that would help?

I could use Dypgen (extensible GLR parser) for this, but it is 
probably too slow.

A system using combinators (rather than strings) would be preferred,
and Unicode support would be nice too. Ideally the lexer automaton
could be saved with Marshal, in which case the time to build the
automaton isn't important.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net