Version française
Home     About     Download     Resources     Contact us    
Browse thread
The lexer hack
[ Home ] [ Index: by date | by threads ]
[ Search: ]

[ Message by date: previous | next ] [ Message in thread: previous | next ] [ Thread: previous | next ]
Date: -- (:)
From: Dario Teixeira <darioteixeira@y...>
Subject: RE: [Caml-list] The lexer hack
Hi,

> Out of interest, how LaTeX-ish do you mean? I would hazard
> a guess that it's impossible to parse an unrestricted TeX
> file using an LR grammar (or at least no more clear than a
> hand-coded automaton) because you have to execute the macro
> expander in order to parse the file *completely* correctly.
> However, if you only mean LaTeX-ish in syntax (i.e. the
> files aren't actually TeX files) then you don't have to
> worry about TeX's elegant (by which I mean terrifying)
> \catcode mechanism and macro language!

I developed the language's syntax in tandem with the parser/lexer
so I made sure it was LR-friendly and Ulex-friendly (the verbatim
environments are the only parsing-unfriendly features).  The language
looks and feels like LaTeX, but without the hairy stuff...

Incidentally, the dummy token/action trick seems to be working
fine with Menhir.  Since the parser will look ahead one token,
I just have a tokenizer sitting between the lexer and the parser,
and inserting a DUMMY token into the stream after any token that
precedes a dummy action:

inline:
  | (...)
  | BEGIN_VERBATIM enter_verb DUMMY RAW exit_verb END_VERBATIM {...}
  | (...)

enter_verb: /*empty*/ {Global.context := Global.Verbatim}
exit_verb: /*empty*/  {Global.context := Global.General}


It's not the prettiest thing in the world (and I suspect I might
still find some problem with it), but as far as lexer hacks go
it's not bad and a lot better than building a state machine.

Cheers,
Dario Teixeira