Version française
Home     About     Download     Resources     Contact us    
Browse thread
using different lexers with one parser?
[ Home ] [ Index: by date | by threads ]
[ Search: ]

[ Message by date: previous | next ] [ Message in thread: previous | next ] [ Thread: previous | next ]
Date: -- (:)
From: skaller <skaller@u...>
Subject: Re: [Caml-list] using different lexers with one parser?
On Wed, 2007-04-18 at 12:01 +0200, Oliver Bandel wrote:
> On Wed, Apr 18, 2007 at 09:38:20AM +0200, Hendrik Tews wrote:
> > Oliver Bandel <oliver@first.in-berlin.de> writes:
> > 
> >    Is it possible (without too much effort) to switch
> >    the lexer during parsing (from within the parser)?
> > 
> > Yes, see
> > http://caml.inria.fr/pub/ml-archives/caml-list/2003/09/3e7f3495840e2bc851b91c3dba8abab9.en.html
> > 
> [...]
> 
> OK; I hoped to find a solution without the global-switching-var hacks;
> something that is built in in ocamlyacc.

There is a solution without global variables.

Referring to Hendrick's example, write:

rule lexer global_var =
   parse
     | ""          { match !global_var with
	                | Xlex -> xlex global_var lexbuf
                        | Ylex -> ylex global_var lexbuf
                   }

instead. Now, you must change at least some --
but I recommend all -- of your tokens to include
the global_var, for example

%token TOKEN<bool ref * int>

and xlex global var = 
  parse 
    | ... { TOKEN (global_var,42) }  

Now in any ocamlyacc action code you can match write this

  | TOKEN ... { let global_var, attrib = $1 in ... }

to get the variable into the action code, and now you
can change it.

This leaves the problem of the lookahead token, which
may or may not exist. You can predict whether it exists
for each reduction. You can do this 'mentally' by asking
"Does this production have a definite set of terminators,
(no lookahead) or does it rely on bumping 
into something unrecognizable (lookahead)"?

[If you can't tell .. you must refactor your grammar
so you can]

If the lookahead exists, you must 'put it back' into the
lexbuf. At present you can do this by studying the implementation
details in Lexing module. The lexeme is certain to be IN the
buffer, so it is safe to reset the pointers to the lexbuf state
before it was lexed.

To get at the lexbuf from the parser .. you have to modify
your tokens AGAIN to include the lexbuf.

I recommend considering using an Ocaml class to encapsulate
all the state information you need. Passing state from the
lexer to the parser this way is ugly but it does work
and is fully re-entrant.

-- 
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net