Hello, people--

In a lexer definition with two or more entry points, is there a way to
emit a lexeme and pass control to another entrypoint in one action?

The specific problem I am trying to deal with is a configuration file
format that includes comments denoted with an initial '#' character. I
would like to support the typical usage of '#', where a comment may
begin either at the beginning of the line, or after a declaration that I
want to capture, and in either case it extends to the end of the line.

So in general, anything after '#' up to the end of a line should be
ignored, which I think requires a separate 'comment' entrypoint. At the
end of the line, control returns to the main entry point. So my first
cut looks like this:

  rule dict = parse
      [' ']                             { dict lexbuf }
    | '#'                               { comment lexbuf }
    | word                              { WORD (Lexing.lexeme lexbuf) }
    | ':'                               { COLON }
    | '{'                               { DS }
    | '}'                               { DE }
    | ',' | '\n'                        { SEP }
    | eof                               { EOF }
  and comment = parse
      [ ^ '\n' ]                        { comment lexbuf }
    | '\n'                              { dict lexbuf }

So far so good. BUT, for the sake of simplicity (for users, not for me
;-)), my syntax has line endings as separators, and in order to support
comments following non-comments on the same line, a line ending after a
comment should be interpreted as a separator. So what I want to do is
something like:

  and comment = parse
      [ ^ '\n' ]                        { comment lexbuf }
    | '\n'                              { SEP; dict lexbuf }

But that doesn't work, of course. Maybe the solution is to push SEP back
onto the head of the buffer, but I don't see a way to do that.

Or would it be better to simply tag the comment text with, say, a
COMMENT symbol and pass it through to the parser?

--
Matt Gushee
Englewood, CO, USA