Module Token

module Token: sig  end
Lexers for Camlp4 grammars.

This module defines the Camlp4 lexer type to be used in extensible grammars (see module Grammar). It also provides some useful functions to create lexers (this module should be renamed Glexer one day).

type pattern = string * string 
Token patterns come from the EXTEND statement.
exception Error of string
An lexing error exception to be used by lexers.

Lexer type

type location = int * int 
type location_function = int -> location 
The type for a function associating a number of a token in a stream (starting from 0) to its source location.
type 'a lexer_func = char Stream.t -> 'a Stream.t * location_function 
The type for a lexer function. The character stream is the input stream to be lexed. The result is a pair of a token stream and a location function for this tokens stream.

type 'a glexer = {
   tok_func : 'a lexer_func;
   tok_using : pattern -> unit;
   tok_removing : pattern -> unit;
   tok_match : pattern -> 'a -> string;
   tok_text : pattern -> string;
   mutable tok_comm : location list option;
The type for a lexer used by Camlp4 grammars.
val lexer_text : pattern -> string
A simple tok_text function for lexers
val default_match : pattern -> string * string -> string
A simple tok_match function for lexers, appling to token type (string * string)

Lexers from char stream parsers or ocamllex function

The functions below create lexer functions either from a char stream parser or for an ocamllex function. With the returned function f, the simplest Token.lexer can be written:

          { Token.tok_func = f;
            Token.tok_using = (fun _ -> ());
            Token.tok_removing = (fun _ -> ());
            Token.tok_match = Token.default_match;
            Token.tok_text = Token.lexer_text }
Note that a better tok_using function should check the used tokens and raise Token.Error for incorrect ones. The other functions tok_removing, tok_match and tok_text may have other implementations as well.

val lexer_func_of_parser : (char Stream.t -> 'a * location) -> 'a lexer_func
A lexer function from a lexer written as a char stream parser returning the next token and its location.
val lexer_func_of_ocamllex : (Lexing.lexbuf -> 'a) -> 'a lexer_func
A lexer function from a lexer created by ocamllex
val make_stream_and_location : (unit -> 'a * location) -> 'a Stream.t * location_function
General function

Useful functions

val eval_char : string -> char
val eval_string : string -> string
Convert a char or a string token, where the backslashes had not been interpreted into a real char or string; raise Failure if bad backslash sequence found; Token.eval_char (Char.escaped c) returns c and Token.eval_string (String.escaped s) returns s