Original bug ID: 900
Reporter: administrator
Status: closed
Resolution: fixed
Priority: normal
Severity: minor
Category: ~DO NOT USE (was: OCaml general)

Bug description

Full_Name: Gerd Stolpmann
Version: 3.04
OS: Linux
Submission from: pd9e26ad1.dip.t-dialin.net (217.226.106.209)

Hello,

I recently noticed a problem with ocamllex and very long tokens. Although
the lexing buffer grows as needed, this is done in a very unfortunate way
(see lexing.ml in stdlib). The lex_refill function fills the buffer from
the end to the beginning, and moves the whole contents every time the
next 512 bytes are added. Are there any good reasons for doing so? It
seems to be that it would be almost trivial to fill the buffer from the
beginning to the end.

The real world problem are XML files with large text sections that are
usually parsed as one token. A user of my XML parser has complained that
he cannot parse such files that seem to actually occur in practice.

Thank you in advance,

Gerd