Comment author: @alainfrisch

FWIW, using "ocamllex -ml" seems to work (at least, no segfault).

Comment author: @xclerc

The latest trunk (db6891f) does not segfault on this code.

Comment author: @nojb

db6891f segfaults for me, looks like some bad indexing on the Lexing.lex_buffer field.

Comment author: @let-def

I started investigating this issue.

The problem triggers when one branch capture sub-values (the (_ as x) in Stephen's example), while another branch catch all cases (the _ as y; note that this is not a sub-capture, the whole lexeme is returned, this doesn't stress the same code path).

The automaton produced is correct (though not minimal :)), that's why the -ml output works. However the bytecode generated is wrong: there will be out of bounds write to the lexbuf.Lexing.lex_mem fields (it is a vector that is used to store the locations of capture groups).

If you don't capture sub-values, the lexer will use the caml_lex_engine primitive for interpretation which is correct as far as I can tell.

However, if one of the branch capture sub-values, caml_new_lex_engine is used, which can do arbitrary writes (via the run_tag C function).

Btw, this is not an initialization issue (one could think that the position vector is too short), it is because of the wrong interpretation of a tag which consumes garbage values and writes at some arbitrary offset of lex_mem.

My next step will be to instrument bytecode generation to understand what goes wrong, but I progress slowly as I found few resources on that part :).

Comment author: @let-def

xclerc: sometimes the random write corrupts the heap, sometimes it doesn't. You will have to test in different memory conditions (and for good measures, put an assertion in run_tag to check for the bounds).

Comment author: @xavierleroy

Maybe @maranget could look into this issue as well.

Comment author: @maranget

I am having a look.

Comment author: @maranget

I think I have found the bug, but I am lacking time to submit
a pull request now.

Basically, the problem originates from the table compaction function being
able to optimize away memory instructions [in compact.ml]; while the
main output function does not notice it (and thus emits a call to caml_new_lex_engine, while a call to caml_lex_engin would be approriate).

Comment author: @maranget

There is now a pull request that corrects this PR,
see #1713