0005127: wrong location after directive

(0006512)
dim (developer)
2011-12-22 21:16

Unfortunately it is going to be very difficult to fix. The only simple fix i have in mind is going to degrade performances a lot.

By the way note that locations are correct after directives when you use the toplevel. And since directives are for the toplevel, I don't think it is a big problem. So you should avoid using them in programs. As a replacement you can use camlp4.macro:

INCLUDE "x.ml"

With this syntax there is no problem.

(0006654)
Hendrik Tews (reporter)
2012-01-12 09:04

May I suggest to add a note to the documentation?

(0006661)
dim (developer)
2012-01-12 11:38

Sure. But i don't know where it should go...

(0007445)
Hendrik Tews (reporter)
2012-05-21 22:00

I would suggest to put it as ocamldoc comment into module Loc in file camlp4/Camlp4/Sig.ml. That's at least where I look when I want to know something about camlp4 locations.

(0007446)
Hendrik Tews (reporter)
2012-05-21 23:30

After looking again at the problem I noticed that we might have a
misunderstanding: The problem is present with all directives (eg,
also with #load), even if I am using a dummy directive handler
for parsing, such as

  let directive_handler x = Some x

The problem is obviously that

  value stopped_at _loc =
    Some (Loc.move_line 1 _loc) (* FIXME be more precise *);

in camlp4/Camlp4Parsers/Camlp4OCamlRevisedParser.ml does not move
the offsets.

(0007510)
hongboz (developer)
2012-06-04 04:57

I was bitten by this bug too.
there's a very useful directive, #default_quotation "blabla", which has parsing time effects, and
help me to remove a lot of duplicated code, <:expr< >> and <:patt< >> can share the same structure. The problem came the documentation generated by ocamldoc is totally wrong.
It would be very nice to fix this bug.

(0007515)
hongboz (developer)
2012-06-04 16:16

I defined my own add hoc syntax extension to work around this problem. I used syntax extension only for ad hoc usage.
EXTEND Gram GLOBAL: str_item ;
str_item:
  [[ "<^"; quot=STRING; "^>"-> begin
    Quotation.default.val:= quot;
    <:str_item< >>
  end
   ]];
END;

(0007522)
dim (developer)
2012-06-07 11:52

> After looking again at the problem I noticed that we might have a
> misunderstanding: The problem is present with all directives (eg,
> also with #load), even if I am using a dummy directive handler
> for parsing, such as

That's what i understood.

> The problem is obviously that
>
> value stopped_at _loc =
> Some (Loc.move_line 1 _loc) (* FIXME be more precise *);
>
> in camlp4/Camlp4Parsers/Camlp4OCamlRevisedParser.ml does not move
> the offsets.

It is a more complicated problem; the parser used by camlp4 for an implementation/interface file has type:

Loc.t -> char Stream.t -> Ast.(str_item/sig_item) * Loc.t option

Where Loc.t option is the the location after the current directive if any.

This parser internally creates a lexer, token stream, ... for this character stream, and when it stops after a directive, pending characters in the lexer or pending tokens in the token streams or any other stuff in internal data structures are just lost, so there is no way to tell what will be the next location.

Here is for example another bug with directives and camlp4:

$ cat > foo.ml
#use "bar.ml"
let x = 1
$ touch bar.ml
$ camlp4o foo.ml
x = 1

The "let" keyword has been dropped.

One way to fix this would be to change the API to use parsers that takes token streams but it will break programs.

If you really want to use directives you can have a look at the optcomp project, it handles them at the token level (before the parser) so it does not have this problem.

(0007967)
hongboz (developer)
2012-08-22 12:33

@dim, the bug you mentioned will makes ocamldoc generate a messy documentation?

(0007972)
dim (developer)
2012-08-22 13:21

Not if you don't use directives.

(0008544)
hongboz (developer)
2012-11-28 21:38

FYI, this problem disappeared in my branch after I changed stopped_at, just pass the same position

Issue History
Date Modified	Username	Field	Change
2010-08-15 21:59	Hendrik Tews	New Issue
2010-08-15 21:59	Hendrik Tews	Status	new => assigned
2010-08-15 21:59	Hendrik Tews	Assigned To	=> ertai
2011-10-26 08:39	ertai	Assigned To	ertai => xclerc
2011-12-22 21:16	dim	Note Added: 0006512
2011-12-22 21:17	dim	Assigned To	xclerc => dim
2011-12-22 21:17	dim	Status	assigned => feedback
2012-01-11 22:48	Hendrik Tews	Note Added: 0006648
2012-01-11 22:48	Hendrik Tews	Status	feedback => assigned
2012-01-12 09:03	Hendrik Tews	Note Deleted: 0006648
2012-01-12 09:04	Hendrik Tews	Note Added: 0006654
2012-01-12 11:38	dim	Note Added: 0006661
2012-05-21 22:00	Hendrik Tews	Note Added: 0007445
2012-05-21 23:30	Hendrik Tews	Note Added: 0007446
2012-06-04 04:57	hongboz	Note Added: 0007510
2012-06-04 16:16	hongboz	Note Added: 0007515
2012-06-07 11:52	dim	Note Added: 0007522
2012-07-10 19:58	doligez	Target Version	=> 4.01.0+dev
2012-07-31 13:36	doligez	Target Version	4.01.0+dev => 4.00.1+dev
2012-08-22 12:33	hongboz	Note Added: 0007967
2012-08-22 13:21	dim	Note Added: 0007972
2012-09-19 13:55	doligez	Target Version	4.00.1+dev => 4.01.0+dev
2012-11-28 21:38	hongboz	Note Added: 0008544

Notes
(0006512) dim (developer) 2011-12-22 21:16	Unfortunately it is going to be very difficult to fix. The only simple fix i have in mind is going to degrade performances a lot. By the way note that locations are correct after directives when you use the toplevel. And since directives are for the toplevel, I don't think it is a big problem. So you should avoid using them in programs. As a replacement you can use camlp4.macro: INCLUDE "x.ml" With this syntax there is no problem.

(0006654) Hendrik Tews (reporter) 2012-01-12 09:04	May I suggest to add a note to the documentation?

(0006661) dim (developer) 2012-01-12 11:38	Sure. But i don't know where it should go...

(0007445) Hendrik Tews (reporter) 2012-05-21 22:00	I would suggest to put it as ocamldoc comment into module Loc in file camlp4/Camlp4/Sig.ml. That's at least where I look when I want to know something about camlp4 locations.

(0007446) Hendrik Tews (reporter) 2012-05-21 23:30	After looking again at the problem I noticed that we might have a misunderstanding: The problem is present with all directives (eg, also with #load), even if I am using a dummy directive handler for parsing, such as let directive_handler x = Some x The problem is obviously that value stopped_at _loc = Some (Loc.move_line 1 _loc) (* FIXME be more precise *); in camlp4/Camlp4Parsers/Camlp4OCamlRevisedParser.ml does not move the offsets.

(0007510) hongboz (developer) 2012-06-04 04:57	I was bitten by this bug too. there's a very useful directive, #default_quotation "blabla", which has parsing time effects, and help me to remove a lot of duplicated code, <:expr< >> and <:patt< >> can share the same structure. The problem came the documentation generated by ocamldoc is totally wrong. It would be very nice to fix this bug.

(0007515) hongboz (developer) 2012-06-04 16:16	I defined my own add hoc syntax extension to work around this problem. I used syntax extension only for ad hoc usage. EXTEND Gram GLOBAL: str_item ; str_item: [[ "<^"; quot=STRING; "^>"-> begin Quotation.default.val:= quot; <:str_item< >> end ]]; END;

(0007522) dim (developer) 2012-06-07 11:52	> After looking again at the problem I noticed that we might have a > misunderstanding: The problem is present with all directives (eg, > also with #load), even if I am using a dummy directive handler > for parsing, such as That's what i understood. > The problem is obviously that > > value stopped_at _loc = > Some (Loc.move_line 1 _loc) (* FIXME be more precise ); > > in camlp4/Camlp4Parsers/Camlp4OCamlRevisedParser.ml does not move > the offsets. It is a more complicated problem; the parser used by camlp4 for an implementation/interface file has type: Loc.t -> char Stream.t -> Ast.(str_item/sig_item) Loc.t option Where Loc.t option is the the location after the current directive if any. This parser internally creates a lexer, token stream, ... for this character stream, and when it stops after a directive, pending characters in the lexer or pending tokens in the token streams or any other stuff in internal data structures are just lost, so there is no way to tell what will be the next location. Here is for example another bug with directives and camlp4: $ cat > foo.ml #use "bar.ml" let x = 1 $ touch bar.ml $ camlp4o foo.ml x = 1 The "let" keyword has been dropped. One way to fix this would be to change the API to use parsers that takes token streams but it will break programs. If you really want to use directives you can have a look at the optcomp project, it handles them at the token level (before the parser) so it does not have this problem.

(0007967) hongboz (developer) 2012-08-22 12:33	@dim, the bug you mentioned will makes ocamldoc generate a messy documentation?

(0007972) dim (developer) 2012-08-22 13:21	Not if you don't use directives.

(0008544) hongboz (developer) 2012-11-28 21:38	FYI, this problem disappeared in my branch after I changed stopped_at, just pass the same position

Relationships