Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0007200OCamllexing and parsingpublic2016-03-30 14:222017-10-05 18:10
Reporterfrisch 
Assigned To 
PrioritynormalSeverityminorReproducibilityhave not tried
StatusfeedbackResolutionopen 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version 
Summary0007200: Testing parsing/printing/parsing roundtripping shows problems with Pprintast / -dsource
DescriptionI'm attached a small script which parses all .ml/.mli files from its command-line (recursing into subdirectories) and for each of them does the following:

  1. Parse.
  2. Print with Pprintast.
  3. Parse the result.
  4. Compare the two parse trees after removing locations).

Applying this tool to the OCaml source tree discovers multiple issues with Pprintast (cases where it produces code that cannot be parsed, and cases where it produces valid code which results in a different Parsetree). I'm also attaching the result. (Some files cannot be parsed at all as implementation files because they are to be interpreted by the toplevel; but we cannot always parse as the toplevel since this would reject valid files, those with multiple ";;" in a row.)

Interpretation of issues (I'll edit this list as I go through the output).

 - ~-1 being printed as -1, reparsed as a literal. [FIXED]

 - (function x | (_ as x) -> x) being printed as (function x | _ as x -> x), invalid precedence of "as" vs "|".

 - type expression [>] printed as [], which is invalid. [FIXED]

 - "let foo : type t'. t' = assert false" printed as "let foo : 't' . 't'= fun (type t') -> (assert false : t')", but 't' is not a valid type variable (parsed as a character literal token). Not clear what to do here.

 - " type t = (::) : int -> t" printed as "type t = | ::: int -> t". [FIXED, also in Printyp]

 - "type t = < foo: int [@foo] >" printed as "type t = < foo[@foo ] :int >".

 - "[%foo: < foo : t > ]" printed as "[%foo :< foo :t >]", cannot be parsed because ">]" is recognized as a token.

 - "type foo += private A of int" printed as "type foo += | A of int" (private is dropped).

 - "let f : 'a . < .. > = assert false" printed as "let f : 'a . < .. >= assert false", cannot be parsed because ">=" is recognized as a token.

 - "let module M = (functor (T : sig end) -> struct end)(struct end) in ()"
printed as "let module M = functor (T : sig end) -> struct end(struct end) in ()".

 - "object method f : int = 42 end" printed as "object method f : int= 42 end", which stores the type annotation in the Pexp_poly node. (Should we fix the Parsetree definition to rely on a Pexp_constraint for that form?).

 - "class c = object inherit ((fun () -> object end) ()) end" printed as "class c = object inherit (fun () -> object end ()) end".
TagsNo tags attached.
Attached Files? file icon checkparse.ml [^] (3,847 bytes) 2016-03-30 14:22 [Show Content]
txt file icon checkparse.txt [^] (135,060 bytes) 2016-03-30 14:23 [Show Content]

- Relationships

-  Notes
(0015646)
runhang (reporter)
2016-03-30 14:26

dsheets and I are improving -dsource (see: https://github.com/ocamllabs/compiler-hacking/wiki/Things-to-work-on#improve--dsource [^]), and I think this script can help
(0015647)
frisch (developer)
2016-03-30 14:41

Another useful test would be to try extracting the source fragments based on locations in the Parsetree (for various kinds of items) and check that these fragments can be parsed correctly. This would allow to detect problems with locations in the parser.
(0015726)
runhang (reporter)
2016-04-13 10:52

"type t = < foo: int [@foo] >" printed as "type t = < foo[@foo ] :int >". - FIXED

""[%foo: < foo : t > ]" printed as "[%foo :< foo :t >]", cannot be parsed because ">]" is recognized as a token." - FIXED

" "type foo += private A of int" printed as "type foo += | A of int" (private is dropped)" - FIXED

""let f : 'a . < .. > = assert false" printed as "let f : 'a . < .. >= assert false", cannot be parsed because ">=" is recognized as a token" - FIXED
(0015727)
runhang (reporter)
2016-04-13 11:18

""let module M = (functor (T : sig end) -> struct end)(struct end) in ()"
printed as "let module M = functor (T : sig end) -> struct end(struct end) in ()"." - FIXED

""class c = object inherit ((fun () -> object end) ()) end" printed as "class c = object inherit (fun () -> object end ()) end"." - FIXED
(0015728)
frisch (developer)
2016-04-13 15:42

Cf https://github.com/ocaml/ocaml/pull/539 [^]
(0015937)
runhang (reporter)
2016-05-17 07:21

class ['a] c () = object
  constraint 'a = < .. > -> unit
  method m = (fun x -> () : 'a)
end

is printed as

class ['a] c () =
  object constraint 'a = < .. > -> unit method m : 'a = fun x -> () end
(0017415)
doligez (administrator)
2017-02-23 17:24

@frisch GPR#539 was merged. Could you give an update on the status of this PR (including @runhang's latest example)?
(0017419)
runhang (reporter)
2017-02-23 19:21

@doligez that example has been addressed in GPR#915 (https://github.com/ocaml/ocaml/pull/915 [^] [^])
(0017856)
doligez (administrator)
2017-06-09 12:10

I tried it on the 4.05 branch and Alain's tool is still complaining about a few discrepancies. Output file attached.

- Issue History
Date Modified Username Field Change
2016-03-30 14:22 frisch New Issue
2016-03-30 14:22 frisch File Added: checkparse.ml
2016-03-30 14:23 frisch File Added: checkparse.txt
2016-03-30 14:26 runhang Note Added: 0015646
2016-03-30 14:39 frisch Description Updated View Revisions
2016-03-30 14:40 frisch Description Updated View Revisions
2016-03-30 14:41 frisch Note Added: 0015647
2016-03-30 14:45 frisch Description Updated View Revisions
2016-03-30 14:52 frisch Description Updated View Revisions
2016-03-30 14:54 frisch Description Updated View Revisions
2016-03-30 14:58 frisch Description Updated View Revisions
2016-03-30 15:59 frisch Description Updated View Revisions
2016-03-30 16:02 frisch Description Updated View Revisions
2016-03-30 16:04 frisch Description Updated View Revisions
2016-03-30 16:11 frisch Description Updated View Revisions
2016-03-30 16:14 frisch Description Updated View Revisions
2016-03-30 16:20 frisch Description Updated View Revisions
2016-03-30 16:23 frisch Description Updated View Revisions
2016-04-06 13:04 doligez Status new => acknowledged
2016-04-06 13:04 doligez Target Version => 4.03.1+dev
2016-04-13 10:52 runhang Note Added: 0015726
2016-04-13 11:18 runhang Note Added: 0015727
2016-04-13 15:42 frisch Note Added: 0015728
2016-05-17 07:21 runhang Note Added: 0015937
2017-02-16 14:00 doligez Target Version 4.03.1+dev => undecided
2017-02-23 16:36 doligez Category OCaml general => -OCaml general
2017-02-23 17:24 doligez Note Added: 0017415
2017-02-23 17:24 doligez Status acknowledged => feedback
2017-02-23 17:24 doligez Category -OCaml general => lexing and parsing
2017-02-23 17:24 doligez Target Version undecided => 4.05.0 +dev/beta1/beta2/beta3/rc1
2017-02-23 19:19 marklrh Note Added: 0017418
2017-02-23 19:21 marklrh Note Deleted: 0017418
2017-02-23 19:21 runhang Note Added: 0017419
2017-06-09 12:10 doligez Note Added: 0017856
2017-06-09 12:11 doligez Target Version 4.05.0 +dev/beta1/beta2/beta3/rc1 => 4.06.0 +dev/beta1/beta2/rc1
2017-10-05 18:10 doligez Target Version 4.06.0 +dev/beta1/beta2/rc1 =>


Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker