Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0006069OCamlocamldocpublic2013-07-09 06:272013-08-05 09:58
Assigned Toguesdon 
PrioritynormalSeverityminorReproducibilityhave not tried
PlatformOSOS Version
Product Version4.00.1 
Target Version4.01.0+devFixed in Version 
Summary0006069: ocamldoc: lexing: empty token
DescriptionJust stumbled upon the case when ocamldoc aborts with unhelpful error message. Consider :

$ cat
(** test *)
(** ok @@ *)
(** @deprecated @@ *)
(** ok [@@] *)
$ ocamldoc
/home/ygrek/work/arena/test/ : lexing: empty token

1 error(s) encountered

In the real code it is hard to figure out what line causes an error. It appears all the code to track line numbers is in place, but not used when reporting errors. Simple patch below.
Additional Informationdiff --git a/ocamldoc/ b/ocamldoc/
index ce96f2a..61b8f9a 100644
--- a/ocamldoc/
+++ b/ocamldoc/
@@ -90,7 +90,7 @@ module Info_retriever =
                  Failure s ->
                    incr Odoc_global.errors ;
- prerr_endline (file^" : "^s^"\n");
+ Printf.eprintf "%s : line %d : %s\n%!" file (!Odoc_lexer.line_number + 1) s;
                    (0, None)
                | Odoc_text.Text_syntax (l, c, s) ->
                    incr Odoc_global.errors ;
TagsNo tags attached.
Attached Filespatch file icon ocamldoc-better-error-message-for-Desc-token.-Fix-6069.patch [^] (2,941 bytes) 2013-07-23 13:17 [Show Content]
patch file icon ocamldoc-better-error-message-for-Desc-token-v2.patch [^] (1,682 bytes) 2013-07-24 12:08 [Show Content]

- Relationships
related to 0004518acknowledged Better location format for reporting errors in ocamldoc 

-  Notes
doligez (administrator)
2013-07-12 17:13

It's not only the error message, but also this "empty token" situation that needs to be fixed.
hnrgrgr (developer)
2013-07-23 13:16

The attached patch introduces an explicit error message, saying: in arguments @-tags, others @-sign should be escaped.
gasche (developer)
2013-07-23 21:19

There are several different things in this patch:
(1) a way to report line errors for lexing errors in ocamldoc
(2) a new lexing rule to accept single "@" in inputs instead of having a lexer failure
(3) an error message for this particular error

I think (1) could be done better, but that would require more effort. The patch format is an improvement over the previous format, but it does not respect the usual OCaml convention (so no emacs location parsing for example), and in particular it does not report character positions. The nice way to fix that would be to call "Location.curr lexbuf" at error points inside the lexer (as done in the usual OCaml parser), or at least when catching the Failure exception in (but then you need to change this code so that you know which "lexbuf" value provoked the error).

I'd say (2) can be easily improved: instead of turning @foo+ into @foo* and then raise an error on the empty foo* case, why not just handle @ alone specifically? That would be more readable. It seems to me that the longest-match precedence rule of lexers should make that work nicely.

I'm not sure I understand (3). Why does it say "in @-tags arguments"? I didn't know about the problem before reading your patch, but I'd say that there is an error for any isolated @ in the comment, not only after, say, a @raise keyword (which expects arguments). Besides, it would be helpful if the error message mentioned explicitly mentioned how to escape (rather than just the need for an escape). I would suggest something like "The character @ has a special meaning in ocamldoc comments, for commands such as @raise or @since. If you want to write a single @, you must escape it as \@.".

(Do we really need to fail when we read @ followed by something else than a lowercase ? I think it is prudent to do so, as it leaves the door open to extending the lexical structure of @-commands in the future.)
hnrgrgr (developer)
2013-07-24 12:07

Well, my intention was just to track down the error. Hence, fixing point (2) was my sole intention. To improve readability, I've updated this part of the patch as suggested.

For point (3), I tried to be coherent with the ocamldoc manual: [^]

As said in the manual, there is not an error for every isolated @-sign, it appears only after the first @-tags. Still, one may probably improve the error message.

> (Do we really need to fail when we read @ followed by something else than a lowercase ? I think it is prudent to do so, as it leaves the door open to extending the lexical structure of @-commands in the future.)

I agree.

I remove point (1) from the modified patch.
gasche (developer)
2013-07-24 18:19

> As said in the manual, there is not an error for every isolated @-sign

I'm not sure I understand the same thing from the documentation. If
you are referring to this part:

> The inside of documentation comments (**…*) consists of free-form text
> with optional formatting annotations, followed by optional tags giving
> more specific information about parameters, version, authors, … The
> tags are distinguished by a leading @ character. Thus, a documentation
> comment has the following shape:
> (** The comment begins with a description, which is text formatted
> according to the rules described in the next section.
> The description continues until the first non-escaped '@' character.
> @author Mr Smith
> @param x description for parameter x
> *)

Then my understanding is that the first non-escaped '@', followed by
a command *or otherwise*, delimits the end of the description
section. There is no mention that non-escaped '@' would be allowed,
when put alone, in the description section (and in fact the
documentation comment example below would be wrong according to this
rule, because of the explicitly mentioned '@'. It should read "until
the first non-escaped \@ character." to be consistent with the
described rule.
guesdon (manager)
2013-08-05 09:46

Improved the error message, displaying also the unexpected character.
Rev 13974 in branch 4.O1.
Rev 13975 in trunk.
guesdon (manager)
2013-08-05 09:58

Oups, did not see the whole discussion before. Added the specific error message for not escaped '@'.
Rev 13976 in 4.01.
Rev 13977 in trunk.

- Issue History
Date Modified Username Field Change
2013-07-09 06:27 ygrek New Issue
2013-07-09 06:27 ygrek Status new => assigned
2013-07-09 06:27 ygrek Assigned To => guesdon
2013-07-12 17:13 doligez Note Added: 0009769
2013-07-12 17:13 doligez Target Version => 4.01.0+dev
2013-07-23 13:16 hnrgrgr Note Added: 0009832
2013-07-23 13:17 hnrgrgr File Added: ocamldoc-better-error-message-for-Desc-token.-Fix-6069.patch
2013-07-23 21:19 gasche Note Added: 0009837
2013-07-24 12:07 hnrgrgr Note Added: 0009847
2013-07-24 12:08 hnrgrgr File Added: ocamldoc-better-error-message-for-Desc-token-v2.patch
2013-07-24 18:19 gasche Note Added: 0009857
2013-07-28 22:30 gasche Relationship added related to 0004518
2013-08-05 09:46 guesdon Note Added: 0010102
2013-08-05 09:46 guesdon Status assigned => closed
2013-08-05 09:46 guesdon Resolution open => fixed
2013-08-05 09:58 guesdon Note Added: 0010103
2017-02-23 16:46 doligez Category OCamldoc => ocamldoc

Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker