Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0005348OCamlOCaml generalpublic2011-08-25 12:462013-10-09 14:23
Reporterpoirriez 
Assigned To 
PrioritynormalSeverityfeatureReproducibilityalways
StatusfeedbackResolutionopen 
PlatformOSOS Version
Product Version3.12.0 
Target VersionFixed in Version 
Summary0005348: no more identifiers with accented characters
DescriptionJust update to 3.12.0 and, on MAC OS 10.6.8 the accented characters are no longer usable:


$ ocaml
        Objective Caml version 3.12.0

# let carr?? x = x*x;;
Error: Illegal character (?)

The ?? was é

And under emacs:

        Objective Caml version 3.12.0

# Characters 9-10:
  let carré n = n * n;;
           ^
Error: Illegal character (\251)

Vincent
TagsNo tags attached.
Attached Files

- Relationships

-  Notes
(0006104)
poirriez (reporter)
2011-08-25 14:44

same with ocaml 3.12.1
(0006112)
xclerc (developer)
2011-09-05 11:24

Well, I am not sure it is an OCaml issue.
I can locally reproduce the problem by setting encoding
of the terminal to UTF8. When switching back to ISO Latin 1
(which is the encoding actually used by OCaml), everything is
fine.

Your can change the terminal encoding in the "Advanced" pane
of the "Settings" page of "Terminal.app" preferences.

(0006174)
gerd (reporter)
2011-10-23 14:53

Maybe the real issue is: ocaml should support UTF8 identifiers (instead of Latin1). Almost the whole world switched to UTF-8 in the meantime (e.g. many Linux distros use it as default now), and it becomes more and more painful that ocaml is so old-fashioned.

The full solution is complicated to implement, though - ocamllex would have to be changed so it can deal with multi-byte encodings. But as a tiny step into this direction, ocaml could at least allow UTF-8 as external encoding but keep Latin1 as internal encoding. That would mean a recoding step for every identifier that is read or written.
(0006849)
doligez (administrator)
2012-01-30 17:38

The real solution is to disallow accented letters in identifiers and accept only ASCII letters, but of course we cannot do that without breaking some existing programs.
(0006864)
frisch (developer)
2012-01-31 22:46

Damien: What about a warning that reports accented letters in identifiers as a deprecated feature?

Gerd: It would be weird to accept UTF8 identifiers, while still parsing string literals as sequences of bytes (i.e. if we assume the source code to be utf8-encoded, String.length on a string literal would not return the length of the literal seen as a sequence of Unicode code points, unless you also change the semantics of strings, but I don't think you propose that).
(0006866)
gerd (reporter)
2012-02-01 00:27

Alain: I agree that a complete solution would also include a Unicode version of "string", maybe called ustring, and with literals like U"xyz". I don't think we should redefine "string", because there is also a need for byte arrays, and we would run into endless compatibility problems. So, having both string and ustring would be the ideal world. I see that there are currently not enough resources for getting there, and the question is how many elements we can nevertheless implement. That could also mean to only deprecate accented letters at the moment.

Btw, for ustring we won't need that much, given that we accept that ocaml only provides basic Unicode support (string literals, one possible representation (ustring = int array), basic input/output, ocamllex), and leave the rest (alternate representations, character classes, transformations, ...) to add-on libraries like Camomile.

But anyway, I hope there is at least consensus that Unicode support is essential nowadays. The world is changing, and it has become irrelevant that Latin1 is still sufficient for most languages.
(0006868)
doligez (administrator)
2012-02-01 11:28

Alain: I would like such a warning but I'm not sure we have a consensus among OCaml developers at this point.
(0006890)
guesdon (manager)
2012-02-06 17:45

+1 for Gerd's proposition on Unicode support in ocaml distro ;-)
(0007680)
doligez (administrator)
2012-07-10 13:57

What we have here are several feature wishes:

1. Deprecate Latin-1 accents in source code and add the corresponding warning
2. Support Unicode strings
3. Have a separate type for mutable byte arrays
(0007683)
ygrek (reporter)
2012-07-10 14:54

Concerning point 3 - what about adding module Bytes equal to current String and provide the compiler switch to expose only read-only access to String module. This way interested people can start migrating some code right now.
(0007948)
dbuenzli (reporter)
2012-08-16 16:43

In my opinion, a first good step would be to

1) Mandate that sources should be UTF-8 encoded.
2) Disallow non-ascii identifiers (which just implies banning any octet
   greater than 0x7F outside string literals and comments).
3) Indicate that an OCaml string is just a sequence of octets
   and that *if* it needs to be interpreted as text it should
   be understood as UTF-8 encoded text (for better library
   interoperability).
3) Deprecate latin-1 functionality (uppercase, lowercase) from
   the String module and indicate that all the function of the
   string module operate octet-wise.

The main of advantage of this is that it allows to write UTF-8
string literals and documentation (by passing -charset utf-8 to
ocamldoc). If one needs to work with a unicode string type it's
easy to invoke a suitable of_utf8 on the UTF-8 string literals to
get a kind of U"bla" notation, at the cost of a small runtime
penalty.

To me UTF-8 identifiers are more a curse than a benefit because
of keyboard and normalization issues (é can be encoded in more
than one way, and I'm not sure the dev team wants to bring in
that whole issue into the compiler). If however some think that's
a good idea note that the Unicode standard has recommendations
for what a unicode identifier should be made of
http://unicode.org/reports/tr31/ [^]

I have written many programs that deal with unicode by just UTF-8
encoding my sources without any problem. I think that this brings
in more unicode compatibility without having to include a unicode
library in the distribution.

Regarding having unicode string support in the distribution. I'd
rather have nothing than a toy Unicode implementation in the
standard library. I also have written many programs that didn't
need a full blown unicode library and only needed to just
blindly pass around and concatenate UTF-8 encoded strings.
(0010443)
doligez (administrator)
2013-10-09 14:23

Note: 4.01.0 adds a warning signaling Latin1 characters in identifiers as a deprecated feature.

- Issue History
Date Modified Username Field Change
2011-08-25 12:46 poirriez New Issue
2011-08-25 14:44 poirriez Note Added: 0006104
2011-09-05 11:24 xclerc Note Added: 0006112
2011-09-05 11:24 xclerc Assigned To => xclerc
2011-09-05 11:24 xclerc Status new => feedback
2011-10-23 14:53 gerd Note Added: 0006174
2012-01-30 17:38 doligez Note Added: 0006849
2012-01-31 22:46 frisch Note Added: 0006864
2012-02-01 00:27 gerd Note Added: 0006866
2012-02-01 11:28 doligez Note Added: 0006868
2012-02-06 17:45 guesdon Note Added: 0006890
2012-07-10 13:57 doligez Note Added: 0007680
2012-07-10 13:57 doligez Severity minor => feature
2012-07-10 14:54 ygrek Note Added: 0007683
2012-08-09 08:49 doligez Assigned To xclerc =>
2012-08-16 16:43 dbuenzli Note Added: 0007948
2013-10-09 14:23 doligez Note Added: 0010443


Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker