Comment author: furuse

The spec of setlocale I found is here: http://pubs.opengroup.org/onlinepubs/7908799/xsh/setlocale.html

Comment author: @mshinwell

Jun, is this a new bug in 4.02?

Comment author: @dbuenzli

The bug is also present before 4.02.

If I read the documentation of String.escape and given the encoding of OCaml's string I expect String.escape to escape only the unprintable characters of ISO-8859-1 (i.e. the gray unlabelled boxes here: http://en.wikipedia.org/wiki/ISO/IEC_8859-1#Codepage_layout). That is those that are not in (0x20 <= c && c <= 0x7E || 0xA0 <= c && c <= 0xFF)

So the current behaviour with LANG=ja_JP.UTF-8 seems fine to me it's just the behaviour with LANG=C that is not. Now if historically (what did the original authors expect ?) and statistically the function has given us the behaviour of LANG=C then I also suggest to hard code the check to ASCII printable characters.

Example with my own locale (right behaviour to me) and then LANG=C on osx 10.9.4 with 4.01.0

echo $LANG
fr_CH.UTF-8
ocaml
OCaml version 4.01.0

let s = String.escaped "\233\171\152";;

val s : string = "?\152"

Char.code s.[0];;

• : int = 233

Char.code s.[1];;

• : int = 171

export LANG=C
ocaml
OCaml version 4.01.0

let s = String.escaped "\233\171\152";;

val s : string = "\233\171\152"

Comment author: @dbuenzli

Well actually given the doc:

Return a copy of the argument, with special characters represented by escape sequences, following the lexical conventions of OCaml.

We can say that both answers in the example above are in fact valid, in the sense that interpreted by an OCaml compiler both strings denote the same sequence of bytes.

Comment author: furuse

Yes the both outputs are valid... I think we should choose one of them. It is very confusing that the runtime of OCaml is affected by LANG.

I prefer escaping non-ASCII printables, since

• I do not check it thoroughly, but it seems to be the behaviour of Linux + any LANG, which many are used to for long.
• Escaping all the non-ASCII printables make the result valid also as UTF-8 and other encodings.
• Many use OCaml string to store UTF-8 data knowing or not knowing it is officially in ISO-8859-1. Escaping non ASCII printables is meaningful also to them.
• I am selfish and I live in Asia :-)

We can choose quote only non-printable ISO-8859-1, but in that case, I would like to have escaped_to_ASCII too.

Comment author: @dbuenzli

Yes to everything (even you being selfish).

I we choose one I'd also be in absolute favour of escaping all the non-ASCII printable characters. UTF-8 compatibility of the returned string is the argument. There's no real point against that if we want a forward looking solution.

Comment author: @alainfrisch

I'm also in favor of the change. FWIW, we already have it in LexiFi's version, to avoid introducing different behaviors between platforms for such a basic function.

Comment author: @damiendoligez

Strings are supposed to be encoding-agnostic, and certainly not officially iso-8859-1.

Definitely change it to escape all non-ascii-printable.

Comment author: @damiendoligez

This will be an incompatible changes, so I'm pushing it back to 4.03.

Comment author: @damiendoligez

Fixed in trunk (commit 15901).

Note that I also changed Bytes.escaped and Char.escaped.