<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>Le 2 déc. 08 à 14:37, Neil Booth a écrit :</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Chris Lattner wrote:-<br><br><blockquote type="cite"><br></blockquote><blockquote type="cite">On Nov 29, 2008, at 1:00 AM, Paolo Bolzoni wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">I need to convert the strings literals to other encoding, I was <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">planning to<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">use iconv.h's functions, but I need to know the encoding of the <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">input strings.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">So the question is, what encoding have the strings returned by<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">clang::StringLiteral::getStrData(), overall wide ones?<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Hi Paolo,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I really have no idea. We're just reading in the raw bytes from the <br></blockquote><blockquote type="cite">source file, so I guess it depends on whatever the source encoding <br></blockquote><blockquote type="cite">is. In practice, this sounds like a really bad idea :).<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Clang doesn't have any notion of an input character set at present, <br></blockquote><blockquote type="cite">and doesn't handle unicode escapes. How do other compilers handle <br></blockquote><blockquote type="cite">input character sets? Are there command line options to specify it? <br></blockquote><blockquote type="cite">Should the AST hold the string in a canonical form like UTF8?<br></blockquote><br>Clang should have an idea of the encoding of its input, otherwise<br>it cannot reason about the characters that appear in a string<br>literal. The standard imposes constraints on those characters,<br>and requires input source to be in the current locale. Of course<br>this latter bit could be overridden with a command line switch.<br><br>Realistically I don't think there is much alternative to an internal<br>representation in some form of Unicode, or at least reasoning about<br>the input in Unicode. This is essentially enforced by requiring<br>UCNs to be accepted.<br><br>As for execution charset, GCC's -fexec-charset seems a very reasonable<br>approach, with some kind of error character for characters not<br>representable in said charset.<br><br>Note that accepting UCNs in identifiers, as both C99 and C++ require,<br>mandates converting to some kind of canonical Unicode form for<br>identifiers internally, before hashing, too.<br></div></blockquote><div><br></div>I didn't know that C99 supports UCN in identifier. </div><div>I don't see a lot of informations about it in the C99 spec (except that UCN may appear in an identifier). Does this mean that this code is valid ? </div><div><br></div><div><div>---------- test.c -------</div><div><br></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "><span style="color: #aa0d91">int</span> main (<span style="color: #aa0d91">int</span> argc, <span style="color: #aa0d91">char</span> **argv) {</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "><span class="Apple-tab-span" style="white-space:pre"> </span><span style="color: #aa0d91">int</span> h\u00e9 = <span style="color: #1c00cf">0</span>; <span class="Apple-style-span" style="color: rgb(0, 116, 0); ">// hé</span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "><span class="Apple-tab-span" style="white-space:pre"> </span><span style="color: #aa0d91">return</span> he\u0301; <span class="Apple-style-span" style="color: rgb(0, 116, 0); ">// hé - using decomposed form</span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; ">}</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "><font class="Apple-style-span" color="#000000" face="Helvetica" size="3"><span class="Apple-style-span" style="background-color: transparent; ">--------------------------</span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;">Actually, GCC does not support combining character (like COMBINING ACUTE ACCENT: 0x0301) :</span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; "><font class="Apple-style-span" face="Helvetica" size="3"><span class="Apple-style-span" style="font-size: 12px;"><br></span></font></div></div><div><div>test.c:4:9: error: universal character \u0301 is not valid in an identifier</div><div>test.c: In function ‘main’:</div><div>test.c:4: error: ‘hé’ undeclared (first use in this function)</div><div>test.c:4: error: (Each undeclared identifier is reported only once</div><div>test.c:4: error: for each function it appears in.)</div><div><br></div><div>Note that the error is correctly displayed anyway.</div><div><br></div><div><br></div></div><blockquote type="cite"><div>I've got some experience implementing all the above, so can give some<br>advice if necessary.<br><br>Neil.</div></blockquote><br></div><br></body></html>