<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Mar 30, 2012, at 8:06 AM, Sean Hunt wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div class="gmail_quote">On Sat, Dec 31, 2011 at 22:11, Sean Hunt <span dir="ltr"><<a href="mailto:scshunt@csclub.uwaterloo.ca">scshunt@csclub.uwaterloo.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hey folks,<br>
<br>
Attached is a proof of concept for the handling of UTF-8 in<br>
identifiers. Aside from the terrible isIdentifierBody function, which<br>
should be optimized where possible (possibly into a lookup table for<br>
the BMP, since that would be 8kb, and using the simple bitwise<br>
operation in there for other planes), I think the approach is the<br>
correct one. Given that this is sensitive code, however, I would like<br>
to ensure no one has any issues with this approach before I convert<br>
more of the lexer code over.<br>
<span class="HOEnZb"><font color="#888888"><br>
Sean<br>
</font></span></blockquote></div><br>This patch still applies reasonably cleanly; any feedback?<br></blockquote></div><br><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Menlo; ">+/// identifier, which is [a-zA-Z0-9_], or a Unicode character defined by</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Menlo; ">+/// Annex E of the C++ standard (note: pretty sure this is different in C).</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Menlo; ">+static inline bool isIdentifierBody(UTF32 c) {</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Menlo; ">+ if (c <= 127)</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Menlo; ">+ return (CharInfo[c] & (CHAR_LETTER|CHAR_NUMBER|CHAR_UNDER)) ? true : false;</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Menlo; ">+ else if (c & 0xffff0000)</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 11px/normal Menlo; ">+ return ~c & 0xfffd;</div></div><div><br></div><div>This should be split into two functions: isIdentifierBody() which just handles the <= 127 case, and the rest in a attribute(no_inline) function to handle the slowpath.</div><div><br></div><div>We don't want the slow path to prevent inlining of the fast path. If we have a macro for builtin_expect, it would be worth using it on the "c <= 127" branch.</div><div><br></div><div>I would also prefer the loop in <span class="Apple-style-span" style="font-family: Menlo; font-size: 11px; ">Lexer::LexIdentifier to be written something like this, which might make the idea above irrelevant:</span></div><div><span class="Apple-style-span" style="font-family: Menlo; font-size: 11px; "><br></span></div><div><span class="Apple-style-span" style="font-family: Menlo; font-size: 11px; "><br></span></div><div><span class="Apple-style-span" style="font-family: Menlo; font-size: 11px; ">do {</span></div><div><span class="Apple-style-span" style="font-family: Menlo; font-size: 11px; "> C = *CurPtr;</span></div><div><font class="Apple-style-span" face="Menlo"><span class="Apple-style-span" style="font-size: 11px;"> if (C > 127) goto UTFIdentifier;</span></font></div><div><font class="Apple-style-span" face="Menlo"><span class="Apple-style-span" style="font-size: 11px;"> ++CurPtr;</span></font></div><div><font class="Apple-style-span" face="Menlo"><span class="Apple-style-span" style="font-size: 11px;">} while (is</span></font><span class="Apple-style-span" style="font-family: Menlo; font-size: 11px; ">NonUTF</span><span class="Apple-style-span" style="font-family: Menlo; font-size: 11px; ">IdentifierBody(C))</span></div><div><span class="Apple-style-span" style="font-family: Menlo; font-size: 11px; "><br></span></div><div><font class="Apple-style-span" face="Menlo"><span class="Apple-style-span" style="font-size: 11px;">...</span></font></div><div><font class="Apple-style-span" face="Menlo"><span class="Apple-style-span" style="font-size: 11px;"><br></span></font></div><div><font class="Apple-style-span" face="Menlo"><span class="Apple-style-span" style="font-size: 11px;">UTFIdentifier:</span></font></div><div><font class="Apple-style-span" face="Menlo"><span class="Apple-style-span" style="font-size: 11px;"> ... handle the general case here.</span></font></div><div><font class="Apple-style-span" face="Menlo"><span class="Apple-style-span" style="font-size: 11px;"><br></span></font></div><div><font class="Apple-style-span" face="Menlo"><span class="Apple-style-span" style="font-size: 11px;">-Chris</span></font></div><div><span class="Apple-style-span" style="font-family: Menlo; font-size: 11px; "><br></span></div><div><br></div><div><br></div><div><br></div></body></html>