<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div>On Nov 24, 2008, at 7:53 AM, Roberto Bagnara wrote:</div><blockquote type="cite"><div><blockquote type="cite">It would be interesting to know a little bit more about what you are <br></blockquote><blockquote type="cite">doing. In general, re-lexing/re-parsing isn't very desirable (though <br></blockquote><blockquote type="cite">it may be appropriate if it's uncommon).<br></blockquote><br>Hi Steve,<br><br>there are program analyses that, for instance:<br><br>1) need to reason on an exact representation of floating-point literals<br> (any approximation may result into an unsound analysis);</div></blockquote><div><br></div><div>Hi Roberto,</div><div><br></div><div>You don't need the original Token to get this. Clang's source location information is so precise that (given a SourceLocation) you can arbitrarily re-lex a token later. This property is actually inherent to how we handle SourceRange's: the source range points to the *start* of the first/end token in the range. To get render through the end of the token, the diagnostics machinery re-lexes the token, which gives exactly the original spelling and length.</div><div><br></div><div>You can see this in action with the command line driver. Consider this program:</div><div><br></div><div><div>struct s;</div><div>void foo(struct s *s) {</div><div> *s + 0.12321e-42;</div><div>}</div><br></div><div>-fsyntax-only prints:</div><div><br></div><div><div><font class="Apple-style-span" face="Courier" size="2"><span class="Apple-style-span" style="font-size: 10px;">t.c:7:7: error: invalid operands to binary expression ('struct s' and 'double')</span></font></div><div><font class="Apple-style-span" face="Courier" size="2"><span class="Apple-style-span" style="font-size: 10px;"> *s + 0.12321e-42;</span></font></div><div><font class="Apple-style-span" face="Courier" size="2"><span class="Apple-style-span" style="font-size: 10px;"> ~~ ^ ~~~~~~~~~~~</span></font></div><br></div><div>The way it gets the end of the fp literal is to relex the token with code in <span class="Apple-style-span" style="color: rgb(38, 71, 75); font-family: Monaco; font-size: 10px; "><span style="color: #3f6e74">Lexer</span><span style="color: #000000">::</span>MeasureTokenLength</span>.</div><div><br></div><div>Given a SourceLocation and the length of the token (as returned by MeasureTokenLength) you can get the exact original spelling (including trigraphs and escaped newlines, beware). The pointer to the start of the string is obtained with:</div><div><br></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 10px/normal Monaco; color: rgb(38, 71, 75); "><span style="color: #aa0d91">const</span><span style="color: #000000"> </span><span style="color: #aa0d91">char</span><span style="color: #000000"> *</span><span style="color: #3f6e74">StrData</span><span style="color: #000000"> = </span><span style="color: #3f6e74">SourceMgr</span><span style="color: #000000">.</span>getCharacterData<span style="color: #000000">(</span><span style="color: #3f6e74">Loc</span><span style="color: #000000">);</span></div><div><font class="Apple-style-span" face="Monaco" size="2"><span class="Apple-style-span" style="font-size: 10px;"><br></span></font></div></div><blockquote type="cite"><div><br>2) need to reason on the textual representation that was used in the<br> program also for integer literals (for example, there are coding<br> rules that forbid the use of octal constants: the analyzer should<br> flag their use in the source program).<br></div></blockquote></div><br><div>Sure, Clang can handle this sort of thing with no problem.</div><div><br></div><div>-Chris</div></body></html>