[cfe-dev] from source location to token?

[Chris Lattner]

On Nov 24, 2008, at 7:53 AM, Roberto Bagnara wrote:
>> It would be interesting to know a little bit more about what you are
>> doing. In general, re-lexing/re-parsing isn't very desirable (though
>> it may be appropriate if it's uncommon).
>
> Hi Steve,
>
> there are program analyses that, for instance:
>
> 1) need to reason on an exact representation of floating-point  
> literals
>    (any approximation may result into an unsound analysis);

Hi Roberto,

You don't need the original Token to get this.  Clang's source  
location information is so precise that (given a SourceLocation) you  
can arbitrarily re-lex a token later.  This property is actually  
inherent to how we handle SourceRange's: the source range points to  
the *start* of the first/end token in the range.  To get render  
through the end of the token, the diagnostics machinery re-lexes the  
token, which gives exactly the original spelling and length.

You can see this in action with the command line driver.  Consider  
this program:

struct s;
void foo(struct s *s) {
    *s + 0.12321e-42;
}

-fsyntax-only prints:

t.c:7:7: error: invalid operands to binary expression ('struct s' and  
'double')
    *s + 0.12321e-42;
    ~~ ^ ~~~~~~~~~~~

The way it gets the end of the fp literal is to relex the token with  
code in Lexer::MeasureTokenLength.

Given a SourceLocation and the length of the token (as returned by  
MeasureTokenLength) you can get the exact original spelling (including  
trigraphs and escaped newlines, beware).  The pointer to the start of  
the string is obtained with:

const char *StrData = SourceMgr.getCharacterData(Loc);

>
> 2) need to reason on the textual representation that was used in the
>    program also for integer literals (for example, there are coding
>    rules that forbid the use of octal constants: the analyzer should
>    flag their use in the source program).

Sure, Clang can handle this sort of thing with no problem.

-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20081124/a4406d8b/attachment.html>