[cfe-dev] Relexing more than one tokens

[Chris Lattner]

On Jul 7, 2009, at 11:07 PM, Abramo Bagnara wrote:
>> We can't do that, SourceLocation has to stay 32-bits, it is very  
>> pervasive.
>
> SourceLocation size would not change, the link to next SourceLocation
> would be added to SLocEntry (similarly to IncludeLoc and SpellingLoc).

That is just moving the cost to another place.  Adding 32-bits per  
token is too much.

>>>> However, not all hope is lost.  It is very reasonable for an  
>>>> ASTConsumer
>>>> to construct ASTs for a translation unit *AND* then preprocess  
>>>> the whole
>>>> file again to get the tokens in a big vector.  Given that, you  
>>>> could map
>>>> from the AST node to an index in the vector, then scan around in  
>>>> the
>>>> vector of tokens looking for what you want.
>>>
>>> How you'd map the AST node to the relexed tokens vector?
>>
>> Just compare the source locations.
>
> Are you meaning that preprocessing the whole translation unit again  
> I'd
> get the same SourceLocation opaque ID?

Hopefully.  If not, you can do a "deep comparison" along the lines of  
SourceManager::isBeforeInTranslationUnit.

-Chris