[cfe-dev] Relexing more than one tokens

[Chris Lattner]

On Jul 6, 2009, at 4:59 AM, Abramo Bagnara wrote:

> Once obtained a SourceLocation (or a SourceRange) we'd like to relex  
> the
> preprocessed stream to check for the presence of some tokens.
>
> An example of use would be to check if the int type in an AST
> declaration was written with "signed" or not.
>
> We are able to relex a single token from a given SourceLocation, but  
> we
> haven't found a way to use a SourceRange to relex all the tokens
> included in the range.
>
> Is there a way?
>
> Do you have hints for alternative way to accomplish the same aim?

We don't have a great way to do this right now.  The basic problem is  
that you could have something like this:

   foo bar baz

In order to lex from "foo" to "baz", you need to know (e.g.) if bar is  
a macro that expands to zero (or many) tokens.  From an arbitrary  
point in an ASTConsumer, you don't have this information, because the  
macros could be undef'd etc.

However, not all hope is lost.  It is very reasonable for an  
ASTConsumer to construct ASTs for a translation unit *AND* then  
preprocess the whole file again to get the tokens in a big vector.   
Given that, you could map from the AST node to an index in the vector,  
then scan around in the vector of tokens looking for what you want.

This is the idea of how the TokenRewriter works: it assumes that the  
clients of the rewriter have an AST or something else, and it works on  
a big array of tokens that it separately preprocesses.

-Chris