[cfe-dev] Relexing more than one tokens
clattner at apple.com
Tue Jul 7 22:48:36 PDT 2009
On Jul 7, 2009, at 10:19 PM, Abramo Bagnara wrote:
>> In order to lex from "foo" to "baz", you need to know (e.g.) if bar
>> is a
>> macro that expands to zero (or many) tokens. From an arbitrary
>> point in
>> an ASTConsumer, you don't have this information, because the macros
>> could be undef'd etc.
> Taken for granted that currently from any SourceLocation I can obtain
> the related token, a possibility could be to have in class SLocEntry a
> reference to next token's SourceLocation in preprocessed stream. It
> should not be too hard to implement, but this means to add 32 bits to
> each SLocEntry and to keep all translation unit source locations in
We can't do that, SourceLocation has to stay 32-bits, it is very
>> However, not all hope is lost. It is very reasonable for an
>> to construct ASTs for a translation unit *AND* then preprocess the
>> file again to get the tokens in a big vector. Given that, you
>> could map
>> from the AST node to an index in the vector, then scan around in the
>> vector of tokens looking for what you want.
> How you'd map the AST node to the relexed tokens vector?
Just compare the source locations.
More information about the cfe-dev