[cfe-dev] Traversing the (macro-expanded) tokens that make up a Decl?

[Stephan Bergmann via cfe-dev]

On 03/03/2017 01:05 AM, James Dennett wrote:
> On Thu, Mar 2, 2017 at 1:10 AM, Stephan Bergmann via cfe-dev
> <cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>> wrote:
>
>     My problem is as follows:  I have a FunctionDecl d for which I know
>     that d->isConstexpr(), so somewhere in d->getSourceRange() the token
>     "constexpr" should appear.  Now, I want to know whether that
>     "constexpr" is from the expansion of a specific object-like macro
>
>       #define M constexpr
>
>     (whose use could be nested in some other macros, so it's important
>     here to look into macro invocations in general).
>
>     I'm trying to traverse d->getSourceRange() with
>     Lexer::MeasureTokenLength/SourceManager::getCharacterData to get at
>     a token's content, then SourceLocation::getLocWithOffset to get at
>     the next token.  Which works more-or-less well.  However:
>
>     * When d->getSourceRange() happens to start off within a macro
>     expansion, I could fall off its end before reaching the
>     declaration's end.  This one is easy, I can use
>     SourceManager::isAtEndOfImmediateMacroExpansion to climb back out of
>     the macro expansion and continue traversal one level further out.
>
>     * But when the traversal reaches a macro invocation, I get the token
>     of that invocation (multiple tokens in case of a function-like macro
>     with its parentheses and arguments), without a clue that it is a
>     macro and without a way to step into its expansion.  Am I missing
>     anything, or is there just no way to do that?

(Another issue to watch out for would be preprocessing directives, e.g., 
skipping over conditional groups when encountering an #else line.)

> You might try to make a lexer to Lex(token) for you until you've reached
> the end of the range, rather than navigating by source locations.

But I assume I'd need to re-lex all from the beginning, so that the 
lexer has an accurate idea of what macros are currently defined and 
whether we're in an #if group (so it wouldn't fail with an error if it 
encounters a---semingly unmatched---#endif line, if that actually is an 
issue).

> Though actually I think we should make the AST record the location of
> the `constexpr` keyword, so that applications don't have to jump through
> these hoops.

Yeah, I guess I understand now why new "get SourceLocation of ..." 
functions keep appearing in include/clang/AST/ as needed (e.g., 
FunctionDecl::getExceptionSpecSourceRange).  But would we want to track 
the locations of e.g. all specifiers in Decl instances?  (Or would that 
be considered making such instances "too fat"?)

Anyway, my immediate need for the original issue is meanwhile gone 
again, so I at least won't pursue this further for now.