[cfe-dev] Using the Lexer...

[Alexei Svitkine]

I'm trying to use the Lexer, but am running into an issue. My code
essentially looks like this:

clang::Token Tok;
clang::Lexer Lexer(...);

Lexer->Lex(Tok);
while (Tok.isNot(clang::tok::eof)) {
   // logic based on Tok.is(...) checks, without touching Lexer or any
other clang objects
   Lexer->Lex(Tok);
}

This should work, right?

When I run this, it gets each token fine until the point where it's
supposed to get the EOF token. Instead, it just crashes with the
following info:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00000078
0x001fb455 in clang::Token::is (this=0x6c, K=clang::tok::eof) at Token.h:84
84	  bool is(tok::TokenKind K) const { return Kind == (unsigned) K; }

(gdb) bt
#0  0x001fb455 in clang::Token::is (this=0x6c, K=clang::tok::eof) at Token.h:84
#1  0x000b0c29 in clang::PTHLexer::getEOF (this=0x0, Tok=@0xbfffeec8)
at PTHLexer.cpp:177
#2  0x000adfdd in clang::Preprocessor::HandleEndOfFile
(this=0x1d04490, Result=@0xbfffeec8, isEndOfMacro=false) at
PPLexerChange.cpp:232
#3  0x00098d25 in clang::Lexer::LexEndOfFile (this=0xbfffece0,
Result=@0xbfffeec8, CurPtr=0x1d04267 "") at Lexer.cpp:1221
#4  0x0009a727 in clang::Lexer::LexTokenInternal (this=0xbfffece0,
Result=@0xbfffeec8) at Lexer.cpp:1302
#5  0x001d5758 in clang::Lexer::Lex (this=0xbfffece0,
Result=@0xbfffeec8) at Lexer.h:128

Is this a bug or am I doing something wrong?

-Alexei