[cfe-dev] Using the Lexer...

[Chris Lattner]

On Feb 11, 2009, at 10:11 AM, Alexei Svitkine wrote:

> I'm trying to use the Lexer, but am running into an issue. My code
> essentially looks like this:
>
> clang::Token Tok;
> clang::Lexer Lexer(...);
>
> Lexer->Lex(Tok);
> while (Tok.isNot(clang::tok::eof)) {
>   // logic based on Tok.is(...) checks, without touching Lexer or any
> other clang objects
>   Lexer->Lex(Tok);
> }
>
> This should work, right?

It depends on the "..." in the lexer set up.  If you want to just use  
the lexer without using a preprocessor then you need to set it up to  
lex in "raw" mode.  Otherwise it will crash at EOF and when trying to  
expand macros.  clang/Driver/DiagChecker.cpp has code that uses the  
raw lexer to pull out comments from a file, I'd use it as an example.

-Chris