[cfe-dev] Using the Lexer...

[Alexei Svitkine]

I'm using a preprocessor and passing in the correct FileID from the
SourceManager.

The preprocessor is created in the same way that I make it as when I
make it to compile and codegen the code (which works right).

-Alexei

On Wed, Feb 11, 2009 at 7:53 PM, Chris Lattner <clattner at apple.com> wrote:
>
> On Feb 11, 2009, at 10:11 AM, Alexei Svitkine wrote:
>
>> I'm trying to use the Lexer, but am running into an issue. My code
>> essentially looks like this:
>>
>> clang::Token Tok;
>> clang::Lexer Lexer(...);
>>
>> Lexer->Lex(Tok);
>> while (Tok.isNot(clang::tok::eof)) {
>>  // logic based on Tok.is(...) checks, without touching Lexer or any
>> other clang objects
>>  Lexer->Lex(Tok);
>> }
>>
>> This should work, right?
>
> It depends on the "..." in the lexer set up.  If you want to just use the
> lexer without using a preprocessor then you need to set it up to lex in
> "raw" mode.  Otherwise it will crash at EOF and when trying to expand
> macros.  clang/Driver/DiagChecker.cpp has code that uses the raw lexer to
> pull out comments from a file, I'd use it as an example.
>
> -Chris
>
>