[cfe-dev] Using the Lexer...

[Chris Lattner]

On Feb 11, 2009, at 6:01 PM, Alexei Svitkine wrote:

> I'm using a preprocessor and passing in the correct FileID from the
> SourceManager.
>
> The preprocessor is created in the same way that I make it as when I
> make it to compile and codegen the code (which works right).

Ok, if you're using a preprocessor and you want it to expand macros  
etc, then the preprocessor has to own the lexer.  You should not  
create a lexer on the stack, you should do what the clang driver does.

-Chris

>
>
> -Alexei
>
> On Wed, Feb 11, 2009 at 7:53 PM, Chris Lattner <clattner at apple.com>  
> wrote:
>>
>> On Feb 11, 2009, at 10:11 AM, Alexei Svitkine wrote:
>>
>>> I'm trying to use the Lexer, but am running into an issue. My code
>>> essentially looks like this:
>>>
>>> clang::Token Tok;
>>> clang::Lexer Lexer(...);
>>>
>>> Lexer->Lex(Tok);
>>> while (Tok.isNot(clang::tok::eof)) {
>>> // logic based on Tok.is(...) checks, without touching Lexer or any
>>> other clang objects
>>> Lexer->Lex(Tok);
>>> }
>>>
>>> This should work, right?
>>
>> It depends on the "..." in the lexer set up.  If you want to just  
>> use the
>> lexer without using a preprocessor then you need to set it up to  
>> lex in
>> "raw" mode.  Otherwise it will crash at EOF and when trying to expand
>> macros.  clang/Driver/DiagChecker.cpp has code that uses the raw  
>> lexer to
>> pull out comments from a file, I'd use it as an example.
>>
>> -Chris
>>
>>