[cfe-dev] Using the Lexer...

[Alexei Svitkine]

Well obviously I wasn't using it right - I'm just saying it would be
helpful (for people just starting with the API) if this was detected
rather than waiting to crash.

Basically what was happening was I was creating the Lexer on the stack
and passing in the preprocessor as a parameter, rather than using the
Preprocessor to lex like the driver does.

Since EnterMainSourceFile() was never called, CurLexer was never set
in PPLexerChange.cpp, and when Preprocessor::HandleEndOfFile() was
called it assumed since CurLexer was NULL, that there was a
CurPTHLexer and tried to use that, resulting in a crash. This would
only happen when we got to the EOF since before that everything was
running fine.

I'm just suggesting that since such a use is clearly wrong, this
should be detected early on. Perhaps in the constructor of Lexer that
takes the Preprocessor parameter. Or when trying to get a token
without first calling EnterMainSourceFile(). Or some other even better
place.

-Alexei

On Wed, Feb 11, 2009 at 9:10 PM, Chris Lattner <clattner at apple.com> wrote:
>
> On Feb 11, 2009, at 6:09 PM, Alexei Svitkine wrote:
>
>>> Ok, if you're using a preprocessor and you want it to expand macros etc,
>>> then the preprocessor has to own the lexer.  You should not create a
>>> lexer
>>> on the stack, you should do what the clang driver does.
>>
>> Ah. Shouldn't there be an assert for this or something?
>
> You haven't shown me enough code to know what you're doing.
>
> -Chris
>