[cfe-commits] r59724 - /cfe/trunk/lib/Lex/PTHLexer.cpp

[Chris Lattner]

On Nov 19, 2008, at 11:58 PM, Ted Kremenek wrote:
> URL: http://llvm.org/viewvc/llvm-project?rev=59724&view=rev
> Log:
> Rewrote PTHLexer::Lex by digging through the sources of Lexer  
> again.  Now we can do basic macro expansion using the PTHLexer.

Ok cool.

> +++ cfe/trunk/lib/Lex/PTHLexer.cpp Thu Nov 20 01:58:05 2008
> void PTHLexer::Lex(Token& Tok) {
> +LexNextToken:
> +  if (CurToken == LastToken) {
> +    if (ParsingPreprocessorDirective) {
>       ParsingPreprocessorDirective = false;
> +      Tok = Tokens[LastToken];
> +      Tok.setKind(tok::eom);
> +      MIOpt.ReadToken();

This code correctly matches what the Lexer does.  However, I'm not  
sure the lexer is right.  A \n isn't a non-whitespace token.  Does the  
lexer appear to work if you change:

       IsAtStartOfLine = true;

   // Update the location of token as well as BufferPtr.
   FormTokenWithChars(Result, CurPtr, Kind);

       Kind = tok::eom;
       break;

to:
       IsAtStartOfLine = true;

       // Update the location of token as well as BufferPtr.
       FormTokenWithChars(Result, CurPtr, tok::eom);
       return;

?

>   Tok = Tokens[CurToken];
>
> +  // Don't advance to the next token yet.  Check if we are at the
> +  // start of a new line and we're processing a directive.  If so, we
> +  // consume this token twice, once as an tok::eom.
> +  if (Tok.isAtStartOfLine() && ParsingPreprocessorDirective) {
> +    ParsingPreprocessorDirective = false;
> +    Tok.setKind(tok::eom);
>     MIOpt.ReadToken();
>     return;
>   }

Likewise.

-Chris