[cfe-commits] r61172 - /cfe/trunk/lib/Lex/PTHLexer.cpp

[Daniel Dunbar]

I'm not sure this is a good idea. At the least, I think it should be
behind a routine like PeekNextTokenKind(). The file format should be
kept separate from the internal implementation, and this is spreading
the parsing of the file format to new places.

In addition, what does this save? In the common case won't we end up
discarding 0 tokens? If the PTHLexer always had the next token
available, it seems to me that this means that in the common case we
would never be doing any unnecessary work. But I don't really grok
this code, so...

 - Daniel

On Wed, Dec 17, 2008 at 3:52 PM, Ted Kremenek <kremenek at apple.com> wrote:
> Author: kremenek
> Date: Wed Dec 17 17:52:11 2008
> New Revision: 61172
>
> URL: http://llvm.org/viewvc/llvm-project?rev=61172&view=rev
> Log:
> Rewrite PTHLexer::DiscardToEndOfLine() to not use GetToken and instead only read the bytes needed to determine if a token is not at the start of the line.
>
> Modified:
>    cfe/trunk/lib/Lex/PTHLexer.cpp
>
> Modified: cfe/trunk/lib/Lex/PTHLexer.cpp
> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Lex/PTHLexer.cpp?rev=61172&r1=61171&r2=61172&view=diff
>
> ==============================================================================
> --- cfe/trunk/lib/Lex/PTHLexer.cpp (original)
> +++ cfe/trunk/lib/Lex/PTHLexer.cpp Wed Dec 17 17:52:11 2008
> @@ -131,15 +131,24 @@
>   assert(ParsingPreprocessorDirective && ParsingFilename == false &&
>          "Must be in a preprocessing directive!");
>
> -  // Already at end-of-file?
> -  if (AtLastToken())
> -    return;
> -
> -  // Find the first token that is not the start of the *current* line.
> -  Token T;
> -  for (Lex(T); !AtLastToken(); Lex(T))
> -    if (GetToken().isAtStartOfLine())
> -      return;
> +  // Skip tokens by only peeking at their token kind and the flags.
> +  // We don't need to actually reconstruct full tokens from the token buffer.
> +  // This saves some copies and it also reduces IdentifierInfo* lookup.
> +  const char* p = CurPtr;
> +  while (1) {
> +    // Read the token kind.  Are we at the end of the file?
> +    tok::TokenKind x = (tok::TokenKind) (uint8_t) *p;
> +    if (x == tok::eof) break;
> +
> +    // Read the token flags.  Are we at the start of the next line?
> +    Token::TokenFlags y = (Token::TokenFlags) (uint8_t) p[1];
> +    if (y == Token::StartOfLine) break;
> +
> +    // Skip to the next token.
> +    p += DISK_TOKEN_SIZE;
> +  }
> +
> +  CurPtr = p;
>  }
>
>  //===----------------------------------------------------------------------===//
>
>
> _______________________________________________
> cfe-commits mailing list
> cfe-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-commits
>