[cfe-commits] r61172 - /cfe/trunk/lib/Lex/PTHLexer.cpp

[Chris Lattner]

On Dec 17, 2008, at 4:09 PM, Daniel Dunbar wrote:

> I'm not sure this is a good idea. At the least, I think it should be
> behind a routine like PeekNextTokenKind(). The file format should be
> kept separate from the internal implementation, and this is spreading
> the parsing of the file format to new places.

I agree that factoring this better would be nice.

> In addition, what does this save? In the common case won't we end up
> discarding 0 tokens? If the PTHLexer always had the next token
> available, it seems to me that this means that in the common case we
> would never be doing any unnecessary work. But I don't really grok
> this code, so...

It saves identifier lookup, and reduce dependencies on GetToken.  Ted  
and I discussed that GetToken and the local one-token cache should go  
away for perf reasons.  The PTH generator should compute and insert an  
EOM token explicitly in the stream where needed.

-Chris