[cfe-dev] Preprocessor::LookNext

[Argiris Kirtzidis]

Chris Lattner wrote:
>>> +++ lib/Lex/PPLexerChange.cpp    (working copy)
>>> @@ -85,6 +85,10 @@
>>>   // size we new to a multiple of 16 tokens.  If the previous buffer 
>>> has space
>>>   // left, we can just grow it.  This means we only have to do the 
>>> new 1/16th as
>>>   // often.
>>> +
>>> +  // Optimized LookAhead(0) case.
>>> +  if (N == 0)
>>> +    return LookNext();
>>>
>>>   Token *LookaheadTokens = new Token[N+1];
>>>
>>> Is this correct for the LookAhead(1) case?
>>
>> I don't quite understand what you mean, LookAhead(1) isn't affected 
>> by this change, can you elaborate ?
>
> I haven't looked at the code, I just want to make sure that 
> LookAhead(1) or LookAhead(2) doesn't get off-by-one due to the peeked 
> token.

LookAhead isn't called recursively, it collects all tokens through 
Lex(), into the LookaheadTokens array.
If a peeked token exists, like when calling "LookAhead(0); 
LookAhead(1);", it goes like this:

LookAhead(0);

There's a peeked token now.

LookAhead(1);

-LookAhead consumes through Lex() the peeked token
-Peeked token is invalidated, Lex() will not use it again.
-LookAhead consumes one more token by 'normal' Lex().
-Previously peeked token plus one more enter the token stream.
-Previously peeked token will be the next available token.

So, everything works as expected :-)


-Argiris