[LLVMbugs] [Bug 16862] New: EnterTokenStream after lookahead creates broken lexer state

[bugzilla-daemon at llvm.org]

http://llvm.org/bugs/show_bug.cgi?id=16862

            Bug ID: 16862
           Summary: EnterTokenStream after lookahead creates broken lexer
                    state
           Product: clang
           Version: trunk
          Hardware: PC
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P
         Component: Frontend
          Assignee: unassignedclangbugs at nondot.org
          Reporter: richard-llvm at metafoo.co.uk
                CC: llvmbugs at cs.uiuc.edu
            Blocks: 10101
    Classification: Unclassified

If Preprocessor::EnterTokenStream is called after we've performed lookahead and
cached one or more tokens, and then we try to perform lookahead while
processing the entered tokens, we get incorrect tokens back. In detail:

1) We add tokens to CachedTokens.
2) We add a TokenLexer layer to hold the entered tokens.
3) We try to perform lookahead and determine that the tokens we added in step
(1) are still the appropriate lookahead.

We seem to be on the very edge of this being a problem. We perform arbitrary
parsing (including lookahead) on entered tokens in a few contexts (late-parsed
portions of class bodies in C++, late-parsed attributes, ...), but we happen to
not perform lookahead past the } of a class body or ; of a function, so we get
away with it.

Another case where we nearly have a problem is #pragma handling: if we perform
lookahead across a #pragma, the lexing of the pragma may itself call
EnterTokenStream internally, and then parse some tokens. If any of that parsing
performs lookahead, it will fail (and receive the token we're in the middle of
trying to lex).

This blocks PR10101, where we want to perform lookahead past the ; of a
function. (This fails if the function declaration has a late-parsed attribute,
because we parse it and re-enter its tokens into the token stream after
performing the lookahead past the ';'.)

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20130812/7f313261/attachment.html>