[cfe-dev] Token lookahead without the preprocessor

[Jordan Rose]

On Jun 26, 2012, at 12:26 AM, Richard Smith wrote:

> Hi Jordy,
> 
> This is PR10101, and was fixed in r145372, but the fix was backed out due to the #pragma weak (and, at the time, #pragma visibility) issue. The problem is that the implementation of this pragma is incorrect, since it takes effect when the pragma is lexed, rather than when it is parsed, and the point at which the pragma occurs has a semantic impact. We shouldn't be hacking around that in the parser by avoiding lookahead; the right fix is for the lexer to produce an annotation token when it encounters such a pragma, as it does for #pragma visibility, #pragma pack, and #pragma unused.
> 
> This issue also makes our parsing of "#pragma weak" accept code which GCC rejects (though I'm hesitant to call it an accepts-invalid since I can't find a precise spec for this pragma): GCC (as far as I can determine) only accepts the pragma in places where it would parse a declaration.
> 
> Incidentally, I wonder whether it'd make sense to provide a more general framework for such cases, rather than adding ad-hoc pragma annotations. Perhaps, for all pragmas which can only appear in specific places in the grammar, we could lex them as a tok::annot_pragma followed by the tokens in the pragma and an tok::eod, and perform the pragma parsing in the parser.
> 
> Richard

Whoops, guess I should have been more proactive in looking for a PR. (I searched "semi" and didn't see it turn up.) Or tracking cfe-commits more closely.

I'd be fine with implementing tok::annot_pragma, but since I'm fairly new to Lex/Parse I'll wait to see if someone else wants to weigh in first.

Thanks for giving me the right context for this,
Jordan