[cfe-dev] Confusing comment on LexTokenInternal
clattner at apple.com
Tue Jul 7 14:02:20 PDT 2009
On Jul 7, 2009, at 11:32 AM, AlisdairM(public) wrote:
>>> I want to kill tok::wide_string_literal and somehow stuff the
>>> encoding into tok::string_literal (char, char16_t, char32_t, wchar_t
>>> or u8 special. Options for other languages may be appropriate too).
>>> Any advice on how to approach this appreciated.
>> Makes sense to me! Do you actually need to encode this in the
>> *Token*? Could you just have StringLiteralParser determine these
> I guess that makes sense. The lexer doesn't want to attribute any
> meaning to the prefix/suffix on the literal, merely find meaningful
> bounds. String literal concatenation happens in the parser, and
> this is probably the first time we really care about representation.
Right. It also uses the Literal stuff to parse it.
> So I guess the first step is to kill tok::wide_string_literal, kill
> the Boolean flag to Lexer::LexStringLiteral, and carry the prefix
> (in the token's SourceRange) through the APIs into StringParser.
> Then perform a matching change for character literals.
> Then we can look into adding Unicode character types, or recognising
> suffices for user-defined literals as part of the same token. Note:
> A user-defined literal is effectively a disguised function call
> syntax, although they might become more 'literal-like' when someone
> (not me!) gets around to implementing constexpr.
>> I think that the main lexer should just get the full extent of the
>> token and store it in the token (as it does with all other tokens).
>> The string literal parser would then "relex" the contents when needed
>> to remove the delimiters etc.
> This means duplicating some work, but probably not too much as the
> delimiters are limited to 16 chars max. The premature optimiser in
> me want to do the work (and maintain it!) once and no more, but I'm
> not about to fight the data structures to make it happen - that is
> rarely a good sign.
> If that sounds right, then it is time for me to stop talking and
> start cleaning up some patches ;¬)
Sounds great, thanks!
More information about the cfe-dev