[cfe-dev] Confusing comment on LexTokenInternal

[Chris Lattner]

On Jul 7, 2009, at 11:32 AM, AlisdairM(public) wrote:
>>> I want to kill tok::wide_string_literal and somehow stuff the
>>> encoding into tok::string_literal (char, char16_t, char32_t, wchar_t
>>> or u8 special. Options for other languages may be appropriate too).
>>> Any advice on how to approach this appreciated.
>>
>> Makes sense to me!  Do you actually need to encode this in the
>> *Token*?  Could you just have StringLiteralParser determine these
>> properties?
>
> I guess that makes sense.  The lexer doesn't want to attribute any  
> meaning to the prefix/suffix on the literal, merely find meaningful  
> bounds.  String literal concatenation happens in the parser, and  
> this is probably the first time we really care about representation.

Right.  It also uses the Literal stuff to parse it.

> So I guess the first step is to kill tok::wide_string_literal, kill  
> the Boolean flag to Lexer::LexStringLiteral, and carry the prefix  
> (in the token's SourceRange) through the APIs into StringParser.

Yep.

> Then perform a matching change for character literals.

Yep.

> Then we can look into adding Unicode character types, or recognising  
> suffices for user-defined literals as part of the same token.  Note:  
> A user-defined literal is effectively a disguised function call  
> syntax, although they might become more 'literal-like' when someone  
> (not me!) gets around to implementing constexpr.

sounds great!

>> I think that the main lexer should just get the full extent of the
>> token and store it in the token (as it does with all other tokens).
>> The string literal parser would then "relex" the contents when needed
>> to remove the delimiters etc.
>
> This means duplicating some work, but probably not too much as the  
> delimiters are limited to 16 chars max.  The premature optimiser in  
> me want to do the work (and maintain it!) once and no more, but I'm  
> not about to fight the data structures to make it happen - that is  
> rarely a good sign.
>
> If that sounds right, then it is time for me to stop talking and  
> start cleaning up some patches ;¬)

Sounds great, thanks!

-Chris