[cfe-dev] Confusing comment on LexTokenInternal

[Chris Lattner]

On Jul 8, 2009, at 11:36 AM, AlisdairM(public) wrote:

>> Makes sense to me!  Do you actually need to encode this in the
>> *Token*?  Could you just have StringLiteralParser determine these
>> properties?
>
> OK, tried it, and time to scratch that plan already!
> The problem is not that string_literal cannot handle the  
> wide_string_literal cases, that was easy to fix up.  However, there  
> are a few places in the grammar that require string literal be  
> exactly that - a narrow string literal.  Examples are #include  
> "myfile" and extern "C".
>
> Now I could try and stuff a flag into the token to indicate it truly  
> is a narrow string literal - but we already have that effect with  
> the two separate tokens.  That seems to be working and is quite well  
> tested by now so I think we should keep this in place.
>
> The new plan is to repurpose wide_string_literal to cover any  
> annotated string literal i.e. with any prefix or suffix.   
> 'Annotated' seems to have other connotations though so I'm looking  
> for a better term.  In the meantime I'll put the foundation in for  
> wide_string_literal to handle the 19 other cases that string_literal  
> does not.

Sounds like a good compromise to me!  How about string_literal and  
complex_string_literal or something like that?

-Chris