[cfe-dev] End SourceLocation for multi-line Tokens?

[Chris Lattner]

On Apr 24, 2008, at 9:18 AM, Emerson Murphy-Hill wrote:

> On Apr 24, 2008, at 9:13 AM, Chris Lattner wrote:
>
>> On Apr 24, 2008, at 8:53 AM, Emerson Murphy-Hill wrote:
>>> How would one get the end of a Token if it spans several lines, such
>>> as a BCPL comment?  I can clearly get the start position and length,
>>> but the token may contain any number of line breaks. I suppose I  
>>> could
>>> use the preprocessor to get the text of the token, count the  
>>> number of
>>> line breaks, and calculate the end line number and column, but is
>>> there an easier way?
>>
>> Are you asking about something like:
>>
>>
>> foo\
>> bar??/
>> baz
>>
>> ?  If so, the source location for the token will point to the start  
>> of foo.  Lexer::getTokenLength will return the full extent of the  
>> token.  Adding it to the start location will give you a location  
>> that points to the space after the 'z'.
>>
>> -Chris
>
> Oops, sorry, no I misspoke - mean c-style comments:
>
> /*
> * Comment here
> */

Same thing works for those.  An example s HTMLRewriter.cpp,  
html::SyntaxHighlight:

     unsigned TokOffs = SourceMgr.getFullFilePos(Tok.getLocation());
     unsigned TokLen = Tok.getLength();
...
     case tok::comment:
       HighlightRange(RB, TokOffs, TokOffs+TokLen, BufferStart,
                      "<span class='comment'>", "</span>");

Note that the lexer doesn't return comments unless specifically in  
keep-comment-mode (which corresponds to -C when using -E mode in the  
driver).  See html::SyntaxHighlight as an example that cons's up a  
lexer to pull out comments on the fly.

-Chris