[cfe-dev] Source rewrite

[Abramo Bagnara]

>> As I write above, what I need is that the AST built from:
>>
>> int p() {
>>  int a = 3;
>>  /* #instrument(a)# */
>> }
>>
>> is as if the source read was:
>>
>> int p() {
>>  int a = 3;
>>  instrument(a);
>> }
> 
> 
> Oh, interesting. You'll probably need to each the preprocessor how to
> parse inside these comments. One option might be to treat such comments
> similarly to macro expansion, so that processing the comment
> 
>   /* #instrument(a)# */
> 
> consumes the comment and then pushes a new lexer that will point into a
> buffer containing
> 
>   instrument(a)
> 
> just like processing
> 
>   FOO
> 
> where there is a macro definition
> 
>   #define FOO instrument(a)
> 
> will create a new lexer pointing into a buffer containing
> 
>   instrument(a)

I've done it and it "almost" works...

The problem I see is that the pushed TokenLexer is used only *after* the
first token after the comment is Lexed and not just after the skipped
comment (because usually the comment is not a token to return).

This means that:

int x /* = 0 */;
int z;

is lexed as:

int x; = 0 int z;

instead of the wished way.

Now I'm stuck...

There is a way to cope with that?

My CommentHandler is written in this way:

void Comment_Converter::HandleComment(clang::Preprocessor &PP,
                                      clang::SourceRange Comment) {
  const clang::SourceManager &sm = PP.getSourceManager();
  const clang::LangOptions &lo = PP.getLangOptions();
  clang::SourceLocation begin = Comment.getBegin();
  clang::FileID fid = sm.getFileID(begin);
  const char* start = sm.getCharacterData(begin);
  const char* end = sm.getCharacterData(Comment.getEnd());
  if (start[1] == '*')
    end -= 2;
  start += 2;
  char saved = *end;
  *const_cast<char*>(end) = 0;
  clang::Lexer lexer(sm.getLocForStartOfFile(fid), lo,
                     sm.getBufferData(fid).first,
                     start, end);
  static std::vector<clang::Token> tokens;
  tokens.clear();
  clang::Token tok;
  while (1) {
    lexer.LexFromRawLexer(tok);
    if (tok.is(clang::tok::eof))
      break;
    if (tok.is(clang::tok::identifier))
      tok.setKind(PP.LookUpIdentifierInfo(tok)->getTokenID());
    tokens.push_back(tok);
  }
  *const_cast<char*>(end) = saved;
  PP.EnterTokenStream(tokens.data(), tokens.size(), false, false);
}