[cfe-dev] Keeping tokens in memory.

[Paolo Bolzoni]

I need to keep all the raw tokens representing a file in memory.
But I could not understand where clang memorizes spelling and flags.
Just filling a vector of clang::Token does not work, this is what I am doing
(scrap):

---->
void tokenize_whole_file(std::vector<clang::Token>& raw_tokens,
                         clang::FileID fileID) {

  clang::Lexer raw_lexer(fileID, source_manager(), lang_info());
  clang::Token current_token;

  raw_tokens.clear();
  raw_tokens.reserve(0x4000);

  raw_lexer.SetCommentRetentionState(true);
  raw_lexer.SetKeepWhitespaceMode(true);

  while (raw_lexer.LexFromRawLexer(current_token),
         current_token.isNot(clang::tok::eof)) {
    raw_tokens.push_back()
  }
}
----<
There is also a preprocessor correlated to the source manager and lang_info,
but later when I try to get the spelling or the flags of any token in the
raw_tokens vector it does not work.

I tried giving the Lexer a static duration, but it does not help.

pb