[cfe-dev] Advanced Rewriting

Jonas Toth via cfe-dev cfe-dev at lists.llvm.org
Mon Jul 16 10:06:25 PDT 2018


Hi Rafael,

wouldn't your usecase be a task for clang-refactor?

Best,  Jonas


Am 16.07.2018 um 17:08 schrieb Rafael·Stahl via cfe-dev:
> Hey everyone
>
> The rewriting API of Clang operates on the source code in textual
> form. The user can use AST nodes to figure out what to replace, but in
> the end he has to remove and insert snippets in a linear piece of text.
>
> This is very inconvenient when it is required to restructure and nest
> replacements. The involvement of macros makes a manual process even
> more difficult. See some recent threads expressing difficulty with the
> API [1][2].
>
> What do I mean by "nested replacements"? For example in the following:
>
>     int i = x + s->a;
>
> I would want to replace the BinaryOperator with a function call and
> the MemberExpr with some constant:
>
>     int i = Addition(x, 7);
>
> When keeping the two replacement rules independent of each other,
> achieving this with the current API is extremely difficult. More so
> when macros are involved.
>
> I am proposing some kind of helper that aims to solve these issues by
> providing an interface that offers to directly replace AST nodes and a
> mechanism to nest AST node replacements - without having to worry
> about macros.
>
> Potential usage:
>
> - Develop a class that derives from StmtToRewrite to define how
> replacements should happen:
>
>     class RewriteAdds : public cu::StmtToRewrite
>     {
>     public:
>         std::string makeReplaceStr() const override
>         {
>             auto binOp = dyn_cast<BinaryOperator>(replaceS);
>             return "Addition(" +
> getMgr()->getReplaced(binOp->getLHS()).strToInsert + ", " +
> getMgr()->getReplaced(binOp->getRHS()).strToInsert + ")";
>         }
>     };
>
>     class RewriteMembs : public cu::StmtToRewrite
>     {
>     public:
>         std::string makeReplaceStr() const override
>         {
>             return "7";
>         }
>     };
>
> - Construct a RewriteManager:
>
>     cu::RewriteManager mgr(ACtx, PP);
>
> - Add rewriting operations to the manager:
>
>     // std::vector<const Stmt *> AddStmts = /* matched from
> binaryOperator() with plus */
>     // std::vector<const Stmt *> MembStmts = /* matched from
> memberExpr() */
>     for (const auto &S : AddStmts) mgr.registerStmt<RewriteAdds>(S);
>     for (const auto &S : MembStmts) mgr.registerStmt<RewriteMembs>(S);
>
> - Retrieve and apply the results:
>
>     clang::Rewriter rewriter(SM, LangOpts);
>     for (const auto &r : mgr.getReplacements()) {
>         rewriter.RemoveText(r.rangeToRemove);
>         rewriter.InsertText(r.rangeToRemove.getBegin(), r.strToInsert);
>     }
>
>
> At the end of this mail is my low quality code that kind-of implements
> this. TLDR:
>
> - Build a hierarchy of stmts to replace and keep track of which
> replacements must be combined
> - Move further up in the AST if these replacements are inside a macro
> - Recursively lex the file and look for replacements outside-in by
> spelling locations. Expand any macros that are encountered during
> this. The re-lexing idea is based on the hint in [3].
>
> The code has the following shortcomings:
>
> - I do not know how to distinguish macro argument expansions within
> macros. For example in "#define FOO(a) a + a" the two "a"s expand to
> different AST nodes that could be replaced with different rules. This
> is an important issue, because it can lead to completely broken code
> with nesting.
> - Limited to Stmts, when Decls should be supported too.
> - Very un-optimized with lexing the entire source file many times.
> Easy to solve, but didn't want to raise the complexity further for now.
> - Could keep written code more clean by only expanding macros if
> required. For example not required if just a macro arg is replaced and
> all expansions would be the same.
>
>
> I am very interested in your general thoughts. I'm not very
> experienced with clang, but this was my vision how I would want to do
> replacements. Are you interested in getting this into clang? I would
> need help with ironing out the remaining issues.
>
> -Rafael
>
>
> [1] http://lists.llvm.org/pipermail/cfe-dev/2018-July/058430.html
> [2] http://lists.llvm.org/pipermail/cfe-dev/2018-June/058213.html
> [3] http://lists.llvm.org/pipermail/cfe-dev/2017-August/055079.html
>
>
>
> ----------------------------------------
>
> RewriteManager.h
>
> ----------------------------------------
>
> #ifndef CLANGUTIL_REWRITEMANAGER_H
> #define CLANGUTIL_REWRITEMANAGER_H
>
> #include "ClangUtil/SourceRangeLess.h"
> #include "make_unique.h"
> #include "clang/AST/AST.h"
> #include <vector>
> #include <map>
>
>
> // TODO extend to decls
>
>
> namespace cu
> {
> // Represents a statement in the original AST that should be
> rewritten. To implement recursive replacements, call
> // getMgr()->getReplaced() on any AST node within the makeReplaceStr
> callback.
> class StmtToRewrite
> {
>     friend class RewriteManager;
>
> public:
>     // Returns the enclosing RewriteManager.
>     class RewriteManager *getMgr() const;
>     // Override this to build a replacement string. Implement
> recursive replacements with RewriteManager::getReplaced.
>     virtual std::string makeReplaceStr() const = 0;
>
>     // The statement to replace.
>     const clang::Stmt *replaceS = nullptr;
>
> private:
>     RewriteManager *m_mgr;
> };
>
> struct RewriteOperation
> {
>     clang::SourceRange rangeToRemove;
>     std::string strToInsert;
> };
>
> // A class for managing replacements of AST nodes. It allows to
> specifically target AST nodes instead of raw source
> // locations to enable easy replacements involving macros and nested
> replacements.
> // For extended documentation see: doc/rewriting.md
> class RewriteManager
> {
> public:
>     RewriteManager(clang::ASTContext &ACtx, clang::Preprocessor &PP);
>
>     clang::ASTContext &getACtx() const { return ACtx; }
>
>     // Registers a StmtToRewrite for use with getReplacements. Call
> this on all
>     // statements that should be rewritten before calling any
> rewriting functions.
>     void registerStmt(std::unique_ptr<StmtToRewrite> S);
>
>     // Helper for constructing the custom type from a Stmt.
>     template <typename T, typename... Args>
>     void registerStmt(const clang::Stmt *S, Args... args)
>     {
>         auto p = std::make_unique<T>(std::forward<Args>(args)...);
>         p->replaceS = S;
>         registerStmt(std::move(p));
>     }
>
>     // Get the full replacement of an AST node. Note that this
> function removes any replaced statements from the work
>     // list, so calling it twice will only replace the first time.
>     RewriteOperation getReplaced(const clang::Stmt *S);
>     // Get all replacements. These may be fewer than the requested
> ones because of nesting.
>     std::vector<RewriteOperation> getReplacements();
>
> private:
>     std::string getExpandedCode(const clang::Stmt *toReplaceS);
>
> private:
>     clang::ASTContext &ACtx;
>     const clang::LangOptions &LangOpts;
>     clang::SourceManager &SM;
>     clang::Preprocessor &PP;
>
>     // Manages the pending replacements.
>     class WorkList
>     {
>     public:
>         typedef std::map<clang::SourceRange, std::vector<const
> StmtToRewrite *>> RangeToRepMap;
>
>         WorkList(clang::ASTContext &ACtx, clang::SourceManager &SM);
>
>         bool isStmtPending(const clang::Stmt *S) const;
>         void addStmt(std::unique_ptr<StmtToRewrite> S);
>         const RangeToRepMap &getRangeToReplacementsMap() const;
>         std::vector<const StmtToRewrite *> getSortedReplacements() const;
>         void markDone(const StmtToRewrite *S);
>         void cleanup();
>
>     private:
>         clang::ASTContext &ACtx;
>         clang::SourceManager &SM;
>         std::vector<std::unique_ptr<StmtToRewrite>> m_pending;
>         std::vector<std::unique_ptr<StmtToRewrite>> m_done;
>         RangeToRepMap m_rangeToReplacements;
>     };
>
>     WorkList m_workList;
> };
>
> } // namespace cu
>
> #endif
>
>
>
> ----------------------------------------
>
> RewriteManager.cpp
>
> ----------------------------------------
>
> #include "ClangUtil/RewriteManager.h"
> #include "ClangUtil/ASTUtil.h"
> #include "clang/Lex/Lexer.h"
> #include "clang/Lex/Preprocessor.h"
> #include "clang/Lex/PreprocessorOptions.h"
> #include "clang/Lex/TokenConcatenation.h"
> #include "clang/Lex/MacroArgs.h"
>
>
> using namespace cu;
>
>
> // Returns a Stmt that is the first parent of startS whose expansion
> range is within the given range.
> static const clang::Stmt *GetFullMacroStmt(clang::SourceRange range,
> const clang::Stmt *startS, clang::ASTContext &ACtx)
> {
>     auto &SM = ACtx.getSourceManager();
>
>     // Walk the tree upwards until ST does no longer expand to within
> range.
>     const clang::Stmt *ST = startS;
>     while (true)
>     {
>         const auto &parents = ACtx.getParents(*ST);
>         if (parents.empty())
>         {
>             break;
>         }
>         auto childS = ST;
>         ST = parents[0].get<clang::Stmt>();
>         if (!ST)
>         {
>             if (auto D = parents[0].get<clang::Decl>())
>             {
>                 const auto &parentsD = ACtx.getParents(*D);
>                 if (parentsD.empty())
>                 {
>                     break;
>                 }
>                 ST = parentsD[0].get<clang::Stmt>();
>                 if (!ST)
>                 {
>                     break;
>                 }
>             }
>             else
>             {
>                 break;
>             }
>         }
>
>         auto exLocS = SM.getExpansionLoc(ST->getLocStart());
>         auto exLocE = SM.getExpansionLoc(ST->getLocEnd());
>         if (SM.isBeforeInTranslationUnit(exLocS, range.getBegin()) ||
>             SM.isBeforeInTranslationUnit(range.getEnd(), exLocE))
>         {
>             return childS;
>         }
>     }
>
>     return nullptr;
> }
>
>
> RewriteManager *StmtToRewrite::getMgr() const
> {
>     return m_mgr;
> }
>
>
> RewriteManager::WorkList::WorkList(clang::ASTContext &ACtx,
> clang::SourceManager &SM) : ACtx(ACtx), SM(SM) {}
> bool RewriteManager::WorkList::isStmtPending(const clang::Stmt *S) const
> {
>     for (const auto &r : m_pending)
>     {
>         if (r->replaceS == S)
>         {
>             return true;
>         }
>     }
>     return false;
> }
> void RewriteManager::WorkList::addStmt(std::unique_ptr<StmtToRewrite> S)
> {
>     // Use the expansion range for maximal replacement flexibility in
> macros.
>     auto replaceRange =
> SM.getExpansionRange(S->replaceS->getSourceRange());
>
>     // TODO not quite correct.
>     /*auto sortRanges = [&](std::vector<const StmtToRewrite *> &vec) {
>         std::sort(vec.begin(), vec.end(), [&](const StmtToRewrite
> *lhs, const StmtToRewrite *rhs) {
>             auto lhsRange =
> SM.getExpansionRange(lhs->replaceS->getSourceRange());
>             auto rhsRange =
> SM.getExpansionRange(rhs->replaceS->getSourceRange());
>             return IsContained(rhsRange, lhsRange, SM);
>         });
>     };*/
>
>     // Establish hierarchical relation between all ranges.
>     bool found = false;
>     // First, check if this range is within one we already have.
>     for (auto &r : m_rangeToReplacements)
>     {
>         if (IsContained(replaceRange, r.first, SM))
>         {
>             // Insert in a sorted order.
>             for (auto it = r.second.begin(); it != r.second.end(); ++it)
>             {
>                 //auto testRange =
> SM.getExpansionRange((*it)->replaceS->getSourceRange());
>                 // if (IsContained(testRange, replaceRange, SM))
>                 if (IsParent(S->replaceS, (*it)->replaceS, ACtx))
>                 {
>                     r.second.insert(it, S.get());
>                     found = true;
>                     break;
>                 }
>             }
>             if (!found)
>             {
>                 r.second.push_back(S.get());
>                 found = true;
>             }
>             break;
>         }
>     }
>     // Not within existing range, add as new top-level range.
>     if (!found)
>     {
>         // Check if any existing ranges are contained within the new one.
>         std::vector<const StmtToRewrite *> moveThese;
>         auto it = m_rangeToReplacements.begin();
>         while (it != m_rangeToReplacements.end())
>         {
>             if (IsContained(it->first, replaceRange, SM))
>             {
>                 moveThese.insert(moveThese.end(), it->second.begin(),
> it->second.end());
>                 it = m_rangeToReplacements.erase(it);
>             }
>             else
>             {
>                 ++it;
>             }
>         }
>         auto &accesses = m_rangeToReplacements[replaceRange];
>         // The order is important here. We want the first element to
> be the one that spans the full range.
>         accesses.push_back(S.get());
>         // TODO sort "moveThese".
>         accesses.insert(accesses.end(), moveThese.begin(),
> moveThese.end());
>     }
>
>     int count = 0;
>     for (const auto &r : m_rangeToReplacements)
>     {
>         printf("range %i\n", count++);
>         for (const auto &a : r.second)
>         {
>             printf("replacement:\n");
>             a->replaceS->dump();
>         }
>     }
>
>     m_pending.push_back(std::move(S));
> }
> const RewriteManager::WorkList::RangeToRepMap
> &RewriteManager::WorkList::getRangeToReplacementsMap() const
> {
>     return m_rangeToReplacements;
> }
> std::vector<const StmtToRewrite *>
> RewriteManager::WorkList::getSortedReplacements() const
> {
>     std::vector<const StmtToRewrite *> result;
>     for (auto &r : m_rangeToReplacements)
>     {
>         result.insert(result.end(), r.second.begin(), r.second.end());
>     }
>     return result;
> }
> void RewriteManager::WorkList::markDone(const StmtToRewrite *S)
> {
>     // Remove from hierarchy.
>     for (auto &r : m_rangeToReplacements)
>     {
>         r.second.erase(std::remove(r.second.begin(), r.second.end(),
> S), r.second.end());
>     }
>
>     // Move from pending to done list.
>     auto it = std::find_if(m_pending.begin(), m_pending.end(),
>                            [&](const std::unique_ptr<StmtToRewrite>
> &rep) { return rep.get() == S; });
>     if (it == m_pending.end())
>     {
>         throw std::runtime_error("Did not find replacement to mark as
> done");
>     }
>     m_done.push_back(std::move(*it));
>     m_pending.erase(it);
> }
> void RewriteManager::WorkList::cleanup()
> {
>     m_done.clear();
> }
>
>
> RewriteManager::RewriteManager(clang::ASTContext &ACtx,
> clang::Preprocessor &PP)
>     : ACtx(ACtx), LangOpts(ACtx.getLangOpts()),
> SM(ACtx.getSourceManager()), PP(PP), m_workList(ACtx, SM)
> {
> }
>
> void RewriteManager::registerStmt(std::unique_ptr<StmtToRewrite> S)
> {
>     if (!S->replaceS)
>     {
>         throw std::runtime_error("Must set replaceS");
>     }
>
>     if (m_workList.isStmtPending(S->replaceS))
>     {
>         throw std::runtime_error("This Stmt will already be replaced");
>     }
>
>     S->m_mgr = this;
>     m_workList.addStmt(std::move(S));
> }
>
> RewriteOperation RewriteManager::getReplaced(const clang::Stmt *S)
> {
>     auto range = SM.getExpansionRange(S->getSourceRange());
>     return { range, getExpandedCode(S) };
> }
>
> std::vector<RewriteOperation> RewriteManager::getReplacements()
> {
>     std::vector<RewriteOperation> results;
>
>     for (auto &rangeAndAccesses : m_workList.getRangeToReplacementsMap())
>     {
>         auto &range = rangeAndAccesses.first;
>         auto &accesses = rangeAndAccesses.second;
>
>         // Cannot replace something inside a macro because it would
> replace all expansions instead of just the selected
>         // AST node. So in a first step, get an enclosing statement
> that is no longer inside a macro.
>         // TODO we could keep the original code more clean by not
> expanding macro args if the whole expansion does not
>         // contain the macro arg more than once.
>         auto macroS = GetFullMacroStmt(range, accesses[0]->replaceS,
> ACtx);
>
>         results.push_back(getReplaced(macroS));
>
>         // TODO we could run clang-format on the replacements. this
> would especially benefit long macro expansions.
>     }
>
>     m_workList.cleanup();
>
>     return results;
> }
>
> std::string RewriteManager::getExpandedCode(const clang::Stmt
> *toReplaceS)
> {
>     // TODO performance optimization. this is parsing way more than
> required.
>
>     using namespace clang;
>
>     printf("getExpandedCode:\n");
>     toReplaceS->dump();
>
>     std::string out;
>
>     auto toReplaceExpStart =
> SM.getExpansionLoc(toReplaceS->getLocStart());
>     auto toReplaceExpEnd = SM.getExpansionLoc(toReplaceS->getLocEnd());
>     auto toReplaceSpellStart =
> SM.getSpellingLoc(toReplaceS->getLocStart());
>     auto toReplaceSpellEnd = SM.getSpellingLoc(toReplaceS->getLocEnd());
>
>     auto FID =
> SM.getFileID(SM.getExpansionLoc(toReplaceS->getLocStart()));
>
>     // The following is inspired by:
> clang/Rewrite/HTMLRewrite.cpp:HighlightMacros
>
>     // Re-lex the raw token stream into a token buffer.
>     std::vector<Token> TokenStream;
>
>     const llvm::MemoryBuffer *FromFile = SM.getBuffer(FID);
>     Lexer L(FID, FromFile, SM, PP.getLangOpts());
>
>     // Lex all the tokens in raw mode, to avoid entering #includes or
> expanding
>     // macros.
>     while (1)
>     {
>         Token Tok;
>         L.LexFromRawLexer(Tok);
>
>         // If this is a # at the start of a line, discard it from the
> token stream.
>         // We don't want the re-preprocess step to see #defines,
> #includes or other
>         // preprocessor directives.
>         if (Tok.is(tok::hash) && Tok.isAtStartOfLine())
>             continue;
>
>         // If this is a ## token, change its kind to unknown so that
> repreprocessing
>         // it will not produce an error.
>         if (Tok.is(tok::hashhash))
>             Tok.setKind(tok::unknown);
>
>         // If this raw token is an identifier, the raw lexer won't
> have looked up
>         // the corresponding identifier info for it.  Do this now so
> that it will be
>         // macro expanded when we re-preprocess it.
>         if (Tok.is(tok::raw_identifier))
>             PP.LookUpIdentifierInfo(Tok);
>
>         TokenStream.push_back(Tok);
>
>         for (auto &rep : m_workList.getSortedReplacements())
>         {
>             auto repS = rep->replaceS;
>             auto spellLoc = SM.getSpellingLoc(repS->getLocStart());
>             if (SM.getSpellingLoc(Tok.getLocation()) == spellLoc)
>             {
>                 //
>             }
>         }
>
>         if (Tok.is(tok::eof))
>             break;
>     }
>
>     // Temporarily change the diagnostics object so that we ignore any
> generated
>     // diagnostics from this pass.
>     DiagnosticsEngine TmpDiags(PP.getDiagnostics().getDiagnosticIDs(),
> &PP.getDiagnostics().getDiagnosticOptions(),
>                                new IgnoringDiagConsumer);
>
>     // Copy the preprocessor and all of its state.
>     auto PPOpts =
> std::make_shared<PreprocessorOptions>(PP.getPreprocessorOpts());
>     LangOptions LO = PP.getLangOpts();
>     Preprocessor TmpPP(PPOpts, TmpDiags, LO, SM, PP.getPCMCache(),
> PP.getHeaderSearchInfo(), PP.getModuleLoader(),
> PP.getIdentifierTable().getExternalIdentifierLookup());
>     TmpPP.Initialize(PP.getTargetInfo(), PP.getAuxTargetInfo());
>     TmpPP.setExternalSource(PP.getExternalSource());
>     TmpPP.setPreprocessedOutput(true);
>
>     std::map<const clang::IdentifierInfo *, bool> MacroPreviouslyEnabled;
>     for (const auto &m : PP.macros())
>     {
>         // printf("PREDEF MACRO: %s\n",
> m.first->getName().str().c_str());
>         TmpPP.getMacroDefinition(m.first);
>
>         for (const auto &tmpm : TmpPP.macros())
>         {
>             if (tmpm.first == m.first)
>             {
>                 auto MD = m.second.getLatest();
>                 auto MI = MD->getMacroInfo();
>                 // If this is a recursive call we might be in a macro
> expansion and the macro might be disabled. We need
>                 // to enable it for now so that all expansions work.
> Restore it later.
>                 MacroPreviouslyEnabled[tmpm.first] = MI->isEnabled();
>                 if (!MI->isEnabled())
>                 {
>                     MD->getMacroInfo()->EnableMacro();
>                 }
>
>                 // This should not change anything since we just copy
> data over.
>                 auto &mutableState =
> const_cast<std::remove_const<decltype(tmpm.second)>::type
> &>(tmpm.second);
>                 mutableState.setLatest(MD);
>                 break;
>             }
>         }
>     }
>
>     class MacroArgCollector : public clang::PPCallbacks
>     {
>     public:
>         MacroArgCollector(Preprocessor &TmpPP) : TmpPP(TmpPP) {}
>
>         void MacroExpands(const Token &Tok, const MacroDefinition &MD,
> SourceRange Range, const MacroArgs *Args) override
>         {
>             if (!Args)
>             {
>                 return;
>             }
>             printf("GOT MACRO ARGS EXPANSION CALLBACK\n");
>             for (int i = 0; i < (int)Args->getNumMacroArguments(); i++)
>             {
>                 auto TokUnex = Args->getUnexpArgument(i);
>                 // Thats just non-const for a cache, so should be fine.
>                 auto TokPreExp = const_cast<MacroArgs
> *>(Args)->getPreExpArgument(i, TmpPP);
>                 printf("unexp: %s\n",
> TmpPP.getSpelling(*TokUnex).c_str());
>                 for (const auto &T : TokPreExp)
>                 {
>                     printf("preexp: %s\n", TmpPP.getSpelling(T).c_str());
>                 }
>             }
>         }
>
>         Preprocessor &TmpPP;
>     };
> TmpPP.addPPCallbacks(std::make_unique<MacroArgCollector>(TmpPP));
>     // Instead: collect the macro arg info in the law lexing step
> above. or do another pass that uses the PP but without expansions.
>
>     /*printf("DUMP MACRO INFO\n");
>     for (const auto &m : PP.macros())
>         PP.dumpMacroInfo(m.first);
>     printf("---\n");
>     for (const auto &m : TmpPP.macros())
>         TmpPP.dumpMacroInfo(m.first);
>     printf("DUMP MACRO INFO END\n");*/
>
>     DiagnosticsEngine *OldDiags = &TmpPP.getDiagnostics();
>
>     // Inform the preprocessor that we don't want comments.
>     TmpPP.SetCommentRetentionState(false, false);
>
>     // We don't want pragmas either. Although we filtered out #pragma,
> removing
>     // _Pragma and __pragma is much harder.
>     bool PragmasPreviouslyEnabled = TmpPP.getPragmasEnabled();
>     TmpPP.setPragmasEnabled(false);
>
>     // Enter the tokens we just lexed.  This will cause them to be
> macro expanded
>     // but won't enter sub-files (because we removed #'s).
>     TmpPP.EnterTokenStream(TokenStream, false);
>
>     TokenConcatenation ConcatInfo(TmpPP);
>
>     // Lex all the tokens.
>     Token Tok;
>     TmpPP.Lex(Tok);
>
>     std::map<SourceLocation, int> slocIdx;
>
>     auto checkReplacement = [&]() {
>         for (auto &rep : m_workList.getSortedReplacements())
>         {
>             // auto rep = r.second.get();
>             auto repS = rep->replaceS;
>             auto spellLoc = SM.getSpellingLoc(repS->getLocStart());
>             // TODO we need to check here if the repS spans the full
> range (or largest?)
>             if (SM.getSpellingLoc(Tok.getLocation()) == spellLoc)
>             {
>                 if (slocIdx[spellLoc] == 7)
>                 {
>                     // replace
>                 }
>                 slocIdx[spellLoc]++;
>
>                 // Done replacing that one, but have to keep it alive
> until we're done with it.
>                 m_workList.markDone(rep);
>
>                 printf("[[[\n");
>                 auto repStr = rep->makeReplaceStr();
>                 printf("REPLACED: %s ]]]\n", repStr.c_str());
>                 out += repStr;
>
>                 // Skip ahead until after the whole replacement.
>                 auto repEnd = SM.getSpellingLoc(repS->getLocEnd());
>                 while (repEnd != SM.getSpellingLoc(Tok.getLocation()))
>                 {
>                     TmpPP.Lex(Tok);
>                     assert(!Tok.is(tok::eof) && "End not found");
>                 }
>
>                 // Eat one more since we stopped at the end token and
> we want to continue after it.
>                 TmpPP.Lex(Tok);
>
>                 return true;
>             }
>         }
>         return false;
>     };
>
>     while (Tok.isNot(tok::eof))
>     {
>         printf("TOKEN: %s\n", TmpPP.getSpelling(Tok).c_str());
>
>         auto TokLoc = Tok.getLocation();
>         auto TokExp = SM.getExpansionLoc(TokLoc);
>         if (SM.isBeforeInTranslationUnit(toReplaceExpEnd, TokExp))
>         {
>             // Anything after the Stmt we want to replace is not
> interesting.
>             break;
>         }
>
>         // Skip ahead until we are at the expansion start of the Stmt
> we want to replace.
>         if (!SM.isBeforeInTranslationUnit(TokLoc, toReplaceExpStart))
>         {
>             if (TokLoc.isMacroID())
>             {
>                 // This is the first token of a macro expansion.
>                 auto LLoc = SM.getExpansionRange(TokLoc);
>
>                 // Ignore tokens whose instantiation location was not
> the main file.
>                 if (SM.getFileID(LLoc.first) != FID)
>                 {
>                     TmpPP.Lex(Tok);
>                     continue;
>                 }
>
>                 assert(SM.getFileID(LLoc.second) == FID &&
>                        "Start and end of expansion must be in the same
> ultimate file!");
>
>                 bool stopOutputOnNextToken = false;
>                 bool toReplaceStartsInMacro = toReplaceExpStart ==
> TokExp;
>                 bool toReplaceEndsInMacro = toReplaceExpEnd == TokExp;
>                 bool startedOutput = false;
>
>                 Token PrevPrevTok;
>                 Token PrevTok = Tok;
>
>                 while (!Tok.is(tok::eof) &&
> SM.getExpansionLoc(Tok.getLocation()) == LLoc.first)
>                 {
>                     printf("TOKEN (in macro): %s\n",
> TmpPP.getSpelling(Tok).c_str());
>
>                     auto TokSpell = SM.getSpellingLoc(Tok.getLocation());
>                     if (stopOutputOnNextToken)
>                     {
>                         break;
>                     }
>                     if (toReplaceEndsInMacro && TokSpell ==
> toReplaceSpellEnd)
>                     {
>                         stopOutputOnNextToken = true;
>                     }
>
>                     if (toReplaceStartsInMacro && !startedOutput)
>                     {
>                         if (TokSpell == toReplaceSpellStart)
>                         {
>                             startedOutput = true;
>                         }
>                         else
>                         {
>                             TmpPP.Lex(Tok);
>                             continue;
>                         }
>                     }
>
>                     // If the tokens were already space separated, or
> if they must be to avoid
>                     // them being implicitly pasted, add a space
> between them.
>                     if (Tok.hasLeadingSpace() ||
> ConcatInfo.AvoidConcat(PrevPrevTok, PrevTok, Tok))
>                         out += ' ';
>
>                     if (checkReplacement())
>                     {
>                         continue;
>                     }
>
>                     out += TmpPP.getSpelling(Tok);
>                     TmpPP.Lex(Tok);
>                 }
>                 if (stopOutputOnNextToken)
>                 {
>                     break;
>                 }
>             }
>             else
>             {
>                 if (checkReplacement())
>                 {
>                     continue;
>                 }
>
>                 // Output original code because we are outside of a
> replacement.
>                 out += TmpPP.getSpelling(Tok);
>                 TmpPP.Lex(Tok);
>             }
>         }
>         else
>         {
>             TmpPP.Lex(Tok);
>         }
>     }
>
>     // Restore the preprocessor's old state.
>     TmpPP.setDiagnostics(*OldDiags);
>     TmpPP.setPragmasEnabled(PragmasPreviouslyEnabled);
>
>     for (const auto &tmpm : TmpPP.macros())
>     {
>         auto it = MacroPreviouslyEnabled.find(tmpm.first);
>         if (it != MacroPreviouslyEnabled.end())
>         {
>             auto MD = tmpm.second.getLatest();
>             auto MI = MD->getMacroInfo();
>             if (MI->isEnabled() && !it->second)
>             {
>                 MI->DisableMacro();
>             }
>             else if (!MI->isEnabled() && it->second)
>             {
>                 MI->EnableMacro();
>             }
>         }
>     }
>
>     return out;
> }
>
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20180716/bf53a905/attachment-0001.html>


More information about the cfe-dev mailing list