[PATCH] D93222: [RFC] Introduce MacroExpansionContext to libAnalysis

[Balázs Benics via Phabricator via cfe-commits]

steakhal created this revision.
steakhal added reviewers: NoQ, vsavchenko, xazax.hun, martong, balazske, Szelethus, ilya-biryukov, rsmith.
Herald added subscribers: Charusso, mgrang, rnkovacs, mgorny.
steakhal requested review of this revision.
Herald added a project: clang.
Herald added a subscriber: cfe-commits.

Introduce `MacroExpansionContext` to track what and how macros in a translation unit expand.
This is the first element of the patch-stack in this direction.

The main goal is to substitute the current macro expansion generator in the `PlistsDiagnostics`, but all the other `DiagnosticsConsumer` could benefit from this.
This provides 2 functions, which token sequence (string) participated in a macro expansion, and what the substituted token sequence (string) will be.
`getExpandedMacroForLocation` and `getSubstitutedTextForLocation` repectively.

Here is an example:

  void bar();
  #define retArg(x) x
  #define retArgUnclosed retArg(bar()
  #define BB CC
  #define applyInt BB(int)
  #define CC(x) retArgUnclosed
  
  void unbalancedMacros() {
    applyInt  );
  //^~~~~~~~~~^ is the substituted range
  // Substituted text is "applyInt  )"
  // Expanded text is "bar()"
  }
  
  #define expandArgUnclosedCommaExpr(x) (x, bar(), 1
  #define f expandArgUnclosedCommaExpr
  
  void unbalancedMacros2() {
    int x =  f(f(1))  ));  // Look at the parenthesis!
  //         ^~~~~~^ is the substituted range
  // Substituted text is "f(f(1))"
  // Expanded text is "((1,bar(),1,bar(),1"
  }

Might worth investigating how to provide a reusable component, which could be used for example by a standalone tool eg. expanding all macros to their definitions.
I borrowed the main idea from the `PrintPreprocessedOutput.cpp` Frontend component, providing a `PreprocessorCallbacks` instance hooking the preprocessor events.
I'm using that for calculating the source range where tokens will be expanded to.
I'm also using the `Preprocessor`'s `OnToken` callback, via the `Preprocessor::setTokenWatcher` to reconstruct the expanded text.
Unfortunately, I concatenate the token's string representation without any whitespace except if the token is an identifier when I emit an extra space to produce valid code for `int var` token sequences. This could be improved if needed.

This proposed API is subject to change.
If you have a better idea, let me know.

---

Patch-stack:

1. [NFC] (this one) Introduces the MacroExpansionContext class and unittests.
2. [NFC] Add a field of this type to the AnalysisConsumer class, bind it to the main Preprocessor, pass a reference to it down to the diagnostic consumers - but we won't use this for the macro expansions in the PlistsDiagnostics.
3. Switch to the new macro expansion mechanism in the PlistsDiagnostics, remove the dead-code after this change. This would temporarily disable macro expansions in CTU mode.
4. (not yet implemented) Introduce a function querying the MacroExpansionContext for an imported source location. Using that context, the given macro could be expanded as well. This function needs to be virtual, probably substituting the `getImportedFromSourceLocation` in the interface described at D92432 <https://reviews.llvm.org/D92432>. This might require changes in `ASTUnit`, `LoadFromCommandLine`, or in the AST exporting/importing mechanisms.

---

It would also relieve us from bugs like:

- [fixed] D86135 <https://reviews.llvm.org/D86135>
- [confirmed] The `__VA_ARGS__` and other macro nitty-gritty, such as how to stringify macro parameters, where to put or swallow commas, etc. are not handled correctly.
- [confirmed] Unbalanced parenthesis are not well handled - resulting in incorrect expansions or even crashes.
- [confirmed][crashing] https://bugs.llvm.org/show_bug.cgi?id=48358


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D93222

Files:
  clang/include/clang/Analysis/MacroExpansionContext.h
  clang/lib/Analysis/CMakeLists.txt
  clang/lib/Analysis/MacroExpansionContext.cpp
  clang/unittests/Analysis/CMakeLists.txt
  clang/unittests/Analysis/MacroExpansionContextTest.cpp

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D93222.311571.patch
Type: text/x-patch
Size: 26495 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20201214/6e5e1a90/attachment-0001.bin>