[all-commits] [llvm/llvm-project] 6e3071: [analyzer] Introduce MacroExpansionContext to libA...

Mon Feb 22 02:07:22 PST 2021

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 6e3071007b4c9438d2ae49476de87db30d6d24e9
      https://github.com/llvm/llvm-project/commit/6e3071007b4c9438d2ae49476de87db30d6d24e9
  Author: Balazs Benics <benicsbalazs at gmail.com>
  Date:   2021-02-22 (Mon, 22 Feb 2021)

  Changed paths:
    A clang/include/clang/Analysis/MacroExpansionContext.h
    M clang/lib/Analysis/CMakeLists.txt
    A clang/lib/Analysis/MacroExpansionContext.cpp
    M clang/unittests/Analysis/CMakeLists.txt
    A clang/unittests/Analysis/MacroExpansionContextTest.cpp

  Log Message:
  -----------
  [analyzer] Introduce MacroExpansionContext to libAnalysis

Introduce `MacroExpansionContext` to track what and how macros in a translation
unit expand. This is the first element of the patch-stack in this direction.

The main goal is to substitute the current macro expansion generator in the
`PlistsDiagnostics`, but all the other `DiagnosticsConsumer` could benefit from
this.

`getExpandedText` and `getOriginalText` are the primary functions of this class.
The former can provide you the text that was the result of the macro expansion
chain starting from a `SourceLocation`.
While the latter will tell you **what text** was in the original source code
replaced by the macro expansion chain from that location.

Here is an example:

  void bar();
  #define retArg(x) x
  #define retArgUnclosed retArg(bar()
  #define BB CC
  #define applyInt BB(int)
  #define CC(x) retArgUnclosed

  void unbalancedMacros() {
    applyInt  );
  //^~~~~~~~~~^ is the substituted range
  // Original text is "applyInt  )"
  // Expanded text is "bar()"
  }

  #define expandArgUnclosedCommaExpr(x) (x, bar(), 1
  #define f expandArgUnclosedCommaExpr

  void unbalancedMacros2() {
    int x =  f(f(1))  ));  // Look at the parenthesis!
  //         ^~~~~~^ is the substituted range
  // Original text is "f(f(1))"
  // Expanded text is "((1,bar(),1,bar(),1"
  }

Might worth investigating how to provide a reusable component, which could be
used for example by a standalone tool eg. expanding all macros to their
definitions.

I borrowed the main idea from the `PrintPreprocessedOutput.cpp` Frontend
component, providing a `PPCallbacks` instance hooking the preprocessor events.
I'm using that for calculating the source range where tokens will be expanded
to. I'm also using the `Preprocessor`'s `OnToken` callback, via the
`Preprocessor::setTokenWatcher` to reconstruct the expanded text.

Unfortunately, I concatenate the token's string representation without any
whitespaces except if the token is an identifier when I emit an extra space
to produce valid code for `int var` token sequences.
This could be improved later if needed.

Patch-stack:
  1) D93222 (this one) Introduces the MacroExpansionContext class and unittests

  2) D93223 Create MacroExpansionContext member in AnalysisConsumer and pass
     down to the diagnostics consumers

  3) D93224 Use the MacroExpansionContext for macro expansions in plists
     It replaces the 'old' macro expansion mechanism.

  4) D94673 API for CTU macro expansions
     You should be able to get a `MacroExpansionContext` for each imported TU.
     Right now it will just return `llvm::None` as this is not implemented yet.

  5) FIXME: Implement macro expansion tracking for imported TUs as well.

It would also relieve us from bugs like:
  - [fixed] D86135
  - [confirmed] The `__VA_ARGS__` and other macro nitty-gritty, such as how to
    stringify macro parameters, where to put or swallow commas, etc. are not
    handled correctly.
  - [confirmed] Unbalanced parenthesis are not well handled - resulting in
    incorrect expansions or even crashes.
  - [confirmed][crashing] https://bugs.llvm.org/show_bug.cgi?id=48358

Reviewed By: martong, Szelethus

Differential Revision: https://reviews.llvm.org/D93222

  Commit: 7c58fb6ba04e28e594587bb27f13849cc1f2d305
      https://github.com/llvm/llvm-project/commit/7c58fb6ba04e28e594587bb27f13849cc1f2d305
  Author: Balazs Benics <benicsbalazs at gmail.com>
  Date:   2021-02-22 (Mon, 22 Feb 2021)

  Changed paths:
    M clang/include/clang/StaticAnalyzer/Core/PathDiagnosticConsumers.h
    M clang/lib/StaticAnalyzer/Core/HTMLDiagnostics.cpp
    M clang/lib/StaticAnalyzer/Core/PlistDiagnostics.cpp
    M clang/lib/StaticAnalyzer/Core/SarifDiagnostics.cpp
    M clang/lib/StaticAnalyzer/Core/TextDiagnostics.cpp
    M clang/lib/StaticAnalyzer/Frontend/AnalysisConsumer.cpp

  Log Message:
  -----------
  [analyzer] Create MacroExpansionContext member in AnalysisConsumer

Adds a `MacroExpansionContext` member to the `AnalysisConsumer` class.
Tracks macro expansions only if the `ShouldDisplayMacroExpansions` is set.
Passes a reference down the pipeline letting AnalysisConsumers query macro
expansions during bugreport construction.

Reviewed By: martong, Szelethus

Differential Revision: https://reviews.llvm.org/D93223

  Commit: 170c67d5b8cc58dd8a4bd0ea7c5ca02290fac39c
      https://github.com/llvm/llvm-project/commit/170c67d5b8cc58dd8a4bd0ea7c5ca02290fac39c
  Author: Balazs Benics <benicsbalazs at gmail.com>
  Date:   2021-02-22 (Mon, 22 Feb 2021)

  Changed paths:
    M clang/lib/StaticAnalyzer/Core/PlistDiagnostics.cpp
    R clang/test/Analysis/Inputs/expected-plists/plist-macros-with-expansion.cpp.plist
    M clang/test/Analysis/plist-macros-with-expansion-ctu.c
    A clang/test/Analysis/plist-macros-with-expansion.c
    M clang/test/Analysis/plist-macros-with-expansion.cpp

  Log Message:
  -----------
  [analyzer] Use the MacroExpansionContext for macro expansions in plists

Removes the obsolete ad-hoc macro expansions during bugreport constructions.
It will skip the macro expansion if the expansion happened in an imported TU.

Also removes the expected plist file, while expanding matching context for
the tests.
Adds a previously crashing `plist-macros-with-expansion.c` testfile.
Temporarily marks `plist-macros-with-expansion-ctu.c ` to `XFAIL`.

Reviewed By: xazax.hun, Szelethus

Differential Revision: https://reviews.llvm.org/D93224

  Commit: 38b185832e04ed0588c43161f5c3a8c0ce267497
      https://github.com/llvm/llvm-project/commit/38b185832e04ed0588c43161f5c3a8c0ce267497
  Author: Balazs Benics <benicsbalazs at gmail.com>
  Date:   2021-02-22 (Mon, 22 Feb 2021)

  Changed paths:
    M clang/include/clang/AST/ASTImporter.h
    M clang/include/clang/CrossTU/CrossTranslationUnit.h
    M clang/lib/AST/ASTImporter.cpp
    M clang/lib/CrossTU/CrossTranslationUnit.cpp
    M clang/lib/StaticAnalyzer/Core/PlistDiagnostics.cpp
    M clang/test/Analysis/plist-macros-with-expansion-ctu.c
    M clang/unittests/CrossTU/CrossTranslationUnitTest.cpp

  Log Message:
  -----------
  [analyzer][CTU] API for CTU macro expansions

Removes `CrossTranslationUnitContext::getImportedFromSourceLocation`
Removes the corresponding unit-test segment.

Introduces the `CrossTranslationUnitContext::getMacroExpansionContextForSourceLocation`
which will return the macro expansion context for an imported TU. Also adds a
few implementation FIXME notes where applicable, since this feature is
not implemented yet. This fact is also noted as Doxygen comments.

Uplifts a few CTU LIT test to match the current **incomplete** behavior.

It is a regression to some extent since now we don't expand any
macros in imported TUs. At least we don't crash anymore.

Note that the introduced function is already covered by LIT tests.
Eg.: Analysis/plist-macros-with-expansion-ctu.c

Reviewed By: balazske, Szelethus

Differential Revision: https://reviews.llvm.org/D94673

Compare: https://github.com/llvm/llvm-project/compare/15a74b64dfa9...38b185832e04