[clang] 97e07d0 - [analyzer] On-demand parsing capability for CTU

Endre Fülöp via cfe-commits cfe-commits at lists.llvm.org
Wed Jun 10 04:45:40 PDT 2020


Author: Endre Fülöp
Date: 2020-06-10T13:43:51+02:00
New Revision: 97e07d0c352ca469eb07a0cb3162c2807ff1099d

URL: https://github.com/llvm/llvm-project/commit/97e07d0c352ca469eb07a0cb3162c2807ff1099d
DIFF: https://github.com/llvm/llvm-project/commit/97e07d0c352ca469eb07a0cb3162c2807ff1099d.diff

LOG: [analyzer] On-demand parsing capability for CTU

Summary:
Introduce on-demand parsing of needed ASTs during CTU analysis.
The index-file format is extended, and analyzer-option CTUInvocationList
is added to specify the exact invocations needed to parse the needed
source-files.

Reviewers: martong, balazske, Szelethus, xazax.hun, whisperity

Reviewed By: martong, xazax.hun

Subscribers: gribozavr2, thakis, ASDenysPetrov, ormris, mgorny, whisperity, xazax.hun, baloghadamsoftware, szepet, rnkovacs, a.sidorin, mikhail.ramalho, Szelethus, donat.nagy, dkrupp, Charusso, steakhal, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D75665

Added: 
    clang/test/Analysis/Inputs/ctu-other.c.externalDefMap.ast-dump.txt
    clang/test/Analysis/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt
    clang/test/Analysis/ctu-on-demand-parsing.c
    clang/test/Analysis/ctu-on-demand-parsing.cpp

Modified: 
    clang/docs/analyzer/user-docs/CrossTranslationUnit.rst
    clang/include/clang/CrossTU/CrossTranslationUnit.h
    clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def
    clang/lib/CrossTU/CrossTranslationUnit.cpp
    clang/test/Analysis/Inputs/ctu-other.c
    clang/test/Analysis/analyzer-config.c
    clang/test/Analysis/ctu-different-triples.cpp
    clang/test/Analysis/ctu-main.c
    clang/test/Analysis/ctu-main.cpp
    clang/test/Analysis/ctu-unknown-parts-in-triples.cpp
    clang/unittests/CrossTU/CrossTranslationUnitTest.cpp

Removed: 
    clang/test/Analysis/Inputs/ctu-other.c.externalDefMap.txt
    clang/test/Analysis/Inputs/ctu-other.cpp.externalDefMap.txt


################################################################################
diff  --git a/clang/docs/analyzer/user-docs/CrossTranslationUnit.rst b/clang/docs/analyzer/user-docs/CrossTranslationUnit.rst
index 86f972b63e31..36be82f209ef 100644
--- a/clang/docs/analyzer/user-docs/CrossTranslationUnit.rst
+++ b/clang/docs/analyzer/user-docs/CrossTranslationUnit.rst
@@ -3,14 +3,35 @@ Cross Translation Unit (CTU) Analysis
 =====================================
 
 Normally, static analysis works in the boundary of one translation unit (TU).
-However, with additional steps and configuration we can enable the analysis to inline the definition of a function from another TU.
+However, with additional steps and configuration we can enable the analysis to inline the definition of a function from
+another TU.
 
 .. contents::
    :local:
 
-Manual CTU Analysis
--------------------
+Overview
+________
+CTU analysis can be used in a variety of ways. The importing of external TU definitions can work with pre-dumped PCH
+files or generating the necessary AST structure on-demand, during the analysis of the main TU. Driving the static
+analysis can also be implemented in multiple ways. The most direct way is to specify the necessary commandline options
+of the Clang frontend manually (and generate the prerequisite dependencies of the specific import method by hand). This
+process can be automated by other tools, like `CodeChecker <https://github.com/Ericsson/codechecker>`_ and scan-build-py
+(preference for the former).
+
+PCH-based analysis
+__________________
+The analysis needs the PCH dumps of all the translations units used in the project.
+These can be generated by the Clang Frontend itself, and must be arranged in a specific way in the filesystem.
+The index, which maps symbols' USR names to PCH dumps containing them must also be generated by the
+`clang-extdef-mapping`. Entries in the index *must* have an `.ast` suffix if the goal
+is to use PCH-based analysis, as the lack of that extension signals that the entry is to be used as a source-file, and parsed on-demand.
+This tool uses a :doc:`compilation database <../../JSONCompilationDatabase>` to
+determine the compilation flags used.
+The analysis invocation must be provided with the directory which contains the dumps and the mapping files.
+
 
+Manual CTU Analysis
+###################
 Let's consider these source files in our minimal example:
 
 .. code-block:: cpp
@@ -47,7 +68,8 @@ And a compilation database:
   ]
 
 We'd like to analyze `main.cpp` and discover the division by zero bug.
-In order to be able to inline the definition of `foo` from `foo.cpp` first we have to generate the `AST` (or `PCH`) file of `foo.cpp`:
+In order to be able to inline the definition of `foo` from `foo.cpp` first we have to generate the `AST` (or `PCH`) file
+of `foo.cpp`:
 
 .. code-block:: bash
 
@@ -58,7 +80,8 @@ In order to be able to inline the definition of `foo` from `foo.cpp` first we ha
   compile_commands.json  foo.cpp.ast  foo.cpp  main.cpp
   $
 
-The next step is to create a CTU index file which holds the `USR` name and location of external definitions in the source files:
+The next step is to create a CTU index file which holds the `USR` name and location of external definitions in the
+source files:
 
 .. code-block:: bash
 
@@ -85,47 +108,33 @@ We have to feed Clang with CTU specific extra arguments:
 
   $ pwd
   /path/to/your/project
-  $ clang++ --analyze -Xclang -analyzer-config -Xclang experimental-enable-naive-ctu-analysis=true -Xclang -analyzer-config -Xclang ctu-dir=. -Xclang -analyzer-output=plist-multi-file main.cpp
+  $ clang++ --analyze \
+      -Xclang -analyzer-config -Xclang experimental-enable-naive-ctu-analysis=true \
+      -Xclang -analyzer-config -Xclang ctu-dir=. \
+      -Xclang -analyzer-output=plist-multi-file \
+      main.cpp
   main.cpp:5:12: warning: Division by zero
     return 3 / foo();
            ~~^~~~~~~
   1 warning generated.
   $ # The plist file with the result is generated.
-  $ ls
+  $ ls -F
   compile_commands.json  externalDefMap.txt  foo.ast  foo.cpp  foo.cpp.ast  main.cpp  main.plist
   $
 
-This manual procedure is error-prone and not scalable, therefore to analyze real projects it is recommended to use `CodeChecker` or `scan-build-py`.
+This manual procedure is error-prone and not scalable, therefore to analyze real projects it is recommended to use
+`CodeChecker` or `scan-build-py`.
 
 Automated CTU Analysis with CodeChecker
----------------------------------------
+#######################################
 The `CodeChecker <https://github.com/Ericsson/codechecker>`_ project fully supports automated CTU analysis with Clang.
 Once we have set up the `PATH` environment variable and we activated the python `venv` then it is all it takes:
 
 .. code-block:: bash
 
   $ CodeChecker analyze --ctu compile_commands.json -o reports
-  [INFO 2019-07-16 17:21] - Pre-analysis started.
-  [INFO 2019-07-16 17:21] - Collecting data for ctu analysis.
-  [INFO 2019-07-16 17:21] - [1/2] foo.cpp
-  [INFO 2019-07-16 17:21] - [2/2] main.cpp
-  [INFO 2019-07-16 17:21] - Pre-analysis finished.
-  [INFO 2019-07-16 17:21] - Starting static analysis ...
-  [INFO 2019-07-16 17:21] - [1/2] clangsa analyzed foo.cpp successfully.
-  [INFO 2019-07-16 17:21] - [2/2] clangsa analyzed main.cpp successfully.
-  [INFO 2019-07-16 17:21] - ----==== Summary ====----
-  [INFO 2019-07-16 17:21] - Successfully analyzed
-  [INFO 2019-07-16 17:21] -   clangsa: 2
-  [INFO 2019-07-16 17:21] - Total analyzed compilation commands: 2
-  [INFO 2019-07-16 17:21] - ----=================----
-  [INFO 2019-07-16 17:21] - Analysis finished.
-  [INFO 2019-07-16 17:21] - To view results in the terminal use the "CodeChecker parse" command.
-  [INFO 2019-07-16 17:21] - To store results use the "CodeChecker store" command.
-  [INFO 2019-07-16 17:21] - See --help and the user guide for further options about parsing and storing the reports.
-  [INFO 2019-07-16 17:21] - ----=================----
-  [INFO 2019-07-16 17:21] - Analysis length: 0.659618854523 sec.
-  $ ls
-  compile_commands.json  foo.cpp  foo.cpp.ast  main.cpp  reports
+  $ ls -F
+  compile_commands.json  foo.cpp  foo.cpp.ast  main.cpp  reports/
   $ tree reports
   reports
   ├── compile_cmd.json
@@ -174,9 +183,9 @@ Or we can use `CodeChecker parse -e html` to export the results into HTML format
   $ firefox html_out/index.html
 
 Automated CTU Analysis with scan-build-py (don't do it)
--------------------------------------------------------
-We actively develop CTU with CodeChecker as a "runner" script, `scan-build-py` is not actively developed for CTU.
-`scan-build-py` has various errors and issues, expect it to work with the very basic projects only.
+#############################################################
+We actively develop CTU with CodeChecker as the driver for this feature, `scan-build-py` is not actively developed for CTU.
+`scan-build-py` has various errors and issues, expect it to work only with the very basic projects only.
 
 Example usage of scan-build-py:
 
@@ -191,3 +200,176 @@ Example usage of scan-build-py:
   Opening in existing browser session.
   ^C
   $
+
+On-demand analysis
+__________________
+The analysis produces the necessary AST structure of external TUs during analysis. This requires the
+exact compiler invocations for each TU, which can be generated by hand, or by tools driving the analyzer.
+The compiler invocation is a shell command that could be used to compile the TU-s main source file.
+The mapping from absolute source file paths of a TU to lists of compilation command segments used to
+compile said TU are given in YAML format referred to as `invocation list`, and must be passed as an
+analyer-config argument.
+The index, which maps function USR names to source files containing them must also be generated by the
+`clang-extdef-mapping`. Entries in the index must *not* have an `.ast` suffix if the goal
+is to use On-demand analysis, as that extension signals that the entry is to be used as an PCH-dump.
+The mapping of external definitions implicitly uses a
+:doc:`compilation database <../../JSONCompilationDatabase>` to determine the compilation flags used.
+The analysis invocation must be provided with the directory which contains the mapping
+files, and the `invocation list` which is used to determine compiler flags.
+
+
+Manual CTU Analysis
+###################
+
+Let's consider these source files in our minimal example:
+
+.. code-block:: cpp
+
+  // main.cpp
+  int foo();
+
+  int main() {
+    return 3 / foo();
+  }
+
+.. code-block:: cpp
+
+  // foo.cpp
+  int foo() {
+    return 0;
+  }
+
+The compilation database:
+
+.. code-block:: bash
+
+  [
+    {
+      "directory": "/path/to/your/project",
+      "command": "clang++ -c foo.cpp -o foo.o",
+      "file": "foo.cpp"
+    },
+    {
+      "directory": "/path/to/your/project",
+      "command": "clang++ -c main.cpp -o main.o",
+      "file": "main.cpp"
+    }
+  ]
+
+The `invocation list`:
+
+.. code-block:: bash
+  
+  "/path/to/your/project/foo.cpp": 
+    - "clang++"
+    - "-c"
+    - "/path/to/your/project/foo.cpp"
+    - "-o"
+    - "/path/to/your/project/foo.o"
+
+  "/path/to/your/project/main.cpp": 
+    - "clang++"
+    - "-c"
+    - "/path/to/your/project/main.cpp"
+    - "-o"
+    - "/path/to/your/project/main.o"
+
+We'd like to analyze `main.cpp` and discover the division by zero bug.
+As we are using On-demand mode, we only need to create a CTU index file which holds the `USR` name and location of
+external definitions in the source files:
+
+.. code-block:: bash
+
+  $ clang-extdef-mapping -p . foo.cpp
+  c:@F at foo# /path/to/your/project/foo.cpp
+  $ clang-extdef-mapping -p . foo.cpp > externalDefMap.txt
+
+Now everything is available for the CTU analysis.
+We have to feed Clang with CTU specific extra arguments:
+
+.. code-block:: bash
+
+  $ pwd
+  /path/to/your/project
+  $ clang++ --analyze \
+      -Xclang -analyzer-config -Xclang experimental-enable-naive-ctu-analysis=true \
+      -Xclang -analyzer-config -Xclang ctu-dir=. \
+      -Xclang -analyzer-config -Xclang ctu-invocation-list=invocations.yaml \
+      -Xclang -analyzer-output=plist-multi-file \
+      main.cpp
+  main.cpp:5:12: warning: Division by zero
+    return 3 / foo();
+           ~~^~~~~~~
+  1 warning generated.
+  $ # The plist file with the result is generated.
+  $ ls -F
+  compile_commands.json  externalDefMap.txt  foo.cpp  main.cpp  main.plist
+  $
+
+This manual procedure is error-prone and not scalable, therefore to analyze real projects it is recommended to use
+`CodeChecker` or `scan-build-py`.
+
+Automated CTU Analysis with CodeChecker
+#######################################
+The `CodeChecker <https://github.com/Ericsson/codechecker>`_ project fully supports automated CTU analysis with Clang.
+Once we have set up the `PATH` environment variable and we activated the python `venv` then it is all it takes:
+
+.. code-block:: bash
+
+  $ CodeChecker analyze --ctu --ctu-ast-loading-mode on-demand compile_commands.json -o reports
+  $ ls -F
+  compile_commands.json  foo.cpp main.cpp  reports/
+  $ tree reports
+  reports
+  ├── compile_cmd.json
+  ├── compiler_info.json
+  ├── foo.cpp_53f6fbf7ab7ec9931301524b551959e2.plist
+  ├── main.cpp_23db3d8df52ff0812e6e5a03071c8337.plist
+  ├── metadata.json
+  └── unique_compile_commands.json
+
+  0 directories, 6 files
+  $
+
+The `plist` files contain the results of the analysis, which may be viewed with the regular analysis tools.
+E.g. one may use `CodeChecker parse` to view the results in command line:
+
+.. code-block:: bash
+
+  $ CodeChecker parse reports
+  [HIGH] /home/egbomrt/ctu_mini_raw_project/main.cpp:5:12: Division by zero [core.DivideZero]
+    return 3 / foo();
+             ^
+
+  Found 1 defect(s) in main.cpp
+
+
+  ----==== Summary ====----
+  -----------------------
+  Filename | Report count
+  -----------------------
+  main.cpp |            1
+  -----------------------
+  -----------------------
+  Severity | Report count
+  -----------------------
+  HIGH     |            1
+  -----------------------
+  ----=================----
+  Total number of reports: 1
+  ----=================----
+
+Or we can use `CodeChecker parse -e html` to export the results into HTML format:
+
+.. code-block:: bash
+
+  $ CodeChecker parse -e html -o html_out reports
+  $ firefox html_out/index.html
+
+Automated CTU Analysis with scan-build-py (don't do it)
+#######################################################
+We actively develop CTU with CodeChecker as the driver for feature, `scan-build-py` is not actively developed for CTU.
+`scan-build-py` has various errors and issues, expect it to work only with the very basic projects only.
+
+Currently On-demand analysis is not supported with `scan-build-py`.
+

diff  --git a/clang/include/clang/CrossTU/CrossTranslationUnit.h b/clang/include/clang/CrossTU/CrossTranslationUnit.h
index 4d2b7109c62a..027c6f16430b 100644
--- a/clang/include/clang/CrossTU/CrossTranslationUnit.h
+++ b/clang/include/clang/CrossTU/CrossTranslationUnit.h
@@ -21,6 +21,7 @@
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/StringMap.h"
 #include "llvm/Support/Error.h"
+#include "llvm/Support/Path.h"
 
 namespace clang {
 class CompilerInstance;
@@ -47,7 +48,12 @@ enum class index_error_code {
   triple_mismatch,
   lang_mismatch,
   lang_dialect_mismatch,
-  load_threshold_reached
+  load_threshold_reached,
+  invocation_list_ambiguous,
+  invocation_list_file_not_found,
+  invocation_list_empty,
+  invocation_list_wrong_format,
+  invocation_list_lookup_unsuccessful
 };
 
 class IndexError : public llvm::ErrorInfo<IndexError> {
@@ -78,7 +84,8 @@ class IndexError : public llvm::ErrorInfo<IndexError> {
 };
 
 /// This function parses an index file that determines which
-///        translation unit contains which definition.
+/// translation unit contains which definition. The IndexPath is not prefixed
+/// with CTUDir, so an absolute path is expected for consistent results.
 ///
 /// The index file format is the following:
 /// each line consists of an USR and a filepath separated by a space.
@@ -86,17 +93,27 @@ class IndexError : public llvm::ErrorInfo<IndexError> {
 /// \return Returns a map where the USR is the key and the filepath is the value
 ///         or an error.
 llvm::Expected<llvm::StringMap<std::string>>
-parseCrossTUIndex(StringRef IndexPath, StringRef CrossTUDir);
+parseCrossTUIndex(StringRef IndexPath);
 
 std::string createCrossTUIndexString(const llvm::StringMap<std::string> &Index);
 
+using InvocationListTy = llvm::StringMap<llvm::SmallVector<std::string, 32>>;
+/// Parse the YAML formatted invocation list file content \p FileContent.
+/// The format is expected to be a mapping from from absolute source file
+/// paths in the filesystem to a list of command-line parts, which
+/// constitute the invocation needed to compile that file. That invocation
+/// will be used to produce the AST of the TU.
+llvm::Expected<InvocationListTy> parseInvocationList(
+    StringRef FileContent,
+    llvm::sys::path::Style PathStyle = llvm::sys::path::Style::posix);
+
 // Returns true if the variable or any field of a record variable is const.
 bool containsConst(const VarDecl *VD, const ASTContext &ACtx);
 
 /// This class is used for tools that requires cross translation
 ///        unit capability.
 ///
-/// This class can load definitions from external AST files.
+/// This class can load definitions from external AST sources.
 /// The loaded definition will be merged back to the original AST using the
 /// AST Importer.
 /// In order to use this class, an index file is required that describes
@@ -116,7 +133,7 @@ class CrossTranslationUnitContext {
   /// the current translation unit. A function definition with the same
   /// declaration will be looked up in the index file which should be in the
   /// \p CrossTUDir directory, called \p IndexName. In case the declaration is
-  /// found in the index the corresponding AST file will be loaded and the
+  /// found in the index the corresponding AST will be loaded and the
   /// definition will be merged into the original AST using the AST Importer.
   ///
   /// \return The declaration with the definition will be returned.
@@ -136,7 +153,7 @@ class CrossTranslationUnitContext {
   /// A definition with the same declaration will be looked up in the
   /// index file which should be in the \p CrossTUDir directory, called
   /// \p IndexName. In case the declaration is found in the index the
-  /// corresponding AST file will be loaded. If the number of TUs imported
+  /// corresponding AST will be loaded. If the number of TUs imported
   /// reaches \p CTULoadTreshold, no loading is performed.
   ///
   /// \return Returns a pointer to the ASTUnit that contains the definition of
@@ -209,14 +226,43 @@ class CrossTranslationUnitContext {
   /// imported the FileID.
   ImportedFileIDMap ImportedFileIDs;
 
-  /// Functor for loading ASTUnits from AST-dump files.
-  class ASTFileLoader {
+  using LoadResultTy = llvm::Expected<std::unique_ptr<ASTUnit>>;
+
+  /// Loads ASTUnits from AST-dumps or source-files.
+  class ASTLoader {
   public:
-    ASTFileLoader(const CompilerInstance &CI);
-    std::unique_ptr<ASTUnit> operator()(StringRef ASTFilePath);
+    ASTLoader(CompilerInstance &CI, StringRef CTUDir,
+              StringRef InvocationListFilePath);
+
+    /// Load the ASTUnit by its identifier found in the index file. If the
+    /// indentifier is suffixed with '.ast' it is considered a dump. Otherwise
+    /// it is treated as source-file, and on-demand parsed. Relative paths are
+    /// prefixed with CTUDir.
+    LoadResultTy load(StringRef Identifier);
+
+    /// Lazily initialize the invocation list information, which is needed for
+    /// on-demand parsing.
+    llvm::Error lazyInitInvocationList();
 
   private:
-    const CompilerInstance &CI;
+    /// The style used for storage and lookup of filesystem paths.
+    /// Defaults to posix.
+    const llvm::sys::path::Style PathStyle = llvm::sys::path::Style::posix;
+
+    /// Loads an AST from a pch-dump.
+    LoadResultTy loadFromDump(StringRef Identifier);
+    /// Loads an AST from a source-file.
+    LoadResultTy loadFromSource(StringRef Identifier);
+
+    CompilerInstance &CI;
+    StringRef CTUDir;
+    /// The path to the file containing the invocation list, which is in YAML
+    /// format, and contains a mapping from source files to compiler invocations
+    /// that produce the AST used for analysis.
+    StringRef InvocationListFilePath;
+    /// In case of on-demand parsing, the invocations for parsing the source
+    /// files is stored.
+    llvm::Optional<InvocationListTy> InvocationList;
   };
 
   /// Maintain number of AST loads and check for reaching the load limit.
@@ -242,7 +288,7 @@ class CrossTranslationUnitContext {
   /// are the concerns of ASTUnitStorage class.
   class ASTUnitStorage {
   public:
-    ASTUnitStorage(const CompilerInstance &CI);
+    ASTUnitStorage(CompilerInstance &CI);
     /// Loads an ASTUnit for a function.
     ///
     /// \param FunctionName USR name of the function.
@@ -287,18 +333,17 @@ class CrossTranslationUnitContext {
     using IndexMapTy = BaseMapTy<std::string>;
     IndexMapTy NameFileMap;
 
-    ASTFileLoader FileAccessor;
+    /// Loads the AST based on the identifier found in the index.
+    ASTLoader Loader;
 
-    /// Limit the number of loaded ASTs. Used to limit the  memory usage of the
-    /// CrossTranslationUnitContext.
-    /// The ASTUnitStorage has the knowledge about if the AST to load is
-    /// actually loaded or returned from cache. This information is needed to
-    /// maintain the counter.
+    /// Limit the number of loaded ASTs. It is used to limit the  memory usage
+    /// of the CrossTranslationUnitContext. The ASTUnitStorage has the
+    /// information whether the AST to load is actually loaded or returned from
+    /// cache. This information is needed to maintain the counter.
     ASTLoadGuard LoadGuard;
   };
 
   ASTUnitStorage ASTStorage;
-
 };
 
 } // namespace cross_tu

diff  --git a/clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def b/clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def
index 597a65c21318..8944dfe0f749 100644
--- a/clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def
+++ b/clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def
@@ -378,9 +378,24 @@ ANALYZER_OPTION(StringRef, CTUDir, "ctu-dir",
                 "The directory containing the CTU related files.", "")
 
 ANALYZER_OPTION(StringRef, CTUIndexName, "ctu-index-name",
-                "the name of the file containing the CTU index of definitions.",
+                "The name of the file containing the CTU index of definitions. "
+                "The index file maps USR-names to identifiers. An identifier "
+                "can end with an '.ast' suffix, indicating the indentifier is "
+                "a path to a pch-dump. Otherwise the identifier is regarded as "
+                "path to a source file which is parsed on-demand. Relative "
+                "paths are prefixed with ctu-dir, absolute paths are used "
+                "unmodified during lookup.",
                 "externalDefMap.txt")
 
+ANALYZER_OPTION(
+    StringRef, CTUInvocationList, "ctu-invocation-list",
+    "The path to the YAML format file containing a mapping from source file "
+    "paths to command-line invocations represented as a list of arguments. "
+    "This invocation is used produce the source-file's AST in case on-demand "
+    "loading is performed. Example file-content: "
+    "{/main.cpp: [clang++, /main.cpp], other.cpp: [clang++, /other.cpp]}",
+    "invocations.yaml")
+
 ANALYZER_OPTION(
     StringRef, ModelPath, "model-path",
     "The analyzer can inline an alternative implementation written in C at the "

diff  --git a/clang/lib/CrossTU/CrossTranslationUnit.cpp b/clang/lib/CrossTU/CrossTranslationUnit.cpp
index 689c988d0b36..24c3143db410 100644
--- a/clang/lib/CrossTU/CrossTranslationUnit.cpp
+++ b/clang/lib/CrossTU/CrossTranslationUnit.cpp
@@ -18,14 +18,19 @@
 #include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/TextDiagnosticPrinter.h"
 #include "clang/Index/USRGeneration.h"
-#include "llvm/ADT/Triple.h"
+#include "llvm/ADT/Optional.h"
 #include "llvm/ADT/Statistic.h"
+#include "llvm/ADT/Triple.h"
+#include "llvm/Option/ArgList.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/ManagedStatic.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/YAMLParser.h"
 #include "llvm/Support/raw_ostream.h"
+#include <algorithm>
 #include <fstream>
 #include <sstream>
+#include <tuple>
 
 namespace clang {
 namespace cross_tu {
@@ -110,6 +115,17 @@ class IndexErrorCategory : public std::error_category {
       return "Language dialect mismatch";
     case index_error_code::load_threshold_reached:
       return "Load threshold reached";
+    case index_error_code::invocation_list_ambiguous:
+      return "Invocation list file contains multiple references to the same "
+             "source file.";
+    case index_error_code::invocation_list_file_not_found:
+      return "Invocation list file is not found.";
+    case index_error_code::invocation_list_empty:
+      return "Invocation list file is empty.";
+    case index_error_code::invocation_list_wrong_format:
+      return "Invocation list file is in wrong format.";
+    case index_error_code::invocation_list_lookup_unsuccessful:
+      return "Invocation list file does not contain the requested source file.";
     }
     llvm_unreachable("Unrecognized index_error_code.");
   }
@@ -129,7 +145,7 @@ std::error_code IndexError::convertToErrorCode() const {
 }
 
 llvm::Expected<llvm::StringMap<std::string>>
-parseCrossTUIndex(StringRef IndexPath, StringRef CrossTUDir) {
+parseCrossTUIndex(StringRef IndexPath) {
   std::ifstream ExternalMapFile{std::string(IndexPath)};
   if (!ExternalMapFile)
     return llvm::make_error<IndexError>(index_error_code::missing_index_file,
@@ -139,21 +155,26 @@ parseCrossTUIndex(StringRef IndexPath, StringRef CrossTUDir) {
   std::string Line;
   unsigned LineNo = 1;
   while (std::getline(ExternalMapFile, Line)) {
-    const size_t Pos = Line.find(" ");
-    if (Pos > 0 && Pos != std::string::npos) {
-      StringRef LineRef{Line};
-      StringRef LookupName = LineRef.substr(0, Pos);
-      if (Result.count(LookupName))
+    StringRef LineRef{Line};
+    const size_t Delimiter = LineRef.find(" ");
+    if (Delimiter > 0 && Delimiter != std::string::npos) {
+      StringRef LookupName = LineRef.substr(0, Delimiter);
+
+      // Store paths with posix-style directory separator.
+      SmallVector<char, 32> FilePath;
+      llvm::Twine{LineRef.substr(Delimiter + 1)}.toVector(FilePath);
+      llvm::sys::path::native(FilePath, llvm::sys::path::Style::posix);
+
+      bool InsertionOccured;
+      std::tie(std::ignore, InsertionOccured) =
+          Result.try_emplace(LookupName, FilePath.begin(), FilePath.end());
+      if (!InsertionOccured)
         return llvm::make_error<IndexError>(
             index_error_code::multiple_definitions, IndexPath.str(), LineNo);
-      StringRef FileName = LineRef.substr(Pos + 1);
-      SmallString<256> FilePath = CrossTUDir;
-      llvm::sys::path::append(FilePath, FileName);
-      Result[LookupName] = std::string(FilePath);
     } else
       return llvm::make_error<IndexError>(
           index_error_code::invalid_index_format, IndexPath.str(), LineNo);
-    LineNo++;
+    ++LineNo;
   }
   return Result;
 }
@@ -341,30 +362,11 @@ void CrossTranslationUnitContext::emitCrossTUDiagnostics(const IndexError &IE) {
   }
 }
 
-CrossTranslationUnitContext::ASTFileLoader::ASTFileLoader(
-    const CompilerInstance &CI)
-    : CI(CI) {}
-
-std::unique_ptr<ASTUnit>
-CrossTranslationUnitContext::ASTFileLoader::operator()(StringRef ASTFilePath) {
-  // Load AST from ast-dump.
-  IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts = new DiagnosticOptions();
-  TextDiagnosticPrinter *DiagClient =
-      new TextDiagnosticPrinter(llvm::errs(), &*DiagOpts);
-  IntrusiveRefCntPtr<DiagnosticIDs> DiagID(new DiagnosticIDs());
-  IntrusiveRefCntPtr<DiagnosticsEngine> Diags(
-      new DiagnosticsEngine(DiagID, &*DiagOpts, DiagClient));
-
-  return ASTUnit::LoadFromASTFile(
-      std::string(ASTFilePath), CI.getPCHContainerOperations()->getRawReader(),
-      ASTUnit::LoadEverything, Diags, CI.getFileSystemOpts());
-}
-
 CrossTranslationUnitContext::ASTUnitStorage::ASTUnitStorage(
-    const CompilerInstance &CI)
-    : FileAccessor(CI), LoadGuard(const_cast<CompilerInstance &>(CI)
-                                      .getAnalyzerOpts()
-                                      ->CTUImportThreshold) {}
+    CompilerInstance &CI)
+    : Loader(CI, CI.getAnalyzerOpts()->CTUDir,
+             CI.getAnalyzerOpts()->CTUInvocationList),
+      LoadGuard(CI.getAnalyzerOpts()->CTUImportThreshold) {}
 
 llvm::Expected<ASTUnit *>
 CrossTranslationUnitContext::ASTUnitStorage::getASTUnitForFile(
@@ -380,8 +382,12 @@ CrossTranslationUnitContext::ASTUnitStorage::getASTUnitForFile(
           index_error_code::load_threshold_reached);
     }
 
-    // Load the ASTUnit from the pre-dumped AST file specified by ASTFileName.
-    std::unique_ptr<ASTUnit> LoadedUnit = FileAccessor(FileName);
+    auto LoadAttempt = Loader.load(FileName);
+
+    if (!LoadAttempt)
+      return LoadAttempt.takeError();
+
+    std::unique_ptr<ASTUnit> LoadedUnit = std::move(LoadAttempt.get());
 
     // Need the raw pointer and the unique_ptr as well.
     ASTUnit *Unit = LoadedUnit.get();
@@ -461,7 +467,7 @@ llvm::Error CrossTranslationUnitContext::ASTUnitStorage::ensureCTUIndexLoaded(
   else
     llvm::sys::path::append(IndexFile, IndexName);
 
-  if (auto IndexMapping = parseCrossTUIndex(IndexFile, CrossTUDir)) {
+  if (auto IndexMapping = parseCrossTUIndex(IndexFile)) {
     // Initialize member map.
     NameFileMap = *IndexMapping;
     return llvm::Error::success();
@@ -494,6 +500,193 @@ llvm::Expected<ASTUnit *> CrossTranslationUnitContext::loadExternalAST(
   return Unit;
 }
 
+CrossTranslationUnitContext::ASTLoader::ASTLoader(
+    CompilerInstance &CI, StringRef CTUDir, StringRef InvocationListFilePath)
+    : CI(CI), CTUDir(CTUDir), InvocationListFilePath(InvocationListFilePath) {}
+
+CrossTranslationUnitContext::LoadResultTy
+CrossTranslationUnitContext::ASTLoader::load(StringRef Identifier) {
+  llvm::SmallString<256> Path;
+  if (llvm::sys::path::is_absolute(Identifier, PathStyle)) {
+    Path = Identifier;
+  } else {
+    Path = CTUDir;
+    llvm::sys::path::append(Path, PathStyle, Identifier);
+  }
+
+  // The path is stored in the InvocationList member in posix style. To
+  // successfully lookup an entry based on filepath, it must be converted.
+  llvm::sys::path::native(Path, PathStyle);
+
+  // Normalize by removing relative path components.
+  llvm::sys::path::remove_dots(Path, /*remove_dot_dot*/ true, PathStyle);
+
+  if (Path.endswith(".ast"))
+    return loadFromDump(Path);
+  else
+    return loadFromSource(Path);
+}
+
+CrossTranslationUnitContext::LoadResultTy
+CrossTranslationUnitContext::ASTLoader::loadFromDump(StringRef ASTDumpPath) {
+  IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts = new DiagnosticOptions();
+  TextDiagnosticPrinter *DiagClient =
+      new TextDiagnosticPrinter(llvm::errs(), &*DiagOpts);
+  IntrusiveRefCntPtr<DiagnosticIDs> DiagID(new DiagnosticIDs());
+  IntrusiveRefCntPtr<DiagnosticsEngine> Diags(
+      new DiagnosticsEngine(DiagID, &*DiagOpts, DiagClient));
+  return ASTUnit::LoadFromASTFile(
+      std::string(ASTDumpPath.str()),
+      CI.getPCHContainerOperations()->getRawReader(), ASTUnit::LoadEverything,
+      Diags, CI.getFileSystemOpts());
+}
+
+/// Load the AST from a source-file, which is supposed to be located inside the
+/// YAML formatted invocation list file under the filesystem path specified by
+/// \p InvocationList. The invocation list should contain absolute paths.
+/// \p SourceFilePath is the absolute path of the source file that contains the
+/// function definition the analysis is looking for. The Index is built by the
+/// \p clang-extdef-mapping tool, which is also supposed to be generating
+/// absolute paths.
+///
+/// Proper diagnostic emission requires absolute paths, so even if a future
+/// change introduces the handling of relative paths, this must be taken into
+/// consideration.
+CrossTranslationUnitContext::LoadResultTy
+CrossTranslationUnitContext::ASTLoader::loadFromSource(
+    StringRef SourceFilePath) {
+
+  if (llvm::Error InitError = lazyInitInvocationList())
+    return std::move(InitError);
+  assert(InvocationList);
+
+  auto Invocation = InvocationList->find(SourceFilePath);
+  if (Invocation == InvocationList->end())
+    return llvm::make_error<IndexError>(
+        index_error_code::invocation_list_lookup_unsuccessful);
+
+  const InvocationListTy::mapped_type &InvocationCommand = Invocation->second;
+
+  SmallVector<const char *, 32> CommandLineArgs(InvocationCommand.size());
+  std::transform(InvocationCommand.begin(), InvocationCommand.end(),
+                 CommandLineArgs.begin(),
+                 [](auto &&CmdPart) { return CmdPart.c_str(); });
+
+  IntrusiveRefCntPtr<DiagnosticOptions> DiagOpts{&CI.getDiagnosticOpts()};
+  auto *DiagClient = new ForwardingDiagnosticConsumer{CI.getDiagnosticClient()};
+  IntrusiveRefCntPtr<DiagnosticIDs> DiagID{
+      CI.getDiagnostics().getDiagnosticIDs()};
+  IntrusiveRefCntPtr<DiagnosticsEngine> Diags(
+      new DiagnosticsEngine{DiagID, &*DiagOpts, DiagClient});
+
+  return std::unique_ptr<ASTUnit>(ASTUnit::LoadFromCommandLine(
+      CommandLineArgs.begin(), (CommandLineArgs.end()),
+      CI.getPCHContainerOperations(), Diags,
+      CI.getHeaderSearchOpts().ResourceDir));
+}
+
+llvm::Expected<InvocationListTy>
+parseInvocationList(StringRef FileContent, llvm::sys::path::Style PathStyle) {
+  InvocationListTy InvocationList;
+
+  /// LLVM YAML parser is used to extract information from invocation list file.
+  llvm::SourceMgr SM;
+  llvm::yaml::Stream InvocationFile(FileContent, SM);
+
+  /// Only the first document is processed.
+  llvm::yaml::document_iterator FirstInvocationFile = InvocationFile.begin();
+
+  /// There has to be at least one document available.
+  if (FirstInvocationFile == InvocationFile.end())
+    return llvm::make_error<IndexError>(
+        index_error_code::invocation_list_empty);
+
+  llvm::yaml::Node *DocumentRoot = FirstInvocationFile->getRoot();
+  if (!DocumentRoot)
+    return llvm::make_error<IndexError>(
+        index_error_code::invocation_list_wrong_format);
+
+  /// According to the format specified the document must be a mapping, where
+  /// the keys are paths to source files, and values are sequences of invocation
+  /// parts.
+  auto *Mappings = dyn_cast<llvm::yaml::MappingNode>(DocumentRoot);
+  if (!Mappings)
+    return llvm::make_error<IndexError>(
+        index_error_code::invocation_list_wrong_format);
+
+  for (auto &NextMapping : *Mappings) {
+    /// The keys should be strings, which represent a source-file path.
+    auto *Key = dyn_cast<llvm::yaml::ScalarNode>(NextMapping.getKey());
+    if (!Key)
+      return llvm::make_error<IndexError>(
+          index_error_code::invocation_list_wrong_format);
+
+    SmallVector<char, 32> ValueStorage;
+    StringRef SourcePath = Key->getValue(ValueStorage);
+
+    // Store paths with PathStyle directory separator.
+    SmallVector<char, 32> NativeSourcePath;
+    llvm::Twine{SourcePath}.toVector(NativeSourcePath);
+    llvm::sys::path::native(NativeSourcePath, PathStyle);
+
+    StringRef InvocationKey{NativeSourcePath.begin(), NativeSourcePath.size()};
+
+    if (InvocationList.find(InvocationKey) != InvocationList.end())
+      return llvm::make_error<IndexError>(
+          index_error_code::invocation_list_ambiguous);
+
+    /// The values should be sequences of strings, each representing a part of
+    /// the invocation.
+    auto *Args = dyn_cast<llvm::yaml::SequenceNode>(NextMapping.getValue());
+    if (!Args)
+      return llvm::make_error<IndexError>(
+          index_error_code::invocation_list_wrong_format);
+
+    for (auto &Arg : *Args) {
+      auto *CmdString = dyn_cast<llvm::yaml::ScalarNode>(&Arg);
+      if (!CmdString)
+        return llvm::make_error<IndexError>(
+            index_error_code::invocation_list_wrong_format);
+      /// Every conversion starts with an empty working storage, as it is not
+      /// clear if this is a requirement of the YAML parser.
+      ValueStorage.clear();
+      InvocationList[InvocationKey].emplace_back(
+          CmdString->getValue(ValueStorage));
+    }
+
+    if (InvocationList[InvocationKey].empty())
+      return llvm::make_error<IndexError>(
+          index_error_code::invocation_list_wrong_format);
+  }
+
+  return InvocationList;
+}
+
+llvm::Error CrossTranslationUnitContext::ASTLoader::lazyInitInvocationList() {
+  /// Lazily initialize the invocation list member used for on-demand parsing.
+  if (InvocationList)
+    return llvm::Error::success();
+
+  llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>> FileContent =
+      llvm::MemoryBuffer::getFile(InvocationListFilePath);
+  if (!FileContent)
+    return llvm::make_error<IndexError>(
+        index_error_code::invocation_list_file_not_found);
+  std::unique_ptr<llvm::MemoryBuffer> ContentBuffer = std::move(*FileContent);
+  assert(ContentBuffer && "If no error was produced after loading, the pointer "
+                          "should not be nullptr.");
+
+  llvm::Expected<InvocationListTy> ExpectedInvocationList =
+      parseInvocationList(ContentBuffer->getBuffer(), PathStyle);
+
+  if (!ExpectedInvocationList)
+    return ExpectedInvocationList.takeError();
+
+  InvocationList = *ExpectedInvocationList;
+
+  return llvm::Error::success();
+}
+
 template <typename T>
 llvm::Expected<const T *>
 CrossTranslationUnitContext::importDefinitionImpl(const T *D, ASTUnit *Unit) {

diff  --git a/clang/test/Analysis/Inputs/ctu-other.c b/clang/test/Analysis/Inputs/ctu-other.c
index 82d29c6edb30..16ebb3506f60 100644
--- a/clang/test/Analysis/Inputs/ctu-other.c
+++ b/clang/test/Analysis/Inputs/ctu-other.c
@@ -31,10 +31,12 @@ int g(struct S *ctx) {
 }
 
 // Test that asm import does not fail.
+// TODO: Support the GNU extension asm keyword as well.
+// Example using the GNU extension: asm("mov $42, %0" : "=r"(res));
 int inlineAsm() {
   int res;
-  asm("mov $42, %0"
-      : "=r"(res));
+  __asm__("mov $42, %0"
+          : "=r"(res));
   return res;
 }
 

diff  --git a/clang/test/Analysis/Inputs/ctu-other.c.externalDefMap.txt b/clang/test/Analysis/Inputs/ctu-other.c.externalDefMap.ast-dump.txt
similarity index 100%
rename from clang/test/Analysis/Inputs/ctu-other.c.externalDefMap.txt
rename to clang/test/Analysis/Inputs/ctu-other.c.externalDefMap.ast-dump.txt

diff  --git a/clang/test/Analysis/Inputs/ctu-other.cpp.externalDefMap.txt b/clang/test/Analysis/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt
similarity index 100%
rename from clang/test/Analysis/Inputs/ctu-other.cpp.externalDefMap.txt
rename to clang/test/Analysis/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt

diff  --git a/clang/test/Analysis/analyzer-config.c b/clang/test/Analysis/analyzer-config.c
index cb3d40688e91..1d4a96389fb3 100644
--- a/clang/test/Analysis/analyzer-config.c
+++ b/clang/test/Analysis/analyzer-config.c
@@ -43,6 +43,7 @@
 // CHECK-NEXT: ctu-dir = ""
 // CHECK-NEXT: ctu-import-threshold = 100
 // CHECK-NEXT: ctu-index-name = externalDefMap.txt
+// CHECK-NEXT: ctu-invocation-list = invocations.yaml
 // CHECK-NEXT: deadcode.DeadStores:ShowFixIts = false
 // CHECK-NEXT: deadcode.DeadStores:WarnForDeadNestedAssignments = true
 // CHECK-NEXT: debug.AnalysisOrder:* = false

diff  --git a/clang/test/Analysis/ctu-
diff erent-triples.cpp b/clang/test/Analysis/ctu-
diff erent-triples.cpp
index 20acc318e2e7..68c7b0dd7dc3 100644
--- a/clang/test/Analysis/ctu-
diff erent-triples.cpp
+++ b/clang/test/Analysis/ctu-
diff erent-triples.cpp
@@ -2,7 +2,7 @@
 // RUN: mkdir -p %t/ctudir
 // RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -emit-pch -o %t/ctudir/ctu-other.cpp.ast %S/Inputs/ctu-other.cpp
-// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.txt %t/ctudir/externalDefMap.txt
+// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt %t/ctudir/externalDefMap.txt
 // RUN: %clang_analyze_cc1 -std=c++14 -triple powerpc64-montavista-linux-gnu \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \

diff  --git a/clang/test/Analysis/ctu-main.c b/clang/test/Analysis/ctu-main.c
index 114d694020a1..d991eb73a95c 100644
--- a/clang/test/Analysis/ctu-main.c
+++ b/clang/test/Analysis/ctu-main.c
@@ -2,7 +2,7 @@
 // RUN: mkdir -p %t/ctudir2
 // RUN: %clang_cc1 -triple x86_64-pc-linux-gnu \
 // RUN:   -emit-pch -o %t/ctudir2/ctu-other.c.ast %S/Inputs/ctu-other.c
-// RUN: cp %S/Inputs/ctu-other.c.externalDefMap.txt %t/ctudir2/externalDefMap.txt
+// RUN: cp %S/Inputs/ctu-other.c.externalDefMap.ast-dump.txt %t/ctudir2/externalDefMap.txt
 // RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -fsyntax-only -std=c89 -analyze \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
@@ -50,6 +50,10 @@ void testMacro(void) {
 void testImplicit() {
   int res = identImplicit(6);   // external implicit functions are not inlined
   clang_analyzer_eval(res == 6); // expected-warning{{TRUE}}
+  // Call something with uninitialized from the same function in which the implicit was called.
+  // This is necessary to reproduce a special bug in NoStoreFuncVisitor.
+  int uninitialized;
+  h(uninitialized); // expected-warning{{1st function call argument is an uninitialized value}}
 }
 
 // Tests the import of functions that have a struct parameter

diff  --git a/clang/test/Analysis/ctu-main.cpp b/clang/test/Analysis/ctu-main.cpp
index 3f095a0aabc3..d76b3984401e 100644
--- a/clang/test/Analysis/ctu-main.cpp
+++ b/clang/test/Analysis/ctu-main.cpp
@@ -4,7 +4,7 @@
 // RUN:   -emit-pch -o %t/ctudir/ctu-other.cpp.ast %S/Inputs/ctu-other.cpp
 // RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -emit-pch -o %t/ctudir/ctu-chain.cpp.ast %S/Inputs/ctu-chain.cpp
-// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.txt %t/ctudir/externalDefMap.txt
+// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt %t/ctudir/externalDefMap.txt
 // RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \

diff  --git a/clang/test/Analysis/ctu-on-demand-parsing.c b/clang/test/Analysis/ctu-on-demand-parsing.c
new file mode 100644
index 000000000000..ebea19282949
--- /dev/null
+++ b/clang/test/Analysis/ctu-on-demand-parsing.c
@@ -0,0 +1,83 @@
+// RUN: rm -rf %t
+// RUN: mkdir -p %t
+// RUN: cp "%s" "%t/ctu-on-demand-parsing.c"
+// RUN: cp "%S/Inputs/ctu-other.c" "%t/ctu-other.c"
+//
+// Path substitutions on Windows platform could contain backslashes. These are escaped in the json file.
+// compile_commands.json is only needed for extdef_mapping, not for the analysis itself.
+// RUN: echo '[{"directory":"%t","command":"gcc -std=c89 -Wno-visibility ctu-other.c","file":"ctu-other.c"}]' | sed -e 's/\\/\\\\/g' > %t/compile_commands.json
+//
+// RUN: echo '"%t/ctu-other.c": ["gcc", "-std=c89", "-Wno-visibility", "ctu-other.c"]' | sed -e 's/\\/\\\\/g' > %t/invocations.yaml
+//
+// RUN: cd "%t" && %clang_extdef_map "%t/ctu-other.c" > externalDefMap.txt
+//
+// RUN: cd "%t" && %clang_cc1 -fsyntax-only -std=c89 -analyze \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=. \
+// RUN:   -analyzer-config ctu-invocation-list=invocations.yaml \
+// RUN:   -verify ctu-on-demand-parsing.c
+//
+// FIXME: remove xfail when PS4-windows buildslave can be satisfied
+// UNSUPPORTED: windows
+
+void clang_analyzer_eval(int);
+
+// Test typedef and global variable in function.
+typedef struct {
+  int a;
+  int b;
+} FooBar;
+extern FooBar fb;
+int f(int);
+void testGlobalVariable() {
+  clang_analyzer_eval(f(5) == 1); // expected-warning{{TRUE}}
+}
+
+// Test enums.
+int enumCheck(void);
+enum A { x,
+         y,
+         z };
+void testEnum() {
+  clang_analyzer_eval(x == 0);            // expected-warning{{TRUE}}
+  clang_analyzer_eval(enumCheck() == 42); // expected-warning{{TRUE}}
+}
+
+// Test that asm import does not fail.
+int inlineAsm();
+int testInlineAsm() { return inlineAsm(); }
+
+// Test reporting error in a macro.
+struct S;
+int g(struct S *);
+void testMacro(void) {
+  g(0);
+  // expected-warning at ctu-other.c:29 {{Access to field 'a' results in a dereference of a null pointer (loaded from variable 'ctx')}}
+}
+
+// The external function prototype is incomplete.
+// warning:implicit functions are prohibited by c99
+void testImplicit() {
+  int res = identImplicit(6);    // external implicit functions are not inlined
+  clang_analyzer_eval(res == 6); // expected-warning{{TRUE}}
+  // Call something with uninitialized from the same function in which the
+  // implicit was called. This is necessary to reproduce a special bug in
+  // NoStoreFuncVisitor.
+  int uninitialized;
+  h(uninitialized); // expected-warning{{1st function call argument is an uninitialized value}}
+}
+
+// Tests the import of functions that have a struct parameter
+// defined in its prototype.
+struct DataType {
+  int a;
+  int b;
+};
+int structInProto(struct DataType *d);
+void testStructDefInArgument() {
+  struct DataType d;
+  d.a = 1;
+  d.b = 0;
+  clang_analyzer_eval(structInProto(&d) == 0); // expected-warning{{TRUE}} expected-warning{{FALSE}}
+}

diff  --git a/clang/test/Analysis/ctu-on-demand-parsing.cpp b/clang/test/Analysis/ctu-on-demand-parsing.cpp
new file mode 100644
index 000000000000..6fc757cb11b0
--- /dev/null
+++ b/clang/test/Analysis/ctu-on-demand-parsing.cpp
@@ -0,0 +1,111 @@
+// RUN: rm -rf %t
+// RUN: mkdir -p %t/Inputs
+// RUN: cp %s %t/ctu-on-demand-parsing.cpp
+// RUN: cp %S/ctu-hdr.h %t/ctu-hdr.h
+// RUN: cp %S/Inputs/ctu-chain.cpp %t/Inputs/ctu-chain.cpp
+// RUN: cp %S/Inputs/ctu-other.cpp %t/Inputs/ctu-other.cpp
+//
+// Path substitutions on Windows platform could contain backslashes. These are escaped in the json file.
+// compile_commands.json is only needed for the extdef_mapping, not for the analysis itself.
+// RUN: echo '[{"directory":"%t/Inputs","command":"clang++ ctu-chain.cpp","file":"ctu-chain.cpp"},{"directory":"%t/Inputs","command":"clang++ ctu-other.cpp","file":"ctu-other.cpp"}]' | sed -e 's/\\/\\\\/g' > %t/compile_commands.json
+//
+// RUN: echo '{"%t/Inputs/ctu-chain.cpp": ["g++", "%t/Inputs/ctu-chain.cpp"], "%t/Inputs/ctu-other.cpp": ["g++", "%t/Inputs/ctu-other.cpp"]}' | sed -e 's/\\/\\\\/g' > %t/invocations.yaml
+//
+// RUN: cd "%t" && %clang_extdef_map Inputs/ctu-chain.cpp Inputs/ctu-other.cpp > externalDefMap.txt
+//
+// RUN: cd "%t" && %clang_analyze_cc1 \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=. \
+// RUN:   -analyzer-config ctu-invocation-list=invocations.yaml \
+// RUN:   -verify ctu-on-demand-parsing.cpp
+// RUN: cd "%t" && %clang_analyze_cc1 \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=. \
+// RUN:   -analyzer-config ctu-invocation-list=invocations.yaml \
+// RUN:   -analyzer-config display-ctu-progress=true ctu-on-demand-parsing.cpp 2>&1 | FileCheck %t/ctu-on-demand-parsing.cpp
+//
+// CHECK: CTU loaded AST file: {{.*}}ctu-other.cpp
+// CHECK: CTU loaded AST file: {{.*}}ctu-chain.cpp
+//
+// FIXME: remove xfail when PS4-windows buildslave can be satisfied
+// UNSUPPORTED: windows
+
+#include "ctu-hdr.h"
+
+void clang_analyzer_eval(int);
+
+int f(int);
+int g(int);
+int h(int);
+
+int callback_to_main(int x) { return x + 1; }
+
+namespace myns {
+int fns(int x);
+
+namespace embed_ns {
+int fens(int x);
+}
+
+class embed_cls {
+public:
+  int fecl(int x);
+};
+} // namespace myns
+
+class mycls {
+public:
+  int fcl(int x);
+  virtual int fvcl(int x);
+  static int fscl(int x);
+
+  class embed_cls2 {
+  public:
+    int fecl2(int x);
+  };
+};
+
+class derived : public mycls {
+public:
+  virtual int fvcl(int x) override;
+};
+
+namespace chns {
+int chf1(int x);
+}
+
+int fun_using_anon_struct(int);
+int other_macro_diag(int);
+
+void test_virtual_functions(mycls *obj) {
+  // The dynamic type is known.
+  clang_analyzer_eval(mycls().fvcl(1) == 8);   // expected-warning{{TRUE}}
+  clang_analyzer_eval(derived().fvcl(1) == 9); // expected-warning{{TRUE}}
+  // We cannot decide about the dynamic type.
+  clang_analyzer_eval(obj->fvcl(1) == 8); // expected-warning{{FALSE}} expected-warning{{TRUE}}
+  clang_analyzer_eval(obj->fvcl(1) == 9); // expected-warning{{FALSE}} expected-warning{{TRUE}}
+}
+
+int main() {
+  clang_analyzer_eval(f(3) == 2); // expected-warning{{TRUE}}
+  clang_analyzer_eval(f(4) == 3); // expected-warning{{TRUE}}
+  clang_analyzer_eval(f(5) == 3); // expected-warning{{FALSE}}
+  clang_analyzer_eval(g(4) == 6); // expected-warning{{TRUE}}
+  clang_analyzer_eval(h(2) == 8); // expected-warning{{TRUE}}
+
+  clang_analyzer_eval(myns::fns(2) == 9);                   // expected-warning{{TRUE}}
+  clang_analyzer_eval(myns::embed_ns::fens(2) == -1);       // expected-warning{{TRUE}}
+  clang_analyzer_eval(mycls().fcl(1) == 6);                 // expected-warning{{TRUE}}
+  clang_analyzer_eval(mycls::fscl(1) == 7);                 // expected-warning{{TRUE}}
+  clang_analyzer_eval(myns::embed_cls().fecl(1) == -6);     // expected-warning{{TRUE}}
+  clang_analyzer_eval(mycls::embed_cls2().fecl2(0) == -11); // expected-warning{{TRUE}}
+
+  clang_analyzer_eval(chns::chf1(4) == 12);           // expected-warning{{TRUE}}
+  clang_analyzer_eval(fun_using_anon_struct(8) == 8); // expected-warning{{TRUE}}
+
+  clang_analyzer_eval(other_macro_diag(1) == 1); // expected-warning{{TRUE}}
+  // expected-warning at Inputs/ctu-other.cpp:93{{REACHABLE}}
+  MACRODIAG(); // expected-warning{{REACHABLE}}
+}

diff  --git a/clang/test/Analysis/ctu-unknown-parts-in-triples.cpp b/clang/test/Analysis/ctu-unknown-parts-in-triples.cpp
index 6bcbd709b5ef..f41f07d1edc9 100644
--- a/clang/test/Analysis/ctu-unknown-parts-in-triples.cpp
+++ b/clang/test/Analysis/ctu-unknown-parts-in-triples.cpp
@@ -5,7 +5,7 @@
 // RUN: mkdir -p %t/ctudir
 // RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -emit-pch -o %t/ctudir/ctu-other.cpp.ast %S/Inputs/ctu-other.cpp
-// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.txt %t/ctudir/externalDefMap.txt
+// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt %t/ctudir/externalDefMap.txt
 // RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-unknown-linux-gnu \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \

diff  --git a/clang/unittests/CrossTU/CrossTranslationUnitTest.cpp b/clang/unittests/CrossTU/CrossTranslationUnitTest.cpp
index 86ede5e319cb..b8bf7363af7f 100644
--- a/clang/unittests/CrossTU/CrossTranslationUnitTest.cpp
+++ b/clang/unittests/CrossTU/CrossTranslationUnitTest.cpp
@@ -7,10 +7,11 @@
 //===----------------------------------------------------------------------===//
 
 #include "clang/CrossTU/CrossTranslationUnit.h"
-#include "clang/Frontend/CompilerInstance.h"
 #include "clang/AST/ASTConsumer.h"
+#include "clang/Frontend/CompilerInstance.h"
 #include "clang/Frontend/FrontendAction.h"
 #include "clang/Tooling/Tooling.h"
+#include "llvm/ADT/Optional.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/ToolOutputFile.h"
@@ -162,7 +163,7 @@ TEST(CrossTranslationUnit, IndexFormatCanBeParsed) {
   IndexFile.os().flush();
   EXPECT_TRUE(llvm::sys::fs::exists(IndexFileName));
   llvm::Expected<llvm::StringMap<std::string>> IndexOrErr =
-      parseCrossTUIndex(IndexFileName, "");
+      parseCrossTUIndex(IndexFileName);
   EXPECT_TRUE((bool)IndexOrErr);
   llvm::StringMap<std::string> ParsedIndex = IndexOrErr.get();
   for (const auto &E : Index) {
@@ -173,24 +174,98 @@ TEST(CrossTranslationUnit, IndexFormatCanBeParsed) {
     EXPECT_TRUE(Index.count(E.getKey()));
 }
 
-TEST(CrossTranslationUnit, CTUDirIsHandledCorrectly) {
-  llvm::StringMap<std::string> Index;
-  Index["a"] = "/b/c/d";
-  std::string IndexText = createCrossTUIndexString(Index);
+TEST(CrossTranslationUnit, EmptyInvocationListIsNotValid) {
+  auto Input = "";
 
-  int IndexFD;
-  llvm::SmallString<256> IndexFileName;
-  ASSERT_FALSE(llvm::sys::fs::createTemporaryFile("index", "txt", IndexFD,
-                                                  IndexFileName));
-  llvm::ToolOutputFile IndexFile(IndexFileName, IndexFD);
-  IndexFile.os() << IndexText;
-  IndexFile.os().flush();
-  EXPECT_TRUE(llvm::sys::fs::exists(IndexFileName));
-  llvm::Expected<llvm::StringMap<std::string>> IndexOrErr =
-      parseCrossTUIndex(IndexFileName, "/ctudir");
-  EXPECT_TRUE((bool)IndexOrErr);
-  llvm::StringMap<std::string> ParsedIndex = IndexOrErr.get();
-  EXPECT_EQ(ParsedIndex["a"], "/ctudir/b/c/d");
+  llvm::Expected<InvocationListTy> Result = parseInvocationList(Input);
+  EXPECT_FALSE(static_cast<bool>(Result));
+  bool IsWrongFromatError = false;
+  llvm::handleAllErrors(Result.takeError(), [&](IndexError &Err) {
+    IsWrongFromatError =
+        Err.getCode() == index_error_code::invocation_list_wrong_format;
+  });
+  EXPECT_TRUE(IsWrongFromatError);
+}
+
+TEST(CrossTranslationUnit, AmbiguousInvocationListIsDetected) {
+  // The same source file occurs twice (for two 
diff erent architecture) in
+  // this test case. The disambiguation is the responsibility of the user.
+  auto Input = R"(
+  /tmp/main.cpp:
+    - clang++
+    - -c
+    - -m32
+    - -o
+    - main32.o
+    - /tmp/main.cpp
+  /tmp/main.cpp:
+    - clang++
+    - -c
+    - -m64
+    - -o
+    - main64.o
+    - /tmp/main.cpp
+  )";
+
+  llvm::Expected<InvocationListTy> Result = parseInvocationList(Input);
+  EXPECT_FALSE(static_cast<bool>(Result));
+  bool IsAmbiguousError = false;
+  llvm::handleAllErrors(Result.takeError(), [&](IndexError &Err) {
+    IsAmbiguousError =
+        Err.getCode() == index_error_code::invocation_list_ambiguous;
+  });
+  EXPECT_TRUE(IsAmbiguousError);
+}
+
+TEST(CrossTranslationUnit, SingleInvocationCanBeParsed) {
+  auto Input = R"(
+  /tmp/main.cpp:
+    - clang++
+    - /tmp/main.cpp
+  )";
+  llvm::Expected<InvocationListTy> Result = parseInvocationList(Input);
+  EXPECT_TRUE(static_cast<bool>(Result));
+
+  EXPECT_EQ(Result->size(), 1);
+
+  auto It = Result->find("/tmp/main.cpp");
+  EXPECT_TRUE(It != Result->end());
+  EXPECT_EQ(It->getValue()[0], "clang++");
+  EXPECT_EQ(It->getValue()[1], "/tmp/main.cpp");
+}
+
+TEST(CrossTranslationUnit, MultipleInvocationsCanBeParsed) {
+  auto Input = R"(
+  /tmp/main.cpp:
+    - clang++
+    - /tmp/other.o
+    - /tmp/main.cpp
+  /tmp/other.cpp:
+    - g++
+    - -c
+    - -o
+    - /tmp/other.o
+    - /tmp/other.cpp
+  )";
+  llvm::Expected<InvocationListTy> Result = parseInvocationList(Input);
+  EXPECT_TRUE(static_cast<bool>(Result));
+
+  EXPECT_EQ(Result->size(), 2);
+
+  auto It = Result->find("/tmp/main.cpp");
+  EXPECT_TRUE(It != Result->end());
+  EXPECT_EQ(It->getKey(), "/tmp/main.cpp");
+  EXPECT_EQ(It->getValue()[0], "clang++");
+  EXPECT_EQ(It->getValue()[1], "/tmp/other.o");
+  EXPECT_EQ(It->getValue()[2], "/tmp/main.cpp");
+
+  It = Result->find("/tmp/other.cpp");
+  EXPECT_TRUE(It != Result->end());
+  EXPECT_EQ(It->getValue()[0], "g++");
+  EXPECT_EQ(It->getValue()[1], "-c");
+  EXPECT_EQ(It->getValue()[2], "-o");
+  EXPECT_EQ(It->getValue()[3], "/tmp/other.o");
+  EXPECT_EQ(It->getValue()[4], "/tmp/other.cpp");
 }
 
 } // end namespace cross_tu


        


More information about the cfe-commits mailing list