[clang] 33a745e - [clang][clang-scan-deps] Add support for extracting full module dependencies.
Reid Kleckner via cfe-commits
cfe-commits at lists.llvm.org
Thu Oct 31 14:49:05 PDT 2019
You could hack it to standardize the yaml output to forward slashes with
sed, or clang might wish to do it internally.
My current position is that we should pick the path style and slash
direction closest to the point at which the path will be emitted. In the
past we've tried to make arguments that slashes should be internally one
way or another depending on if it's Windows, MinGW, MSVC, or some other
criteria, but the codebase is inconsistent. Now I lean towards
canonicalizing when printing based on the format in use.
- For example, if emitting codeview, try to standardize to backslashes to
match MSVC.
- If emitting DWARF, maybe use forward slashes since the consumer will
probably be a gnu tool.
- When implementing -print-*-path in the driver, the user is probably a
Makefile, so use forward slashes.
- When emitting this YAML stuff, it's our own format, so do whatever is
convenient.
On Wed, Oct 30, 2019 at 4:03 PM Michael Spencer <bigcheesegs at gmail.com>
wrote:
> On Wed, Oct 30, 2019 at 3:54 PM Reid Kleckner via cfe-commits <
> cfe-commits at lists.llvm.org> wrote:
>
>> I XFAILED the test on Windows in 52194350cfe. The [/\\] regexes
>> might need to match two slashes, and the PREFIX variable ends up having
>> inconsistent slash direction.
>>
>
> Hmm, fixing the slashes should be easy, but I'm not sure there's a good
> way to get the PREFIX right. I feel like FileCheck needs an ignore slash
> direction mode.
>
> - Michael Spencer
>
>
>>
>> On Wed, Oct 30, 2019 at 3:27 PM Michael Spencer via cfe-commits <
>> cfe-commits at lists.llvm.org> wrote:
>>
>>>
>>> Author: Michael Spencer
>>> Date: 2019-10-30T15:27:27-07:00
>>> New Revision: 33a745e6fe7e81d3793f7831d2832aa0785ef327
>>>
>>> URL:
>>> https://github.com/llvm/llvm-project/commit/33a745e6fe7e81d3793f7831d2832aa0785ef327
>>> DIFF:
>>> https://github.com/llvm/llvm-project/commit/33a745e6fe7e81d3793f7831d2832aa0785ef327.diff
>>>
>>> LOG: [clang][clang-scan-deps] Add support for extracting full module
>>> dependencies.
>>>
>>> This is a recommit of d8a4ef0e685c with the nondeterminism fixed.
>>>
>>> This adds experimental support for extracting a Clang module dependency
>>> graph
>>> from a compilation database. The output format is experimental and will
>>> change.
>>> It is currently a concatenation of JSON outputs for each compilation.
>>> Future
>>> patches will change this to deduplicate modules between compilations.
>>>
>>> Differential Revision: https://reviews.llvm.org/D69420
>>>
>>> Added:
>>> clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
>>> clang/lib/Tooling/DependencyScanning/ModuleDepCollector.cpp
>>> clang/test/ClangScanDeps/modules-full.cpp
>>>
>>> Modified:
>>>
>>> clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
>>>
>>> clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
>>>
>>> clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
>>> clang/lib/Tooling/DependencyScanning/CMakeLists.txt
>>> clang/lib/Tooling/DependencyScanning/DependencyScanningService.cpp
>>> clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp
>>> clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
>>> clang/tools/clang-scan-deps/ClangScanDeps.cpp
>>>
>>> Removed:
>>>
>>>
>>>
>>>
>>> ################################################################################
>>> diff --git
>>> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
>>> b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
>>> index fd8ed80b143c..76edf150dbee 100644
>>> ---
>>> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
>>> +++
>>> b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
>>> @@ -30,15 +30,30 @@ enum class ScanningMode {
>>> MinimizedSourcePreprocessing
>>> };
>>>
>>> +/// The format that is output by the dependency scanner.
>>> +enum class ScanningOutputFormat {
>>> + /// This is the Makefile compatible dep format. This will include all
>>> of the
>>> + /// deps necessary for an implicit modules build, but won't include
>>> any
>>> + /// intermodule dependency information.
>>> + Make,
>>> +
>>> + /// This outputs the full module dependency graph suitable for use for
>>> + /// explicitly building modules.
>>> + Full,
>>> +};
>>> +
>>> /// The dependency scanning service contains the shared state that is
>>> used by
>>> /// the invidual dependency scanning workers.
>>> class DependencyScanningService {
>>> public:
>>> - DependencyScanningService(ScanningMode Mode, bool ReuseFileManager =
>>> true,
>>> + DependencyScanningService(ScanningMode Mode, ScanningOutputFormat
>>> Format,
>>> + bool ReuseFileManager = true,
>>> bool SkipExcludedPPRanges = true);
>>>
>>> ScanningMode getMode() const { return Mode; }
>>>
>>> + ScanningOutputFormat getFormat() const { return Format; }
>>> +
>>> bool canReuseFileManager() const { return ReuseFileManager; }
>>>
>>> bool canSkipExcludedPPRanges() const { return SkipExcludedPPRanges; }
>>> @@ -49,6 +64,7 @@ class DependencyScanningService {
>>>
>>> private:
>>> const ScanningMode Mode;
>>> + const ScanningOutputFormat Format;
>>> const bool ReuseFileManager;
>>> /// Set to true to use the preprocessor optimization that skips
>>> excluded PP
>>> /// ranges by bumping the buffer pointer in the lexer instead of
>>> lexing the
>>>
>>> diff --git
>>> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
>>> b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
>>> index c950cbe167cd..78b49e4fa0c5 100644
>>> ---
>>> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
>>> +++
>>> b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningTool.h
>>> @@ -40,6 +40,7 @@ class DependencyScanningTool {
>>> StringRef CWD);
>>>
>>> private:
>>> + const ScanningOutputFormat Format;
>>> DependencyScanningWorker Worker;
>>> const tooling::CompilationDatabase &Compilations;
>>> };
>>>
>>> diff --git
>>> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
>>> b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
>>> index 45c9fb4f029d..689119330c41 100644
>>> ---
>>> a/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
>>> +++
>>> b/clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
>>> @@ -15,6 +15,8 @@
>>> #include "clang/Frontend/PCHContainerOperations.h"
>>> #include
>>> "clang/Lex/PreprocessorExcludedConditionalDirectiveSkipMapping.h"
>>> #include "clang/Tooling/CompilationDatabase.h"
>>> +#include "clang/Tooling/DependencyScanning/DependencyScanningService.h"
>>> +#include "clang/Tooling/DependencyScanning/ModuleDepCollector.h"
>>> #include "llvm/Support/Error.h"
>>> #include "llvm/Support/FileSystem.h"
>>> #include <string>
>>> @@ -26,7 +28,6 @@ class DependencyOutputOptions;
>>> namespace tooling {
>>> namespace dependencies {
>>>
>>> -class DependencyScanningService;
>>> class DependencyScanningWorkerFilesystem;
>>>
>>> class DependencyConsumer {
>>> @@ -36,7 +37,9 @@ class DependencyConsumer {
>>> virtual void handleFileDependency(const DependencyOutputOptions &Opts,
>>> StringRef Filename) = 0;
>>>
>>> - // FIXME: Add support for reporting modular dependencies.
>>> + virtual void handleModuleDependency(ModuleDeps MD) = 0;
>>> +
>>> + virtual void handleContextHash(std::string Hash) = 0;
>>> };
>>>
>>> /// An individual dependency scanning worker that is able to run on its
>>> own
>>> @@ -73,6 +76,7 @@ class DependencyScanningWorker {
>>> /// The file manager that is reused accross multiple invocations by
>>> this
>>> /// worker. If null, the file manager will not be reused.
>>> llvm::IntrusiveRefCntPtr<FileManager> Files;
>>> + ScanningOutputFormat Format;
>>> };
>>>
>>> } // end namespace dependencies
>>>
>>> diff --git
>>> a/clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
>>> b/clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
>>> new file mode 100644
>>> index 000000000000..7a9fc276fcaa
>>> --- /dev/null
>>> +++ b/clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
>>> @@ -0,0 +1,94 @@
>>> +//===- ModuleDepCollector.h - Callbacks to collect deps ---------*- C++
>>> -*-===//
>>> +//
>>> +// The LLVM Compiler Infrastructure
>>> +//
>>> +// This file is distributed under the University of Illinois Open Source
>>> +// License. See LICENSE.TXT for details.
>>> +//
>>>
>>> +//===----------------------------------------------------------------------===//
>>> +
>>> +#ifndef LLVM_CLANG_TOOLING_DEPENDENCY_SCANNING_MODULE_DEP_COLLECTOR_H
>>> +#define LLVM_CLANG_TOOLING_DEPENDENCY_SCANNING_MODULE_DEP_COLLECTOR_H
>>> +
>>> +#include "clang/Basic/LLVM.h"
>>> +#include "clang/Basic/SourceManager.h"
>>> +#include "clang/Frontend/Utils.h"
>>> +#include "clang/Lex/HeaderSearch.h"
>>> +#include "clang/Lex/PPCallbacks.h"
>>> +#include "clang/Serialization/ASTReader.h"
>>> +#include "llvm/ADT/DenseMap.h"
>>> +#include "llvm/ADT/StringSet.h"
>>> +#include "llvm/Support/raw_ostream.h"
>>> +
>>> +#include <string>
>>> +
>>> +namespace clang {
>>> +namespace tooling {
>>> +namespace dependencies {
>>> +
>>> +class DependencyConsumer;
>>> +
>>> +struct ModuleDeps {
>>> + std::string ModuleName;
>>> + std::string ClangModuleMapFile;
>>> + std::string ModulePCMPath;
>>> + std::string ContextHash;
>>> + llvm::StringSet<> FileDeps;
>>> + llvm::StringSet<> ClangModuleDeps;
>>> + bool ImportedByMainFile = false;
>>> +};
>>> +
>>> +class ModuleDepCollector;
>>> +
>>> +class ModuleDepCollectorPP final : public PPCallbacks {
>>> +public:
>>> + ModuleDepCollectorPP(CompilerInstance &I, ModuleDepCollector &MDC)
>>> + : Instance(I), MDC(MDC) {}
>>> +
>>> + void FileChanged(SourceLocation Loc, FileChangeReason Reason,
>>> + SrcMgr::CharacteristicKind FileType,
>>> + FileID PrevFID) override;
>>> + void InclusionDirective(SourceLocation HashLoc, const Token
>>> &IncludeTok,
>>> + StringRef FileName, bool IsAngled,
>>> + CharSourceRange FilenameRange, const
>>> FileEntry *File,
>>> + StringRef SearchPath, StringRef RelativePath,
>>> + const Module *Imported,
>>> + SrcMgr::CharacteristicKind FileType) override;
>>> +
>>> + void EndOfMainFile() override;
>>> +
>>> +private:
>>> + CompilerInstance &Instance;
>>> + ModuleDepCollector &MDC;
>>> + llvm::DenseSet<const Module *> DirectDeps;
>>> +
>>> + void handleTopLevelModule(const Module *M);
>>> + void addAllSubmoduleDeps(const Module *M, ModuleDeps &MD);
>>> + void addModuleDep(const Module *M, ModuleDeps &MD);
>>> +
>>> + void addDirectDependencies(const Module *Mod);
>>> +};
>>> +
>>> +class ModuleDepCollector final : public DependencyCollector {
>>> +public:
>>> + ModuleDepCollector(CompilerInstance &I, DependencyConsumer &C);
>>> +
>>> + void attachToPreprocessor(Preprocessor &PP) override;
>>> + void attachToASTReader(ASTReader &R) override;
>>> +
>>> +private:
>>> + friend ModuleDepCollectorPP;
>>> +
>>> + CompilerInstance &Instance;
>>> + DependencyConsumer &Consumer;
>>> + std::string MainFile;
>>> + std::string ContextHash;
>>> + std::vector<std::string> MainDeps;
>>> + std::unordered_map<std::string, ModuleDeps> Deps;
>>> +};
>>> +
>>> +} // end namespace dependencies
>>> +} // end namespace tooling
>>> +} // end namespace clang
>>> +
>>> +#endif // LLVM_CLANG_TOOLING_DEPENDENCY_SCANNING_MODULE_DEP_COLLECTOR_H
>>>
>>> diff --git a/clang/lib/Tooling/DependencyScanning/CMakeLists.txt
>>> b/clang/lib/Tooling/DependencyScanning/CMakeLists.txt
>>> index 05e1aa54f8d4..c6fe207ab2f2 100644
>>> --- a/clang/lib/Tooling/DependencyScanning/CMakeLists.txt
>>> +++ b/clang/lib/Tooling/DependencyScanning/CMakeLists.txt
>>> @@ -8,6 +8,7 @@ add_clang_library(clangDependencyScanning
>>> DependencyScanningService.cpp
>>> DependencyScanningWorker.cpp
>>> DependencyScanningTool.cpp
>>> + ModuleDepCollector.cpp
>>>
>>> DEPENDS
>>> ClangDriverOptions
>>>
>>> diff --git
>>> a/clang/lib/Tooling/DependencyScanning/DependencyScanningService.cpp
>>> b/clang/lib/Tooling/DependencyScanning/DependencyScanningService.cpp
>>> index e5cebe381000..93bb0cde439d 100644
>>> --- a/clang/lib/Tooling/DependencyScanning/DependencyScanningService.cpp
>>> +++ b/clang/lib/Tooling/DependencyScanning/DependencyScanningService.cpp
>>> @@ -12,8 +12,8 @@ using namespace clang;
>>> using namespace tooling;
>>> using namespace dependencies;
>>>
>>> -DependencyScanningService::DependencyScanningService(ScanningMode Mode,
>>> - bool
>>> ReuseFileManager,
>>> - bool
>>> SkipExcludedPPRanges)
>>> - : Mode(Mode), ReuseFileManager(ReuseFileManager),
>>> +DependencyScanningService::DependencyScanningService(
>>> + ScanningMode Mode, ScanningOutputFormat Format, bool
>>> ReuseFileManager,
>>> + bool SkipExcludedPPRanges)
>>> + : Mode(Mode), Format(Format), ReuseFileManager(ReuseFileManager),
>>> SkipExcludedPPRanges(SkipExcludedPPRanges) {}
>>>
>>> diff --git
>>> a/clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp
>>> b/clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp
>>> index d2af1a9d110c..bffd7c338124 100644
>>> --- a/clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp
>>> +++ b/clang/lib/Tooling/DependencyScanning/DependencyScanningTool.cpp
>>> @@ -8,6 +8,15 @@
>>>
>>> #include "clang/Tooling/DependencyScanning/DependencyScanningTool.h"
>>> #include "clang/Frontend/Utils.h"
>>> +#include "llvm/Support/JSON.h"
>>> +
>>> +static llvm::json::Array toJSONSorted(const llvm::StringSet<> &Set) {
>>> + std::vector<llvm::StringRef> Strings;
>>> + for (auto &&I : Set)
>>> + Strings.push_back(I.getKey());
>>> + std::sort(Strings.begin(), Strings.end());
>>> + return llvm::json::Array(Strings);
>>> +}
>>>
>>> namespace clang{
>>> namespace tooling{
>>> @@ -16,13 +25,14 @@ namespace dependencies{
>>> DependencyScanningTool::DependencyScanningTool(
>>> DependencyScanningService &Service,
>>> const tooling::CompilationDatabase &Compilations)
>>> - : Worker(Service), Compilations(Compilations) {}
>>> + : Format(Service.getFormat()), Worker(Service),
>>> Compilations(Compilations) {
>>> +}
>>>
>>> llvm::Expected<std::string>
>>> DependencyScanningTool::getDependencyFile(const std::string &Input,
>>> StringRef CWD) {
>>> /// Prints out all of the gathered dependencies into a string.
>>> - class DependencyPrinterConsumer : public DependencyConsumer {
>>> + class MakeDependencyPrinterConsumer : public DependencyConsumer {
>>> public:
>>> void handleFileDependency(const DependencyOutputOptions &Opts,
>>> StringRef File) override {
>>> @@ -31,6 +41,14 @@ DependencyScanningTool::getDependencyFile(const
>>> std::string &Input,
>>> Dependencies.push_back(File);
>>> }
>>>
>>> + void handleModuleDependency(ModuleDeps MD) override {
>>> + // These are ignored for the make format as it can't support the
>>> full
>>> + // set of deps, and handleFileDependency handles enough for
>>> implicitly
>>> + // built modules to work.
>>> + }
>>> +
>>> + void handleContextHash(std::string Hash) override {}
>>> +
>>> void printDependencies(std::string &S) {
>>> if (!Opts)
>>> return;
>>> @@ -59,14 +77,88 @@ DependencyScanningTool::getDependencyFile(const
>>> std::string &Input,
>>> std::vector<std::string> Dependencies;
>>> };
>>>
>>> - DependencyPrinterConsumer Consumer;
>>> - auto Result =
>>> - Worker.computeDependencies(Input, CWD, Compilations, Consumer);
>>> - if (Result)
>>> - return std::move(Result);
>>> - std::string Output;
>>> - Consumer.printDependencies(Output);
>>> - return Output;
>>> + class FullDependencyPrinterConsumer : public DependencyConsumer {
>>> + public:
>>> + void handleFileDependency(const DependencyOutputOptions &Opts,
>>> + StringRef File) override {
>>> + Dependencies.push_back(File);
>>> + }
>>> +
>>> + void handleModuleDependency(ModuleDeps MD) override {
>>> + ModuleDeps[MD.ContextHash + MD.ModuleName] = std::move(MD);
>>> + }
>>> +
>>> + void handleContextHash(std::string Hash) override {
>>> + ContextHash = std::move(Hash);
>>> + }
>>> +
>>> + void printDependencies(std::string &S, StringRef MainFile) {
>>> + // Sort the modules by name to get a deterministic order.
>>> + std::vector<StringRef> Modules;
>>> + for (auto &&Dep : ModuleDeps)
>>> + Modules.push_back(Dep.first);
>>> + std::sort(Modules.begin(), Modules.end());
>>> +
>>> + llvm::raw_string_ostream OS(S);
>>> +
>>> + using namespace llvm::json;
>>> +
>>> + Array Imports;
>>> + for (auto &&ModName : Modules) {
>>> + auto &MD = ModuleDeps[ModName];
>>> + if (MD.ImportedByMainFile)
>>> + Imports.push_back(MD.ModuleName);
>>> + }
>>> +
>>> + Array Mods;
>>> + for (auto &&ModName : Modules) {
>>> + auto &MD = ModuleDeps[ModName];
>>> + Object Mod{
>>> + {"name", MD.ModuleName},
>>> + {"file-deps", toJSONSorted(MD.FileDeps)},
>>> + {"clang-module-deps", toJSONSorted(MD.ClangModuleDeps)},
>>> + {"clang-modulemap-file", MD.ClangModuleMapFile},
>>> + };
>>> + Mods.push_back(std::move(Mod));
>>> + }
>>> +
>>> + Object O{
>>> + {"input-file", MainFile},
>>> + {"clang-context-hash", ContextHash},
>>> + {"file-deps", Dependencies},
>>> + {"clang-module-deps", std::move(Imports)},
>>> + {"clang-modules", std::move(Mods)},
>>> + };
>>> +
>>> + S = llvm::formatv("{0:2},\n", Value(std::move(O))).str();
>>> + return;
>>> + }
>>> +
>>> + private:
>>> + std::vector<std::string> Dependencies;
>>> + std::unordered_map<std::string, ModuleDeps> ModuleDeps;
>>> + std::string ContextHash;
>>> + };
>>> +
>>> + if (Format == ScanningOutputFormat::Make) {
>>> + MakeDependencyPrinterConsumer Consumer;
>>> + auto Result =
>>> + Worker.computeDependencies(Input, CWD, Compilations, Consumer);
>>> + if (Result)
>>> + return std::move(Result);
>>> + std::string Output;
>>> + Consumer.printDependencies(Output);
>>> + return Output;
>>> + } else {
>>> + FullDependencyPrinterConsumer Consumer;
>>> + auto Result =
>>> + Worker.computeDependencies(Input, CWD, Compilations, Consumer);
>>> + if (Result)
>>> + return std::move(Result);
>>> + std::string Output;
>>> + Consumer.printDependencies(Output, Input);
>>> + return Output;
>>> + }
>>> }
>>>
>>> } // end namespace dependencies
>>>
>>> diff --git
>>> a/clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
>>> b/clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
>>> index f382c202f8c2..edf2cf8bd70f 100644
>>> --- a/clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
>>> +++ b/clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
>>> @@ -14,6 +14,7 @@
>>> #include "clang/Frontend/Utils.h"
>>> #include "clang/Lex/PreprocessorOptions.h"
>>> #include "clang/Tooling/DependencyScanning/DependencyScanningService.h"
>>> +#include "clang/Tooling/DependencyScanning/ModuleDepCollector.h"
>>> #include "clang/Tooling/Tooling.h"
>>>
>>> using namespace clang;
>>> @@ -72,9 +73,11 @@ class DependencyScanningAction : public
>>> tooling::ToolAction {
>>> DependencyScanningAction(
>>> StringRef WorkingDirectory, DependencyConsumer &Consumer,
>>> llvm::IntrusiveRefCntPtr<DependencyScanningWorkerFilesystem>
>>> DepFS,
>>> - ExcludedPreprocessorDirectiveSkipMapping *PPSkipMappings)
>>> + ExcludedPreprocessorDirectiveSkipMapping *PPSkipMappings,
>>> + ScanningOutputFormat Format)
>>> : WorkingDirectory(WorkingDirectory), Consumer(Consumer),
>>> - DepFS(std::move(DepFS)), PPSkipMappings(PPSkipMappings) {}
>>> + DepFS(std::move(DepFS)), PPSkipMappings(PPSkipMappings),
>>> + Format(Format) {}
>>>
>>> bool runInvocation(std::shared_ptr<CompilerInvocation> Invocation,
>>> FileManager *FileMgr,
>>> @@ -131,9 +134,20 @@ class DependencyScanningAction : public
>>> tooling::ToolAction {
>>> // We need at least one -MT equivalent for the generator to work.
>>> if (Opts->Targets.empty())
>>> Opts->Targets = {"clang-scan-deps dependency"};
>>> - Compiler.addDependencyCollector(
>>> - std::make_shared<DependencyConsumerForwarder>(std::move(Opts),
>>> - Consumer));
>>> +
>>> + switch (Format) {
>>> + case ScanningOutputFormat::Make:
>>> + Compiler.addDependencyCollector(
>>> + std::make_shared<DependencyConsumerForwarder>(std::move(Opts),
>>> + Consumer));
>>> + break;
>>> + case ScanningOutputFormat::Full:
>>> + Compiler.addDependencyCollector(
>>> + std::make_shared<ModuleDepCollector>(Compiler, Consumer));
>>> + break;
>>> + }
>>> +
>>> +
>>> Consumer.handleContextHash(Compiler.getInvocation().getModuleHash());
>>>
>>> auto Action = std::make_unique<PreprocessOnlyAction>();
>>> const bool Result = Compiler.ExecuteAction(*Action);
>>> @@ -147,12 +161,14 @@ class DependencyScanningAction : public
>>> tooling::ToolAction {
>>> DependencyConsumer &Consumer;
>>> llvm::IntrusiveRefCntPtr<DependencyScanningWorkerFilesystem> DepFS;
>>> ExcludedPreprocessorDirectiveSkipMapping *PPSkipMappings;
>>> + ScanningOutputFormat Format;
>>> };
>>>
>>> } // end anonymous namespace
>>>
>>> DependencyScanningWorker::DependencyScanningWorker(
>>> - DependencyScanningService &Service) {
>>> + DependencyScanningService &Service)
>>> + : Format(Service.getFormat()) {
>>> DiagOpts = new DiagnosticOptions();
>>> PCHContainerOps = std::make_shared<PCHContainerOperations>();
>>> RealFS = new
>>> ProxyFileSystemWithoutChdir(llvm::vfs::getRealFileSystem());
>>> @@ -195,7 +211,7 @@ llvm::Error
>>> DependencyScanningWorker::computeDependencies(
>>> Tool.setPrintErrorMessage(false);
>>> Tool.setDiagnosticConsumer(&DC);
>>> DependencyScanningAction Action(WorkingDirectory, Consumer, DepFS,
>>> - PPSkipMappings.get());
>>> + PPSkipMappings.get(), Format);
>>> return !Tool.run(&Action);
>>> });
>>> }
>>>
>>> diff --git
>>> a/clang/lib/Tooling/DependencyScanning/ModuleDepCollector.cpp
>>> b/clang/lib/Tooling/DependencyScanning/ModuleDepCollector.cpp
>>> new file mode 100644
>>> index 000000000000..7f20ec7056c6
>>> --- /dev/null
>>> +++ b/clang/lib/Tooling/DependencyScanning/ModuleDepCollector.cpp
>>> @@ -0,0 +1,136 @@
>>> +//===- ModuleDepCollector.cpp - Callbacks to collect deps -------*- C++
>>> -*-===//
>>> +//
>>> +// The LLVM Compiler Infrastructure
>>> +//
>>> +// This file is distributed under the University of Illinois Open Source
>>> +// License. See LICENSE.TXT for details.
>>> +//
>>>
>>> +//===----------------------------------------------------------------------===//
>>> +
>>> +#include "clang/Tooling/DependencyScanning/ModuleDepCollector.h"
>>> +
>>> +#include "clang/Frontend/CompilerInstance.h"
>>> +#include "clang/Lex/Preprocessor.h"
>>> +#include "clang/Tooling/DependencyScanning/DependencyScanningWorker.h"
>>> +
>>> +using namespace clang;
>>> +using namespace tooling;
>>> +using namespace dependencies;
>>> +
>>> +void ModuleDepCollectorPP::FileChanged(SourceLocation Loc,
>>> + FileChangeReason Reason,
>>> + SrcMgr::CharacteristicKind
>>> FileType,
>>> + FileID PrevFID) {
>>> + if (Reason != PPCallbacks::EnterFile)
>>> + return;
>>> +
>>> + SourceManager &SM = Instance.getSourceManager();
>>> +
>>> + // Dependency generation really does want to go all the way to the
>>> + // file entry for a source location to find out what is depended on.
>>> + // We do not want #line markers to affect dependency generation!
>>> + Optional<FileEntryRef> File =
>>> + SM.getFileEntryRefForID(SM.getFileID(SM.getExpansionLoc(Loc)));
>>> + if (!File)
>>> + return;
>>> +
>>> + StringRef FileName =
>>> + llvm::sys::path::remove_leading_dotslash(File->getName());
>>> +
>>> + MDC.MainDeps.push_back(FileName);
>>> +}
>>> +
>>> +void ModuleDepCollectorPP::InclusionDirective(
>>> + SourceLocation HashLoc, const Token &IncludeTok, StringRef FileName,
>>> + bool IsAngled, CharSourceRange FilenameRange, const FileEntry *File,
>>> + StringRef SearchPath, StringRef RelativePath, const Module
>>> *Imported,
>>> + SrcMgr::CharacteristicKind FileType) {
>>> + if (!File && !Imported) {
>>> + // This is a non-modular include that HeaderSearch failed to find.
>>> Add it
>>> + // here as `FileChanged` will never see it.
>>> + MDC.MainDeps.push_back(FileName);
>>> + }
>>> +
>>> + if (!Imported)
>>> + return;
>>> +
>>> + MDC.Deps[MDC.ContextHash +
>>> Imported->getTopLevelModule()->getFullModuleName()]
>>> + .ImportedByMainFile = true;
>>> + DirectDeps.insert(Imported->getTopLevelModule());
>>> +}
>>> +
>>> +void ModuleDepCollectorPP::EndOfMainFile() {
>>> + FileID MainFileID = Instance.getSourceManager().getMainFileID();
>>> + MDC.MainFile =
>>> +
>>> Instance.getSourceManager().getFileEntryForID(MainFileID)->getName();
>>> +
>>> + for (const Module *M : DirectDeps) {
>>> + handleTopLevelModule(M);
>>> + }
>>> +
>>> + for (auto &&I : MDC.Deps)
>>> + MDC.Consumer.handleModuleDependency(I.second);
>>> +
>>> + DependencyOutputOptions Opts;
>>> + for (auto &&I : MDC.MainDeps)
>>> + MDC.Consumer.handleFileDependency(Opts, I);
>>> +}
>>> +
>>> +void ModuleDepCollectorPP::handleTopLevelModule(const Module *M) {
>>> + assert(M == M->getTopLevelModule() && "Expected top level module!");
>>> +
>>> + auto ModI = MDC.Deps.insert(
>>> + std::make_pair(MDC.ContextHash + M->getFullModuleName(),
>>> ModuleDeps{}));
>>> +
>>> + if (!ModI.first->second.ModuleName.empty())
>>> + return;
>>> +
>>> + ModuleDeps &MD = ModI.first->second;
>>> +
>>> + const FileEntry *ModuleMap = Instance.getPreprocessor()
>>> + .getHeaderSearchInfo()
>>> + .getModuleMap()
>>> + .getContainingModuleMapFile(M);
>>> +
>>> + MD.ClangModuleMapFile = ModuleMap ? ModuleMap->getName() : "";
>>> + MD.ModuleName = M->getFullModuleName();
>>> + MD.ModulePCMPath = M->getASTFile()->getName();
>>> + MD.ContextHash = MDC.ContextHash;
>>> + serialization::ModuleFile *MF =
>>> + MDC.Instance.getModuleManager()->getModuleManager().lookup(
>>> + M->getASTFile());
>>> + MDC.Instance.getModuleManager()->visitInputFiles(
>>> + *MF, true, true, [&](const serialization::InputFile &IF, bool
>>> isSystem) {
>>> + MD.FileDeps.insert(IF.getFile()->getName());
>>> + });
>>> +
>>> + addAllSubmoduleDeps(M, MD);
>>> +}
>>> +
>>> +void ModuleDepCollectorPP::addAllSubmoduleDeps(const Module *M,
>>> + ModuleDeps &MD) {
>>> + addModuleDep(M, MD);
>>> +
>>> + for (const Module *SubM : M->submodules())
>>> + addAllSubmoduleDeps(SubM, MD);
>>> +}
>>> +
>>> +void ModuleDepCollectorPP::addModuleDep(const Module *M, ModuleDeps
>>> &MD) {
>>> + for (const Module *Import : M->Imports) {
>>> + if (Import->getTopLevelModule() != M->getTopLevelModule()) {
>>> + MD.ClangModuleDeps.insert(Import->getTopLevelModuleName());
>>> + handleTopLevelModule(Import->getTopLevelModule());
>>> + }
>>> + }
>>> +}
>>> +
>>> +ModuleDepCollector::ModuleDepCollector(CompilerInstance &I,
>>> + DependencyConsumer &C)
>>> + : Instance(I), Consumer(C),
>>> ContextHash(I.getInvocation().getModuleHash()) {
>>> +}
>>> +
>>> +void ModuleDepCollector::attachToPreprocessor(Preprocessor &PP) {
>>> + PP.addPPCallbacks(std::make_unique<ModuleDepCollectorPP>(Instance,
>>> *this));
>>> +}
>>> +
>>> +void ModuleDepCollector::attachToASTReader(ASTReader &R) {}
>>>
>>> diff --git a/clang/test/ClangScanDeps/modules-full.cpp
>>> b/clang/test/ClangScanDeps/modules-full.cpp
>>> new file mode 100644
>>> index 000000000000..f8bff06d8f74
>>> --- /dev/null
>>> +++ b/clang/test/ClangScanDeps/modules-full.cpp
>>> @@ -0,0 +1,74 @@
>>> +// RUN: rm -rf %t.dir
>>> +// RUN: rm -rf %t.cdb
>>> +// RUN: rm -rf %t.module-cache
>>> +// RUN: mkdir -p %t.dir
>>> +// RUN: cp %s %t.dir/modules_cdb_input.cpp
>>> +// RUN: cp %s %t.dir/modules_cdb_input2.cpp
>>> +// RUN: mkdir %t.dir/Inputs
>>> +// RUN: cp %S/Inputs/header.h %t.dir/Inputs/header.h
>>> +// RUN: cp %S/Inputs/header2.h %t.dir/Inputs/header2.h
>>> +// RUN: cp %S/Inputs/module.modulemap %t.dir/Inputs/module.modulemap
>>> +// RUN: sed -e "s|DIR|%/t.dir|g" %S/Inputs/modules_cdb.json > %t.cdb
>>> +//
>>> +// RUN: echo %t.dir > %t.result
>>> +// RUN: clang-scan-deps -compilation-database %t.cdb -j 1 \
>>> +// RUN: -mode preprocess-minimized-sources -format experimental-full
>>> >> %t.result
>>> +// RUN: cat %t.result | FileCheck --check-prefixes=CHECK %s
>>> +
>>> +#include "header.h"
>>> +
>>> +// CHECK: [[PREFIX:(.*[/\\])+[a-zA-Z0-9.-]+]]
>>> +// CHECK-NEXT: {
>>> +// CHECK-NEXT: "clang-context-hash": "[[CONTEXT_HASH:[A-Z0-9]+]]",
>>> +// CHECK-NEXT: "clang-module-deps": [
>>> +// CHECK-NEXT: "header1"
>>> +// CHECK-NEXT: ],
>>> +// CHECK-NEXT: "clang-modules": [
>>> +// CHECK-NEXT: {
>>> +// CHECK-NEXT: "clang-module-deps": [
>>> +// CHECK-NEXT: "header2"
>>> +// CHECK-NEXT: ],
>>> +// CHECK-NEXT: "clang-modulemap-file":
>>> "[[PREFIX]]{{[/\\]}}Inputs{{[/\\]}}module.modulemap",
>>> +// CHECK-NEXT: "file-deps": [
>>> +// CHECK-NEXT: "[[PREFIX]]{{[/\\]}}Inputs{{[/\\]}}header.h",
>>> +// CHECK-NEXT:
>>> "[[PREFIX]]{{[/\\]}}Inputs{{[/\\]}}module.modulemap"
>>> +// CHECK-NEXT: ],
>>> +// CHECK-NEXT: "name": "header1"
>>> +// CHECK-NEXT: },
>>> +// CHECK-NEXT: {
>>> +// CHECK-NEXT: "clang-module-deps": [],
>>> +// CHECK-NEXT: "clang-modulemap-file":
>>> "[[PREFIX]]{{[/\\]}}Inputs{{[/\\]}}module.modulemap",
>>> +// CHECK-NEXT: "file-deps": [
>>> +// CHECK-NEXT: "[[PREFIX]]{{[/\\]}}Inputs{{[/\\]}}header2.h",
>>> +// CHECK-NEXT:
>>> "[[PREFIX]]{{[/\\]}}Inputs{{[/\\]}}module.modulemap"
>>> +// CHECK-NEXT: ],
>>> +// CHECK-NEXT: "name": "header2"
>>> +// CHECK-NEXT: }
>>> +// CHECK-NEXT: ],
>>> +// CHECK-NEXT: "file-deps": [
>>> +// CHECK-NEXT: "[[PREFIX]]{{[/\\]}}modules_cdb_input2.cpp"
>>> +// CHECK-NEXT: ],
>>> +// CHECK-NEXT: "input-file":
>>> "[[PREFIX]]{{[/\\]}}modules_cdb_input2.cpp"
>>> +// CHECK-NEXT:},
>>> +// CHECK-NEXT:{
>>> +// CHECK-NOT: "clang-context-hash": "[[CONTEXT_HASH]]",
>>> +// CHECK-NEXT: "clang-context-hash": "{{[A-Z0-9]+}}",
>>> +// CHECK-NEXT: "clang-module-deps": [
>>> +// CHECK-NEXT: "header1"
>>> +// CHECK-NEXT: ],
>>> +// CHECK-NEXT: "clang-modules": [
>>> +// CHECK-NEXT: {
>>> +// CHECK-NEXT: "clang-module-deps": [],
>>> +// CHECK-NEXT: "clang-modulemap-file":
>>> "[[PREFIX]]{{[/\\]}}Inputs{{[/\\]}}module.modulemap",
>>> +// CHECK-NEXT: "file-deps": [
>>> +// CHECK-NEXT: "[[PREFIX]]{{[/\\]}}Inputs{{[/\\]}}header.h",
>>> +// CHECK-NEXT:
>>> "[[PREFIX]]{{[/\\]}}Inputs{{[/\\]}}module.modulemap"
>>> +// CHECK-NEXT: ],
>>> +// CHECK-NEXT: "name": "header1"
>>> +// CHECK-NEXT: }
>>> +// CHECK-NEXT: ],
>>> +// CHECK-NEXT: "file-deps": [
>>> +// CHECK-NEXT: "[[PREFIX]]{{[/\\]}}modules_cdb_input.cpp"
>>> +// CHECK-NEXT: ],
>>> +// CHECK-NEXT: "input-file": "[[PREFIX]]{{[/\\]}}modules_cdb_input.cpp"
>>> +// CHECK-NEXT:},
>>>
>>> diff --git a/clang/tools/clang-scan-deps/ClangScanDeps.cpp
>>> b/clang/tools/clang-scan-deps/ClangScanDeps.cpp
>>> index d57983ed1664..a6abb4a9600b 100644
>>> --- a/clang/tools/clang-scan-deps/ClangScanDeps.cpp
>>> +++ b/clang/tools/clang-scan-deps/ClangScanDeps.cpp
>>> @@ -59,6 +59,17 @@ static llvm::cl::opt<ScanningMode> ScanMode(
>>> llvm::cl::init(ScanningMode::MinimizedSourcePreprocessing),
>>> llvm::cl::cat(DependencyScannerCategory));
>>>
>>> +static llvm::cl::opt<ScanningOutputFormat> Format(
>>> + "format", llvm::cl::desc("The output format for the dependencies"),
>>> + llvm::cl::values(clEnumValN(ScanningOutputFormat::Make, "make",
>>> + "Makefile compatible dep file"),
>>> + clEnumValN(ScanningOutputFormat::Full,
>>> "experimental-full",
>>> + "Full dependency graph suitable"
>>> + " for explicitly building modules. This
>>> format "
>>> + "is experimental and will change.")),
>>> + llvm::cl::init(ScanningOutputFormat::Make),
>>> + llvm::cl::cat(DependencyScannerCategory));
>>> +
>>> llvm::cl::opt<unsigned>
>>> NumThreads("j", llvm::cl::Optional,
>>> llvm::cl::desc("Number of worker threads to use
>>> (default: use "
>>> @@ -200,7 +211,7 @@ int main(int argc, const char **argv) {
>>> // Print out the dependency results to STDOUT by default.
>>> SharedStream DependencyOS(llvm::outs());
>>>
>>> - DependencyScanningService Service(ScanMode, ReuseFileManager,
>>> + DependencyScanningService Service(ScanMode, Format, ReuseFileManager,
>>> SkipExcludedPPRanges);
>>> #if LLVM_ENABLE_THREADS
>>> unsigned NumWorkers =
>>>
>>>
>>>
>>> _______________________________________________
>>> cfe-commits mailing list
>>> cfe-commits at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
>>>
>> _______________________________________________
>> cfe-commits mailing list
>> cfe-commits at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20191031/3e361c4c/attachment-0001.html>
More information about the cfe-commits
mailing list