<div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 12 Jun 2019 at 14:29, Alex Lorenz via cfe-commits <<a href="mailto:cfe-commits@lists.llvm.org">cfe-commits@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Author: arphaman<br>
Date: Wed Jun 12 14:32:49 2019<br>
New Revision: 363204<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=363204&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project?rev=363204&view=rev</a><br>
Log:<br>
[clang-scan-deps] initial outline of the tool that runs preprocessor to find<br>
dependencies over a JSON compilation database<br>
<br>
This commit introduces an outline for the clang-scan-deps tool that will be<br>
used to implement fast dependency discovery phase using implicit modules for<br>
explicit module builds.<br>
<br>
The initial version of the tool works by computing non-modular header dependencies<br>
for files in the compilation database without any optimizations<br>
(i.e. without source minimization from r362459).<br>
The tool spawns a number of worker threads to run the clang compiler workers in parallel.<br>
<br>
The immediate goal for clang-scan-deps is to create a ClangScanDeps library<br>
which will be used to build up this tool to use the source minimization and<br>
caching multi-threaded filesystem to implement the optimized non-incremental<br>
dependency scanning phase for a non-modular build. This will allow us to do<br>
benchmarks and comparisons for performance that the minimization and caching give us<br>
<br>
Differential Revision: <a href="https://reviews.llvm.org/D60233" rel="noreferrer" target="_blank">https://reviews.llvm.org/D60233</a><br>
<br>
Added:<br>
cfe/trunk/test/ClangScanDeps/<br>
cfe/trunk/test/ClangScanDeps/Inputs/<br>
cfe/trunk/test/ClangScanDeps/Inputs/header.h<br>
cfe/trunk/test/ClangScanDeps/Inputs/header2.h<br>
cfe/trunk/test/ClangScanDeps/Inputs/regular_cdb.json<br>
cfe/trunk/test/ClangScanDeps/regular_cdb.cpp<br>
cfe/trunk/tools/clang-scan-deps/<br>
cfe/trunk/tools/clang-scan-deps/CMakeLists.txt<br>
cfe/trunk/tools/clang-scan-deps/ClangScanDeps.cpp<br>
Modified:<br>
cfe/trunk/test/CMakeLists.txt<br>
cfe/trunk/tools/CMakeLists.txt<br>
<br>
Modified: cfe/trunk/test/CMakeLists.txt<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/test/CMakeLists.txt?rev=363204&r1=363203&r2=363204&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/cfe/trunk/test/CMakeLists.txt?rev=363204&r1=363203&r2=363204&view=diff</a><br>
==============================================================================<br>
--- cfe/trunk/test/CMakeLists.txt (original)<br>
+++ cfe/trunk/test/CMakeLists.txt Wed Jun 12 14:32:49 2019<br>
@@ -57,6 +57,7 @@ list(APPEND CLANG_TEST_DEPS<br>
clang-rename<br>
clang-refactor<br>
clang-diff<br>
+ clang-scan-deps<br>
diagtool<br>
hmaptool<br>
)<br>
<br>
Added: cfe/trunk/test/ClangScanDeps/Inputs/header.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/test/ClangScanDeps/Inputs/header.h?rev=363204&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/cfe/trunk/test/ClangScanDeps/Inputs/header.h?rev=363204&view=auto</a><br>
==============================================================================<br>
--- cfe/trunk/test/ClangScanDeps/Inputs/header.h (added)<br>
+++ cfe/trunk/test/ClangScanDeps/Inputs/header.h Wed Jun 12 14:32:49 2019<br>
@@ -0,0 +1,3 @@<br>
+#ifdef INCLUDE_HEADER2<br>
+#include "header2.h"<br>
+#endif<br>
<br>
Added: cfe/trunk/test/ClangScanDeps/Inputs/header2.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/test/ClangScanDeps/Inputs/header2.h?rev=363204&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/cfe/trunk/test/ClangScanDeps/Inputs/header2.h?rev=363204&view=auto</a><br>
==============================================================================<br>
--- cfe/trunk/test/ClangScanDeps/Inputs/header2.h (added)<br>
+++ cfe/trunk/test/ClangScanDeps/Inputs/header2.h Wed Jun 12 14:32:49 2019<br>
@@ -0,0 +1 @@<br>
+// header 2.<br>
<br>
Added: cfe/trunk/test/ClangScanDeps/Inputs/regular_cdb.json<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/test/ClangScanDeps/Inputs/regular_cdb.json?rev=363204&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/cfe/trunk/test/ClangScanDeps/Inputs/regular_cdb.json?rev=363204&view=auto</a><br>
==============================================================================<br>
--- cfe/trunk/test/ClangScanDeps/Inputs/regular_cdb.json (added)<br>
+++ cfe/trunk/test/ClangScanDeps/Inputs/regular_cdb.json Wed Jun 12 14:32:49 2019<br>
@@ -0,0 +1,12 @@<br>
+[<br>
+{<br>
+ "directory": "DIR",<br>
+ "command": "clang -c DIR/regular_cdb.cpp -IInputs -MD -MF DIR/regular_cdb.d",<br>
+ "file": "DIR/regular_cdb.cpp"<br>
+},<br>
+{<br>
+ "directory": "DIR",<br>
+ "command": "clang -c DIR/regular_cdb.cpp -IInputs -D INCLUDE_HEADER2 -MD -MF DIR/regular_cdb2.d",<br>
+ "file": "DIR/regular_cdb.cpp"<br>
+}<br>
+]<br>
<br>
Added: cfe/trunk/test/ClangScanDeps/regular_cdb.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/test/ClangScanDeps/regular_cdb.cpp?rev=363204&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/cfe/trunk/test/ClangScanDeps/regular_cdb.cpp?rev=363204&view=auto</a><br>
==============================================================================<br>
--- cfe/trunk/test/ClangScanDeps/regular_cdb.cpp (added)<br>
+++ cfe/trunk/test/ClangScanDeps/regular_cdb.cpp Wed Jun 12 14:32:49 2019<br>
@@ -0,0 +1,27 @@<br>
+// RUN: rm -rf %t.dir<br>
+// RUN: rm -rf %t.cdb<br>
+// RUN: mkdir -p %t.dir<br>
+// RUN: cp %s %t.dir/regular_cdb.cpp<br>
+// RUN: mkdir %t.dir/Inputs<br>
+// RUN: cp %S/Inputs/header.h %t.dir/Inputs/header.h<br>
+// RUN: cp %S/Inputs/header2.h %t.dir/Inputs/header2.h<br>
+// RUN: sed -e "s|DIR|%/t.dir|g" %S/Inputs/regular_cdb.json > %t.cdb<br>
+//<br>
+// RUN: clang-scan-deps -compilation-database %t.cdb -j 1<br>
+// RUN: cat %t.dir/regular_cdb.d | FileCheck %s<br>
+// RUN: cat %t.dir/regular_cdb2.d | FileCheck --check-prefix=CHECK2 %s<br>
+// RUN: rm -rf %t.dir/regular_cdb.d %t.dir/regular_cdb2.d<br>
+//<br>
+// RUN: clang-scan-deps -compilation-database %t.cdb -j 2<br>
+// RUN: cat %t.dir/regular_cdb.d | FileCheck %s<br>
+// RUN: cat %t.dir/regular_cdb2.d | FileCheck --check-prefix=CHECK2 %s<br>
+<br>
+#include "header.h"<br>
+<br>
+// CHECK: regular_cdb.cpp<br>
+// CHECK-NEXT: Inputs{{/|\\}}header.h<br>
+// CHECK-NOT: header2<br>
+<br>
+// CHECK2: regular_cdb.cpp<br>
+// CHECK2-NEXT: Inputs{{/|\\}}header.h<br>
+// CHECK2-NEXT: Inputs{{/|\\}}header2.h<br>
<br>
Modified: cfe/trunk/tools/CMakeLists.txt<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/CMakeLists.txt?rev=363204&r1=363203&r2=363204&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/CMakeLists.txt?rev=363204&r1=363203&r2=363204&view=diff</a><br>
==============================================================================<br>
--- cfe/trunk/tools/CMakeLists.txt (original)<br>
+++ cfe/trunk/tools/CMakeLists.txt Wed Jun 12 14:32:49 2019<br>
@@ -8,6 +8,7 @@ add_clang_subdirectory(clang-format-vs)<br>
add_clang_subdirectory(clang-fuzzer)<br>
add_clang_subdirectory(clang-import-test)<br>
add_clang_subdirectory(clang-offload-bundler)<br>
+add_clang_subdirectory(clang-scan-deps)<br>
<br>
add_clang_subdirectory(c-index-test)<br>
<br>
<br>
Added: cfe/trunk/tools/clang-scan-deps/CMakeLists.txt<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/clang-scan-deps/CMakeLists.txt?rev=363204&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/clang-scan-deps/CMakeLists.txt?rev=363204&view=auto</a><br>
==============================================================================<br>
--- cfe/trunk/tools/clang-scan-deps/CMakeLists.txt (added)<br>
+++ cfe/trunk/tools/clang-scan-deps/CMakeLists.txt Wed Jun 12 14:32:49 2019<br>
@@ -0,0 +1,26 @@<br>
+set(LLVM_LINK_COMPONENTS<br>
+ Core<br>
+ Support<br>
+)<br>
+<br>
+add_clang_tool(clang-scan-deps<br>
+ ClangScanDeps.cpp<br>
+ )<br>
+<br>
+set(CLANG_SCAN_DEPS_LIB_DEPS<br>
+ clangAST<br>
+ clangBasic<br>
+ clangCodeGen<br>
+ clangDriver<br>
+ clangFrontend<br>
+ clangFrontendTool<br>
+ clangLex<br>
+ clangParse<br>
+ clangTooling<br>
+ )<br>
+<br>
+target_link_libraries(clang-scan-deps<br>
+ PRIVATE<br>
+ ${CLANG_SCAN_DEPS_LIB_DEPS}<br>
+ )<br>
+<br>
<br>
Added: cfe/trunk/tools/clang-scan-deps/ClangScanDeps.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/clang-scan-deps/ClangScanDeps.cpp?rev=363204&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/cfe/trunk/tools/clang-scan-deps/ClangScanDeps.cpp?rev=363204&view=auto</a><br>
==============================================================================<br>
--- cfe/trunk/tools/clang-scan-deps/ClangScanDeps.cpp (added)<br>
+++ cfe/trunk/tools/clang-scan-deps/ClangScanDeps.cpp Wed Jun 12 14:32:49 2019<br>
@@ -0,0 +1,218 @@<br>
+//===-- ClangScanDeps.cpp - Implementation of clang-scan-deps -------------===//<br>
+//<br>
+// The LLVM Compiler Infrastructure<br>
+//<br>
+// This file is distributed under the University of Illinois Open Source<br>
+// License. See LICENSE.TXT for details.<br>
+//<br>
+//===----------------------------------------------------------------------===//<br></blockquote><div><br></div><div>Sorry, I forgot to update the old license comment in this file. I updated it in r363207.</div><div><br></div><div>Thanks,</div><div>Alex</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
+<br>
+#include "clang/Frontend/CompilerInstance.h"<br>
+#include "clang/Frontend/CompilerInvocation.h"<br>
+#include "clang/Frontend/FrontendActions.h"<br>
+#include "clang/Frontend/PCHContainerOperations.h"<br>
+#include "clang/FrontendTool/Utils.h"<br>
+#include "clang/Tooling/CommonOptionsParser.h"<br>
+#include "clang/Tooling/JSONCompilationDatabase.h"<br>
+#include "clang/Tooling/Tooling.h"<br>
+#include "llvm/Support/FileSystem.h"<br>
+#include "llvm/Support/InitLLVM.h"<br>
+#include "llvm/Support/JSON.h"<br>
+#include "llvm/Support/Options.h"<br>
+#include "llvm/Support/Path.h"<br>
+#include "llvm/Support/Program.h"<br>
+#include "llvm/Support/Signals.h"<br>
+#include "llvm/Support/Threading.h"<br>
+#include <thread><br>
+<br>
+using namespace clang;<br>
+<br>
+namespace {<br>
+<br>
+/// A clang tool that runs the preprocessor only for the given compiler<br>
+/// invocation.<br>
+class PreprocessorOnlyTool : public tooling::ToolAction {<br>
+public:<br>
+ PreprocessorOnlyTool(StringRef WorkingDirectory)<br>
+ : WorkingDirectory(WorkingDirectory) {}<br>
+<br>
+ bool runInvocation(std::shared_ptr<CompilerInvocation> Invocation,<br>
+ FileManager *FileMgr,<br>
+ std::shared_ptr<PCHContainerOperations> PCHContainerOps,<br>
+ DiagnosticConsumer *DiagConsumer) override {<br>
+ // Create a compiler instance to handle the actual work.<br>
+ CompilerInstance Compiler(std::move(PCHContainerOps));<br>
+ Compiler.setInvocation(std::move(Invocation));<br>
+ FileMgr->getFileSystemOpts().WorkingDir = WorkingDirectory;<br>
+ Compiler.setFileManager(FileMgr);<br>
+<br>
+ // Create the compiler's actual diagnostics engine.<br>
+ Compiler.createDiagnostics(DiagConsumer, /*ShouldOwnClient=*/false);<br>
+ if (!Compiler.hasDiagnostics())<br>
+ return false;<br>
+<br>
+ Compiler.createSourceManager(*FileMgr);<br>
+<br>
+ auto Action = llvm::make_unique<PreprocessOnlyAction>();<br>
+ const bool Result = Compiler.ExecuteAction(*Action);<br>
+ FileMgr->clearStatCache();<br>
+ return Result;<br>
+ }<br>
+<br>
+private:<br>
+ StringRef WorkingDirectory;<br>
+};<br>
+<br>
+/// A proxy file system that doesn't call `chdir` when changing the working<br>
+/// directory of a clang tool.<br>
+class ProxyFileSystemWithoutChdir : public llvm::vfs::ProxyFileSystem {<br>
+public:<br>
+ ProxyFileSystemWithoutChdir(<br>
+ llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> FS)<br>
+ : ProxyFileSystem(std::move(FS)) {}<br>
+<br>
+ llvm::ErrorOr<std::string> getCurrentWorkingDirectory() const override {<br>
+ assert(!CWD.empty() && "empty CWD");<br>
+ return CWD;<br>
+ }<br>
+<br>
+ std::error_code setCurrentWorkingDirectory(const Twine &Path) override {<br>
+ CWD = Path.str();<br>
+ return {};<br>
+ }<br>
+<br>
+private:<br>
+ std::string CWD;<br>
+};<br>
+<br>
+/// The high-level implementation of the dependency discovery tool that runs on<br>
+/// an individual worker thread.<br>
+class DependencyScanningTool {<br>
+public:<br>
+ /// Construct a dependency scanning tool.<br>
+ ///<br>
+ /// \param Compilations The reference to the compilation database that's<br>
+ /// used by the clang tool.<br>
+ DependencyScanningTool(const tooling::CompilationDatabase &Compilations)<br>
+ : Compilations(Compilations) {<br>
+ PCHContainerOps = std::make_shared<PCHContainerOperations>();<br>
+ BaseFS = new ProxyFileSystemWithoutChdir(llvm::vfs::getRealFileSystem());<br>
+ }<br>
+<br>
+ /// Computes the dependencies for the given file.<br>
+ ///<br>
+ /// \returns True on error.<br>
+ bool runOnFile(const std::string &Input, StringRef CWD) {<br>
+ BaseFS->setCurrentWorkingDirectory(CWD);<br>
+ tooling::ClangTool Tool(Compilations, Input, PCHContainerOps, BaseFS);<br>
+ Tool.clearArgumentsAdjusters();<br>
+ Tool.setRestoreWorkingDir(false);<br>
+ PreprocessorOnlyTool Action(CWD);<br>
+ return Tool.run(&Action);<br>
+ }<br>
+<br>
+private:<br>
+ const tooling::CompilationDatabase &Compilations;<br>
+ std::shared_ptr<PCHContainerOperations> PCHContainerOps;<br>
+ /// The real filesystem used as a base for all the operations performed by the<br>
+ /// tool.<br>
+ llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> BaseFS;<br>
+};<br>
+<br>
+llvm::cl::opt<bool> Help("h", llvm::cl::desc("Alias for -help"),<br>
+ llvm::cl::Hidden);<br>
+<br>
+llvm::cl::OptionCategory DependencyScannerCategory("Tool options");<br>
+<br>
+llvm::cl::opt<unsigned><br>
+ NumThreads("j", llvm::cl::Optional,<br>
+ llvm::cl::desc("Number of worker threads to use (default: use "<br>
+ "all concurrent threads)"),<br>
+ llvm::cl::init(0));<br>
+<br>
+llvm::cl::opt<std::string><br>
+ CompilationDB("compilation-database",<br>
+ llvm::cl::desc("Compilation database"), llvm::cl::Required,<br>
+ llvm::cl::cat(DependencyScannerCategory));<br>
+<br>
+} // end anonymous namespace<br>
+<br>
+int main(int argc, const char **argv) {<br>
+ llvm::InitLLVM X(argc, argv);<br>
+ llvm::cl::HideUnrelatedOptions(DependencyScannerCategory);<br>
+ if (!llvm::cl::ParseCommandLineOptions(argc, argv))<br>
+ return 1;<br>
+<br>
+ std::string ErrorMessage;<br>
+ std::unique_ptr<tooling::JSONCompilationDatabase> Compilations =<br>
+ tooling::JSONCompilationDatabase::loadFromFile(<br>
+ CompilationDB, ErrorMessage,<br>
+ tooling::JSONCommandLineSyntax::AutoDetect);<br>
+ if (!Compilations) {<br>
+ llvm::errs() << "error: " << ErrorMessage << "\n";<br>
+ return 1;<br>
+ }<br>
+<br>
+ llvm::cl::PrintOptionValues();<br>
+<br>
+ // By default the tool runs on all inputs in the CDB.<br>
+ std::vector<std::pair<std::string, std::string>> Inputs;<br>
+ for (const auto &Command : Compilations->getAllCompileCommands())<br>
+ Inputs.emplace_back(Command.Filename, Command.Directory);<br>
+<br>
+ // The command options are rewritten to run Clang in preprocessor only mode.<br>
+ auto AdjustingCompilations =<br>
+ llvm::make_unique<tooling::ArgumentsAdjustingCompilations>(<br>
+ std::move(Compilations));<br>
+ AdjustingCompilations->appendArgumentsAdjuster(<br>
+ [](const tooling::CommandLineArguments &Args, StringRef /*unused*/) {<br>
+ tooling::CommandLineArguments AdjustedArgs = Args;<br>
+ AdjustedArgs.push_back("-o");<br>
+ AdjustedArgs.push_back("/dev/null");<br>
+ AdjustedArgs.push_back("-Xclang");<br>
+ AdjustedArgs.push_back("-Eonly");<br>
+ AdjustedArgs.push_back("-Xclang");<br>
+ AdjustedArgs.push_back("-sys-header-deps");<br>
+ return AdjustedArgs;<br>
+ });<br>
+<br>
+ unsigned NumWorkers =<br>
+ NumThreads == 0 ? llvm::hardware_concurrency() : NumThreads;<br>
+ std::vector<std::unique_ptr<DependencyScanningTool>> WorkerTools;<br>
+ for (unsigned I = 0; I < NumWorkers; ++I)<br>
+ WorkerTools.push_back(<br>
+ llvm::make_unique<DependencyScanningTool>(*AdjustingCompilations));<br>
+<br>
+ std::vector<std::thread> WorkerThreads;<br>
+ std::atomic<bool> HadErrors(false);<br>
+ std::mutex Lock;<br>
+ size_t Index = 0;<br>
+<br>
+ llvm::outs() << "Running clang-scan-deps on " << Inputs.size()<br>
+ << " files using " << NumWorkers << " workers\n";<br>
+ for (unsigned I = 0; I < NumWorkers; ++I) {<br>
+ WorkerThreads.emplace_back(<br>
+ [I, &Lock, &Index, &Inputs, &HadErrors, &WorkerTools]() {<br>
+ while (true) {<br>
+ std::string Input;<br>
+ StringRef CWD;<br>
+ // Take the next input.<br>
+ {<br>
+ std::unique_lock<std::mutex> LockGuard(Lock);<br>
+ if (Index >= Inputs.size())<br>
+ return;<br>
+ const auto &Compilation = Inputs[Index++];<br>
+ Input = Compilation.first;<br>
+ CWD = Compilation.second;<br>
+ }<br>
+ // Run the tool on it.<br>
+ if (WorkerTools[I]->runOnFile(Input, CWD))<br>
+ HadErrors = true;<br>
+ }<br>
+ });<br>
+ }<br>
+ for (auto &W : WorkerThreads)<br>
+ W.join();<br>
+<br>
+ return HadErrors;<br>
+}<br>
<br>
<br>
_______________________________________________<br>
cfe-commits mailing list<br>
<a href="mailto:cfe-commits@lists.llvm.org" target="_blank">cfe-commits@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits</a><br>
</blockquote></div></div></div>