<div dir="ltr">Hi Sam,<div><br></div><div>I've reverted to get the bot green again.</div><div><br></div><div>Regards</div><div>Russ</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 12 Jul 2019 at 15:52, Russell Gallop <<a href="mailto:russell.gallop@gmail.com">russell.gallop@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Sam,<div><br></div><div>This is hitting a test failure on a Windows bot: <a href="http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/builds/26916" target="_blank">http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/builds/26916</a></div><div><br></div><div>Please could you take a look?</div><div><br></div><div>Thanks</div><div>Russ</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 12 Jul 2019 at 11:11, Sam McCall via cfe-commits <<a href="mailto:cfe-commits@lists.llvm.org" target="_blank">cfe-commits@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Author: sammccall<br>
Date: Fri Jul 12 03:11:40 2019<br>
New Revision: 365887<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=365887&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project?rev=365887&view=rev</a><br>
Log:<br>
[JSONCompilationDatabase] Strip distcc/ccache/gomacc wrappers from parsed commands.<br>
<br>
Summary:<br>
It's common to use compiler wrappers by setting CC="gomacc clang++".<br>
This results in both args appearing in compile_commands.json, and clang's driver<br>
can't handle this.<br>
<br>
This patch attempts to recognize this pattern (by looking for well-known<br>
wrappers) and dropping argv0 in this case.<br>
<br>
It conservatively ignores other cases for now:<br>
- wrappers with unknown names<br>
- wrappers that accept -flags<br>
- wrappers where the compiler to use is implied (usually cc or gcc)<br>
<br>
This is done at the JSONCompilationDatabase level rather than somewhere more<br>
fundamental, as (hopefully) this isn't a general conceptual problem, but a messy<br>
aspect of the nature of the ecosystem around compile_commands.json.<br>
i.e. compilation databases more tightly tied to the build system should not have<br>
this problem.<br>
<br>
Reviewers: phosek, klimek<br>
<br>
Subscribers: llvm-commits<br>
<br>
Tags: #llvm<br>
<br>
Differential Revision: <a href="https://reviews.llvm.org/D64297" rel="noreferrer" target="_blank">https://reviews.llvm.org/D64297</a><br>
<br>
Modified:<br>
cfe/trunk/lib/Tooling/JSONCompilationDatabase.cpp<br>
cfe/trunk/unittests/Tooling/CompilationDatabaseTest.cpp<br>
<br>
Modified: cfe/trunk/lib/Tooling/JSONCompilationDatabase.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Tooling/JSONCompilationDatabase.cpp?rev=365887&r1=365886&r2=365887&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Tooling/JSONCompilationDatabase.cpp?rev=365887&r1=365886&r2=365887&view=diff</a><br>
==============================================================================<br>
--- cfe/trunk/lib/Tooling/JSONCompilationDatabase.cpp (original)<br>
+++ cfe/trunk/lib/Tooling/JSONCompilationDatabase.cpp Fri Jul 12 03:11:40 2019<br>
@@ -256,15 +256,57 @@ JSONCompilationDatabase::getAllCompileCo<br>
return Commands;<br>
}<br>
<br>
+static llvm::StringRef stripExecutableExtension(llvm::StringRef Name) {<br>
+ Name.consume_back(".exe");<br>
+ return Name;<br>
+}<br>
+<br>
+// There are compiler-wrappers (ccache, distcc, gomacc) that take the "real"<br>
+// compiler as an argument, e.g. distcc gcc -O3 foo.c.<br>
+// These end up in compile_commands.json when people set CC="distcc gcc".<br>
+// Clang's driver doesn't understand this, so we need to unwrap.<br>
+static bool unwrapCommand(std::vector<std::string> &Args) {<br>
+ if (Args.size() < 2)<br>
+ return false;<br>
+ StringRef Wrapper =<br>
+ stripExecutableExtension(llvm::sys::path::filename(Args.front()));<br>
+ if (Wrapper == "distcc" || Wrapper == "gomacc" || Wrapper == "ccache") {<br>
+ // Most of these wrappers support being invoked 3 ways:<br>
+ // `distcc g++ file.c` This is the mode we're trying to match.<br>
+ // We need to drop `distcc`.<br>
+ // `distcc file.c` This acts like compiler is cc or similar.<br>
+ // Clang's driver can handle this, no change needed.<br>
+ // `g++ file.c` g++ is a symlink to distcc.<br>
+ // We don't even notice this case, and all is well.<br>
+ //<br>
+ // We need to distinguish between the first and second case.<br>
+ // The wrappers themselves don't take flags, so Args[1] is a compiler flag,<br>
+ // an input file, or a compiler. Inputs have extensions, compilers don't.<br>
+ bool HasCompiler =<br>
+ (Args[1][0] != '-') &&<br>
+ !llvm::sys::path::has_extension(stripExecutableExtension(Args[1]));<br>
+ if (HasCompiler) {<br>
+ Args.erase(Args.begin());<br>
+ return true;<br>
+ }<br>
+ // If !HasCompiler, wrappers act like GCC. Fine: so do we.<br>
+ }<br>
+ return false;<br>
+}<br>
+<br>
static std::vector<std::string><br>
nodeToCommandLine(JSONCommandLineSyntax Syntax,<br>
const std::vector<llvm::yaml::ScalarNode *> &Nodes) {<br>
SmallString<1024> Storage;<br>
- if (Nodes.size() == 1)<br>
- return unescapeCommandLine(Syntax, Nodes[0]->getValue(Storage));<br>
std::vector<std::string> Arguments;<br>
- for (const auto *Node : Nodes)<br>
- Arguments.push_back(Node->getValue(Storage));<br>
+ if (Nodes.size() == 1)<br>
+ Arguments = unescapeCommandLine(Syntax, Nodes[0]->getValue(Storage));<br>
+ else<br>
+ for (const auto *Node : Nodes)<br>
+ Arguments.push_back(Node->getValue(Storage));<br>
+ // There may be multiple wrappers: using distcc and ccache together is common.<br>
+ while (unwrapCommand(Arguments))<br>
+ ;<br>
return Arguments;<br>
}<br>
<br>
<br>
Modified: cfe/trunk/unittests/Tooling/CompilationDatabaseTest.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/unittests/Tooling/CompilationDatabaseTest.cpp?rev=365887&r1=365886&r2=365887&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/cfe/trunk/unittests/Tooling/CompilationDatabaseTest.cpp?rev=365887&r1=365886&r2=365887&view=diff</a><br>
==============================================================================<br>
--- cfe/trunk/unittests/Tooling/CompilationDatabaseTest.cpp (original)<br>
+++ cfe/trunk/unittests/Tooling/CompilationDatabaseTest.cpp Fri Jul 12 03:11:40 2019<br>
@@ -370,6 +370,30 @@ TEST(findCompileArgsInJsonDatabase, Find<br>
EXPECT_EQ("command4", FoundCommand.CommandLine[0]) << ErrorMessage;<br>
}<br>
<br>
+TEST(findCompileArgsInJsonDatabase, ParsesCompilerWrappers) {<br>
+ std::vector<std::pair<std::string, std::string>> Cases = {<br>
+ {"distcc gcc foo.c", "gcc foo.c"},<br>
+ {"gomacc clang++ foo.c", "clang++ foo.c"},<br>
+ {"ccache gcc foo.c", "gcc foo.c"},<br>
+ {"ccache.exe gcc foo.c", "gcc foo.c"},<br>
+ {"ccache g++.exe foo.c", "g++.exe foo.c"},<br>
+ {"ccache distcc gcc foo.c", "gcc foo.c"},<br>
+<br>
+ {"distcc foo.c", "distcc foo.c"},<br>
+ {"distcc -I/foo/bar foo.c", "distcc -I/foo/bar foo.c"},<br>
+ };<br>
+ std::string ErrorMessage;<br>
+<br>
+ for (const auto &Case : Cases) {<br>
+ std::string DB = R"([{"directory":".", "file":"/foo.c", "command":")" +<br>
+ Case.first + "\"}]";<br>
+ CompileCommand FoundCommand =<br>
+ findCompileArgsInJsonDatabase("/foo.c", DB, ErrorMessage);<br>
+ EXPECT_EQ(Case.second, llvm::join(FoundCommand.CommandLine, " "))<br>
+ << Case.first;<br>
+ }<br>
+}<br>
+<br>
static std::vector<std::string> unescapeJsonCommandLine(StringRef Command) {<br>
std::string JsonDatabase =<br>
("[{\"directory\":\"//net/root\", \"file\":\"test\", \"command\": \"" +<br>
<br>
<br>
_______________________________________________<br>
cfe-commits mailing list<br>
<a href="mailto:cfe-commits@lists.llvm.org" target="_blank">cfe-commits@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits</a><br>
</blockquote></div>
</blockquote></div>