r365887 - [JSONCompilationDatabase] Strip distcc/ccache/gomacc wrappers from parsed commands.
Russell Gallop via cfe-commits
cfe-commits at lists.llvm.org
Fri Jul 12 08:22:47 PDT 2019
Hi Sam,
I've reverted to get the bot green again.
Regards
Russ
On Fri, 12 Jul 2019 at 15:52, Russell Gallop <russell.gallop at gmail.com>
wrote:
> Hi Sam,
>
> This is hitting a test failure on a Windows bot:
> http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/builds/26916
>
> Please could you take a look?
>
> Thanks
> Russ
>
> On Fri, 12 Jul 2019 at 11:11, Sam McCall via cfe-commits <
> cfe-commits at lists.llvm.org> wrote:
>
>> Author: sammccall
>> Date: Fri Jul 12 03:11:40 2019
>> New Revision: 365887
>>
>> URL: http://llvm.org/viewvc/llvm-project?rev=365887&view=rev
>> Log:
>> [JSONCompilationDatabase] Strip distcc/ccache/gomacc wrappers from parsed
>> commands.
>>
>> Summary:
>> It's common to use compiler wrappers by setting CC="gomacc clang++".
>> This results in both args appearing in compile_commands.json, and clang's
>> driver
>> can't handle this.
>>
>> This patch attempts to recognize this pattern (by looking for well-known
>> wrappers) and dropping argv0 in this case.
>>
>> It conservatively ignores other cases for now:
>> - wrappers with unknown names
>> - wrappers that accept -flags
>> - wrappers where the compiler to use is implied (usually cc or gcc)
>>
>> This is done at the JSONCompilationDatabase level rather than somewhere
>> more
>> fundamental, as (hopefully) this isn't a general conceptual problem, but
>> a messy
>> aspect of the nature of the ecosystem around compile_commands.json.
>> i.e. compilation databases more tightly tied to the build system should
>> not have
>> this problem.
>>
>> Reviewers: phosek, klimek
>>
>> Subscribers: llvm-commits
>>
>> Tags: #llvm
>>
>> Differential Revision: https://reviews.llvm.org/D64297
>>
>> Modified:
>> cfe/trunk/lib/Tooling/JSONCompilationDatabase.cpp
>> cfe/trunk/unittests/Tooling/CompilationDatabaseTest.cpp
>>
>> Modified: cfe/trunk/lib/Tooling/JSONCompilationDatabase.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Tooling/JSONCompilationDatabase.cpp?rev=365887&r1=365886&r2=365887&view=diff
>>
>> ==============================================================================
>> --- cfe/trunk/lib/Tooling/JSONCompilationDatabase.cpp (original)
>> +++ cfe/trunk/lib/Tooling/JSONCompilationDatabase.cpp Fri Jul 12 03:11:40
>> 2019
>> @@ -256,15 +256,57 @@ JSONCompilationDatabase::getAllCompileCo
>> return Commands;
>> }
>>
>> +static llvm::StringRef stripExecutableExtension(llvm::StringRef Name) {
>> + Name.consume_back(".exe");
>> + return Name;
>> +}
>> +
>> +// There are compiler-wrappers (ccache, distcc, gomacc) that take the
>> "real"
>> +// compiler as an argument, e.g. distcc gcc -O3 foo.c.
>> +// These end up in compile_commands.json when people set CC="distcc gcc".
>> +// Clang's driver doesn't understand this, so we need to unwrap.
>> +static bool unwrapCommand(std::vector<std::string> &Args) {
>> + if (Args.size() < 2)
>> + return false;
>> + StringRef Wrapper =
>> + stripExecutableExtension(llvm::sys::path::filename(Args.front()));
>> + if (Wrapper == "distcc" || Wrapper == "gomacc" || Wrapper == "ccache")
>> {
>> + // Most of these wrappers support being invoked 3 ways:
>> + // `distcc g++ file.c` This is the mode we're trying to match.
>> + // We need to drop `distcc`.
>> + // `distcc file.c` This acts like compiler is cc or similar.
>> + // Clang's driver can handle this, no change
>> needed.
>> + // `g++ file.c` g++ is a symlink to distcc.
>> + // We don't even notice this case, and all is
>> well.
>> + //
>> + // We need to distinguish between the first and second case.
>> + // The wrappers themselves don't take flags, so Args[1] is a
>> compiler flag,
>> + // an input file, or a compiler. Inputs have extensions, compilers
>> don't.
>> + bool HasCompiler =
>> + (Args[1][0] != '-') &&
>> +
>> !llvm::sys::path::has_extension(stripExecutableExtension(Args[1]));
>> + if (HasCompiler) {
>> + Args.erase(Args.begin());
>> + return true;
>> + }
>> + // If !HasCompiler, wrappers act like GCC. Fine: so do we.
>> + }
>> + return false;
>> +}
>> +
>> static std::vector<std::string>
>> nodeToCommandLine(JSONCommandLineSyntax Syntax,
>> const std::vector<llvm::yaml::ScalarNode *> &Nodes) {
>> SmallString<1024> Storage;
>> - if (Nodes.size() == 1)
>> - return unescapeCommandLine(Syntax, Nodes[0]->getValue(Storage));
>> std::vector<std::string> Arguments;
>> - for (const auto *Node : Nodes)
>> - Arguments.push_back(Node->getValue(Storage));
>> + if (Nodes.size() == 1)
>> + Arguments = unescapeCommandLine(Syntax, Nodes[0]->getValue(Storage));
>> + else
>> + for (const auto *Node : Nodes)
>> + Arguments.push_back(Node->getValue(Storage));
>> + // There may be multiple wrappers: using distcc and ccache together is
>> common.
>> + while (unwrapCommand(Arguments))
>> + ;
>> return Arguments;
>> }
>>
>>
>> Modified: cfe/trunk/unittests/Tooling/CompilationDatabaseTest.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/cfe/trunk/unittests/Tooling/CompilationDatabaseTest.cpp?rev=365887&r1=365886&r2=365887&view=diff
>>
>> ==============================================================================
>> --- cfe/trunk/unittests/Tooling/CompilationDatabaseTest.cpp (original)
>> +++ cfe/trunk/unittests/Tooling/CompilationDatabaseTest.cpp Fri Jul 12
>> 03:11:40 2019
>> @@ -370,6 +370,30 @@ TEST(findCompileArgsInJsonDatabase, Find
>> EXPECT_EQ("command4", FoundCommand.CommandLine[0]) << ErrorMessage;
>> }
>>
>> +TEST(findCompileArgsInJsonDatabase, ParsesCompilerWrappers) {
>> + std::vector<std::pair<std::string, std::string>> Cases = {
>> + {"distcc gcc foo.c", "gcc foo.c"},
>> + {"gomacc clang++ foo.c", "clang++ foo.c"},
>> + {"ccache gcc foo.c", "gcc foo.c"},
>> + {"ccache.exe gcc foo.c", "gcc foo.c"},
>> + {"ccache g++.exe foo.c", "g++.exe foo.c"},
>> + {"ccache distcc gcc foo.c", "gcc foo.c"},
>> +
>> + {"distcc foo.c", "distcc foo.c"},
>> + {"distcc -I/foo/bar foo.c", "distcc -I/foo/bar foo.c"},
>> + };
>> + std::string ErrorMessage;
>> +
>> + for (const auto &Case : Cases) {
>> + std::string DB = R"([{"directory":".", "file":"/foo.c",
>> "command":")" +
>> + Case.first + "\"}]";
>> + CompileCommand FoundCommand =
>> + findCompileArgsInJsonDatabase("/foo.c", DB, ErrorMessage);
>> + EXPECT_EQ(Case.second, llvm::join(FoundCommand.CommandLine, " "))
>> + << Case.first;
>> + }
>> +}
>> +
>> static std::vector<std::string> unescapeJsonCommandLine(StringRef
>> Command) {
>> std::string JsonDatabase =
>> ("[{\"directory\":\"//net/root\", \"file\":\"test\", \"command\":
>> \"" +
>>
>>
>> _______________________________________________
>> cfe-commits mailing list
>> cfe-commits at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20190712/013c835e/attachment.html>
More information about the cfe-commits
mailing list