<div dir="ltr">Hello Sam,<br><br>It looks like this commit added broken tests to one of our builders:<br><a href="http://lab.llvm.org:8011/builders/llvm-clang-x86_64-expensive-checks-win/builds/8957/steps/test-check-all/logs/stdio">http://lab.llvm.org:8011/builders/llvm-clang-x86_64-expensive-checks-win/builds/8957/steps/test-check-all/logs/stdio</a><br><br>Failing Tests (5):<br> Clang-Unit :: Tooling/./ToolingTests.exe/InterpolateTest.Case<br> Clang-Unit :: Tooling/./ToolingTests.exe/InterpolateTest.Language<br> Clang-Unit :: Tooling/./ToolingTests.exe/InterpolateTest.Nearby<br> Clang-Unit :: Tooling/./ToolingTests.exe/InterpolateTest.Strip<br>. . .<br>Please have a look?<br><br>The builder was red and did not send notifications.<br><br>Thanks<br><br>Galina<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 9, 2018 at 8:17 AM, Sam McCall via cfe-commits <span dir="ltr"><<a href="mailto:cfe-commits@lists.llvm.org" target="_blank">cfe-commits@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Author: sammccall<br>
Date: Mon Apr 9 08:17:39 2018<br>
New Revision: 329580<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=329580&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project?rev=329580&view=rev</a><br>
Log:<br>
[Tooling] A CompilationDatabase wrapper that infers header commands.<br>
<br>
Summary:<br>
The wrapper finds the closest matching compile command using filename heuristics<br>
and makes minimal tweaks so it can be used with the header.<br>
<br>
Subscribers: klimek, mgorny, cfe-commits<br>
<br>
Differential Revision: <a href="https://reviews.llvm.org/D45006" rel="noreferrer" target="_blank">https://reviews.llvm.org/<wbr>D45006</a><br>
<br>
Added:<br>
cfe/trunk/lib/Tooling/<wbr>InterpolatingCompilationDataba<wbr>se.cpp<br>
Modified:<br>
cfe/trunk/include/clang/<wbr>Tooling/CompilationDatabase.h<br>
cfe/trunk/lib/Tooling/<wbr>CMakeLists.txt<br>
cfe/trunk/unittests/Tooling/<wbr>CompilationDatabaseTest.cpp<br>
<br>
Modified: cfe/trunk/include/clang/<wbr>Tooling/CompilationDatabase.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/Tooling/CompilationDatabase.h?rev=329580&r1=329579&r2=329580&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/cfe/trunk/include/<wbr>clang/Tooling/<wbr>CompilationDatabase.h?rev=<wbr>329580&r1=329579&r2=329580&<wbr>view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- cfe/trunk/include/clang/<wbr>Tooling/CompilationDatabase.h (original)<br>
+++ cfe/trunk/include/clang/<wbr>Tooling/CompilationDatabase.h Mon Apr 9 08:17:39 2018<br>
@@ -213,6 +213,13 @@ private:<br>
std::vector<CompileCommand> CompileCommands;<br>
};<br>
<br>
+/// Returns a wrapped CompilationDatabase that defers to the provided one,<br>
+/// but getCompileCommands() will infer commands for unknown files.<br>
+/// The return value of getAllFiles() or getAllCompileCommands() is unchanged.<br>
+/// See InterpolatingCompilationDataba<wbr>se.cpp for details on heuristics.<br>
+std::unique_ptr<<wbr>CompilationDatabase><br>
+ inferMissingCompileCommands(<wbr>std::unique_ptr<<wbr>CompilationDatabase>);<br>
+<br>
} // namespace tooling<br>
} // namespace clang<br>
<br>
<br>
Modified: cfe/trunk/lib/Tooling/<wbr>CMakeLists.txt<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Tooling/CMakeLists.txt?rev=329580&r1=329579&r2=329580&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/cfe/trunk/lib/Tooling/<wbr>CMakeLists.txt?rev=329580&r1=<wbr>329579&r2=329580&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- cfe/trunk/lib/Tooling/<wbr>CMakeLists.txt (original)<br>
+++ cfe/trunk/lib/Tooling/<wbr>CMakeLists.txt Mon Apr 9 08:17:39 2018<br>
@@ -15,6 +15,7 @@ add_clang_library(clangTooling<br>
Execution.cpp<br>
FileMatchTrie.cpp<br>
FixIt.cpp<br>
+ InterpolatingCompilationDataba<wbr>se.cpp<br>
JSONCompilationDatabase.cpp<br>
Refactoring.cpp<br>
RefactoringCallbacks.cpp<br>
<br>
Added: cfe/trunk/lib/Tooling/<wbr>InterpolatingCompilationDataba<wbr>se.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Tooling/InterpolatingCompilationDatabase.cpp?rev=329580&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/cfe/trunk/lib/Tooling/<wbr>InterpolatingCompilationDataba<wbr>se.cpp?rev=329580&view=auto</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- cfe/trunk/lib/Tooling/<wbr>InterpolatingCompilationDataba<wbr>se.cpp (added)<br>
+++ cfe/trunk/lib/Tooling/<wbr>InterpolatingCompilationDataba<wbr>se.cpp Mon Apr 9 08:17:39 2018<br>
@@ -0,0 +1,458 @@<br>
+//===- InterpolatingCompilationDataba<wbr>se.cpp ---------------------*- C++ -*-===//<br>
+//<br>
+// The LLVM Compiler Infrastructure<br>
+//<br>
+// This file is distributed under the University of Illinois Open Source<br>
+// License. See LICENSE.TXT for details.<br>
+//<br>
+//===------------------------<wbr>------------------------------<wbr>----------------===//<br>
+//<br>
+// InterpolatingCompilationDataba<wbr>se wraps another CompilationDatabase and<br>
+// attempts to heuristically determine appropriate compile commands for files<br>
+// that are not included, such as headers or newly created files.<br>
+//<br>
+// Motivating cases include:<br>
+// Header files that live next to their implementation files. These typically<br>
+// share a base filename. (libclang/CXString.h, libclang/CXString.cpp).<br>
+// Some projects separate headers from includes. Filenames still typically<br>
+// match, maybe other path segments too. (include/llvm/IR/Use.h, lib/IR/Use.cc).<br>
+// Matches are sometimes only approximate (Sema.h, SemaDecl.cpp). This goes<br>
+// for directories too (Support/Unix/Process.inc, lib/Support/Process.cpp).<br>
+// Even if we can't find a "right" compile command, even a random one from<br>
+// the project will tend to get important flags like -I and -x right.<br>
+//<br>
+// We "borrow" the compile command for the closest available file:<br>
+// - points are awarded if the filename matches (ignoring extension)<br>
+// - points are awarded if the directory structure matches<br>
+// - ties are broken by length of path prefix match<br>
+//<br>
+// The compile command is adjusted, replacing the filename and removing output<br>
+// file arguments. The -x and -std flags may be affected too.<br>
+//<br>
+// Source language is a tricky issue: is it OK to use a .c file's command<br>
+// for building a .cc file? What language is a .h file in?<br>
+// - We only consider compile commands for c-family languages as candidates.<br>
+// - For files whose language is implied by the filename (e.g. .m, .hpp)<br>
+// we prefer candidates from the same language.<br>
+// If we must cross languages, we drop any -x and -std flags.<br>
+// - For .h files, candidates from any c-family language are acceptable.<br>
+// We use the candidate's language, inserting e.g. -x c++-header.<br>
+//<br>
+// This class is only useful when wrapping databases that can enumerate all<br>
+// their compile commands. If getAllFilenames() is empty, no inference occurs.<br>
+//<br>
+//===------------------------<wbr>------------------------------<wbr>----------------===//<br>
+<br>
+#include "clang/Driver/Options.h"<br>
+#include "clang/Driver/Types.h"<br>
+#include "clang/Frontend/LangStandard.<wbr>h"<br>
+#include "clang/Tooling/<wbr>CompilationDatabase.h"<br>
+#include "llvm/ADT/DenseMap.h"<br>
+#include "llvm/ADT/StringExtras.h"<br>
+#include "llvm/ADT/StringSwitch.h"<br>
+#include "llvm/Option/ArgList.h"<br>
+#include "llvm/Option/OptTable.h"<br>
+#include "llvm/Support/Debug.h"<br>
+#include "llvm/Support/Path.h"<br>
+#include "llvm/Support/StringSaver.h"<br>
+#include "llvm/Support/raw_ostream.h"<br>
+#include <memory><br>
+<br>
+namespace clang {<br>
+namespace tooling {<br>
+namespace {<br>
+using namespace llvm;<br>
+namespace types = clang::driver::types;<br>
+namespace path = llvm::sys::path;<br>
+<br>
+// The length of the prefix these two strings have in common.<br>
+size_t matchingPrefix(StringRef L, StringRef R) {<br>
+ size_t Limit = std::min(L.size(), R.size());<br>
+ for (size_t I = 0; I < Limit; ++I)<br>
+ if (L[I] != R[I])<br>
+ return I;<br>
+ return Limit;<br>
+}<br>
+<br>
+// A comparator for searching SubstringWithIndexes with std::equal_range etc.<br>
+// Optionaly prefix semantics: compares equal if the key is a prefix.<br>
+template <bool Prefix> struct Less {<br>
+ bool operator()(StringRef Key, std::pair<StringRef, size_t> Value) const {<br>
+ StringRef V = Prefix ? Value.first.substr(0, Key.size()) : Value.first;<br>
+ return Key < V;<br>
+ }<br>
+ bool operator()(std::pair<<wbr>StringRef, size_t> Value, StringRef Key) const {<br>
+ StringRef V = Prefix ? Value.first.substr(0, Key.size()) : Value.first;<br>
+ return V < Key;<br>
+ }<br>
+};<br>
+<br>
+// Infer type from filename. If we might have gotten it wrong, set *Certain.<br>
+// *.h will be inferred as a C header, but not certain.<br>
+types::ID guessType(StringRef Filename, bool *Certain = nullptr) {<br>
+ // path::extension is ".cpp", lookupTypeForExtension wants "cpp".<br>
+ auto Lang =<br>
+ types::lookupTypeForExtension(<wbr>path::extension(Filename).<wbr>substr(1));<br>
+ if (Certain)<br>
+ *Certain = Lang != types::TY_CHeader && Lang != types::TY_INVALID;<br>
+ return Lang;<br>
+}<br>
+<br>
+// Return Lang as one of the canonical supported types.<br>
+// e.g. c-header --> c; fortran --> TY_INVALID<br>
+static types::ID foldType(types::ID Lang) {<br>
+ switch (Lang) {<br>
+ case types::TY_C:<br>
+ case types::TY_CHeader:<br>
+ return types::TY_C;<br>
+ case types::TY_ObjC:<br>
+ case types::TY_ObjCHeader:<br>
+ return types::TY_ObjC;<br>
+ case types::TY_CXX:<br>
+ case types::TY_CXXHeader:<br>
+ return types::TY_CXX;<br>
+ case types::TY_ObjCXX:<br>
+ case types::TY_ObjCXXHeader:<br>
+ return types::TY_ObjCXX;<br>
+ default:<br>
+ return types::TY_INVALID;<br>
+ }<br>
+}<br>
+<br>
+// A CompileCommand that can be applied to another file.<br>
+struct TransferableCommand {<br>
+ // Flags that should not apply to all files are stripped from CommandLine.<br>
+ CompileCommand Cmd;<br>
+ // Language detected from -x or the filename.<br>
+ types::ID Type = types::TY_INVALID;<br>
+ // Standard specified by -std.<br>
+ LangStandard::Kind Std = LangStandard::lang_<wbr>unspecified;<br>
+<br>
+ TransferableCommand(<wbr>CompileCommand C)<br>
+ : Cmd(std::move(C)), Type(guessType(Cmd.Filename)) {<br>
+ std::vector<std::string> NewArgs = {Cmd.CommandLine.front()};<br>
+ // Parse the old args in order to strip out and record unwanted flags.<br>
+ auto OptTable = clang::driver::<wbr>createDriverOptTable();<br>
+ std::vector<const char *> Argv;<br>
+ for (unsigned I = 1; I < Cmd.CommandLine.size(); ++I)<br>
+ Argv.push_back(Cmd.<wbr>CommandLine[I].c_str());<br>
+ unsigned MissingI, MissingC;<br>
+ auto ArgList = OptTable->ParseArgs(Argv, MissingI, MissingC);<br>
+ for (const auto *Arg : ArgList) {<br>
+ const auto &option = Arg->getOption();<br>
+ // Strip input and output files.<br>
+ if (option.matches(clang::driver:<wbr>:options::OPT_INPUT) ||<br>
+ option.matches(clang::driver::<wbr>options::OPT_o)) {<br>
+ continue;<br>
+ }<br>
+ // Strip -x, but record the overridden language.<br>
+ if (option.matches(clang::driver:<wbr>:options::OPT_x)) {<br>
+ for (const char *Value : Arg->getValues())<br>
+ Type = types::<wbr>lookupTypeForTypeSpecifier(<wbr>Value);<br>
+ continue;<br>
+ }<br>
+ // Strip --std, but record the value.<br>
+ if (option.matches(clang::driver:<wbr>:options::OPT_std_EQ)) {<br>
+ for (const char *Value : Arg->getValues()) {<br>
+ Std = llvm::StringSwitch<<wbr>LangStandard::Kind>(Value)<br>
+#define LANGSTANDARD(id, name, lang, desc, features) \<br>
+ .Case(name, LangStandard::lang_##id)<br>
+#define LANGSTANDARD_ALIAS(id, alias) .Case(alias, LangStandard::lang_##id)<br>
+#include "clang/Frontend/LangStandards.<wbr>def"<br>
+ .Default(Std);<br>
+ }<br>
+ continue;<br>
+ }<br>
+ llvm::opt::ArgStringList ArgStrs;<br>
+ Arg->render(ArgList, ArgStrs);<br>
+ NewArgs.insert(NewArgs.end(), ArgStrs.begin(), ArgStrs.end());<br>
+ }<br>
+ Cmd.CommandLine = std::move(NewArgs);<br>
+<br>
+ if (Std != LangStandard::lang_<wbr>unspecified) // -std take precedence over -x<br>
+ Type = toType(LangStandard::<wbr>getLangStandardForKind(Std).<wbr>getLanguage());<br>
+ Type = foldType(Type);<br>
+ }<br>
+<br>
+ // Produce a CompileCommand for \p filename, based on this one.<br>
+ CompileCommand transferTo(StringRef Filename) const {<br>
+ CompileCommand Result = Cmd;<br>
+ Result.Filename = Filename;<br>
+ bool TypeCertain;<br>
+ auto TargetType = guessType(Filename, &TypeCertain);<br>
+ // If the filename doesn't determine the language (.h), transfer with -x.<br>
+ if (!TypeCertain) {<br>
+ TargetType = types::onlyPrecompileType(<wbr>TargetType) // header?<br>
+ ? types::<wbr>lookupHeaderTypeForSourceType(<wbr>Type)<br>
+ : Type;<br>
+ Result.CommandLine.push_back("<wbr>-x");<br>
+ Result.CommandLine.push_back(<wbr>types::getTypeName(TargetType)<wbr>);<br>
+ }<br>
+ // --std flag may only be transferred if the language is the same.<br>
+ // We may consider "translating" these, e.g. c++11 -> c11.<br>
+ if (Std != LangStandard::lang_unspecified && foldType(TargetType) == Type) {<br>
+ Result.CommandLine.push_back("<wbr>-std");<br>
+ Result.CommandLine.push_back(<br>
+ LangStandard::<wbr>getLangStandardForKind(Std).<wbr>getName());<br>
+ }<br>
+ Result.CommandLine.push_back(<wbr>Filename);<br>
+ return Result;<br>
+ }<br>
+<br>
+private:<br>
+ // Map the language from the --std flag to that of the -x flag.<br>
+ static types::ID toType(InputKind::Language Lang) {<br>
+ switch (Lang) {<br>
+ case InputKind::C:<br>
+ return types::TY_C;<br>
+ case InputKind::CXX:<br>
+ return types::TY_CXX;<br>
+ case InputKind::ObjC:<br>
+ return types::TY_ObjC;<br>
+ case InputKind::ObjCXX:<br>
+ return types::TY_ObjCXX;<br>
+ default:<br>
+ return types::TY_INVALID;<br>
+ }<br>
+ }<br>
+};<br>
+<br>
+// CommandIndex does the real work: given a filename, it produces the best<br>
+// matching TransferableCommand by matching filenames. Basic strategy:<br>
+// - Build indexes of each of the substrings we want to look up by.<br>
+// These indexes are just sorted lists of the substrings.<br>
+// - Forward requests to the inner CDB. If it fails, we must pick a proxy.<br>
+// - Each criterion corresponds to a range lookup into the index, so we only<br>
+// need O(log N) string comparisons to determine scores.<br>
+// - We then break ties among the candidates with the highest score.<br>
+class CommandIndex {<br>
+public:<br>
+ CommandIndex(std::vector<<wbr>TransferableCommand> AllCommands)<br>
+ : Commands(std::move(<wbr>AllCommands)), Strings(Arena) {<br>
+ // Sort commands by filename for determinism (index is a tiebreaker later).<br>
+ llvm::sort(<br>
+ Commands.begin(), Commands.end(),<br>
+ [](const TransferableCommand &Left, const TransferableCommand &Right) {<br>
+ return Left.Cmd.Filename < Right.Cmd.Filename;<br>
+ });<br>
+ for (size_t I = 0; I < Commands.size(); ++I) {<br>
+ StringRef Path =<br>
+ Strings.save(StringRef(<wbr>Commands[I].Cmd.Filename).<wbr>lower());<br>
+ Paths.push_back({Path, I});<br>
+ Stems.emplace_back(sys::path::<wbr>stem(Path), I);<br>
+ auto Dir = ++sys::path::rbegin(Path), DirEnd = sys::path::rend(Path);<br>
+ for (int J = 0; J < DirectorySegmentsIndexed && Dir != DirEnd; ++J, ++Dir)<br>
+ if (Dir->size() > ShortDirectorySegment) // not trivial ones<br>
+ Components.emplace_back(*Dir, I);<br>
+ }<br>
+ llvm::sort(Paths.begin(), Paths.end());<br>
+ llvm::sort(Stems.begin(), Stems.end());<br>
+ llvm::sort(Components.begin(), Components.end());<br>
+ }<br>
+<br>
+ bool empty() const { return Commands.empty(); }<br>
+<br>
+ // Returns the command that best fits OriginalFilename.<br>
+ // Candidates with PreferLanguage will be chosen over others (unless it's<br>
+ // TY_INVALID, or all candidates are bad).<br>
+ const TransferableCommand &chooseProxy(StringRef OriginalFilename,<br>
+ types::ID PreferLanguage) const {<br>
+ assert(!empty() && "need at least one candidate!");<br>
+ std::string Filename = OriginalFilename.lower();<br>
+ auto Candidates = scoreCandidates(Filename);<br>
+ std::pair<size_t, int> Best =<br>
+ pickWinner(Candidates, Filename, PreferLanguage);<br>
+<br>
+ DEBUG_WITH_TYPE("interpolate",<br>
+ llvm::dbgs()<br>
+ << "interpolate: chose "<br>
+ << Commands[Best.first].Cmd.<wbr>Filename << " as proxy for "<br>
+ << OriginalFilename << " preferring "<br>
+ << (PreferLanguage == types::TY_INVALID<br>
+ ? "none"<br>
+ : types::getTypeName(<wbr>PreferLanguage))<br>
+ << " score=" << Best.second << "\n");<br>
+ return Commands[Best.first];<br>
+ }<br>
+<br>
+private:<br>
+ using SubstringAndIndex = std::pair<StringRef, size_t>;<br>
+ // Directory matching parameters: we look at the last two segments of the<br>
+ // parent directory (usually the semantically significant ones in practice).<br>
+ // We search only the last four of each candidate (for efficiency).<br>
+ constexpr static int DirectorySegmentsIndexed = 4;<br>
+ constexpr static int DirectorySegmentsQueried = 2;<br>
+ constexpr static int ShortDirectorySegment = 1; // Only look at longer names.<br>
+<br>
+ // Award points to candidate entries that should be considered for the file.<br>
+ // Returned keys are indexes into paths, and the values are (nonzero) scores.<br>
+ DenseMap<size_t, int> scoreCandidates(StringRef Filename) const {<br>
+ // Decompose Filename into the parts we care about.<br>
+ // /some/path/complicated/<wbr>project/Interesting.h<br>
+ // [-prefix--][---dir---] [-dir-] [--stem---]<br>
+ StringRef Stem = sys::path::stem(Filename);<br>
+ llvm::SmallVector<StringRef, DirectorySegmentsQueried> Dirs;<br>
+ llvm::StringRef Prefix;<br>
+ auto Dir = ++sys::path::rbegin(Filename),<br>
+ DirEnd = sys::path::rend(Filename);<br>
+ for (int I = 0; I < DirectorySegmentsQueried && Dir != DirEnd; ++I, ++Dir) {<br>
+ if (Dir->size() > ShortDirectorySegment)<br>
+ Dirs.push_back(*Dir);<br>
+ Prefix = Filename.substr(0, Dir - DirEnd);<br>
+ }<br>
+<br>
+ // Now award points based on lookups into our various indexes.<br>
+ DenseMap<size_t, int> Candidates; // Index -> score.<br>
+ auto Award = [&](int Points, ArrayRef<SubstringAndIndex> Range) {<br>
+ for (const auto &Entry : Range)<br>
+ Candidates[Entry.second] += Points;<br>
+ };<br>
+ // Award one point if the file's basename is a prefix of the candidate,<br>
+ // and another if it's an exact match (so exact matches get two points).<br>
+ Award(1, indexLookup</*Prefix=*/true>(<wbr>Stem, Stems));<br>
+ Award(1, indexLookup</*Prefix=*/false>(<wbr>Stem, Stems));<br>
+ // For each of the last few directories in the Filename, award a point<br>
+ // if it's present in the candidate.<br>
+ for (StringRef Dir : Dirs)<br>
+ Award(1, indexLookup</*Prefix=*/false>(<wbr>Dir, Components));<br>
+ // Award one more point if the whole rest of the path matches.<br>
+ if (sys::path::root_directory(<wbr>Prefix) != Prefix)<br>
+ Award(1, indexLookup</*Prefix=*/true>(<wbr>Prefix, Paths));<br>
+ return Candidates;<br>
+ }<br>
+<br>
+ // Pick a single winner from the set of scored candidates.<br>
+ // Returns (index, score).<br>
+ std::pair<size_t, int> pickWinner(const DenseMap<size_t, int> &Candidates,<br>
+ StringRef Filename,<br>
+ types::ID PreferredLanguage) const {<br>
+ struct ScoredCandidate {<br>
+ size_t Index;<br>
+ bool Preferred;<br>
+ int Points;<br>
+ size_t PrefixLength;<br>
+ };<br>
+ // Choose the best candidate by (preferred, points, prefix length, alpha).<br>
+ ScoredCandidate Best = {size_t(-1), false, 0, 0};<br>
+ for (const auto &Candidate : Candidates) {<br>
+ ScoredCandidate S;<br>
+ S.Index = Candidate.first;<br>
+ S.Preferred = PreferredLanguage == types::TY_INVALID ||<br>
+ PreferredLanguage == Commands[S.Index].Type;<br>
+ S.Points = Candidate.second;<br>
+ if (!S.Preferred && Best.Preferred)<br>
+ continue;<br>
+ if (S.Preferred == Best.Preferred) {<br>
+ if (S.Points < Best.Points)<br>
+ continue;<br>
+ if (S.Points == Best.Points) {<br>
+ S.PrefixLength = matchingPrefix(Filename, Paths[S.Index].first);<br>
+ if (S.PrefixLength < Best.PrefixLength)<br>
+ continue;<br>
+ // hidden heuristics should at least be deterministic!<br>
+ if (S.PrefixLength == Best.PrefixLength)<br>
+ if (S.Index > Best.Index)<br>
+ continue;<br>
+ }<br>
+ }<br>
+ // PrefixLength was only set above if actually needed for a tiebreak.<br>
+ // But it definitely needs to be set to break ties in the future.<br>
+ S.PrefixLength = matchingPrefix(Filename, Paths[S.Index].first);<br>
+ Best = S;<br>
+ }<br>
+ // Edge case: no candidate got any points.<br>
+ // We ignore PreferredLanguage at this point (not ideal).<br>
+ if (Best.Index == size_t(-1))<br>
+ return {longestMatch(Filename, Paths).second, 0};<br>
+ return {Best.Index, Best.Points};<br>
+ }<br>
+<br>
+ // Returns the range within a sorted index that compares equal to Key.<br>
+ // If Prefix is true, it's instead the range starting with Key.<br>
+ template <bool Prefix><br>
+ ArrayRef<SubstringAndIndex><br>
+ indexLookup(StringRef Key, const std::vector<SubstringAndIndex> &Idx) const {<br>
+ // Use pointers as iteratiors to ease conversion of result to ArrayRef.<br>
+ auto Range =<br>
+ std::equal_range(&Idx[0], &Idx[Idx.size()], Key, Less<Prefix>());<br>
+ return {Range.first, Range.second};<br>
+ }<br>
+<br>
+ // Performs a point lookup into a nonempty index, returning a longest match.<br>
+ SubstringAndIndex<br>
+ longestMatch(StringRef Key, const std::vector<SubstringAndIndex> &Idx) const {<br>
+ assert(!Idx.empty());<br>
+ // Longest substring match will be adjacent to a direct lookup.<br>
+ auto It =<br>
+ std::lower_bound(Idx.begin(), Idx.end(), SubstringAndIndex{Key, 0});<br>
+ if (It == Idx.begin())<br>
+ return *It;<br>
+ if (It == Idx.end())<br>
+ return *--It;<br>
+ // Have to choose between It and It-1<br>
+ size_t Prefix = matchingPrefix(Key, It->first);<br>
+ size_t PrevPrefix = matchingPrefix(Key, (It - 1)->first);<br>
+ return Prefix > PrevPrefix ? *It : *--It;<br>
+ }<br>
+<br>
+ std::vector<<wbr>TransferableCommand> Commands; // Indexes point into this.<br>
+ BumpPtrAllocator Arena;<br>
+ StringSaver Strings;<br>
+ // Indexes of candidates by certain substrings.<br>
+ // String is lowercase and sorted, index points into OriginalPaths.<br>
+ std::vector<SubstringAndIndex> Paths; // Full path.<br>
+ std::vector<SubstringAndIndex> Stems; // Basename, without extension.<br>
+ std::vector<SubstringAndIndex> Components; // Last path components.<br>
+};<br>
+<br>
+// The actual CompilationDatabase wrapper delegates to its inner database.<br>
+// If no match, looks up a command in CommandIndex and transfers it to the file.<br>
+class InterpolatingCompilationDataba<wbr>se : public CompilationDatabase {<br>
+public:<br>
+ InterpolatingCompilationDataba<wbr>se(std::unique_ptr<<wbr>CompilationDatabase> Inner)<br>
+ : Inner(std::move(Inner)), Index(allCommands()) {}<br>
+<br>
+ std::vector<CompileCommand><br>
+ getCompileCommands(StringRef Filename) const override {<br>
+ auto Known = Inner->getCompileCommands(<wbr>Filename);<br>
+ if (Index.empty() || !Known.empty())<br>
+ return Known;<br>
+ bool TypeCertain;<br>
+ auto Lang = guessType(Filename, &TypeCertain);<br>
+ if (!TypeCertain)<br>
+ Lang = types::TY_INVALID;<br>
+ return {Index.chooseProxy(Filename, foldType(Lang)).transferTo(<wbr>Filename)};<br>
+ }<br>
+<br>
+ std::vector<std::string> getAllFiles() const override {<br>
+ return Inner->getAllFiles();<br>
+ }<br>
+<br>
+ std::vector<CompileCommand> getAllCompileCommands() const override {<br>
+ return Inner->getAllCompileCommands()<wbr>;<br>
+ }<br>
+<br>
+private:<br>
+ std::vector<<wbr>TransferableCommand> allCommands() {<br>
+ std::vector<<wbr>TransferableCommand> Result;<br>
+ for (auto Command : Inner->getAllCompileCommands()<wbr>) {<br>
+ Result.emplace_back(std::move(<wbr>Command));<br>
+ if (Result.back().Type == types::TY_INVALID)<br>
+ Result.pop_back();<br>
+ }<br>
+ return Result;<br>
+ }<br>
+<br>
+ std::unique_ptr<<wbr>CompilationDatabase> Inner;<br>
+ CommandIndex Index;<br>
+};<br>
+<br>
+} // namespace<br>
+<br>
+std::unique_ptr<<wbr>CompilationDatabase><br>
+inferMissingCompileCommands(<wbr>std::unique_ptr<<wbr>CompilationDatabase> Inner) {<br>
+ return llvm::make_unique<<wbr>InterpolatingCompilationDataba<wbr>se>(std::move(Inner));<br>
+}<br>
+<br>
+} // namespace tooling<br>
+} // namespace clang<br>
<br>
Modified: cfe/trunk/unittests/Tooling/<wbr>CompilationDatabaseTest.cpp<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/cfe/trunk/unittests/Tooling/CompilationDatabaseTest.cpp?rev=329580&r1=329579&r2=329580&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/cfe/trunk/unittests/<wbr>Tooling/<wbr>CompilationDatabaseTest.cpp?<wbr>rev=329580&r1=329579&r2=<wbr>329580&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- cfe/trunk/unittests/Tooling/<wbr>CompilationDatabaseTest.cpp (original)<br>
+++ cfe/trunk/unittests/Tooling/<wbr>CompilationDatabaseTest.cpp Mon Apr 9 08:17:39 2018<br>
@@ -626,5 +626,115 @@ TEST(<wbr>ParseFixedCompilationDatabase, Hand<br>
EXPECT_EQ(2, Argc);<br>
}<br>
<br>
+struct MemCDB : public CompilationDatabase {<br>
+ using EntryMap = llvm::StringMap<SmallVector<<wbr>CompileCommand, 1>>;<br>
+ EntryMap Entries;<br>
+ MemCDB(const EntryMap &E) : Entries(E) {}<br>
+<br>
+ std::vector<CompileCommand> getCompileCommands(StringRef F) const override {<br>
+ auto Ret = Entries.lookup(F);<br>
+ return {Ret.begin(), Ret.end()};<br>
+ }<br>
+<br>
+ std::vector<std::string> getAllFiles() const override {<br>
+ std::vector<std::string> Result;<br>
+ for (const auto &Entry : Entries)<br>
+ Result.push_back(Entry.first()<wbr>);<br>
+ return Result;<br>
+ }<br>
+};<br>
+<br>
+class InterpolateTest : public ::testing::Test {<br>
+protected:<br>
+ // Adds an entry to the underlying compilation database.<br>
+ // A flag is injected: -D <File>, so the command used can be identified.<br>
+ void add(llvm::StringRef File, llvm::StringRef Flags = "") {<br>
+ llvm::SmallVector<StringRef, 8> Argv = {"clang", File, "-D", File};<br>
+ llvm::SplitString(Flags, Argv);<br>
+ llvm::SmallString<32> Dir;<br>
+ llvm::sys::path::system_temp_<wbr>directory(false, Dir);<br>
+ Entries[path(File)].push_back(<br>
+ {Dir, path(File), {Argv.begin(), Argv.end()}, "foo.o"});<br>
+ }<br>
+<br>
+ // Turn a unix path fragment (foo/bar.h) into a native path (C:\tmp\foo\bar.h)<br>
+ std::string path(llvm::SmallString<32> File) {<br>
+ llvm::SmallString<32> Dir;<br>
+ llvm::sys::path::system_temp_<wbr>directory(false, Dir);<br>
+ llvm::sys::path::native(File);<br>
+ llvm::SmallString<64> Result;<br>
+ llvm::sys::path::append(<wbr>Result, Dir, File);<br>
+ return Result.str();<br>
+ }<br>
+<br>
+ // Look up the command from a relative path, and return it in string form.<br>
+ // The input file is not included in the returned command.<br>
+ std::string getCommand(llvm::StringRef F) {<br>
+ auto Results =<br>
+ inferMissingCompileCommands(<wbr>llvm::make_unique<MemCDB>(<wbr>Entries))<br>
+ ->getCompileCommands(path(F));<br>
+ if (Results.empty())<br>
+ return "none";<br>
+ // drop the input file argument, so tests don't have to deal with path().<br>
+ EXPECT_EQ(Results[0].<wbr>CommandLine.back(), path(F))<br>
+ << "Last arg should be the file";<br>
+ Results[0].CommandLine.pop_<wbr>back();<br>
+ return llvm::join(Results[0].<wbr>CommandLine, " ");<br>
+ }<br>
+<br>
+ MemCDB::EntryMap Entries;<br>
+};<br>
+<br>
+TEST_F(InterpolateTest, Nearby) {<br>
+ add("dir/foo.cpp");<br>
+ add("dir/bar.cpp");<br>
+ add("an/other/foo.cpp");<br>
+<br>
+ // great: dir and name both match (prefix or full, case insensitive)<br>
+ EXPECT_EQ(getCommand("dir/f.<wbr>cpp"), "clang -D dir/foo.cpp");<br>
+ EXPECT_EQ(getCommand("dir/FOO.<wbr>cpp"), "clang -D dir/foo.cpp");<br>
+ // no name match. prefer matching dir, break ties by alpha<br>
+ EXPECT_EQ(getCommand("dir/a.<wbr>cpp"), "clang -D dir/bar.cpp");<br>
+ // an exact name match beats one segment of directory match<br>
+ EXPECT_EQ(getCommand("some/<wbr>other/bar.h"),<br>
+ "clang -D dir/bar.cpp -x c++-header");<br>
+ // two segments of directory match beat a prefix name match<br>
+ EXPECT_EQ(getCommand("an/<wbr>other/b.cpp"), "clang -D an/other/foo.cpp");<br>
+ // if nothing matches at all, we still get the closest alpha match<br>
+ EXPECT_EQ(getCommand("below/<wbr>some/obscure/path.cpp"),<br>
+ "clang -D an/other/foo.cpp");<br>
+}<br>
+<br>
+TEST_F(InterpolateTest, Language) {<br>
+ add("dir/foo.cpp", "-std=c++17");<br>
+ add("dir/baz.cee", "-x c");<br>
+<br>
+ // .h is ambiguous, so we add explicit language flags<br>
+ EXPECT_EQ(getCommand("foo.h"),<br>
+ "clang -D dir/foo.cpp -x c++-header -std c++17");<br>
+ // and don't add -x if the inferred language is correct.<br>
+ EXPECT_EQ(getCommand("foo.hpp"<wbr>), "clang -D dir/foo.cpp -std c++17");<br>
+ // respect -x if it's already there.<br>
+ EXPECT_EQ(getCommand("baz.h"), "clang -D dir/baz.cee -x c-header");<br>
+ // prefer a worse match with the right language<br>
+ EXPECT_EQ(getCommand("foo.c"), "clang -D dir/baz.cee");<br>
+ Entries.erase(path(StringRef("<wbr>dir/baz.cee")));<br>
+ // Now we transfer across languages, so drop -std too.<br>
+ EXPECT_EQ(getCommand("foo.c"), "clang -D dir/foo.cpp");<br>
+}<br>
+<br>
+TEST_F(InterpolateTest, Strip) {<br>
+ add("dir/foo.cpp", "-o foo.o -Wall");<br>
+ // the -o option and the input file are removed, but -Wall is preserved.<br>
+ EXPECT_EQ(getCommand("dir/bar.<wbr>cpp"), "clang -D dir/foo.cpp -Wall");<br>
+}<br>
+<br>
+TEST_F(InterpolateTest, Case) {<br>
+ add("FOO/BAR/BAZ/SHOUT.cc");<br>
+ add("foo/bar/baz/quiet.cc");<br>
+ // Case mismatches are completely ignored, so we choose the name match.<br>
+ EXPECT_EQ(getCommand("foo/bar/<wbr>baz/shout.C"), "clang -D FOO/BAR/BAZ/SHOUT.cc");<br>
+}<br>
+<br>
} // end namespace tooling<br>
} // end namespace clang<br>
<br>
<br>
______________________________<wbr>_________________<br>
cfe-commits mailing list<br>
<a href="mailto:cfe-commits@lists.llvm.org">cfe-commits@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/cfe-commits</a><br>
</blockquote></div><br></div>