[lld] r296702 - ELF: Add ThinLTO caching support.

Reid Kleckner via llvm-commits llvm-commits at lists.llvm.org
Mon Oct 22 16:11:25 PDT 2018


This test flaked here:
http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu/builds/13465/

The wasm equivalent one also failed three builds later:
http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu/builds/13468

Perhaps we should disable the cache policy test until these issues are
resolved?

On Wed, Oct 17, 2018 at 9:38 AM Chandler Carruth via llvm-commits <
llvm-commits at lists.llvm.org> wrote:

> I don't have more recent bot failures handy, but you can reproduce this by
> running the test repeatedly I suspect....
>
> That said, maybe the patch you mention will address it?
>
> On Mon, Oct 15, 2018 at 9:03 AM Teresa Johnson <tejohnson at google.com>
> wrote:
>
>> Do you have a more recent bot failure you could point at? The previous
>> one has expired. I wonder if this is:
>> https://reviews.llvm.org/D53123 (was fixed for the equivalent gold test
>> recently, and I asked that they fix for lld as well. I just found that they
>> indeed sent a patch to fix that one too last week, and I just LGTMed).
>>
>> Teresa
>>
>> On Mon, Oct 15, 2018 at 2:06 AM Chandler Carruth via llvm-commits <
>> llvm-commits at lists.llvm.org> wrote:
>>
>>> Ping!
>>>
>>> I continue to see spurious failures of the LTO cache tests for LLD on
>>> every architecture that supports it. This is a really frustrating thing to
>>> have just sit in tree and constant break bots and developers.
>>>
>>> On Tue, Sep 4, 2018 at 5:58 AM Chandler Carruth <chandlerc at gmail.com>
>>> wrote:
>>>
>>>> Sorry to dig up an old commit, but...
>>>>
>>>> On Thu, Mar 2, 2017 at 12:11 AM Peter Collingbourne via llvm-commits <
>>>> llvm-commits at lists.llvm.org> wrote:
>>>>
>>>>> Author: pcc
>>>>> Date: Wed Mar  1 17:00:10 2017
>>>>> New Revision: 296702
>>>>>
>>>>> URL: http://llvm.org/viewvc/llvm-project?rev=296702&view=rev
>>>>> Log:
>>>>> ELF: Add ThinLTO caching support.
>>>>>
>>>>> This patch adds an option named --thinlto-cache-dir, which specifies
>>>>> the
>>>>> path to a directory in which to cache native object files for ThinLTO
>>>>> incremental builds.
>>>>>
>>>>> Differential Revision: https://reviews.llvm.org/D30509
>>>>>
>>>>> Added:
>>>>>     lld/trunk/test/ELF/lto/Inputs/cache.ll
>>>>>     lld/trunk/test/ELF/lto/cache.ll
>>>>>
>>>>
>>>> I  have found this test to be frustratingly flaky. Most recent example:
>>>>
>>>> http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu/builds/12451
>>>>
>>>> It is particularly interesting that it is flaky on this linux bot.
>>>>
>>>> Anything else we can do here?
>>>>
>>>>
>>>>> Modified:
>>>>>     lld/trunk/ELF/Config.h
>>>>>     lld/trunk/ELF/Driver.cpp
>>>>>     lld/trunk/ELF/LTO.cpp
>>>>>     lld/trunk/ELF/LTO.h
>>>>>     lld/trunk/ELF/Options.td
>>>>>
>>>>> Modified: lld/trunk/ELF/Config.h
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/Config.h?rev=296702&r1=296701&r2=296702&view=diff
>>>>>
>>>>> ==============================================================================
>>>>> --- lld/trunk/ELF/Config.h (original)
>>>>> +++ lld/trunk/ELF/Config.h Wed Mar  1 17:00:10 2017
>>>>> @@ -86,6 +86,7 @@ struct Configuration {
>>>>>    llvm::StringRef OptRemarksFilename;
>>>>>    llvm::StringRef SoName;
>>>>>    llvm::StringRef Sysroot;
>>>>> +  llvm::StringRef ThinLTOCacheDir;
>>>>>    std::string RPath;
>>>>>    std::vector<VersionDefinition> VersionDefinitions;
>>>>>    std::vector<llvm::StringRef> AuxiliaryList;
>>>>>
>>>>> Modified: lld/trunk/ELF/Driver.cpp
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/Driver.cpp?rev=296702&r1=296701&r2=296702&view=diff
>>>>>
>>>>> ==============================================================================
>>>>> --- lld/trunk/ELF/Driver.cpp (original)
>>>>> +++ lld/trunk/ELF/Driver.cpp Wed Mar  1 17:00:10 2017
>>>>> @@ -583,6 +583,7 @@ void LinkerDriver::readConfigs(opt::Inpu
>>>>>    Config->Sysroot = getString(Args, OPT_sysroot);
>>>>>    Config->Target1Rel = getArg(Args, OPT_target1_rel, OPT_target1_abs,
>>>>> false);
>>>>>    Config->Target2 = getTarget2(Args);
>>>>> +  Config->ThinLTOCacheDir = getString(Args, OPT_thinlto_cache_dir);
>>>>>    Config->ThinLTOJobs = getInteger(Args, OPT_thinlto_jobs, -1u);
>>>>>    Config->Threads = getArg(Args, OPT_threads, OPT_no_threads, true);
>>>>>    Config->Trace = Args.hasArg(OPT_trace);
>>>>>
>>>>> Modified: lld/trunk/ELF/LTO.cpp
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/LTO.cpp?rev=296702&r1=296701&r2=296702&view=diff
>>>>>
>>>>> ==============================================================================
>>>>> --- lld/trunk/ELF/LTO.cpp (original)
>>>>> +++ lld/trunk/ELF/LTO.cpp Wed Mar  1 17:00:10 2017
>>>>> @@ -18,6 +18,7 @@
>>>>>  #include "llvm/ADT/StringRef.h"
>>>>>  #include "llvm/ADT/Twine.h"
>>>>>  #include "llvm/IR/DiagnosticPrinter.h"
>>>>> +#include "llvm/LTO/Caching.h"
>>>>>  #include "llvm/LTO/Config.h"
>>>>>  #include "llvm/LTO/LTO.h"
>>>>>  #include "llvm/Object/SymbolicFile.h"
>>>>> @@ -142,11 +143,24 @@ std::vector<InputFile *> BitcodeCompiler
>>>>>    std::vector<InputFile *> Ret;
>>>>>    unsigned MaxTasks = LTOObj->getMaxTasks();
>>>>>    Buff.resize(MaxTasks);
>>>>> +  Files.resize(MaxTasks);
>>>>>
>>>>> -  checkError(LTOObj->run([&](size_t Task) {
>>>>> -    return llvm::make_unique<lto::NativeObjectStream>(
>>>>> -        llvm::make_unique<raw_svector_ostream>(Buff[Task]));
>>>>> -  }));
>>>>> +  // The --thinlto-cache-dir option specifies the path to a directory
>>>>> in which
>>>>> +  // to cache native object files for ThinLTO incremental builds. If
>>>>> a path was
>>>>> +  // specified, configure LTO to use it as the cache directory.
>>>>> +  lto::NativeObjectCache Cache;
>>>>> +  if (!Config->ThinLTOCacheDir.empty())
>>>>> +    Cache = lto::localCache(Config->ThinLTOCacheDir,
>>>>> +                            [&](size_t Task, StringRef Path) {
>>>>> +                              Files[Task] =
>>>>> check(MemoryBuffer::getFile(Path));
>>>>> +                            });
>>>>> +
>>>>> +  checkError(LTOObj->run(
>>>>> +      [&](size_t Task) {
>>>>> +        return llvm::make_unique<lto::NativeObjectStream>(
>>>>> +            llvm::make_unique<raw_svector_ostream>(Buff[Task]));
>>>>> +      },
>>>>> +      Cache));
>>>>>
>>>>>    for (unsigned I = 0; I != MaxTasks; ++I) {
>>>>>      if (Buff[I].empty())
>>>>> @@ -160,5 +174,10 @@ std::vector<InputFile *> BitcodeCompiler
>>>>>      InputFile *Obj = createObjectFile(MemoryBufferRef(Buff[I],
>>>>> "lto.tmp"));
>>>>>      Ret.push_back(Obj);
>>>>>    }
>>>>> +
>>>>> +  for (std::unique_ptr<MemoryBuffer> &File : Files)
>>>>> +    if (File)
>>>>> +      Ret.push_back(createObjectFile(*File));
>>>>> +
>>>>>    return Ret;
>>>>>  }
>>>>>
>>>>> Modified: lld/trunk/ELF/LTO.h
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/LTO.h?rev=296702&r1=296701&r2=296702&view=diff
>>>>>
>>>>> ==============================================================================
>>>>> --- lld/trunk/ELF/LTO.h (original)
>>>>> +++ lld/trunk/ELF/LTO.h Wed Mar  1 17:00:10 2017
>>>>> @@ -49,6 +49,7 @@ public:
>>>>>  private:
>>>>>    std::unique_ptr<llvm::lto::LTO> LTOObj;
>>>>>    std::vector<SmallString<0>> Buff;
>>>>> +  std::vector<std::unique_ptr<MemoryBuffer>> Files;
>>>>>  };
>>>>>  }
>>>>>  }
>>>>>
>>>>> Modified: lld/trunk/ELF/Options.td
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/Options.td?rev=296702&r1=296701&r2=296702&view=diff
>>>>>
>>>>> ==============================================================================
>>>>> --- lld/trunk/ELF/Options.td (original)
>>>>> +++ lld/trunk/ELF/Options.td Wed Mar  1 17:00:10 2017
>>>>> @@ -392,4 +392,6 @@ def opt_remarks_filename: S<"opt-remarks
>>>>>  def opt_remarks_with_hotness: F<"opt-remarks-with-hotness">,
>>>>>    HelpText<"Include hotness informations in the optimization remarks
>>>>> file">;
>>>>>  def save_temps: F<"save-temps">;
>>>>> +def thinlto_cache_dir: J<"thinlto-cache-dir=">,
>>>>> +  HelpText<"Path to ThinLTO cached object file directory">;
>>>>>  def thinlto_jobs: J<"thinlto-jobs=">, HelpText<"Number of ThinLTO
>>>>> jobs">;
>>>>>
>>>>> Added: lld/trunk/test/ELF/lto/Inputs/cache.ll
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/test/ELF/lto/Inputs/cache.ll?rev=296702&view=auto
>>>>>
>>>>> ==============================================================================
>>>>> --- lld/trunk/test/ELF/lto/Inputs/cache.ll (added)
>>>>> +++ lld/trunk/test/ELF/lto/Inputs/cache.ll Wed Mar  1 17:00:10 2017
>>>>> @@ -0,0 +1,10 @@
>>>>> +target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
>>>>> +target triple = "x86_64-unknown-linux-gnu"
>>>>> +
>>>>> +define i32 @main() {
>>>>> +entry:
>>>>> +  call void (...) @globalfunc()
>>>>> +  ret i32 0
>>>>> +}
>>>>> +
>>>>> +declare void @globalfunc(...)
>>>>>
>>>>> Added: lld/trunk/test/ELF/lto/cache.ll
>>>>> URL:
>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/test/ELF/lto/cache.ll?rev=296702&view=auto
>>>>>
>>>>> ==============================================================================
>>>>> --- lld/trunk/test/ELF/lto/cache.ll (added)
>>>>> +++ lld/trunk/test/ELF/lto/cache.ll Wed Mar  1 17:00:10 2017
>>>>> @@ -0,0 +1,15 @@
>>>>> +; RUN: opt -module-hash -module-summary %s -o %t.o
>>>>> +; RUN: opt -module-hash -module-summary %p/Inputs/cache.ll -o %t2.o
>>>>> +
>>>>> +; RUN: rm -Rf %t.cache && mkdir %t.cache
>>>>> +; RUN: ld.lld --thinlto-cache-dir=%t.cache -o %t3.o %t2.o %t.o
>>>>> +
>>>>> +; RUN: ls %t.cache | count 2
>>>>> +
>>>>> +target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
>>>>> +target triple = "x86_64-unknown-linux-gnu"
>>>>> +
>>>>> +define void @globalfunc() #0 {
>>>>> +entry:
>>>>> +  ret void
>>>>> +}
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> llvm-commits mailing list
>>>>> llvm-commits at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>>>>
>>>> _______________________________________________
>>> llvm-commits mailing list
>>> llvm-commits at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>>
>>
>>
>> --
>> Teresa Johnson |  Software Engineer |  tejohnson at google.com |
>>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20181022/78ac65c0/attachment.html>


More information about the llvm-commits mailing list