[lld] r296702 - ELF: Add ThinLTO caching support.
Teresa Johnson via llvm-commits
llvm-commits at lists.llvm.org
Mon Oct 22 16:47:02 PDT 2018
Peter - do you have time to look into this?
Otherwise I agree they should probably be disabled if it can't be fixed
quickly.
On Mon, Oct 22, 2018 at 4:11 PM Reid Kleckner <rnk at google.com> wrote:
> This test flaked here:
> http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu/builds/13465/
>
> The wasm equivalent one also failed three builds later:
> http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu/builds/13468
>
> Perhaps we should disable the cache policy test until these issues are
> resolved?
>
> On Wed, Oct 17, 2018 at 9:38 AM Chandler Carruth via llvm-commits <
> llvm-commits at lists.llvm.org> wrote:
>
>> I don't have more recent bot failures handy, but you can reproduce this
>> by running the test repeatedly I suspect....
>>
>> That said, maybe the patch you mention will address it?
>>
>> On Mon, Oct 15, 2018 at 9:03 AM Teresa Johnson <tejohnson at google.com>
>> wrote:
>>
>>> Do you have a more recent bot failure you could point at? The previous
>>> one has expired. I wonder if this is:
>>> https://reviews.llvm.org/D53123 (was fixed for the equivalent gold test
>>> recently, and I asked that they fix for lld as well. I just found that they
>>> indeed sent a patch to fix that one too last week, and I just LGTMed).
>>>
>>> Teresa
>>>
>>> On Mon, Oct 15, 2018 at 2:06 AM Chandler Carruth via llvm-commits <
>>> llvm-commits at lists.llvm.org> wrote:
>>>
>>>> Ping!
>>>>
>>>> I continue to see spurious failures of the LTO cache tests for LLD on
>>>> every architecture that supports it. This is a really frustrating thing to
>>>> have just sit in tree and constant break bots and developers.
>>>>
>>>> On Tue, Sep 4, 2018 at 5:58 AM Chandler Carruth <chandlerc at gmail.com>
>>>> wrote:
>>>>
>>>>> Sorry to dig up an old commit, but...
>>>>>
>>>>> On Thu, Mar 2, 2017 at 12:11 AM Peter Collingbourne via llvm-commits <
>>>>> llvm-commits at lists.llvm.org> wrote:
>>>>>
>>>>>> Author: pcc
>>>>>> Date: Wed Mar 1 17:00:10 2017
>>>>>> New Revision: 296702
>>>>>>
>>>>>> URL: http://llvm.org/viewvc/llvm-project?rev=296702&view=rev
>>>>>> Log:
>>>>>> ELF: Add ThinLTO caching support.
>>>>>>
>>>>>> This patch adds an option named --thinlto-cache-dir, which specifies
>>>>>> the
>>>>>> path to a directory in which to cache native object files for ThinLTO
>>>>>> incremental builds.
>>>>>>
>>>>>> Differential Revision: https://reviews.llvm.org/D30509
>>>>>>
>>>>>> Added:
>>>>>> lld/trunk/test/ELF/lto/Inputs/cache.ll
>>>>>> lld/trunk/test/ELF/lto/cache.ll
>>>>>>
>>>>>
>>>>> I have found this test to be frustratingly flaky. Most recent example:
>>>>>
>>>>> http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu/builds/12451
>>>>>
>>>>> It is particularly interesting that it is flaky on this linux bot.
>>>>>
>>>>> Anything else we can do here?
>>>>>
>>>>>
>>>>>> Modified:
>>>>>> lld/trunk/ELF/Config.h
>>>>>> lld/trunk/ELF/Driver.cpp
>>>>>> lld/trunk/ELF/LTO.cpp
>>>>>> lld/trunk/ELF/LTO.h
>>>>>> lld/trunk/ELF/Options.td
>>>>>>
>>>>>> Modified: lld/trunk/ELF/Config.h
>>>>>> URL:
>>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/Config.h?rev=296702&r1=296701&r2=296702&view=diff
>>>>>>
>>>>>> ==============================================================================
>>>>>> --- lld/trunk/ELF/Config.h (original)
>>>>>> +++ lld/trunk/ELF/Config.h Wed Mar 1 17:00:10 2017
>>>>>> @@ -86,6 +86,7 @@ struct Configuration {
>>>>>> llvm::StringRef OptRemarksFilename;
>>>>>> llvm::StringRef SoName;
>>>>>> llvm::StringRef Sysroot;
>>>>>> + llvm::StringRef ThinLTOCacheDir;
>>>>>> std::string RPath;
>>>>>> std::vector<VersionDefinition> VersionDefinitions;
>>>>>> std::vector<llvm::StringRef> AuxiliaryList;
>>>>>>
>>>>>> Modified: lld/trunk/ELF/Driver.cpp
>>>>>> URL:
>>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/Driver.cpp?rev=296702&r1=296701&r2=296702&view=diff
>>>>>>
>>>>>> ==============================================================================
>>>>>> --- lld/trunk/ELF/Driver.cpp (original)
>>>>>> +++ lld/trunk/ELF/Driver.cpp Wed Mar 1 17:00:10 2017
>>>>>> @@ -583,6 +583,7 @@ void LinkerDriver::readConfigs(opt::Inpu
>>>>>> Config->Sysroot = getString(Args, OPT_sysroot);
>>>>>> Config->Target1Rel = getArg(Args, OPT_target1_rel,
>>>>>> OPT_target1_abs, false);
>>>>>> Config->Target2 = getTarget2(Args);
>>>>>> + Config->ThinLTOCacheDir = getString(Args, OPT_thinlto_cache_dir);
>>>>>> Config->ThinLTOJobs = getInteger(Args, OPT_thinlto_jobs, -1u);
>>>>>> Config->Threads = getArg(Args, OPT_threads, OPT_no_threads, true);
>>>>>> Config->Trace = Args.hasArg(OPT_trace);
>>>>>>
>>>>>> Modified: lld/trunk/ELF/LTO.cpp
>>>>>> URL:
>>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/LTO.cpp?rev=296702&r1=296701&r2=296702&view=diff
>>>>>>
>>>>>> ==============================================================================
>>>>>> --- lld/trunk/ELF/LTO.cpp (original)
>>>>>> +++ lld/trunk/ELF/LTO.cpp Wed Mar 1 17:00:10 2017
>>>>>> @@ -18,6 +18,7 @@
>>>>>> #include "llvm/ADT/StringRef.h"
>>>>>> #include "llvm/ADT/Twine.h"
>>>>>> #include "llvm/IR/DiagnosticPrinter.h"
>>>>>> +#include "llvm/LTO/Caching.h"
>>>>>> #include "llvm/LTO/Config.h"
>>>>>> #include "llvm/LTO/LTO.h"
>>>>>> #include "llvm/Object/SymbolicFile.h"
>>>>>> @@ -142,11 +143,24 @@ std::vector<InputFile *> BitcodeCompiler
>>>>>> std::vector<InputFile *> Ret;
>>>>>> unsigned MaxTasks = LTOObj->getMaxTasks();
>>>>>> Buff.resize(MaxTasks);
>>>>>> + Files.resize(MaxTasks);
>>>>>>
>>>>>> - checkError(LTOObj->run([&](size_t Task) {
>>>>>> - return llvm::make_unique<lto::NativeObjectStream>(
>>>>>> - llvm::make_unique<raw_svector_ostream>(Buff[Task]));
>>>>>> - }));
>>>>>> + // The --thinlto-cache-dir option specifies the path to a
>>>>>> directory in which
>>>>>> + // to cache native object files for ThinLTO incremental builds. If
>>>>>> a path was
>>>>>> + // specified, configure LTO to use it as the cache directory.
>>>>>> + lto::NativeObjectCache Cache;
>>>>>> + if (!Config->ThinLTOCacheDir.empty())
>>>>>> + Cache = lto::localCache(Config->ThinLTOCacheDir,
>>>>>> + [&](size_t Task, StringRef Path) {
>>>>>> + Files[Task] =
>>>>>> check(MemoryBuffer::getFile(Path));
>>>>>> + });
>>>>>> +
>>>>>> + checkError(LTOObj->run(
>>>>>> + [&](size_t Task) {
>>>>>> + return llvm::make_unique<lto::NativeObjectStream>(
>>>>>> + llvm::make_unique<raw_svector_ostream>(Buff[Task]));
>>>>>> + },
>>>>>> + Cache));
>>>>>>
>>>>>> for (unsigned I = 0; I != MaxTasks; ++I) {
>>>>>> if (Buff[I].empty())
>>>>>> @@ -160,5 +174,10 @@ std::vector<InputFile *> BitcodeCompiler
>>>>>> InputFile *Obj = createObjectFile(MemoryBufferRef(Buff[I],
>>>>>> "lto.tmp"));
>>>>>> Ret.push_back(Obj);
>>>>>> }
>>>>>> +
>>>>>> + for (std::unique_ptr<MemoryBuffer> &File : Files)
>>>>>> + if (File)
>>>>>> + Ret.push_back(createObjectFile(*File));
>>>>>> +
>>>>>> return Ret;
>>>>>> }
>>>>>>
>>>>>> Modified: lld/trunk/ELF/LTO.h
>>>>>> URL:
>>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/LTO.h?rev=296702&r1=296701&r2=296702&view=diff
>>>>>>
>>>>>> ==============================================================================
>>>>>> --- lld/trunk/ELF/LTO.h (original)
>>>>>> +++ lld/trunk/ELF/LTO.h Wed Mar 1 17:00:10 2017
>>>>>> @@ -49,6 +49,7 @@ public:
>>>>>> private:
>>>>>> std::unique_ptr<llvm::lto::LTO> LTOObj;
>>>>>> std::vector<SmallString<0>> Buff;
>>>>>> + std::vector<std::unique_ptr<MemoryBuffer>> Files;
>>>>>> };
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> Modified: lld/trunk/ELF/Options.td
>>>>>> URL:
>>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/Options.td?rev=296702&r1=296701&r2=296702&view=diff
>>>>>>
>>>>>> ==============================================================================
>>>>>> --- lld/trunk/ELF/Options.td (original)
>>>>>> +++ lld/trunk/ELF/Options.td Wed Mar 1 17:00:10 2017
>>>>>> @@ -392,4 +392,6 @@ def opt_remarks_filename: S<"opt-remarks
>>>>>> def opt_remarks_with_hotness: F<"opt-remarks-with-hotness">,
>>>>>> HelpText<"Include hotness informations in the optimization remarks
>>>>>> file">;
>>>>>> def save_temps: F<"save-temps">;
>>>>>> +def thinlto_cache_dir: J<"thinlto-cache-dir=">,
>>>>>> + HelpText<"Path to ThinLTO cached object file directory">;
>>>>>> def thinlto_jobs: J<"thinlto-jobs=">, HelpText<"Number of ThinLTO
>>>>>> jobs">;
>>>>>>
>>>>>> Added: lld/trunk/test/ELF/lto/Inputs/cache.ll
>>>>>> URL:
>>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/test/ELF/lto/Inputs/cache.ll?rev=296702&view=auto
>>>>>>
>>>>>> ==============================================================================
>>>>>> --- lld/trunk/test/ELF/lto/Inputs/cache.ll (added)
>>>>>> +++ lld/trunk/test/ELF/lto/Inputs/cache.ll Wed Mar 1 17:00:10 2017
>>>>>> @@ -0,0 +1,10 @@
>>>>>> +target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
>>>>>> +target triple = "x86_64-unknown-linux-gnu"
>>>>>> +
>>>>>> +define i32 @main() {
>>>>>> +entry:
>>>>>> + call void (...) @globalfunc()
>>>>>> + ret i32 0
>>>>>> +}
>>>>>> +
>>>>>> +declare void @globalfunc(...)
>>>>>>
>>>>>> Added: lld/trunk/test/ELF/lto/cache.ll
>>>>>> URL:
>>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/test/ELF/lto/cache.ll?rev=296702&view=auto
>>>>>>
>>>>>> ==============================================================================
>>>>>> --- lld/trunk/test/ELF/lto/cache.ll (added)
>>>>>> +++ lld/trunk/test/ELF/lto/cache.ll Wed Mar 1 17:00:10 2017
>>>>>> @@ -0,0 +1,15 @@
>>>>>> +; RUN: opt -module-hash -module-summary %s -o %t.o
>>>>>> +; RUN: opt -module-hash -module-summary %p/Inputs/cache.ll -o %t2.o
>>>>>> +
>>>>>> +; RUN: rm -Rf %t.cache && mkdir %t.cache
>>>>>> +; RUN: ld.lld --thinlto-cache-dir=%t.cache -o %t3.o %t2.o %t.o
>>>>>> +
>>>>>> +; RUN: ls %t.cache | count 2
>>>>>> +
>>>>>> +target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
>>>>>> +target triple = "x86_64-unknown-linux-gnu"
>>>>>> +
>>>>>> +define void @globalfunc() #0 {
>>>>>> +entry:
>>>>>> + ret void
>>>>>> +}
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> llvm-commits mailing list
>>>>>> llvm-commits at lists.llvm.org
>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>>>>>
>>>>> _______________________________________________
>>>> llvm-commits mailing list
>>>> llvm-commits at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>>>
>>>
>>>
>>> --
>>> Teresa Johnson | Software Engineer | tejohnson at google.com |
>>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>
>
--
Teresa Johnson | Software Engineer | tejohnson at google.com |
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20181022/66c19299/attachment.html>
More information about the llvm-commits
mailing list