[lld] r296702 - ELF: Add ThinLTO caching support.
Xin Tong via llvm-commits
llvm-commits at lists.llvm.org
Mon Oct 22 17:49:47 PDT 2018
Hi Teresa, Peter and Reid
We have seen similar problems with this one. I diagnosed the case I have
seen and concluded this is a problem with ATime on Centos in my case. My
failure looks exactly the same. I can try to provide a fix.
More details about the failure:
Say we create file A (1K size), B (1K size), C (2K size) in this order, and
then read them in this order, C, B, A. And we need to prune the total size
to 2KB. We first want to delete files w.r.t access time, i.e. LRU files get
deleted. And in case of tie, we delete the larger file.
If ATime is tracked properly, we end up deleting C because it is LRU.
However, if it is not tracked properly, we may end up delete A and B
because ATime is not tracked and they are created first.
So this is the fundamental issue to cause the test case to fail, i.e. we
deleted the wrong file.
This does not completely explain the intermittence. The non-determinism
happens because ATime is tracked on a second granularity and if their
creation/modification time happen to fall on the same second, we delete C
because its larger and the test passes. Otherwise we delete A and B because
they are created first and test fails.
Thanks
-Xin
On Mon, Oct 22, 2018 at 4:47 PM Teresa Johnson via llvm-commits <
llvm-commits at lists.llvm.org> wrote:
> Peter - do you have time to look into this?
>
> Otherwise I agree they should probably be disabled if it can't be fixed
> quickly.
>
> On Mon, Oct 22, 2018 at 4:11 PM Reid Kleckner <rnk at google.com> wrote:
>
>> This test flaked here:
>> http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu/builds/13465/
>>
>> The wasm equivalent one also failed three builds later:
>> http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu/builds/13468
>>
>> Perhaps we should disable the cache policy test until these issues are
>> resolved?
>>
>> On Wed, Oct 17, 2018 at 9:38 AM Chandler Carruth via llvm-commits <
>> llvm-commits at lists.llvm.org> wrote:
>>
>>> I don't have more recent bot failures handy, but you can reproduce this
>>> by running the test repeatedly I suspect....
>>>
>>> That said, maybe the patch you mention will address it?
>>>
>>> On Mon, Oct 15, 2018 at 9:03 AM Teresa Johnson <tejohnson at google.com>
>>> wrote:
>>>
>>>> Do you have a more recent bot failure you could point at? The previous
>>>> one has expired. I wonder if this is:
>>>> https://reviews.llvm.org/D53123 (was fixed for the equivalent gold
>>>> test recently, and I asked that they fix for lld as well. I just found that
>>>> they indeed sent a patch to fix that one too last week, and I just LGTMed).
>>>>
>>>> Teresa
>>>>
>>>> On Mon, Oct 15, 2018 at 2:06 AM Chandler Carruth via llvm-commits <
>>>> llvm-commits at lists.llvm.org> wrote:
>>>>
>>>>> Ping!
>>>>>
>>>>> I continue to see spurious failures of the LTO cache tests for LLD on
>>>>> every architecture that supports it. This is a really frustrating thing to
>>>>> have just sit in tree and constant break bots and developers.
>>>>>
>>>>> On Tue, Sep 4, 2018 at 5:58 AM Chandler Carruth <chandlerc at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Sorry to dig up an old commit, but...
>>>>>>
>>>>>> On Thu, Mar 2, 2017 at 12:11 AM Peter Collingbourne via llvm-commits <
>>>>>> llvm-commits at lists.llvm.org> wrote:
>>>>>>
>>>>>>> Author: pcc
>>>>>>> Date: Wed Mar 1 17:00:10 2017
>>>>>>> New Revision: 296702
>>>>>>>
>>>>>>> URL: http://llvm.org/viewvc/llvm-project?rev=296702&view=rev
>>>>>>> Log:
>>>>>>> ELF: Add ThinLTO caching support.
>>>>>>>
>>>>>>> This patch adds an option named --thinlto-cache-dir, which specifies
>>>>>>> the
>>>>>>> path to a directory in which to cache native object files for ThinLTO
>>>>>>> incremental builds.
>>>>>>>
>>>>>>> Differential Revision: https://reviews.llvm.org/D30509
>>>>>>>
>>>>>>> Added:
>>>>>>> lld/trunk/test/ELF/lto/Inputs/cache.ll
>>>>>>> lld/trunk/test/ELF/lto/cache.ll
>>>>>>>
>>>>>>
>>>>>> I have found this test to be frustratingly flaky. Most recent
>>>>>> example:
>>>>>>
>>>>>> http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu/builds/12451
>>>>>>
>>>>>> It is particularly interesting that it is flaky on this linux bot.
>>>>>>
>>>>>> Anything else we can do here?
>>>>>>
>>>>>>
>>>>>>> Modified:
>>>>>>> lld/trunk/ELF/Config.h
>>>>>>> lld/trunk/ELF/Driver.cpp
>>>>>>> lld/trunk/ELF/LTO.cpp
>>>>>>> lld/trunk/ELF/LTO.h
>>>>>>> lld/trunk/ELF/Options.td
>>>>>>>
>>>>>>> Modified: lld/trunk/ELF/Config.h
>>>>>>> URL:
>>>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/Config.h?rev=296702&r1=296701&r2=296702&view=diff
>>>>>>>
>>>>>>> ==============================================================================
>>>>>>> --- lld/trunk/ELF/Config.h (original)
>>>>>>> +++ lld/trunk/ELF/Config.h Wed Mar 1 17:00:10 2017
>>>>>>> @@ -86,6 +86,7 @@ struct Configuration {
>>>>>>> llvm::StringRef OptRemarksFilename;
>>>>>>> llvm::StringRef SoName;
>>>>>>> llvm::StringRef Sysroot;
>>>>>>> + llvm::StringRef ThinLTOCacheDir;
>>>>>>> std::string RPath;
>>>>>>> std::vector<VersionDefinition> VersionDefinitions;
>>>>>>> std::vector<llvm::StringRef> AuxiliaryList;
>>>>>>>
>>>>>>> Modified: lld/trunk/ELF/Driver.cpp
>>>>>>> URL:
>>>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/Driver.cpp?rev=296702&r1=296701&r2=296702&view=diff
>>>>>>>
>>>>>>> ==============================================================================
>>>>>>> --- lld/trunk/ELF/Driver.cpp (original)
>>>>>>> +++ lld/trunk/ELF/Driver.cpp Wed Mar 1 17:00:10 2017
>>>>>>> @@ -583,6 +583,7 @@ void LinkerDriver::readConfigs(opt::Inpu
>>>>>>> Config->Sysroot = getString(Args, OPT_sysroot);
>>>>>>> Config->Target1Rel = getArg(Args, OPT_target1_rel,
>>>>>>> OPT_target1_abs, false);
>>>>>>> Config->Target2 = getTarget2(Args);
>>>>>>> + Config->ThinLTOCacheDir = getString(Args, OPT_thinlto_cache_dir);
>>>>>>> Config->ThinLTOJobs = getInteger(Args, OPT_thinlto_jobs, -1u);
>>>>>>> Config->Threads = getArg(Args, OPT_threads, OPT_no_threads, true);
>>>>>>> Config->Trace = Args.hasArg(OPT_trace);
>>>>>>>
>>>>>>> Modified: lld/trunk/ELF/LTO.cpp
>>>>>>> URL:
>>>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/LTO.cpp?rev=296702&r1=296701&r2=296702&view=diff
>>>>>>>
>>>>>>> ==============================================================================
>>>>>>> --- lld/trunk/ELF/LTO.cpp (original)
>>>>>>> +++ lld/trunk/ELF/LTO.cpp Wed Mar 1 17:00:10 2017
>>>>>>> @@ -18,6 +18,7 @@
>>>>>>> #include "llvm/ADT/StringRef.h"
>>>>>>> #include "llvm/ADT/Twine.h"
>>>>>>> #include "llvm/IR/DiagnosticPrinter.h"
>>>>>>> +#include "llvm/LTO/Caching.h"
>>>>>>> #include "llvm/LTO/Config.h"
>>>>>>> #include "llvm/LTO/LTO.h"
>>>>>>> #include "llvm/Object/SymbolicFile.h"
>>>>>>> @@ -142,11 +143,24 @@ std::vector<InputFile *> BitcodeCompiler
>>>>>>> std::vector<InputFile *> Ret;
>>>>>>> unsigned MaxTasks = LTOObj->getMaxTasks();
>>>>>>> Buff.resize(MaxTasks);
>>>>>>> + Files.resize(MaxTasks);
>>>>>>>
>>>>>>> - checkError(LTOObj->run([&](size_t Task) {
>>>>>>> - return llvm::make_unique<lto::NativeObjectStream>(
>>>>>>> - llvm::make_unique<raw_svector_ostream>(Buff[Task]));
>>>>>>> - }));
>>>>>>> + // The --thinlto-cache-dir option specifies the path to a
>>>>>>> directory in which
>>>>>>> + // to cache native object files for ThinLTO incremental builds.
>>>>>>> If a path was
>>>>>>> + // specified, configure LTO to use it as the cache directory.
>>>>>>> + lto::NativeObjectCache Cache;
>>>>>>> + if (!Config->ThinLTOCacheDir.empty())
>>>>>>> + Cache = lto::localCache(Config->ThinLTOCacheDir,
>>>>>>> + [&](size_t Task, StringRef Path) {
>>>>>>> + Files[Task] =
>>>>>>> check(MemoryBuffer::getFile(Path));
>>>>>>> + });
>>>>>>> +
>>>>>>> + checkError(LTOObj->run(
>>>>>>> + [&](size_t Task) {
>>>>>>> + return llvm::make_unique<lto::NativeObjectStream>(
>>>>>>> + llvm::make_unique<raw_svector_ostream>(Buff[Task]));
>>>>>>> + },
>>>>>>> + Cache));
>>>>>>>
>>>>>>> for (unsigned I = 0; I != MaxTasks; ++I) {
>>>>>>> if (Buff[I].empty())
>>>>>>> @@ -160,5 +174,10 @@ std::vector<InputFile *> BitcodeCompiler
>>>>>>> InputFile *Obj = createObjectFile(MemoryBufferRef(Buff[I],
>>>>>>> "lto.tmp"));
>>>>>>> Ret.push_back(Obj);
>>>>>>> }
>>>>>>> +
>>>>>>> + for (std::unique_ptr<MemoryBuffer> &File : Files)
>>>>>>> + if (File)
>>>>>>> + Ret.push_back(createObjectFile(*File));
>>>>>>> +
>>>>>>> return Ret;
>>>>>>> }
>>>>>>>
>>>>>>> Modified: lld/trunk/ELF/LTO.h
>>>>>>> URL:
>>>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/LTO.h?rev=296702&r1=296701&r2=296702&view=diff
>>>>>>>
>>>>>>> ==============================================================================
>>>>>>> --- lld/trunk/ELF/LTO.h (original)
>>>>>>> +++ lld/trunk/ELF/LTO.h Wed Mar 1 17:00:10 2017
>>>>>>> @@ -49,6 +49,7 @@ public:
>>>>>>> private:
>>>>>>> std::unique_ptr<llvm::lto::LTO> LTOObj;
>>>>>>> std::vector<SmallString<0>> Buff;
>>>>>>> + std::vector<std::unique_ptr<MemoryBuffer>> Files;
>>>>>>> };
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> Modified: lld/trunk/ELF/Options.td
>>>>>>> URL:
>>>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/Options.td?rev=296702&r1=296701&r2=296702&view=diff
>>>>>>>
>>>>>>> ==============================================================================
>>>>>>> --- lld/trunk/ELF/Options.td (original)
>>>>>>> +++ lld/trunk/ELF/Options.td Wed Mar 1 17:00:10 2017
>>>>>>> @@ -392,4 +392,6 @@ def opt_remarks_filename: S<"opt-remarks
>>>>>>> def opt_remarks_with_hotness: F<"opt-remarks-with-hotness">,
>>>>>>> HelpText<"Include hotness informations in the optimization
>>>>>>> remarks file">;
>>>>>>> def save_temps: F<"save-temps">;
>>>>>>> +def thinlto_cache_dir: J<"thinlto-cache-dir=">,
>>>>>>> + HelpText<"Path to ThinLTO cached object file directory">;
>>>>>>> def thinlto_jobs: J<"thinlto-jobs=">, HelpText<"Number of ThinLTO
>>>>>>> jobs">;
>>>>>>>
>>>>>>> Added: lld/trunk/test/ELF/lto/Inputs/cache.ll
>>>>>>> URL:
>>>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/test/ELF/lto/Inputs/cache.ll?rev=296702&view=auto
>>>>>>>
>>>>>>> ==============================================================================
>>>>>>> --- lld/trunk/test/ELF/lto/Inputs/cache.ll (added)
>>>>>>> +++ lld/trunk/test/ELF/lto/Inputs/cache.ll Wed Mar 1 17:00:10 2017
>>>>>>> @@ -0,0 +1,10 @@
>>>>>>> +target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
>>>>>>> +target triple = "x86_64-unknown-linux-gnu"
>>>>>>> +
>>>>>>> +define i32 @main() {
>>>>>>> +entry:
>>>>>>> + call void (...) @globalfunc()
>>>>>>> + ret i32 0
>>>>>>> +}
>>>>>>> +
>>>>>>> +declare void @globalfunc(...)
>>>>>>>
>>>>>>> Added: lld/trunk/test/ELF/lto/cache.ll
>>>>>>> URL:
>>>>>>> http://llvm.org/viewvc/llvm-project/lld/trunk/test/ELF/lto/cache.ll?rev=296702&view=auto
>>>>>>>
>>>>>>> ==============================================================================
>>>>>>> --- lld/trunk/test/ELF/lto/cache.ll (added)
>>>>>>> +++ lld/trunk/test/ELF/lto/cache.ll Wed Mar 1 17:00:10 2017
>>>>>>> @@ -0,0 +1,15 @@
>>>>>>> +; RUN: opt -module-hash -module-summary %s -o %t.o
>>>>>>> +; RUN: opt -module-hash -module-summary %p/Inputs/cache.ll -o %t2.o
>>>>>>> +
>>>>>>> +; RUN: rm -Rf %t.cache && mkdir %t.cache
>>>>>>> +; RUN: ld.lld --thinlto-cache-dir=%t.cache -o %t3.o %t2.o %t.o
>>>>>>> +
>>>>>>> +; RUN: ls %t.cache | count 2
>>>>>>> +
>>>>>>> +target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
>>>>>>> +target triple = "x86_64-unknown-linux-gnu"
>>>>>>> +
>>>>>>> +define void @globalfunc() #0 {
>>>>>>> +entry:
>>>>>>> + ret void
>>>>>>> +}
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> llvm-commits mailing list
>>>>>>> llvm-commits at lists.llvm.org
>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>>>>>>
>>>>>> _______________________________________________
>>>>> llvm-commits mailing list
>>>>> llvm-commits at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>>>>
>>>>
>>>>
>>>> --
>>>> Teresa Johnson | Software Engineer | tejohnson at google.com |
>>>>
>>> _______________________________________________
>>> llvm-commits mailing list
>>> llvm-commits at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>>
>>
>
> --
> Teresa Johnson | Software Engineer | tejohnson at google.com |
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20181022/fee95001/attachment-0001.html>
More information about the llvm-commits
mailing list