<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 19, 2019 at 11:15 AM Kostya Serebryany <<a href="mailto:kcc@google.com">kcc@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 19, 2019 at 5:44 AM Peter Smith <<a href="mailto:peter.smith@linaro.org" target="_blank">peter.smith@linaro.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello Kostya, Diana,<br>
<br>
I've taken over from Diana at looking after the AArch64 bots this<br>
week. I've run the cxa_atexit_race.cc test in a loop and it<br>
intermittently hangs on average about 1 in 75 times,</blockquote><div><br></div><div><a class="gmail_plusreply" id="gmail-m_7473641812114484524plusReplyChip-3" href="mailto:vitalybuka@google.com" target="_blank">@Vitaly Buka</a> who added the test. <br></div></div></div></blockquote><div><br></div><div>ah, I see you already removed the test. thanks! </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> a frequency that<br>
seems to go up a lot when the machine is heavily loaded. I've tested<br>
before and after locally reverting r354092 I was able to reproduce a<br>
hang without r354092 so in conjunction with hanging test being in<br>
msan, I agree that it isn't this particular change that is the root<br>
cause.<br>
<br>
Given that the cxa_atexit_race.cc is relatively new (14th Februrary) I<br>
think it may just be that the fix for the atexit problem doesn't work<br>
on AArch64. I'll follow up with the author of the test to see if we<br>
can work something out.<br>
<br>
I don't know who maintains lsan on AArch64 either. I can ask our local<br>
expert on Thursday to see if he knows. </blockquote><div><br></div><div>yes, please! </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Having said that while 18<br>
seconds is slow it is unlikely to cause the builder to run out of<br>
time.<br></blockquote><div><br></div><div>the check-fuzzer test rule (and a few others) runs hundreds of tiny tests that run lsan in the end. <br></div><div>On x86_64 it takes a fraction of a second, but on aarch64 it takes forver, so it might very well be reason for other timeouts. </div><div><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Peter<br>
<br>
On Sat, 16 Feb 2019 at 02:06, Kostya Serebryany via llvm-commits<br>
<<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>> wrote:<br>
><br>
> My change just triggered an older problem: lsan seems to be badly broken on aarch64.<br>
> It spends ridiculous amount of time, and a test that is expected to pass in < 1 second takes 18 seconds.<br>
><br>
> % clang -fsanitize=leak ~/llvm/compiler-rt/test/lsan/TestCases/sanity_check_pure_c.c && time ./a.out<br>
><br>
> real 0m18.313s<br>
> user 0m18.304s<br>
> sys 0m0.008s<br>
><br>
> Perhaps this is <a href="https://github.com/google/sanitizers/issues/703" rel="noreferrer" target="_blank">https://github.com/google/sanitizers/issues/703</a>, or something else.<br>
><br>
> My team doesn't maintain lsan on aarch64.. :(<br>
> I wonder who does?<br>
><br>
><br>
><br>
><br>
> On Fri, Feb 15, 2019 at 10:50 AM Kostya Serebryany <<a href="mailto:kcc@google.com" target="_blank">kcc@google.com</a>> wrote:<br>
>><br>
>> Hi Diana,<br>
>><br>
>> I am not sure how to interpret these bot failures...<br>
>> Can you tell which test times out?<br>
>><br>
>> The only tests that can be affected by this change pass:<br>
>><br>
>> PASS: libFuzzer :: merge.test (48757 of 48763)<br>
>><br>
>> PASS: libFuzzer :: fork.test (48754 of 48763)<br>
>><br>
>> (let me revive my aarch64 development box and see it myself)<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> On Fri, Feb 15, 2019 at 4:22 AM Diana Picus <<a href="mailto:diana.picus@linaro.org" target="_blank">diana.picus@linaro.org</a>> wrote:<br>
>>><br>
>>> Hi Kostya,<br>
>>><br>
>>> It seems that this is causing the tests to hang on AArch64:<br>
>>> <a href="http://lab.llvm.org:8011/builders/clang-cmake-aarch64-full/builds/6571" rel="noreferrer" target="_blank">http://lab.llvm.org:8011/builders/clang-cmake-aarch64-full/builds/6571</a><br>
>>> <a href="http://lab.llvm.org:8011/builders/clang-cmake-aarch64-lld/builds/6038" rel="noreferrer" target="_blank">http://lab.llvm.org:8011/builders/clang-cmake-aarch64-lld/builds/6038</a><br>
>>><br>
>>> It might have something to do with cxa_atexit_race, since I saw it<br>
>>> running for about 8 minutes on one of the machines.<br>
>>><br>
>>> Could you please have a look?<br>
>>><br>
>>> Thanks,<br>
>>> Diana<br>
>>><br>
>>> On Fri, 15 Feb 2019 at 02:21, Kostya Serebryany via llvm-commits<br>
>>> <<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>> wrote:<br>
>>> ><br>
>>> > Author: kcc<br>
>>> > Date: Thu Feb 14 17:22:00 2019<br>
>>> > New Revision: 354092<br>
>>> ><br>
>>> > URL: <a href="http://llvm.org/viewvc/llvm-project?rev=354092&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project?rev=354092&view=rev</a><br>
>>> > Log:<br>
>>> > [libFuzzer] print new functions as they are discovered in the fork mode<br>
>>> ><br>
>>> > Modified:<br>
>>> > compiler-rt/trunk/lib/fuzzer/FuzzerFork.cpp<br>
>>> > compiler-rt/trunk/lib/fuzzer/FuzzerMerge.cpp<br>
>>> > compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp<br>
>>> > compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h<br>
>>> ><br>
>>> > Modified: compiler-rt/trunk/lib/fuzzer/FuzzerFork.cpp<br>
>>> > URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerFork.cpp?rev=354092&r1=354091&r2=354092&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerFork.cpp?rev=354092&r1=354091&r2=354092&view=diff</a><br>
>>> > ==============================================================================<br>
>>> > --- compiler-rt/trunk/lib/fuzzer/FuzzerFork.cpp (original)<br>
>>> > +++ compiler-rt/trunk/lib/fuzzer/FuzzerFork.cpp Thu Feb 14 17:22:00 2019<br>
>>> > @@ -13,6 +13,7 @@<br>
>>> > #include "FuzzerIO.h"<br>
>>> > #include "FuzzerMerge.h"<br>
>>> > #include "FuzzerSHA1.h"<br>
>>> > +#include "FuzzerTracePC.h"<br>
>>> > #include "FuzzerUtil.h"<br>
>>> ><br>
>>> > #include <atomic><br>
>>> > @@ -86,11 +87,13 @@ struct GlobalEnv {<br>
>>> > Cmd.removeArgument(C);<br>
>>> > Cmd.addFlag("reload", "0"); // working in an isolated dir, no reload.<br>
>>> > Cmd.addFlag("print_final_stats", "1");<br>
>>> > + Cmd.addFlag("print_funcs", "0"); // no need to spend time symbolizing.<br>
>>> > Cmd.addFlag("max_total_time", std::to_string(std::min((size_t)300, JobId)));<br>
>>> ><br>
>>> > auto Job = new FuzzJob;<br>
>>> > std::string Seeds;<br>
>>> > - if (size_t CorpusSubsetSize = std::min(Files.size(), (size_t)100))<br>
>>> > + if (size_t CorpusSubsetSize =<br>
>>> > + std::min(Files.size(), (size_t)sqrt(Files.size() + 2)))<br>
>>> > for (size_t i = 0; i < CorpusSubsetSize; i++)<br>
>>> > Seeds += (Seeds.empty() ? "" : ",") +<br>
>>> > Files[Rand->SkewTowardsLast(Files.size())];<br>
>>> > @@ -135,6 +138,12 @@ struct GlobalEnv {<br>
>>> > RmDirRecursive(Job->CorpusDir);<br>
>>> > Features.insert(NewFeatures.begin(), NewFeatures.end());<br>
>>> > Cov.insert(NewCov.begin(), NewCov.end());<br>
>>> > + for (auto Idx : NewCov)<br>
>>> > + if (auto *TE = TPC.PCTableEntryByIdx(Idx))<br>
>>> > + if (TPC.PcIsFuncEntry(TE))<br>
>>> > + PrintPC(" NEW_FUNC: %p %F %L\n", "",<br>
>>> > + TPC.GetNextInstructionPc(TE->PC));<br>
>>> > +<br>
>>> > auto Stats = ParseFinalStatsFromLog(Job->LogPath);<br>
>>> > NumRuns += Stats.number_of_executed_units;<br>
>>> > if (!FilesToAdd.empty())<br>
>>> ><br>
>>> > Modified: compiler-rt/trunk/lib/fuzzer/FuzzerMerge.cpp<br>
>>> > URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerMerge.cpp?rev=354092&r1=354091&r2=354092&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerMerge.cpp?rev=354092&r1=354091&r2=354092&view=diff</a><br>
>>> > ==============================================================================<br>
>>> > --- compiler-rt/trunk/lib/fuzzer/FuzzerMerge.cpp (original)<br>
>>> > +++ compiler-rt/trunk/lib/fuzzer/FuzzerMerge.cpp Thu Feb 14 17:22:00 2019<br>
>>> > @@ -100,7 +100,7 @@ bool Merger::Parse(std::istream &IS, boo<br>
>>> > LastSeenStartMarker = kInvalidStartMarker;<br>
>>> > if (ParseCoverage) {<br>
>>> > TmpFeatures.clear(); // use a vector from outer scope to avoid resizes.<br>
>>> > - while (ISS1 >> std::hex >> N)<br>
>>> > + while (ISS1 >> N)<br>
>>> > TmpFeatures.push_back(N);<br>
>>> > std::sort(TmpFeatures.begin(), TmpFeatures.end());<br>
>>> > Files[CurrentFileIdx].Features = TmpFeatures;<br>
>>> > @@ -108,7 +108,7 @@ bool Merger::Parse(std::istream &IS, boo<br>
>>> > } else if (Marker == "COV") {<br>
>>> > size_t CurrentFileIdx = N;<br>
>>> > if (ParseCoverage)<br>
>>> > - while (ISS1 >> std::hex >> N)<br>
>>> > + while (ISS1 >> N)<br>
>>> > if (PCs.insert(N).second)<br>
>>> > Files[CurrentFileIdx].Cov.push_back(N);<br>
>>> > } else {<br>
>>> > @@ -220,7 +220,7 @@ void Fuzzer::CrashResistantMergeInternal<br>
>>> > }<br>
>>> > std::ostringstream StartedLine;<br>
>>> > // Write the pre-run marker.<br>
>>> > - OF << "STARTED " << std::dec << i << " " << U.size() << "\n";<br>
>>> > + OF << "STARTED " << i << " " << U.size() << "\n";<br>
>>> > OF.flush(); // Flush is important since Command::Execute may crash.<br>
>>> > // Run.<br>
>>> > TPC.ResetMaps();<br>
>>> > @@ -242,9 +242,9 @@ void Fuzzer::CrashResistantMergeInternal<br>
>>> > // Write the post-run marker and the coverage.<br>
>>> > OF << "FT " << i;<br>
>>> > for (size_t F : UniqFeatures)<br>
>>> > - OF << " " << std::hex << F;<br>
>>> > + OF << " " << F;<br>
>>> > OF << "\n";<br>
>>> > - OF << "COV " << std::dec << i;<br>
>>> > + OF << "COV " << i;<br>
>>> > TPC.ForEachObservedPC([&](const TracePC::PCTableEntry *TE) {<br>
>>> > if (AllPCs.insert(TE).second)<br>
>>> > OF << " " << TPC.PCTableEntryIdx(TE);<br>
>>> ><br>
>>> > Modified: compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp<br>
>>> > URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp?rev=354092&r1=354091&r2=354092&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp?rev=354092&r1=354091&r2=354092&view=diff</a><br>
>>> > ==============================================================================<br>
>>> > --- compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp (original)<br>
>>> > +++ compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp Thu Feb 14 17:22:00 2019<br>
>>> > @@ -174,7 +174,7 @@ inline ALWAYS_INLINE uintptr_t GetPrevio<br>
>>> ><br>
>>> > /// \return the address of the next instruction.<br>
>>> > /// Note: the logic is copied from `sanitizer_common/sanitizer_stacktrace.cc`<br>
>>> > -inline ALWAYS_INLINE uintptr_t GetNextInstructionPc(uintptr_t PC) {<br>
>>> > +ALWAYS_INLINE uintptr_t TracePC::GetNextInstructionPc(uintptr_t PC) {<br>
>>> > #if defined(__mips__)<br>
>>> > return PC + 8;<br>
>>> > #elif defined(__powerpc__) || defined(__sparc__) || defined(__arm__) || \<br>
>>> > @@ -196,7 +196,7 @@ void TracePC::UpdateObservedPCs() {<br>
>>> > };<br>
>>> ><br>
>>> > auto Observe = [&](const PCTableEntry *TE) {<br>
>>> > - if (TE->PCFlags & 1)<br>
>>> > + if (PcIsFuncEntry(TE))<br>
>>> > if (++ObservedFuncs[TE->PC] == 1 && NumPrintNewFuncs)<br>
>>> > CoveredFuncs.push_back(TE->PC);<br>
>>> > ObservePC(TE);<br>
>>> > @@ -239,6 +239,16 @@ uintptr_t TracePC::PCTableEntryIdx(const<br>
>>> > return 0;<br>
>>> > }<br>
>>> ><br>
>>> > +const TracePC::PCTableEntry *TracePC::PCTableEntryByIdx(uintptr_t Idx) {<br>
>>> > + for (size_t i = 0; i < NumPCTables; i++) {<br>
>>> > + auto &M = ModulePCTable[i];<br>
>>> > + size_t Size = M.Stop - M.Start;<br>
>>> > + if (Idx < Size) return &M.Start[Idx];<br>
>>> > + Idx -= Size;<br>
>>> > + }<br>
>>> > + return nullptr;<br>
>>> > +}<br>
>>> > +<br>
>>> > static std::string GetModuleName(uintptr_t PC) {<br>
>>> > char ModulePathRaw[4096] = ""; // What's PATH_MAX in portable C++?<br>
>>> > void *OffsetRaw = nullptr;<br>
>>> > @@ -257,10 +267,10 @@ void TracePC::IterateCoveredFunctions(Ca<br>
>>> > auto ModuleName = GetModuleName(M.Start->PC);<br>
>>> > for (auto NextFE = M.Start; NextFE < M.Stop; ) {<br>
>>> > auto FE = NextFE;<br>
>>> > - assert((FE->PCFlags & 1) && "Not a function entry point");<br>
>>> > + assert(PcIsFuncEntry(FE) && "Not a function entry point");<br>
>>> > do {<br>
>>> > NextFE++;<br>
>>> > - } while (NextFE < M.Stop && !(NextFE->PCFlags & 1));<br>
>>> > + } while (NextFE < M.Stop && !(PcIsFuncEntry(NextFE)));<br>
>>> > CB(FE, NextFE, ObservedFuncs[FE->PC]);<br>
>>> > }<br>
>>> > }<br>
>>> > @@ -275,7 +285,7 @@ void TracePC::SetFocusFunction(const std<br>
>>> > auto &PCTE = ModulePCTable[M];<br>
>>> > size_t N = PCTE.Stop - PCTE.Start;<br>
>>> > for (size_t I = 0; I < N; I++) {<br>
>>> > - if (!(PCTE.Start[I].PCFlags & 1)) continue; // not a function entry.<br>
>>> > + if (!(PcIsFuncEntry(&PCTE.Start[I]))) continue; // not a function entry.<br>
>>> > auto Name = DescribePC("%F", GetNextInstructionPc(PCTE.Start[I].PC));<br>
>>> > if (Name[0] == 'i' && Name[1] == 'n' && Name[2] == ' ')<br>
>>> > Name = Name.substr(3, std::string::npos);<br>
>>> ><br>
>>> > Modified: compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h<br>
>>> > URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h?rev=354092&r1=354091&r2=354092&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h?rev=354092&r1=354091&r2=354092&view=diff</a><br>
>>> > ==============================================================================<br>
>>> > --- compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h (original)<br>
>>> > +++ compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h Thu Feb 14 17:22:00 2019<br>
>>> > @@ -127,6 +127,9 @@ class TracePC {<br>
>>> > };<br>
>>> ><br>
>>> > uintptr_t PCTableEntryIdx(const PCTableEntry *TE);<br>
>>> > + const PCTableEntry *PCTableEntryByIdx(uintptr_t Idx);<br>
>>> > + static uintptr_t GetNextInstructionPc(uintptr_t PC);<br>
>>> > + bool PcIsFuncEntry(const PCTableEntry *TE) { return TE->PCFlags & 1; }<br>
>>> ><br>
>>> > private:<br>
>>> > bool UseCounters = false;<br>
>>> ><br>
>>> ><br>
>>> > _______________________________________________<br>
>>> > llvm-commits mailing list<br>
>>> > <a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a><br>
>>> > <a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br>
><br>
> _______________________________________________<br>
> llvm-commits mailing list<br>
> <a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a><br>
> <a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits</a><br>
</blockquote></div></div>
</blockquote></div></div>