[compiler-rt] r354092 - [libFuzzer] print new functions as they are discovered in the fork mode
Peter Smith via llvm-commits
llvm-commits at lists.llvm.org
Tue Feb 19 05:44:05 PST 2019
Hello Kostya, Diana,
I've taken over from Diana at looking after the AArch64 bots this
week. I've run the cxa_atexit_race.cc test in a loop and it
intermittently hangs on average about 1 in 75 times, a frequency that
seems to go up a lot when the machine is heavily loaded. I've tested
before and after locally reverting r354092 I was able to reproduce a
hang without r354092 so in conjunction with hanging test being in
msan, I agree that it isn't this particular change that is the root
cause.
Given that the cxa_atexit_race.cc is relatively new (14th Februrary) I
think it may just be that the fix for the atexit problem doesn't work
on AArch64. I'll follow up with the author of the test to see if we
can work something out.
I don't know who maintains lsan on AArch64 either. I can ask our local
expert on Thursday to see if he knows. Having said that while 18
seconds is slow it is unlikely to cause the builder to run out of
time.
Peter
On Sat, 16 Feb 2019 at 02:06, Kostya Serebryany via llvm-commits
<llvm-commits at lists.llvm.org> wrote:
>
> My change just triggered an older problem: lsan seems to be badly broken on aarch64.
> It spends ridiculous amount of time, and a test that is expected to pass in < 1 second takes 18 seconds.
>
> % clang -fsanitize=leak ~/llvm/compiler-rt/test/lsan/TestCases/sanity_check_pure_c.c && time ./a.out
>
> real 0m18.313s
> user 0m18.304s
> sys 0m0.008s
>
> Perhaps this is https://github.com/google/sanitizers/issues/703, or something else.
>
> My team doesn't maintain lsan on aarch64.. :(
> I wonder who does?
>
>
>
>
> On Fri, Feb 15, 2019 at 10:50 AM Kostya Serebryany <kcc at google.com> wrote:
>>
>> Hi Diana,
>>
>> I am not sure how to interpret these bot failures...
>> Can you tell which test times out?
>>
>> The only tests that can be affected by this change pass:
>>
>> PASS: libFuzzer :: merge.test (48757 of 48763)
>>
>> PASS: libFuzzer :: fork.test (48754 of 48763)
>>
>> (let me revive my aarch64 development box and see it myself)
>>
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Feb 15, 2019 at 4:22 AM Diana Picus <diana.picus at linaro.org> wrote:
>>>
>>> Hi Kostya,
>>>
>>> It seems that this is causing the tests to hang on AArch64:
>>> http://lab.llvm.org:8011/builders/clang-cmake-aarch64-full/builds/6571
>>> http://lab.llvm.org:8011/builders/clang-cmake-aarch64-lld/builds/6038
>>>
>>> It might have something to do with cxa_atexit_race, since I saw it
>>> running for about 8 minutes on one of the machines.
>>>
>>> Could you please have a look?
>>>
>>> Thanks,
>>> Diana
>>>
>>> On Fri, 15 Feb 2019 at 02:21, Kostya Serebryany via llvm-commits
>>> <llvm-commits at lists.llvm.org> wrote:
>>> >
>>> > Author: kcc
>>> > Date: Thu Feb 14 17:22:00 2019
>>> > New Revision: 354092
>>> >
>>> > URL: http://llvm.org/viewvc/llvm-project?rev=354092&view=rev
>>> > Log:
>>> > [libFuzzer] print new functions as they are discovered in the fork mode
>>> >
>>> > Modified:
>>> > compiler-rt/trunk/lib/fuzzer/FuzzerFork.cpp
>>> > compiler-rt/trunk/lib/fuzzer/FuzzerMerge.cpp
>>> > compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp
>>> > compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h
>>> >
>>> > Modified: compiler-rt/trunk/lib/fuzzer/FuzzerFork.cpp
>>> > URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerFork.cpp?rev=354092&r1=354091&r2=354092&view=diff
>>> > ==============================================================================
>>> > --- compiler-rt/trunk/lib/fuzzer/FuzzerFork.cpp (original)
>>> > +++ compiler-rt/trunk/lib/fuzzer/FuzzerFork.cpp Thu Feb 14 17:22:00 2019
>>> > @@ -13,6 +13,7 @@
>>> > #include "FuzzerIO.h"
>>> > #include "FuzzerMerge.h"
>>> > #include "FuzzerSHA1.h"
>>> > +#include "FuzzerTracePC.h"
>>> > #include "FuzzerUtil.h"
>>> >
>>> > #include <atomic>
>>> > @@ -86,11 +87,13 @@ struct GlobalEnv {
>>> > Cmd.removeArgument(C);
>>> > Cmd.addFlag("reload", "0"); // working in an isolated dir, no reload.
>>> > Cmd.addFlag("print_final_stats", "1");
>>> > + Cmd.addFlag("print_funcs", "0"); // no need to spend time symbolizing.
>>> > Cmd.addFlag("max_total_time", std::to_string(std::min((size_t)300, JobId)));
>>> >
>>> > auto Job = new FuzzJob;
>>> > std::string Seeds;
>>> > - if (size_t CorpusSubsetSize = std::min(Files.size(), (size_t)100))
>>> > + if (size_t CorpusSubsetSize =
>>> > + std::min(Files.size(), (size_t)sqrt(Files.size() + 2)))
>>> > for (size_t i = 0; i < CorpusSubsetSize; i++)
>>> > Seeds += (Seeds.empty() ? "" : ",") +
>>> > Files[Rand->SkewTowardsLast(Files.size())];
>>> > @@ -135,6 +138,12 @@ struct GlobalEnv {
>>> > RmDirRecursive(Job->CorpusDir);
>>> > Features.insert(NewFeatures.begin(), NewFeatures.end());
>>> > Cov.insert(NewCov.begin(), NewCov.end());
>>> > + for (auto Idx : NewCov)
>>> > + if (auto *TE = TPC.PCTableEntryByIdx(Idx))
>>> > + if (TPC.PcIsFuncEntry(TE))
>>> > + PrintPC(" NEW_FUNC: %p %F %L\n", "",
>>> > + TPC.GetNextInstructionPc(TE->PC));
>>> > +
>>> > auto Stats = ParseFinalStatsFromLog(Job->LogPath);
>>> > NumRuns += Stats.number_of_executed_units;
>>> > if (!FilesToAdd.empty())
>>> >
>>> > Modified: compiler-rt/trunk/lib/fuzzer/FuzzerMerge.cpp
>>> > URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerMerge.cpp?rev=354092&r1=354091&r2=354092&view=diff
>>> > ==============================================================================
>>> > --- compiler-rt/trunk/lib/fuzzer/FuzzerMerge.cpp (original)
>>> > +++ compiler-rt/trunk/lib/fuzzer/FuzzerMerge.cpp Thu Feb 14 17:22:00 2019
>>> > @@ -100,7 +100,7 @@ bool Merger::Parse(std::istream &IS, boo
>>> > LastSeenStartMarker = kInvalidStartMarker;
>>> > if (ParseCoverage) {
>>> > TmpFeatures.clear(); // use a vector from outer scope to avoid resizes.
>>> > - while (ISS1 >> std::hex >> N)
>>> > + while (ISS1 >> N)
>>> > TmpFeatures.push_back(N);
>>> > std::sort(TmpFeatures.begin(), TmpFeatures.end());
>>> > Files[CurrentFileIdx].Features = TmpFeatures;
>>> > @@ -108,7 +108,7 @@ bool Merger::Parse(std::istream &IS, boo
>>> > } else if (Marker == "COV") {
>>> > size_t CurrentFileIdx = N;
>>> > if (ParseCoverage)
>>> > - while (ISS1 >> std::hex >> N)
>>> > + while (ISS1 >> N)
>>> > if (PCs.insert(N).second)
>>> > Files[CurrentFileIdx].Cov.push_back(N);
>>> > } else {
>>> > @@ -220,7 +220,7 @@ void Fuzzer::CrashResistantMergeInternal
>>> > }
>>> > std::ostringstream StartedLine;
>>> > // Write the pre-run marker.
>>> > - OF << "STARTED " << std::dec << i << " " << U.size() << "\n";
>>> > + OF << "STARTED " << i << " " << U.size() << "\n";
>>> > OF.flush(); // Flush is important since Command::Execute may crash.
>>> > // Run.
>>> > TPC.ResetMaps();
>>> > @@ -242,9 +242,9 @@ void Fuzzer::CrashResistantMergeInternal
>>> > // Write the post-run marker and the coverage.
>>> > OF << "FT " << i;
>>> > for (size_t F : UniqFeatures)
>>> > - OF << " " << std::hex << F;
>>> > + OF << " " << F;
>>> > OF << "\n";
>>> > - OF << "COV " << std::dec << i;
>>> > + OF << "COV " << i;
>>> > TPC.ForEachObservedPC([&](const TracePC::PCTableEntry *TE) {
>>> > if (AllPCs.insert(TE).second)
>>> > OF << " " << TPC.PCTableEntryIdx(TE);
>>> >
>>> > Modified: compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp
>>> > URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp?rev=354092&r1=354091&r2=354092&view=diff
>>> > ==============================================================================
>>> > --- compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp (original)
>>> > +++ compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.cpp Thu Feb 14 17:22:00 2019
>>> > @@ -174,7 +174,7 @@ inline ALWAYS_INLINE uintptr_t GetPrevio
>>> >
>>> > /// \return the address of the next instruction.
>>> > /// Note: the logic is copied from `sanitizer_common/sanitizer_stacktrace.cc`
>>> > -inline ALWAYS_INLINE uintptr_t GetNextInstructionPc(uintptr_t PC) {
>>> > +ALWAYS_INLINE uintptr_t TracePC::GetNextInstructionPc(uintptr_t PC) {
>>> > #if defined(__mips__)
>>> > return PC + 8;
>>> > #elif defined(__powerpc__) || defined(__sparc__) || defined(__arm__) || \
>>> > @@ -196,7 +196,7 @@ void TracePC::UpdateObservedPCs() {
>>> > };
>>> >
>>> > auto Observe = [&](const PCTableEntry *TE) {
>>> > - if (TE->PCFlags & 1)
>>> > + if (PcIsFuncEntry(TE))
>>> > if (++ObservedFuncs[TE->PC] == 1 && NumPrintNewFuncs)
>>> > CoveredFuncs.push_back(TE->PC);
>>> > ObservePC(TE);
>>> > @@ -239,6 +239,16 @@ uintptr_t TracePC::PCTableEntryIdx(const
>>> > return 0;
>>> > }
>>> >
>>> > +const TracePC::PCTableEntry *TracePC::PCTableEntryByIdx(uintptr_t Idx) {
>>> > + for (size_t i = 0; i < NumPCTables; i++) {
>>> > + auto &M = ModulePCTable[i];
>>> > + size_t Size = M.Stop - M.Start;
>>> > + if (Idx < Size) return &M.Start[Idx];
>>> > + Idx -= Size;
>>> > + }
>>> > + return nullptr;
>>> > +}
>>> > +
>>> > static std::string GetModuleName(uintptr_t PC) {
>>> > char ModulePathRaw[4096] = ""; // What's PATH_MAX in portable C++?
>>> > void *OffsetRaw = nullptr;
>>> > @@ -257,10 +267,10 @@ void TracePC::IterateCoveredFunctions(Ca
>>> > auto ModuleName = GetModuleName(M.Start->PC);
>>> > for (auto NextFE = M.Start; NextFE < M.Stop; ) {
>>> > auto FE = NextFE;
>>> > - assert((FE->PCFlags & 1) && "Not a function entry point");
>>> > + assert(PcIsFuncEntry(FE) && "Not a function entry point");
>>> > do {
>>> > NextFE++;
>>> > - } while (NextFE < M.Stop && !(NextFE->PCFlags & 1));
>>> > + } while (NextFE < M.Stop && !(PcIsFuncEntry(NextFE)));
>>> > CB(FE, NextFE, ObservedFuncs[FE->PC]);
>>> > }
>>> > }
>>> > @@ -275,7 +285,7 @@ void TracePC::SetFocusFunction(const std
>>> > auto &PCTE = ModulePCTable[M];
>>> > size_t N = PCTE.Stop - PCTE.Start;
>>> > for (size_t I = 0; I < N; I++) {
>>> > - if (!(PCTE.Start[I].PCFlags & 1)) continue; // not a function entry.
>>> > + if (!(PcIsFuncEntry(&PCTE.Start[I]))) continue; // not a function entry.
>>> > auto Name = DescribePC("%F", GetNextInstructionPc(PCTE.Start[I].PC));
>>> > if (Name[0] == 'i' && Name[1] == 'n' && Name[2] == ' ')
>>> > Name = Name.substr(3, std::string::npos);
>>> >
>>> > Modified: compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h
>>> > URL: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h?rev=354092&r1=354091&r2=354092&view=diff
>>> > ==============================================================================
>>> > --- compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h (original)
>>> > +++ compiler-rt/trunk/lib/fuzzer/FuzzerTracePC.h Thu Feb 14 17:22:00 2019
>>> > @@ -127,6 +127,9 @@ class TracePC {
>>> > };
>>> >
>>> > uintptr_t PCTableEntryIdx(const PCTableEntry *TE);
>>> > + const PCTableEntry *PCTableEntryByIdx(uintptr_t Idx);
>>> > + static uintptr_t GetNextInstructionPc(uintptr_t PC);
>>> > + bool PcIsFuncEntry(const PCTableEntry *TE) { return TE->PCFlags & 1; }
>>> >
>>> > private:
>>> > bool UseCounters = false;
>>> >
>>> >
>>> > _______________________________________________
>>> > llvm-commits mailing list
>>> > llvm-commits at lists.llvm.org
>>> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
More information about the llvm-commits
mailing list