[PATCH] D21218: [LibFuzzer] Avoid using std::random_swap() due to platform differences and implement our own version.

[Kostya Serebryany via llvm-commits]

What problem are you trying to fix and do you really need to fix it?

-- kcc (sent from phone)

On Jun 10, 2016 1:22 AM, "Dan Liew" <dan at su-root.co.uk> wrote:

> delcypher created this revision.
> delcypher added reviewers: kcc, aizatsky.
> delcypher added subscribers: kcc, aizatsky, zaks.anna, kubabrecka,
> dcoughlin, llvm-commits.
>
> [LibFuzzer] Avoid using std::random_swap() due to platform differences
> and implement our own version.
>
> It turns out that the behavior of std::random_swap is different between
> libstdc++ and libcxx (even with the same random number source).
> Therefore if we want consistent behavior between platforms we have to
> use our own implementation.
>
> This change (plus a change to the number of iterations of the Mutator in
> the test required for the particular shuffle implementation used) fixes
> the ``FuzzerMutate.ShuffleBytes2`` unit test on OSX.
>
> I have verified that for the above unittest identical mutations are
> now generated on both Linux and on OSX.
>
>
> http://reviews.llvm.org/D21218
>
> Files:
>   lib/Fuzzer/FuzzerInternal.h
>   lib/Fuzzer/FuzzerLoop.cpp
>   lib/Fuzzer/FuzzerMutate.cpp
>   lib/Fuzzer/test/FuzzerUnittest.cpp
>
> Index: lib/Fuzzer/test/FuzzerUnittest.cpp
> ===================================================================
> --- lib/Fuzzer/test/FuzzerUnittest.cpp
> +++ lib/Fuzzer/test/FuzzerUnittest.cpp
> @@ -259,7 +259,7 @@
>    TestShuffleBytes(&MutationDispatcher::Mutate_ShuffleBytes, 1 << 16);
>  }
>  TEST(FuzzerMutate, ShuffleBytes2) {
> -  TestShuffleBytes(&MutationDispatcher::Mutate, 1 << 19);
> +  TestShuffleBytes(&MutationDispatcher::Mutate, 566171);
>  }
>
>  void TestAddWordFromDictionary(Mutator M, int NumIter) {
> Index: lib/Fuzzer/FuzzerMutate.cpp
> ===================================================================
> --- lib/Fuzzer/FuzzerMutate.cpp
> +++ lib/Fuzzer/FuzzerMutate.cpp
> @@ -96,8 +96,7 @@
>        Rand(std::min(Size, (size_t)8)) + 1; // [1,8] and <= Size.
>    size_t ShuffleStart = Rand(Size - ShuffleAmount);
>    assert(ShuffleStart + ShuffleAmount <= Size);
> -  std::random_shuffle(Data + ShuffleStart, Data + ShuffleStart +
> ShuffleAmount,
> -                      Rand);
> +  Rand.Shuffle(Data + ShuffleStart, Data + ShuffleStart + ShuffleAmount);
>    return Size;
>  }
>
> Index: lib/Fuzzer/FuzzerLoop.cpp
> ===================================================================
> --- lib/Fuzzer/FuzzerLoop.cpp
> +++ lib/Fuzzer/FuzzerLoop.cpp
> @@ -321,7 +321,7 @@
>  }
>
>  void Fuzzer::ShuffleCorpus(UnitVector *V) {
> -  std::random_shuffle(V->begin(), V->end(), MD.GetRand());
> +  MD.GetRand().Shuffle(V->begin(), V->end());
>    if (Options.PreferSmall)
>      std::stable_sort(V->begin(), V->end(), [](const Unit &A, const Unit
> &B) {
>        return A.size() < B.size();
> Index: lib/Fuzzer/FuzzerInternal.h
> ===================================================================
> --- lib/Fuzzer/FuzzerInternal.h
> +++ lib/Fuzzer/FuzzerInternal.h
> @@ -137,6 +137,22 @@
>    size_t RandBool() { return Rand() % 2; }
>    size_t operator()(size_t n) { return n ? Rand() % n : 0; }
>    std::mt19937 &Get_mt19937() { return R; }
> +  // Shuffle data in the range [First, End)
> +  //
> +  // We do not use std::random_shuffle() here because its
> +  // behavior is not consistent across different platforms.
> +  //
> +  // The algorithm used here will pick a permutation at
> +  // random where every permutation has equal probability
> +  // (provided the random source is uniformly distributed).
> +  template<typename RndAccessIt>
> +  void Shuffle(RndAccessIt First, RndAccessIt End) {
> +    typename std::iterator_traits<RndAccessIt>::difference_type Offset, N;
> +    N = End - First;
> +    for (Offset = 0; Offset < N; ++Offset) {
> +      std::swap(First[Offset], First[Offset + (*this)(N - Offset)]);
> +    }
> +  }
>   private:
>    std::mt19937 R;
>  };
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20160611/6fdab31a/attachment.html>