[lld] r288373 - Parallelize ICF to make LLD's ICF really fast.
Rui Ueyama via llvm-commits
llvm-commits at lists.llvm.org
Wed Dec 7 10:32:25 PST 2016
Thank you for testing this. This algorithm is lock-free (and even
wait-free), so it should scale well.
On Wed, Dec 7, 2016 at 10:22 AM, Rafael Avila de Espindola <
rafael.espindola at gmail.com> wrote:
>
> Impressive.
>
> In sigle thread mode the link of chromium goes from 4.140444332 to
> 4.244436379 seconds.
>
> With multiple threads and 2 cores available, the difference is already
> from 4.161336277 to 3.340861790.
>
> With 4 cores it goes from 4.092804448 to 2.921354616 and with 8 from
> 4.047472207 to 2.754971966.
>
> Cheers,
> Rafael
>
>
> Rui Ueyama via llvm-commits <llvm-commits at lists.llvm.org> writes:
>
> > Author: ruiu
> > Date: Thu Dec 1 11:09:04 2016
> > New Revision: 288373
> >
> > URL: http://llvm.org/viewvc/llvm-project?rev=288373&view=rev
> > Log:
> > Parallelize ICF to make LLD's ICF really fast.
> >
> > ICF is short for Identical Code Folding. It is a size optimization to
> > identify two or more functions that happened to have the same contents
> > to merges them. It usually reduces output size by a few percent.
> >
> > ICF is slow because it is computationally intensive process. I tried
> > to paralellize it before but failed because I couldn't make a
> > parallelized version produce consistent outputs. Although it didn't
> > create broken executables, every invocation of the linker generated
> > slightly different output, and I couldn't figure out why.
> >
> > I think I now understand what was going on, and also came up with a
> > simple algorithm to fix it. So is this patch.
> >
> > The result is very exciting. Chromium for example has 780,662 input
> > sections in which 20,774 are reducible by ICF. LLD previously took
> > 7.980 seconds for ICF. Now it finishes in 1.065 seconds.
> >
> > As a result, LLD can now link a Chromium binary (output size 1.59 GB)
> > in 10.28 seconds on my machine with ICF enabled. Compared to gold
> > which takes 40.94 seconds to do the same thing, this is an amazing
> > number.
> >
> > From here, I'll describe what we are doing for ICF, what was the
> > previous problem, and what I did in this patch.
> >
> > In ICF, two sections are considered identical if they have the same
> > section flags, section data, and relocations. Relocations are tricky,
> > becuase two relocations are considered the same if they have the same
> > relocation type, values, and if they point to the same section _in
> > terms of ICF_.
> >
> > Here is an example. If foo and bar defined below are compiled to the
> > same machine instructions, ICF can (and should) merge the two,
> > although their relocations point to each other.
> >
> > void foo() { bar(); }
> > void bar() { foo(); }
> >
> > This is not an easy problem to solve.
> >
> > What we are doing in LLD is some sort of coloring algorithm. We color
> > non-identical sections using different colors repeatedly, and sections
> > in the same color when the algorithm terminates are considered
> > identical. Here is the details:
> >
> > 1. First, we color all sections using their hash values of section
> > types, section contents, and numbers of relocations. At this moment,
> > relocation targets are not taken into account. We just color
> > sections that apparently differ in different colors.
> >
> > 2. Next, for each color C, we visit sections having color C to see
> > if their relocations are the same. Relocations are considered equal
> > if their targets have the same color. We then recolor sections that
> > have different relocation targets in new colors.
> >
> > 3. If we recolor some section in step 2, relocations that were
> > previously pointing to the same color targets may now be pointing to
> > different colors. Therefore, repeat 2 until a convergence is
> > obtained.
> >
> > Step 2 is a heavy operation. For Chromium, the first iteration of step
> > 2 takes 2.882 seconds, and the second iteration takes 1.038 seconds,
> > and in total it needs 23 iterations.
> >
> > Parallelizing step 1 is easy because we can color each section
> > independently. This patch does that.
> >
> > Parallelizing step 2 is tricky. We could work on each color
> > independently, but we cannot recolor sections in place, because it
> > will break the invariance that two possibly-identical sections must
> > have the same color at any moment.
> >
> > Consider sections S1, S2, S3, S4 in the same color C, where S1 and S2
> > are identical, S3 and S4 are identical, but S2 and S3 are not. Thread
> > A is about to recolor S1 and S2 in C'. After thread A recolor S1 in
> > C', but before recolor S2 in C', other thread B might observe S1 and
> > S2. Then thread B will conclude that S1 and S2 are different, and it
> > will split thread B's sections into smaller groups wrongly. Over-
> > splitting doesn't produce broken results, but it loses a chance to
> > merge some identical sections. That was the cause of indeterminism.
> >
> > To fix the problem, I made sections have two colors, namely current
> > color and next color. At the beginning of each iteration, both colors
> > are the same. Each thread reads from current color and writes to next
> > color. In this way, we can avoid threads from reading partial
> > results. After each iteration, we flip current and next.
> >
> > This is a very simple solution and is implemented in less than 50
> > lines of code.
> >
> > I tested this patch with Chromium and confirmed that this parallelized
> > ICF produces the identical output as the non-parallelized one.
> >
> > Differential Revision: https://reviews.llvm.org/D27247
> >
> > Modified:
> > lld/trunk/ELF/ICF.cpp
> > lld/trunk/ELF/InputSection.h
> >
> > Modified: lld/trunk/ELF/ICF.cpp
> > URL: http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/ICF.cpp?
> rev=288373&r1=288372&r2=288373&view=diff
> > ============================================================
> ==================
> > --- lld/trunk/ELF/ICF.cpp (original)
> > +++ lld/trunk/ELF/ICF.cpp Thu Dec 1 11:09:04 2016
> > @@ -59,10 +59,12 @@
> > #include "Config.h"
> > #include "SymbolTable.h"
> >
> > +#include "lld/Core/Parallel.h"
> > #include "llvm/ADT/Hashing.h"
> > #include "llvm/Object/ELF.h"
> > #include "llvm/Support/ELF.h"
> > #include <algorithm>
> > +#include <mutex>
> >
> > using namespace lld;
> > using namespace lld::elf;
> > @@ -95,16 +97,16 @@ private:
> >
> > std::vector<InputSection<ELFT> *> Sections;
> > std::vector<Range> Ranges;
> > + std::mutex Mu;
> >
> > - // The main loop is repeated until we get a convergence.
> > - bool Repeat = false; // If Repeat is true, we need to repeat.
> > - int Cnt = 0; // Counter for the main loop.
> > + uint32_t NextId = 1;
> > + int Cnt = 0;
> > };
> > }
> >
> > // Returns a hash value for S. Note that the information about
> > // relocation targets is not included in the hash value.
> > -template <class ELFT> static uint64_t getHash(InputSection<ELFT> *S) {
> > +template <class ELFT> static uint32_t getHash(InputSection<ELFT> *S) {
> > return hash_combine(S->Flags, S->getSize(), S->NumRelocations);
> > }
> >
> > @@ -128,33 +130,54 @@ template <class ELFT> void ICF<ELFT>::se
> > // issue in practice because the number of the distinct sections in
> > // [R.Begin, R.End] is usually very small.
> > while (R->End - R->Begin > 1) {
> > + size_t Begin = R->Begin;
> > + size_t End = R->End;
> > +
> > // Divide range R into two. Let Mid be the start index of the
> > // second group.
> > auto Bound = std::stable_partition(
> > - Sections.begin() + R->Begin + 1, Sections.begin() + R->End,
> > + Sections.begin() + Begin + 1, Sections.begin() + End,
> > [&](InputSection<ELFT> *S) {
> > if (Constant)
> > - return equalsConstant(Sections[R->Begin], S);
> > - return equalsVariable(Sections[R->Begin], S);
> > + return equalsConstant(Sections[Begin], S);
> > + return equalsVariable(Sections[Begin], S);
> > });
> > size_t Mid = Bound - Sections.begin();
> >
> > - if (Mid == R->End)
> > + if (Mid == End)
> > return;
> >
> > - // Now we split [R.Begin, R.End) into [R.Begin, Mid) and [Mid,
> R.End).
> > - if (Mid - R->Begin > 1)
> > - Ranges.push_back({R->Begin, Mid});
> > - R->Begin = Mid;
> > -
> > - // Update GroupIds for the new group members. We use the index of
> > - // the group first member as a group ID because that is unique.
> > - for (size_t I = Mid; I < R->End; ++I)
> > - Sections[I]->GroupId = Mid;
> > -
> > - // Since we have split a group, we need to repeat the main loop
> > - // later to obtain a convergence. Remember that.
> > - Repeat = true;
> > + // Now we split [Begin, End) into [Begin, Mid) and [Mid, End).
> > + uint32_t Id;
> > + Range *NewRange;
> > + {
> > + std::lock_guard<std::mutex> Lock(Mu);
> > + Ranges.push_back({Mid, End});
> > + NewRange = &Ranges.back();
> > + Id = NextId++;
> > + }
> > + R->End = Mid;
> > +
> > + // Update GroupIds for the new group members.
> > + //
> > + // Note on GroupId[0] and GroupId[1]: we have two storages for
> > + // group IDs. At the beginning of each iteration of the main loop,
> > + // both have the same ID. GroupId[0] contains the current ID, and
> > + // GroupId[1] contains the next ID which will be used in the next
> > + // iteration.
> > + //
> > + // Recall that other threads may be working on other ranges. They
> > + // may be reading group IDs that we are about to update. We cannot
> > + // update group IDs in place because it breaks the invariance that
> > + // all sections in the same group must have the same ID. In other
> > + // words, the following for loop is not an atomic operation, and
> > + // that is observable from other threads.
> > + //
> > + // By writing new IDs to write-only places, we can keep the
> invariance.
> > + for (size_t I = Mid; I < End; ++I)
> > + Sections[I]->GroupId[(Cnt + 1) % 2] = Id;
> > +
> > + R = NewRange;
> > }
> > }
> >
> > @@ -211,7 +234,16 @@ bool ICF<ELFT>::variableEq(const InputSe
> > auto *Y = dyn_cast<InputSection<ELFT>>(DB->Section);
> > if (!X || !Y)
> > return false;
> > - return X->GroupId != 0 && X->GroupId == Y->GroupId;
> > + if (X->GroupId[Cnt % 2] == 0)
> > + return false;
> > +
> > + // Performance hack for single-thread. If no other threads are
> > + // running, we can safely read next GroupIDs as there is no race
> > + // condition. This optimization may reduce the number of
> > + // iterations of the main loop because we can see results of the
> > + // same iteration.
> > + size_t Idx = (Config->Threads ? Cnt : Cnt + 1) % 2;
> > + return X->GroupId[Idx] == Y->GroupId[Idx];
> > };
> >
> > return std::equal(RelsA.begin(), RelsA.end(), RelsB.begin(), Eq);
> > @@ -226,6 +258,14 @@ bool ICF<ELFT>::equalsVariable(const Inp
> > return variableEq(A, A->rels(), B, B->rels());
> > }
> >
> > +template <class IterTy, class FuncTy>
> > +static void foreach(IterTy Begin, IterTy End, FuncTy Fn) {
> > + if (Config->Threads)
> > + parallel_for_each(Begin, End, Fn);
> > + else
> > + std::for_each(Begin, End, Fn);
> > +}
> > +
> > // The main function of ICF.
> > template <class ELFT> void ICF<ELFT>::run() {
> > // Collect sections to merge.
> > @@ -239,14 +279,14 @@ template <class ELFT> void ICF<ELFT>::ru
> > // guaranteed) to have the same static contents in terms of ICF.
> > for (InputSection<ELFT> *S : Sections)
> > // Set MSB to 1 to avoid collisions with non-hash IDs.
> > - S->GroupId = getHash(S) | (uint64_t(1) << 63);
> > + S->GroupId[0] = S->GroupId[1] = getHash(S) | (1 << 31);
> >
> > // From now on, sections in Sections are ordered so that sections in
> > // the same group are consecutive in the vector.
> > std::stable_sort(Sections.begin(), Sections.end(),
> > [](InputSection<ELFT> *A, InputSection<ELFT> *B) {
> > - if (A->GroupId != B->GroupId)
> > - return A->GroupId < B->GroupId;
> > + if (A->GroupId[0] != B->GroupId[0])
> > + return A->GroupId[0] < B->GroupId[0];
> > // Within a group, put the highest alignment
> > // requirement first, so that's the one we'll keep.
> > return B->Alignment < A->Alignment;
> > @@ -260,25 +300,37 @@ template <class ELFT> void ICF<ELFT>::ru
> > for (size_t I = 0, E = Sections.size(); I < E - 1;) {
> > // Let J be the first index whose element has a different ID.
> > size_t J = I + 1;
> > - while (J < E && Sections[I]->GroupId == Sections[J]->GroupId)
> > + while (J < E && Sections[I]->GroupId[0] == Sections[J]->GroupId[0])
> > ++J;
> > if (J - I > 1)
> > Ranges.push_back({I, J});
> > I = J;
> > }
> >
> > + // This function copies new GroupIds from former write-only space to
> > + // former read-only space, so that we can flip GroupId[0] and
> GroupId[1].
> > + // Note that new GroupIds are always be added to end of Ranges.
> > + auto Copy = [&](Range &R) {
> > + for (size_t I = R.Begin; I < R.End; ++I)
> > + Sections[I]->GroupId[Cnt % 2] = Sections[I]->GroupId[(Cnt + 1) %
> 2];
> > + };
> > +
> > // Compare static contents and assign unique IDs for each static
> content.
> > - std::for_each(Ranges.begin(), Ranges.end(),
> > - [&](Range &R) { segregate(&R, true); });
> > + auto End = Ranges.end();
> > + foreach(Ranges.begin(), End, [&](Range &R) { segregate(&R, true); });
> > + foreach(End, Ranges.end(), Copy);
> > ++Cnt;
> >
> > // Split groups by comparing relocations until convergence is
> obtained.
> > - do {
> > - Repeat = false;
> > - std::for_each(Ranges.begin(), Ranges.end(),
> > - [&](Range &R) { segregate(&R, false); });
> > + for (;;) {
> > + auto End = Ranges.end();
> > + foreach(Ranges.begin(), End, [&](Range &R) { segregate(&R, false);
> });
> > + foreach(End, Ranges.end(), Copy);
> > ++Cnt;
> > - } while (Repeat);
> > +
> > + if (End == Ranges.end())
> > + break;
> > + }
> >
> > log("ICF needed " + Twine(Cnt) + " iterations");
> >
> >
> > Modified: lld/trunk/ELF/InputSection.h
> > URL: http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/
> InputSection.h?rev=288373&r1=288372&r2=288373&view=diff
> > ============================================================
> ==================
> > --- lld/trunk/ELF/InputSection.h (original)
> > +++ lld/trunk/ELF/InputSection.h Thu Dec 1 11:09:04 2016
> > @@ -289,7 +289,7 @@ public:
> > void relocateNonAlloc(uint8_t *Buf, llvm::ArrayRef<RelTy> Rels);
> >
> > // Used by ICF.
> > - uint64_t GroupId = 0;
> > + uint32_t GroupId[2] = {0, 0};
> >
> > // Called by ICF to merge two input sections.
> > void replace(InputSection<ELFT> *Other);
> >
> >
> > _______________________________________________
> > llvm-commits mailing list
> > llvm-commits at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20161207/da8ab693/attachment.html>
More information about the llvm-commits
mailing list