[lld] r336790 - Parallelize GdbIndexSection's symbol table creation.
David Blaikie via llvm-commits
llvm-commits at lists.llvm.org
Mon Jul 16 12:05:41 PDT 2018
Do I understand the commit message correctly that this regresses single
threaded performance? Is single threaded the default lld behavior?
On Wed, Jul 11, 2018 at 4:42 AM Rui Ueyama via llvm-commits <
llvm-commits at lists.llvm.org> wrote:
> Author: ruiu
> Date: Wed Jul 11 04:37:10 2018
> New Revision: 336790
>
> URL: http://llvm.org/viewvc/llvm-project?rev=336790&view=rev
> Log:
> Parallelize GdbIndexSection's symbol table creation.
>
> Since .gdb_index sections contain all known symbols, they can be very
> large.
> One of my executables has a .gdb_index section of 1350 GiB. Uniquifying
> symbols by name takes 3.77 seconds on my machine. This patch parallelize
> it.
>
> Time to call createSymbols() with 8.4 million unique symbols:
>
> Without this patch: 3773 ms
> Parallelism = 1: 4374 ms
> Parallelism = 2: 2628 ms
> Parallelism = 16: 837 ms
>
> As you can see above, this algorithm is a bit more inefficient
> than the non-parallelized version, but even with dual-core, it is
> faster than that, so I think it is overall a win.
>
> Differential Revision: https://reviews.llvm.org/D49164
>
> Modified:
> lld/trunk/ELF/SyntheticSections.cpp
> lld/trunk/test/ELF/gdb-index.s
>
> Modified: lld/trunk/ELF/SyntheticSections.cpp
> URL:
> http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/SyntheticSections.cpp?rev=336790&r1=336789&r2=336790&view=diff
>
> ==============================================================================
> --- lld/trunk/ELF/SyntheticSections.cpp (original)
> +++ lld/trunk/ELF/SyntheticSections.cpp Wed Jul 11 04:37:10 2018
> @@ -2366,23 +2366,51 @@ createSymbols(ArrayRef<std::vector<GdbIn
> typedef GdbIndexSection::GdbSymbol GdbSymbol;
> typedef GdbIndexSection::NameTypeEntry NameTypeEntry;
>
> - // A map to uniquify symbols by name.
> - DenseMap<CachedHashStringRef, size_t> Map;
> + // The number of symbols we will handle in this function is of the order
> + // of millions for very large executables, so we use multi-threading to
> + // speed it up.
> + size_t NumShards = 32;
> + size_t Concurrency = 1;
> + if (ThreadsEnabled)
> + Concurrency =
> + std::min<size_t>(PowerOf2Floor(hardware_concurrency()),
> NumShards);
> +
> + // A sharded map to uniquify symbols by name.
> + std::vector<DenseMap<CachedHashStringRef, size_t>> Map(NumShards);
> + size_t Shift = 32 - countTrailingZeros(NumShards);
>
> // Instantiate GdbSymbols while uniqufying them by name.
> - std::vector<GdbSymbol> Ret;
> - for (ArrayRef<NameTypeEntry> Entries : NameTypes) {
> - for (const NameTypeEntry &Ent : Entries) {
> - size_t &Idx = Map[Ent.Name];
> - if (Idx) {
> - Ret[Idx - 1].CuVector.push_back(Ent.Type);
> - continue;
> - }
> + std::vector<std::vector<GdbSymbol>> Symbols(NumShards);
> + parallelForEachN(0, Concurrency, [&](size_t ThreadId) {
> + for (ArrayRef<NameTypeEntry> Entries : NameTypes) {
> + for (const NameTypeEntry &Ent : Entries) {
> + size_t ShardId = Ent.Name.hash() >> Shift;
> + if ((ShardId & (Concurrency - 1)) != ThreadId)
> + continue;
> +
> + size_t &Idx = Map[ShardId][Ent.Name];
> + if (Idx) {
> + Symbols[ShardId][Idx - 1].CuVector.push_back(Ent.Type);
> + continue;
> + }
>
> - Idx = Ret.size() + 1;
> - Ret.push_back({Ent.Name, {Ent.Type}, 0, 0});
> + Idx = Symbols[ShardId].size() + 1;
> + Symbols[ShardId].push_back({Ent.Name, {Ent.Type}, 0, 0});
> + }
> }
> - }
> + });
> +
> + size_t NumSymbols = 0;
> + for (ArrayRef<GdbSymbol> V : Symbols)
> + NumSymbols += V.size();
> +
> + // The return type is a flattened vector, so we'll copy each vector
> + // contents to Ret.
> + std::vector<GdbSymbol> Ret;
> + Ret.reserve(NumSymbols);
> + for (std::vector<GdbSymbol> &Vec : Symbols)
> + for (GdbSymbol &Sym : Vec)
> + Ret.push_back(std::move(Sym));
>
> // CU vectors and symbol names are adjacent in the output file.
> // We can compute their offsets in the output file now.
>
> Modified: lld/trunk/test/ELF/gdb-index.s
> URL:
> http://llvm.org/viewvc/llvm-project/lld/trunk/test/ELF/gdb-index.s?rev=336790&r1=336789&r2=336790&view=diff
>
> ==============================================================================
> --- lld/trunk/test/ELF/gdb-index.s (original)
> +++ lld/trunk/test/ELF/gdb-index.s Wed Jul 11 04:37:10 2018
> @@ -34,16 +34,16 @@
> # DWARF-NEXT: Low/High address = [0x201000, 0x201001) (Size: 0x1), CU
> id = 0
> # DWARF-NEXT: Low/High address = [0x201004, 0x201006) (Size: 0x2), CU
> id = 1
> # DWARF: Symbol table offset = 0x60, size = 1024, filled slots:
> -# DWARF-NEXT: 512: Name offset = 0x2b, CU vector offset = 0x14
> -# DWARF-NEXT: String name: aaaaaaaaaaaaaaaa, CU vector index: 2
> -# DWARF-NEXT: 754: Name offset = 0x27, CU vector offset = 0x8
> -# DWARF-NEXT: String name: int, CU vector index: 1
> -# DWARF-NEXT: 822: Name offset = 0x1c, CU vector offset = 0x0
> -# DWARF-NEXT: String name: entrypoint, CU vector index: 0
> +# DWARF-NEXT: 512: Name offset = 0x1c, CU vector offset = 0x0
> +# DWARF-NEXT: String name: aaaaaaaaaaaaaaaa, CU vector index: 0
> +# DWARF-NEXT: 754: Name offset = 0x38, CU vector offset = 0x10
> +# DWARF-NEXT: String name: int, CU vector index: 2
> +# DWARF-NEXT: 822: Name offset = 0x2d, CU vector offset = 0x8
> +# DWARF-NEXT: String name: entrypoint, CU vector index: 1
> # DWARF: Constant pool offset = 0x2060, has 3 CU vectors:
> -# DWARF-NEXT: 0(0x0): 0x30000000
> -# DWARF-NEXT: 1(0x8): 0x90000000 0x90000001
> -# DWARF-NEXT: 2(0x14): 0x30000001
> +# DWARF-NEXT: 0(0x0): 0x30000001
> +# DWARF-NEXT: 1(0x8): 0x30000000
> +# DWARF-NEXT: 2(0x10): 0x90000000 0x90000001
>
> # SECTION-NOT: debug_gnu_pubnames
>
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20180716/eff15f36/attachment.html>
More information about the llvm-commits
mailing list