[lld] r336790 - Parallelize GdbIndexSection's symbol table creation.
Rui Ueyama via llvm-commits
llvm-commits at lists.llvm.org
Wed Jul 18 10:53:59 PDT 2018
Yes this regresses single threaded performance, but multi threaded is the
default lld behavior.
On Mon, Jul 16, 2018 at 12:05 PM David Blaikie <dblaikie at gmail.com> wrote:
> Do I understand the commit message correctly that this regresses single
> threaded performance? Is single threaded the default lld behavior?
>
> On Wed, Jul 11, 2018 at 4:42 AM Rui Ueyama via llvm-commits <
> llvm-commits at lists.llvm.org> wrote:
>
>> Author: ruiu
>> Date: Wed Jul 11 04:37:10 2018
>> New Revision: 336790
>>
>> URL: http://llvm.org/viewvc/llvm-project?rev=336790&view=rev
>> Log:
>> Parallelize GdbIndexSection's symbol table creation.
>>
>> Since .gdb_index sections contain all known symbols, they can be very
>> large.
>> One of my executables has a .gdb_index section of 1350 GiB. Uniquifying
>> symbols by name takes 3.77 seconds on my machine. This patch parallelize
>> it.
>>
>> Time to call createSymbols() with 8.4 million unique symbols:
>>
>> Without this patch: 3773 ms
>> Parallelism = 1: 4374 ms
>> Parallelism = 2: 2628 ms
>> Parallelism = 16: 837 ms
>>
>> As you can see above, this algorithm is a bit more inefficient
>> than the non-parallelized version, but even with dual-core, it is
>> faster than that, so I think it is overall a win.
>>
>> Differential Revision: https://reviews.llvm.org/D49164
>>
>> Modified:
>> lld/trunk/ELF/SyntheticSections.cpp
>> lld/trunk/test/ELF/gdb-index.s
>>
>> Modified: lld/trunk/ELF/SyntheticSections.cpp
>> URL:
>> http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/SyntheticSections.cpp?rev=336790&r1=336789&r2=336790&view=diff
>>
>> ==============================================================================
>> --- lld/trunk/ELF/SyntheticSections.cpp (original)
>> +++ lld/trunk/ELF/SyntheticSections.cpp Wed Jul 11 04:37:10 2018
>> @@ -2366,23 +2366,51 @@ createSymbols(ArrayRef<std::vector<GdbIn
>> typedef GdbIndexSection::GdbSymbol GdbSymbol;
>> typedef GdbIndexSection::NameTypeEntry NameTypeEntry;
>>
>> - // A map to uniquify symbols by name.
>> - DenseMap<CachedHashStringRef, size_t> Map;
>> + // The number of symbols we will handle in this function is of the
>> order
>> + // of millions for very large executables, so we use multi-threading to
>> + // speed it up.
>> + size_t NumShards = 32;
>> + size_t Concurrency = 1;
>> + if (ThreadsEnabled)
>> + Concurrency =
>> + std::min<size_t>(PowerOf2Floor(hardware_concurrency()),
>> NumShards);
>> +
>> + // A sharded map to uniquify symbols by name.
>> + std::vector<DenseMap<CachedHashStringRef, size_t>> Map(NumShards);
>> + size_t Shift = 32 - countTrailingZeros(NumShards);
>>
>> // Instantiate GdbSymbols while uniqufying them by name.
>> - std::vector<GdbSymbol> Ret;
>> - for (ArrayRef<NameTypeEntry> Entries : NameTypes) {
>> - for (const NameTypeEntry &Ent : Entries) {
>> - size_t &Idx = Map[Ent.Name];
>> - if (Idx) {
>> - Ret[Idx - 1].CuVector.push_back(Ent.Type);
>> - continue;
>> - }
>> + std::vector<std::vector<GdbSymbol>> Symbols(NumShards);
>> + parallelForEachN(0, Concurrency, [&](size_t ThreadId) {
>> + for (ArrayRef<NameTypeEntry> Entries : NameTypes) {
>> + for (const NameTypeEntry &Ent : Entries) {
>> + size_t ShardId = Ent.Name.hash() >> Shift;
>> + if ((ShardId & (Concurrency - 1)) != ThreadId)
>> + continue;
>> +
>> + size_t &Idx = Map[ShardId][Ent.Name];
>> + if (Idx) {
>> + Symbols[ShardId][Idx - 1].CuVector.push_back(Ent.Type);
>> + continue;
>> + }
>>
>> - Idx = Ret.size() + 1;
>> - Ret.push_back({Ent.Name, {Ent.Type}, 0, 0});
>> + Idx = Symbols[ShardId].size() + 1;
>> + Symbols[ShardId].push_back({Ent.Name, {Ent.Type}, 0, 0});
>> + }
>> }
>> - }
>> + });
>> +
>> + size_t NumSymbols = 0;
>> + for (ArrayRef<GdbSymbol> V : Symbols)
>> + NumSymbols += V.size();
>> +
>> + // The return type is a flattened vector, so we'll copy each vector
>> + // contents to Ret.
>> + std::vector<GdbSymbol> Ret;
>> + Ret.reserve(NumSymbols);
>> + for (std::vector<GdbSymbol> &Vec : Symbols)
>> + for (GdbSymbol &Sym : Vec)
>> + Ret.push_back(std::move(Sym));
>>
>> // CU vectors and symbol names are adjacent in the output file.
>> // We can compute their offsets in the output file now.
>>
>> Modified: lld/trunk/test/ELF/gdb-index.s
>> URL:
>> http://llvm.org/viewvc/llvm-project/lld/trunk/test/ELF/gdb-index.s?rev=336790&r1=336789&r2=336790&view=diff
>>
>> ==============================================================================
>> --- lld/trunk/test/ELF/gdb-index.s (original)
>> +++ lld/trunk/test/ELF/gdb-index.s Wed Jul 11 04:37:10 2018
>> @@ -34,16 +34,16 @@
>> # DWARF-NEXT: Low/High address = [0x201000, 0x201001) (Size: 0x1), CU
>> id = 0
>> # DWARF-NEXT: Low/High address = [0x201004, 0x201006) (Size: 0x2), CU
>> id = 1
>> # DWARF: Symbol table offset = 0x60, size = 1024, filled slots:
>> -# DWARF-NEXT: 512: Name offset = 0x2b, CU vector offset = 0x14
>> -# DWARF-NEXT: String name: aaaaaaaaaaaaaaaa, CU vector index: 2
>> -# DWARF-NEXT: 754: Name offset = 0x27, CU vector offset = 0x8
>> -# DWARF-NEXT: String name: int, CU vector index: 1
>> -# DWARF-NEXT: 822: Name offset = 0x1c, CU vector offset = 0x0
>> -# DWARF-NEXT: String name: entrypoint, CU vector index: 0
>> +# DWARF-NEXT: 512: Name offset = 0x1c, CU vector offset = 0x0
>> +# DWARF-NEXT: String name: aaaaaaaaaaaaaaaa, CU vector index: 0
>> +# DWARF-NEXT: 754: Name offset = 0x38, CU vector offset = 0x10
>> +# DWARF-NEXT: String name: int, CU vector index: 2
>> +# DWARF-NEXT: 822: Name offset = 0x2d, CU vector offset = 0x8
>> +# DWARF-NEXT: String name: entrypoint, CU vector index: 1
>> # DWARF: Constant pool offset = 0x2060, has 3 CU vectors:
>> -# DWARF-NEXT: 0(0x0): 0x30000000
>> -# DWARF-NEXT: 1(0x8): 0x90000000 0x90000001
>> -# DWARF-NEXT: 2(0x14): 0x30000001
>> +# DWARF-NEXT: 0(0x0): 0x30000001
>> +# DWARF-NEXT: 1(0x8): 0x30000000
>> +# DWARF-NEXT: 2(0x10): 0x90000000 0x90000001
>>
>> # SECTION-NOT: debug_gnu_pubnames
>>
>>
>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20180718/e14b462e/attachment.html>
More information about the llvm-commits
mailing list