[PATCH] D38266: Parallelize string merging.

Thu Sep 28 19:02:51 PDT 2017

The results are attached.

scylla is probably the most interesting. Without threads it gets 1.11x
slower. With 8 cores it is 1.45x faster.

So this is probably fine, but what do you think of the idea of sorting
by hash to have a reproducible output with any number of shards?

Cheers,
Rafael

On 28 September 2017 at 15:42, Rafael Avila de Espindola
<rafael.espindola at gmail.com> wrote:
> Rafael Avila de Espindola <rafael.espindola at gmail.com> writes:
>
>>> +  // Parallelism. Changing this number causes benign changes in the
>>> +  // order of output section pieces. For build reproducibility, we
>>> +  // always use the same number.
>>> +  static constexpr size_t NumShards = 8;
>>
>> One idea that might allow using a variable number of shards:
>>
>> Sort the strings once they are know to be unique. The sort order can be
>> based on the hash and look at the string itself only if two hashes are
>> identical, so it shouldn't be too slow.
>
> In fact, we could use the N most significant bits of the hash to do the
> sharding. That way we only need to sort inside each shard.
>
> Cheers,
> Rafael
-------------- next part --------------
linux-kernel
  master 0.55544403
  patch  0.559538875 1.00737220094x slower
clang-gdb-index
  master 10.608503082
  patch  10.513981887 1.00899004735x faster
firefox-O0
  master 4.577045704
  patch  4.572261472 1.00104636011x faster
firefox
  master 7.38679043
  patch  8.149984172 1.10331872134x slower
firefox-gc
  master 7.585813073
  patch  8.343501213 1.0998822582x slower
chromium
  master 4.047151992
  patch  4.058094124 1.00270366223x slower
chromium fast
  master 2.18329147
  patch  2.186675613 1.00155001888x slower
the gold plugin
  master 0.379459034
  patch  0.379199075 1.00068554756x faster
clang
  master 0.66072077
  patch  0.658274584 1.00371605719x faster
llvm-as
  master 0.042257896
  patch  0.042159528 1.00233323295x faster
the gold plugin fsds
  master 0.414197155
  patch  0.414573638 1.00090894637x slower
clang fsds
  master 0.758676737
  patch  0.756753377 1.00254159421x faster
llvm-as fsds
  master 0.042162466
  patch  0.042207809 1.0010754352x slower
scylla
  master 3.239592993
  patch  3.604992284 1.11279172778x slower
-------------- next part --------------
linux-kernel
  master 0.612900497
  patch  0.626735729 1.02257337377x slower
clang-gdb-index
  master 7.868864712
  patch  7.824436018 1.00567819762x faster
firefox-O0
  master 3.714728111
  patch  3.699741878 1.00405061583x faster
firefox
  master 6.274184475
  patch  6.062873045 1.03485334897x faster
firefox-gc
  master 6.485669106
  patch  6.261206962 1.03584966051x faster
chromium
  master 3.329566892
  patch  3.333475654 1.00117395509x slower
chromium fast
  master 2.099190258
  patch  2.10405567 1.00231775656x slower
the gold plugin
  master 0.370556415
  patch  0.370375819 1.00048760203x faster
clang
  master 0.654198786
  patch  0.651366426 1.00434833588x faster
llvm-as
  master 0.047591554
  patch  0.048813361 1.02567276958x slower
the gold plugin fsds
  master 0.407458112
  patch  0.408974394 1.00372131995x slower
clang fsds
  master 0.744753862
  patch  0.744995833 1.00032490063x slower
llvm-as fsds
  master 0.048276047
  patch  0.048890697 1.01273198694x slower
scylla
  master 2.40187268
  patch  2.30835122 1.04051439798x faster
-------------- next part --------------
linux-kernel
  master 0.626080534
  patch  0.643165399 1.02728860597x slower
clang-gdb-index
  master 6.426126764
  patch  6.404751752 1.00333736776x faster
firefox-O0
  master 3.264842018
  patch  3.270227588 1.00164956527x slower
firefox
  master 5.47685605
  patch  4.720803832 1.16015328001x faster
firefox-gc
  master 5.694961349
  patch  4.92669938 1.15593847112x faster
chromium
  master 2.997183366
  patch  2.940536752 1.01926403877x faster
chromium fast
  master 2.024303611
  patch  2.030329949 1.00297699316x slower
the gold plugin
  master 0.356583147
  patch  0.357632095 1.00294166454x slower
clang
  master 0.61550761
  patch  0.615536122 1.00004632274x slower
llvm-as
  master 0.049557048
  patch  0.049994632 1.00882990448x slower
the gold plugin fsds
  master 0.390568296
  patch  0.392584591 1.00516246459x slower
clang fsds
  master 0.703107454
  patch  0.702208642 1.00127997855x faster
llvm-as fsds
  master 0.049662001
  patch  0.049993524 1.00667558683x slower
scylla
  master 1.87639585
  patch  1.526806019 1.22896807234x faster
-------------- next part --------------
linux-kernel
  master 0.632168079
  patch  0.653757886 1.03415200438x slower
clang-gdb-index
  master 5.609210645
  patch  5.593429546 1.00282136369x faster
firefox-O0
  master 3.050899817
  patch  3.061637794 1.00351960983x slower
firefox
  master 4.954002541
  patch  3.934472614 1.25912746816x faster
firefox-gc
  master 5.160510855
  patch  4.122687205 1.25173475415x faster
chromium
  master 2.782784982
  patch  2.755903826 1.00975402543x faster
chromium fast
  master 1.992202049
  patch  1.995266382 1.00153816376x slower
the gold plugin
  master 0.342796428
  patch  0.34437438 1.00460317515x slower
clang
  master 0.58580986
  patch  0.585700386 1.00018691126x faster
llvm-as
  master 0.050535743
  patch  0.050949229 1.00818205047x slower
the gold plugin fsds
  master 0.379719707
  patch  0.38113977 1.0037397664x slower
clang fsds
  master 0.669768084
  patch  0.671255731 1.00222113749x slower
llvm-as fsds
  master 0.050649774
  patch  0.051227567 1.0114076126x slower
scylla
  master 1.581009575
  patch  1.085384062 1.45663607045x faster