[PATCH] D27146: Merge strings using a probabilistic algorithm to reduce latency.

[Rui Ueyama via Phabricator via llvm-commits]

ruiu added a comment.

Sean, thank you for the investigation and the numbers! I knew some of the numbers (that's why I could come up with this), but I just implemented, set the threshold to 10%, and it just worked, so I didn't take a close look at each section contents.

One thing I'd like to note in addition to your results is that strings are already uniquified for each input section (for obvious reason... compilers don't emit redundant strings.) So the likelihood of identifying duplicate strings when picking up random samples from input sections is larger than picking up random samples from 30 million strings. Instead of a single bag containing 30 million strings, we have approximately 1000 bags containing 30 million strings in total, and inside each bag all strings are unique. That's why I construct the small string table from random samples of input sections instead of random samples of section pieces.

> One other question: how much of this speedup is due to the parallelizing, and how much due to the improved algorithm?

The speedup is entirely due to the parallelizing. If you run this with a single thread, you'd get the same latency as before. And that's a good property because this algorithm doesn't slow down the linker when available CPU resource is scarce.



================
Comment at: ELF/OutputSections.cpp:633
+            size_t Size = align2(S.size(), StringAlignment);
+            Piece.OutputOff = Offset.fetch_add(Size);
+          } else {
----------------
silvas wrote:
> This will result in non-deterministic output :/
Hmm, that's true, and I have no good idea to make it deterministic at the moment. We can partly serialize it, but it would slow down this part of process.


https://reviews.llvm.org/D27146