<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Sun, Dec 4, 2016 at 1:55 AM, Sean Silva <span dir="ltr"><<a href="mailto:chisophugis@gmail.com" target="_blank">chisophugis@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Sat, Dec 3, 2016 at 3:35 PM, Rui Ueyama via llvm-commits <span dir="ltr"><<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Author: ruiu<br>
Date: Sat Dec 3 17:35:22 2016<br>
New Revision: 288606<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=288606&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-pr<wbr>oject?rev=288606&view=rev</a><br>
Log:<br>
Add comments about the use of threads in LLD.<br>
<br>
Modified:<br>
lld/trunk/ELF/Threads.h<br>
<br>
Modified: lld/trunk/ELF/Threads.h<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/lld/trunk/ELF/Threads.h?rev=288606&r1=288605&r2=288606&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-pr<wbr>oject/lld/trunk/ELF/Threads.h?<wbr>rev=288606&r1=288605&r2=288606<wbr>&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- lld/trunk/ELF/Threads.h (original)<br>
+++ lld/trunk/ELF/Threads.h Sat Dec 3 17:35:22 2016<br>
@@ -6,6 +6,54 @@<br>
// License. See LICENSE.TXT for details.<br>
//<br>
//===------------------------<wbr>------------------------------<wbr>----------------===//<br>
+//<br>
+// LLD supports threads to distribute workloads to multiple cores. Using<br>
+// multicore is most effective when more than one core are idle. At the<br>
+// last step of a build, it is often the case that a linker is the only<br>
+// active process on a computer. So, we are naturally interested in using<br>
+// threads wisely to reduce latency to deliver results to users.<br>
+//<br>
+// That said, we don't want to do "too clever" things using threads.<br>
+// Complex multi-threaded algorithms are sometimes extremely hard to<br>
+// justify the correctness and can easily mess up the entire design.<br>
+//<br>
+// Fortunately, when a linker links large programs (when the link time is<br>
+// most critical), it spends most of the time to work on massive number of<br>
+// small pieces of data of the same kind. Here are examples:<br>
+//<br>
+// - We have hundreds of thousands of input sections that need to be<br>
+// copied to a result file at the last step of link. Once we fix a file<br>
+// layout, each section can be copied to its destination and its<br>
+// relocations can be applied independently.<br>
+//<br>
+// - We have tens of millions of small strings when constructing a<br>
+// mergeable string section.<br>
+//<br>
+// For the cases such as the former, we can just use parallel_for_each<br>
+// instead of std::for_each (or a plain for loop). Because tasks are<br>
+// completely independent from each other, we can run them in parallel<br>
+// without any coordination between them. That's very easy to understand<br>
+// and justify.<br>
+//<br>
+// For the cases such as the latter, we can use parallel algorithms to<br>
+// deal with massive data. We have to write code for a tailored algorithm<br>
+// for each problem, but the complexity of multi-threading is isolated in<br>
+// a single pass and doesn't affect the linker's overall design.<br>
+//<br>
+// The above approach seems to be working fairly well. As an example, when<br>
+// linking Chromium (output size 1.6 GB), using 4 cores reduces latency to<br>
+// 75% compared to single core (from 12.66 seconds to 9.55 seconds) on my<br>
+// machine. Using 40 cores reduces it to 63% (from 12.66 seconds to 7.95<br>
+// seconds). Because of the Amdahl's law, the speedup is not linear, but<br>
+// as you add more cores, it gets faster.<br>
+//<br>
+// On a final note, if you are trying to optimize, keep the axiom "don't<br>
+// guess, measure!" in mind. Some important passes of the linker are not<br>
+// that slow. For example, resolving all symbols is not a very heavy pass,<br>
+// although it would be very hard to parallelize it. You want to first<br>
+// identify a slow pass and then optimize it.<br></blockquote><div><br></div></div></div><div>Actually, LLD's symbol resolution (the approach with Lazy symbols for archives) is a perfect example of a MapReduce type problem, so it is actually quite parallelizable.</div><div>You basically have a huge number of (SymbolName,SymbolValue) pairs and you want to coalesce all values with the same SymbolName into pairs (SymbolName, [SymbolValue1,SymbolValue2,...<wbr>]) which you can then process all the SymbolValueN's to see which is the real definition. This is precisely the problem that MapReduce solves.</div></div></div></div></blockquote><div><br></div><div>How do you handle static archives?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>(note: I don't necessarily mean that it needs to be done in a distributed fashion, just that the core problem is really one of coalescing values with the same keys.</div><div>)</div><div><br></div><div>MapReduce's core abstraction is also a good tool for deduplicating strings.</div><div><br></div><div><br></div><div>Richard Smith and I were actually brainstorming at the latest llvm social a distributed linker may be a good fit for the linking problem at Google (but it was just brainstorming; obviously that would be a huge effort and we would need very serious justification before embarking on that effort).<span class="HOEnZb"><font color="#888888"><br></font></span></div><span class="HOEnZb"><font color="#888888"><div><br></div><div>-- Sean Silva</div></font></span><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
+//<br>
+//===------------------------<wbr>------------------------------<wbr>----------------===//<br>
<br>
#ifndef LLD_ELF_THREADS_H<br>
#define LLD_ELF_THREADS_H<br>
<br>
<br>
______________________________<wbr>_________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/llvm-commits</a><br>
</blockquote></span></div><br></div></div>
</blockquote></div><br></div></div>