[llvm-dev] Multi-Threading Compilers

[Alexandre Ganea via llvm-dev]

In addition to what Chris said, there’s also the case of large TUs / Unity files<https://en.wikipedia.org/wiki/Single_Compilation_Unit>. Given that currently the compilation of a single TU is not multi-thread, you can get “spikes” as the one below, where only one core (out of many) is working. One minute wasted, when I have many cores available:

[cid:image001.png at 01D5F09E.5ED95CA0]

For this specific case, not having Unity files makes the build uniform (no spikes like the one above), but it takes 10x more time to compile (20k TUs and 25k .h files compiled, which reduce down to ~600 Unity).

To fix the spike, you then have to resort to a iterative optimization algorithm running nightly to find the best trade-off between size of unity and build times. This complicates things further.

The point being, +1 for multi-threading the compiler :-)

Alex.

De : llvm-dev <llvm-dev-bounces at lists.llvm.org> De la part de Chris Lattner via llvm-dev
Envoyé : March 2, 2020 1:08 PM
À : Neil Nelson <nnelson at infowest.com>
Cc : llvm-dev at lists.llvm.org
Objet : Re: [llvm-dev] Multi-Threading Compilers

On Mar 1, 2020, at 1:29 AM, Neil Nelson via llvm-dev <llvm-dev at lists.llvm.org<mailto:llvm-dev at lists.llvm.org>> wrote:
The point

When the number of compiles exceeds the number of cores such that all the cores are utilized, nothing is gained by trying to multi-thread the individual compiles. In fact, loading up the cores with more threads or tasks than there are cores will reduce compiling efficiency because of time slicing. And sequencing through more tasks than less when the cores are not overloaded will reduce compiling efficiency because more tasks have to be loaded and unloaded to the cores.

That makes a lot of sense Neil.  Two counterpoints though:

1) In development situations, it is common to rebuild one file (the one you changed) without rebuilding everything.  Speeding that up is useful.

2) LLVM is a library that is used for a lot more than just C compilers.  Many of its use cases are not modeled by bulk “compile all the code” workloads like you describe.


You’re right that multithreading isn’t a panacea, but in my opinion, it is important to be able to provide access to multicore compilation for use cases that do benefit from it.

-Chris

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200302/8fba3cb3/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 262647 bytes
Desc: image001.png
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200302/8fba3cb3/attachment-0001.png>