[llvm-dev] Multi-Threading Compilers

[Neil Nelson via llvm-dev]

This is a recent desktop.
Xubuntu 19.10
Compiling for 10.0.0 clang and llvm. See below.

For this test, running 14 processors in a gui VM. The cores are 
hyperthreaded, processors are twice the cores, but all the cores before 
the run are showing negligible activity.

compile_commands.json has 3022 entries.

The ninja compile run lasted 7 minutes and 43 seconds with 99% all 
processor usage throughout.

We then have 7*60+43 = 463 seconds.

Compile seconds per compile line in compile_commands.json 463/3022 = 
0.1532 seconds. Average compile time per processor would be about 
14*0.1532 seconds.

cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;llvm" -DLLVM_USE_LINKER=lld 
-DCMAKE_BUILD_TYPE="Release" -DLLVM_TARGETS_TO_BUILD=X86 
-DLLVM_ENABLE_LIBPFM=OFF -DRUN_HAVE_GNU_POSIX_REGEX=0 
-DRUN_HAVE_THREAD_SAFETY_ATTRIBUTES=0 -Wno-dev ../llvm &> 
/home/nnelson/Documents/cmake.log

ninja &> /home/nnelson/Documents/ninja.log

Here are some useful pages from Threading Building Blocks.

Task-Based Programming
https://software.intel.com/en-us/node/506100

Appendix A Costs of Time Slicing
https://software.intel.com/en-us/node/506127

The point

When the number of compiles exceeds the number of cores such that all 
the cores are utilized, nothing is gained by trying to multi-thread the 
individual compiles. In fact, loading up the cores with more threads or 
tasks than there are cores will reduce compiling efficiency because of 
time slicing. And sequencing through more tasks than less when the cores 
are not overloaded will reduce compiling efficiency because more tasks 
have to be loaded and unloaded to the cores.

Neil Nelson
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200301/d8a69443/attachment.html>