[PATCH] D52193: RFC: [clang] Multithreaded compilation support

[Alexandre Ganea via Phabricator via cfe-commits]

aganea marked an inline comment as done.
aganea added a comment.

It seems Reid's change has done good: `cmake` is not as significant as before in the build process. See below the improved timings:

The tests consist in a full cleanup (delete build folder), cmake to regenerate, then a full rebuild of LLVM + Clang + LLD (at r342552), Release target, optimized tablegen.
VS2017 15.8.3, Ninja 1.8.2, CMake 3.12.2
For the `clang-cl` tests, I'm not using any official LLVM release, only the binaries I built myself. `lld-link` is used in that case.

//(built with MSVC)// means the LLVM toolchain used to perfom this test was compiled in a previous run with MSVC cl 15.8.3
//(built with Clang)// means the LLVM toolchain used to perform this test was compiled in a previous run with Clang at r342552

I took the best figures from several runs (ran in a random order).

-------

**Config 1 :** Intel Xeon Haswell 6 cores / 12 HW threads, 3.5 GHz, 15M cache, 128 GB RAM, SSD 550 MB/s

__MSBuild :__

| MSVC cl /MP                             | (50min 26sec) | 2 parallel msbuild  |
| MSVC cl /MP                             | (40min 23sec) | 8 parallel msbuild  |
| MSVC cl /MP                             | (40min 5sec)  | 16 parallel msbuild |
| clang-cl //(built with MSVC)//          | (43min 36sec) | 16 parallel msbuild |
| clang-cl //(built with Clang//)         | (43min 42sec) | 16 parallel msbuild |
| clang-cl **/MP** //(built with MSVC)//  | not tested    |                     |
| clang-cl **/MP** //(built with Clang)// | (36min 13sec) | 8 parallel msbuild  |
| clang-cl **/MP** //(built with Clang)// | (34min 57sec) | 16 parallel msbuild |
|

__Ninja:__

| MSVC cl                         | (33min 29sec) |
| clang-cl //(built with MSVC)//  | (30min 2sec)  |
| clang-cl //(built with Clang)// | (28min 29sec) |
|



-----

**Config 2 :** Intel Xeon Skylake 18 cores / 36 HW threads, x2 (Dual CPU), 72 HW threads total, 2.3 GHz, 24.75M cache, 128 GB RAM, NVMe 4.6 GB/s

__MSBuild :__

| MSVC cl /MP                             | (10min 3sec)  | 32 parallel msbuild |
| clang-cl //(built with MSVC)//          | (24min 15sec) | 32 parallel msbuild |
| clang-cl //(built with Clang)//         | (21min 21sec) | 32 parallel msbuild |
| clang-cl **/MP** //(built with MSVC)//  | (7min 52sec)  | 32 parallel msbuild |
| clang-cl **/MP** //(built with Clang)// | (7min 30sec)  | 32 parallel msbuild |
|

__Ninja:__

| MSVC cl                         | (7min 25sec) |
| clang-cl //(built with MSVC)//  | (8min 23sec) |
| clang-cl //(built with Clang)// | (8min)       |
|



================
Comment at: llvm/trunk/lib/Support/Windows/Program.inc:424
 
-ProcessInfo sys::Wait(const ProcessInfo &PI, unsigned SecondsToWait,
-                      bool WaitUntilChildTerminates, std::string *ErrMsg) {
-  assert(PI.Pid && "invalid pid to wait on, process not started?");
-  assert((PI.Process && PI.Process != INVALID_HANDLE_VALUE) &&
-         "invalid process handle to wait on, process not started?");
+bool sys::WaitMany(MutableArrayRef<ProcessInfo *> PIArray, bool WaitOnAll,
+                   unsigned SecondsToWait, bool WaitUntilProcessTerminates) {
----------------
rnk wrote:
> I guess no Posix implementation? It's kind of hard to know if we made the right abstractions without doing it for Windows and *nix.
Yes, I currenly don't have an Unix box at hand. But I will implement it shortly as it needs to be atomically commited with the Windows part.


Repository:
  rC Clang

https://reviews.llvm.org/D52193