[PATCH] D52193: RFC: [clang] Multithreaded compilation support

Tue Sep 18 16:52:29 PDT 2018

zturner added a comment.

In https://reviews.llvm.org/D52193#1238978, @aganea wrote:

> In https://reviews.llvm.org/D52193#1238944, @zturner wrote:
>
> > In https://reviews.llvm.org/D52193#1238923, @aganea wrote:
> >
> > > The goal of this change is frictionless compilation into VS2017 when using `clang-cl` as a compiler. We've realized that compiling Clang+LLD with Clang generates a faster executable that with MSVC (even latest one).
> > >  I currently can't see a good way of generating the Visual Studio solution with CMake, while using Ninja+clang-cl for compilation. They are two orthogonal configurations. Any suggestions?
> >
> >
> > I don't think this is necessary.  I think your updated Matrix is pretty good.
> >
> > I'm surprised to see that Ninja + Clang is slower than Ninja + MSVC.  Are you using lld here?
>
>
> Yes, all the ‘Clang’ rows use both clang-cl and lld-link.
>  Like stated above, I think this is caused by a lot more processes (clang-cl.exe) being invoked. In contrast, cl.exe does not invoke a child process. There are about 3200 files to compiles, which makes 6400 calls to clang-cl. At about ~60ms lead time per process, that adds up to an extra ~3min wall clock time. It is the simplest explanation I could find.

Would you mind syncing to r342002 and trying again?  I don't doubt your results, but I'm interested to see how much of an improvement this patch provides.

  commit 840717872a0e0f03b19040ef143bf45aa1a7f0a0
  Author: Reid Kleckner <rnk at google.com>
  Date:   Tue Sep 11 22:25:13 2018 +0000

      [cmake] Speed up check-llvm 5x by delay loading shell32 and ole32

      Summary:
      Previously, check-llvm on my Windows 10 workstation took about 300s to
      run, and it would lock up my mouse. Now, it takes just under 60s.
      Previously running the tests only used about 15% of the available CPU
      time, now it uses up to 60%.

      Shell32.dll and ole32.dll have direct dependencies on user32.dll and
      gdi32.dll. These DLLs are mostly used to for Windows GUI functionality,
      which most LLVM tools don't need. It seems that these DLLs acquire and
      release some system resources on startup and exit, and that requires
      acquiring the same highly contended lock discussed in this post:
      https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and-i-cant-move-my-mouse/

      Most LLVM tools still have a transitive dependency on
      SHGetKnownFolderPathW, which is used to look up the user home directory
      and local AppData directory. We also use SHFileOperationW to recursively
      delete a directory, but only LLDB and clang-doc use this today. At some
      point, we might consider removing these last shell32.dll deps, but for
      now, I would like to take this free performance.

      Many thanks to Bruce Dawson for suggesting this fix. He looked at an ETW
      trace of an LLVM test run, noticed these DLLs, and suggested avoiding
      them.

      Reviewers: zturner, pcc, thakis

      Subscribers: mgorny, llvm-commits, hiraditya

      Differential Revision: https://reviews.llvm.org/D51952

  Notes:
      git-svn-rev: 342002

Repository:
  rC Clang

https://reviews.llvm.org/D52193