[llvm-dev] RFC: Supporting clang-cl /MP

[Alexandre Ganea via llvm-dev]

Hi folks,

With the recent support for Clang in Visual Studio 2019 [1], I thought we should revive and eventually land the /MP proposal I made a while ago [2].

As it stands now, building with clang-cl is much slower than MSVC when used along MSBuild, see timings in summary [2].
With this patch, the situation is greatly improved, clang-cl /MP performance is better in all cases [5].

----

When the MP option is active (default behavior) in a Visual Studio Project [3], MSBuild issues commands such as:

clang-cl /MP file-A.cpp file-B.cpp file-C.cpp

This creates a root clang-cl process, and then *sequentially* a cc1 sub-process for each CPP file in the list. Evidently this is slow and doesn't use the full potential on modern multi-cores machines.
A workaround is to increase the number of parallel builds [4], but that is sub-optimal and is still slower than MSVC.

The /MP proposal I made in [2] simply maintains a list of cc1 sub-processes for each hardware core (or for the number of cores provided as an option to /MP).
Waiting for termination in the root clang-cl process is done through a single OS primitive (see sys::WaitMany), which temporarily suspends the root process until at least one cc1 sub-process finishes. This ensure no cycles are wasted waiting, and avoids useless context-switches to the root process.

----

Most of the changes in the demo patch in [2] are: modernization of the process launching API (ProcessInfo), and support for sys::WaitMany, which I'd like to commit separately and incrementally.
Support for /MP is very small and mostly located in clang/trunk/lib/Driver/Compilation.cpp in the demo patch.

We could subsequently add a new flag -j to the regular clang driver, which mimics /MP. Further down the road, I'd like to discuss optional support for concrt multithreading which, from some preliminary testing, would be much faster than the current cc1 sub-process invocation, at least on Windows 10 1703+.

Feel free to comment here or in the demo patch.

Thanks,
Alex.

[1] https://devblogs.microsoft.com/cppblog/clang-llvm-support-in-visual-studio/
[2] https://reviews.llvm.org/D52193
[3] https://docs.microsoft.com/en-us/previous-versions/bb385193(v=vs.140)
[4] https://docs.microsoft.com/en-us/visualstudio/ide/reference/options-dialog-box-projects-and-solutions-build-and-run?view=vs-2019
[5] https://reviews.llvm.org/D52193#1241023
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190425/c814f364/attachment-0001.html>