[cfe-dev] RFC: Supporting clang-cl /MP

[Nico Weber via cfe-dev]

On Thu, Apr 25, 2019 at 10:35 AM Alexandre Ganea via cfe-dev <
cfe-dev at lists.llvm.org> wrote:

> Hi folks,
>
>
>
> With the recent support for Clang in Visual Studio 2019 [1], I thought we
> should revive and eventually land the /MP proposal I made a while ago [2].
>
>
>
> As it stands now, building with clang-cl is much slower than MSVC when
> used along MSBuild, see timings in summary [2].
>
> With this patch, the situation is greatly improved, clang-cl /MP
> performance is better in all cases [5].
>
>
>
> ----
>
>
>
> When the MP option is active (default behavior) in a Visual Studio Project
> [3], MSBuild issues commands such as:
>
>
>
> clang-cl /MP file-A.cpp file-B.cpp file-C.cpp
>
>
>
> This creates a root clang-cl process, and then **sequentially** a cc1
> sub-process for each CPP file in the list. Evidently this is slow and
> doesn’t use the full potential on modern multi-cores machines.
>
> A workaround is to increase the number of parallel builds [4], but that is
> sub-optimal and is still slower than MSVC.
>
>
>
> The /MP proposal I made in [2] simply maintains a list of cc1
> sub-processes for each hardware core (or for the number of cores provided
> as an option to /MP).
>
> Waiting for termination in the root clang-cl process is done through a
> single OS primitive (see sys::WaitMany), which temporarily suspends the
> root process until at least one cc1 sub-process finishes. This ensure no
> cycles are wasted waiting, and avoids useless context-switches to the root
> process.
>
>
>
> ----
>
>
>
> Most of the changes in the demo patch in [2] are: modernization of the
> process launching API (ProcessInfo), and support for sys::WaitMany, which
> I’d like to commit separately and incrementally.
>
> Support for /MP is very small and mostly located in
> clang/trunk/lib/Driver/Compilation.cpp in the demo patch.
>
>
>
> We could subsequently add a new flag –j to the regular clang driver, which
> mimics /MP. Further down the road, I’d like to discuss optional support for
> concrt multithreading which, from some preliminary testing, would be much
> faster than the current cc1 sub-process invocation, at least on Windows 10
> 1703+.
>

(Folks working on Microsoft's standard library said they've moved off
concrt, and the one use of it in LLVM is incorrect according to them, see
https://bugs.llvm.org/show_bug.cgi?id=41198)


>
>
> Feel free to comment here or in the demo patch.
>
>
>
> Thanks,
>
> Alex.
>
>
>
> [1]
> https://devblogs.microsoft.com/cppblog/clang-llvm-support-in-visual-studio/
>
> [2] https://reviews.llvm.org/D52193
>
> [3] https://docs.microsoft.com/en-us/previous-versions/bb385193(v=vs.140)
>
> [4]
> https://docs.microsoft.com/en-us/visualstudio/ide/reference/options-dialog-box-projects-and-solutions-build-and-run?view=vs-2019
>
> [5] https://reviews.llvm.org/D52193#1241023
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20190429/b28fd826/attachment.html>