[all-commits] [llvm/llvm-project] 091582: [ThinLTO] Allow usage of all hardware threads in t...

[Alexandre Ganea via All-commits]

  Branch: refs/heads/master
  Home:   https://github.com/llvm/llvm-project
  Commit: 09158252f777c2e2f06a86b154c44abcbcf9bb74
      https://github.com/llvm/llvm-project/commit/09158252f777c2e2f06a86b154c44abcbcf9bb74
  Author: Alexandre Ganea <alexandre.ganea at ubisoft.com>
  Date:   2020-03-27 (Fri, 27 Mar 2020)

  Changed paths:
    M clang/lib/Driver/ToolChains/CommonArgs.cpp
    M clang/lib/Driver/ToolChains/CommonArgs.h
    M clang/lib/Driver/ToolChains/Darwin.cpp
    M lld/COFF/Config.h
    M lld/COFF/Driver.cpp
    M lld/COFF/LTO.cpp
    M lld/ELF/Config.h
    M lld/ELF/Driver.cpp
    M lld/ELF/LTO.cpp
    M lld/test/COFF/thinlto.ll
    M lld/test/ELF/basic.s
    M lld/test/ELF/lto/thinlto.ll
    M lld/test/wasm/lto/thinlto.ll
    M lld/wasm/Config.h
    M lld/wasm/Driver.cpp
    M lld/wasm/LTO.cpp
    M llvm/include/llvm/LTO/LTO.h
    M llvm/include/llvm/Support/Threading.h
    M llvm/lib/LTO/LTO.cpp
    M llvm/lib/Support/Threading.cpp
    M llvm/lib/Support/Unix/Threading.inc
    M llvm/lib/Support/Windows/Threading.inc
    M llvm/tools/gold/gold-plugin.cpp
    M llvm/tools/llvm-lto2/llvm-lto2.cpp

  Log Message:
  -----------
  [ThinLTO] Allow usage of all hardware threads in the system

Before this patch, it wasn't possible to extend the ThinLTO threads to all SMT/CMT threads in the system. Only one thread per core was allowed, instructed by usage of llvm::heavyweight_hardware_concurrency() in the ThinLTO code. Any number passed to the LLD flag /opt:lldltojobs=..., or any other ThinLTO-specific flag, was previously interpreted in the context of llvm::heavyweight_hardware_concurrency(), which means SMT disabled.

One can now say in LLD:
/opt:lldltojobs=0 -- Use one std::thread / hardware core in the system (no SMT). Default value if flag not specified.
/opt:lldltojobs=N -- Limit usage to N threads, regardless of usage of heavyweight_hardware_concurrency().
/opt:lldltojobs=all -- Use all hardware threads in the system. Equivalent to /opt:lldltojobs=$(nproc) on Linux and /opt:lldltojobs=%NUMBER_OF_PROCESSORS% on Windows. When an affinity mask is set for the process, threads will be created only for the cores selected by the mask.

When N > number-of-hardware-threads-in-the-system, the threads in the thread pool will be dispatched equally on all CPU sockets (tested only on Windows).
When N <= number-of-hardware-threads-on-a-CPU-socket, the threads will remain on the CPU socket where the process started (only on Windows).

Differential Revision: https://reviews.llvm.org/D75153