[all-commits] [llvm/llvm-project] 2edd89: Make WriteIndexesThinBackend multi threaded (#109847)

[Nuri Amari via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 2edd897a4227e481af33e8e43090ab088cd9d953
      https://github.com/llvm/llvm-project/commit/2edd897a4227e481af33e8e43090ab088cd9d953
  Author: Nuri Amari <nuri.amari99 at gmail.com>
  Date:   2024-10-07 (Mon, 07 Oct 2024)

  Changed paths:
    M lld/COFF/LTO.cpp
    M lld/ELF/LTO.cpp
    M lld/MachO/LTO.cpp
    M lld/test/COFF/thinlto-emit-imports.ll
    M lld/test/ELF/lto/thinlto-cant-write-index.ll
    M lld/test/ELF/lto/thinlto-emit-imports.ll
    M lld/test/MachO/thinlto-emit-imports.ll
    M llvm/include/llvm/LTO/LTO.h
    M llvm/include/llvm/Support/Threading.h
    M llvm/include/llvm/Transforms/IPO/FunctionImport.h
    M llvm/lib/LTO/LTO.cpp
    M llvm/lib/LTO/ThinLTOCodeGenerator.cpp
    M llvm/lib/Transforms/IPO/FunctionImport.cpp
    M llvm/tools/gold/gold-plugin.cpp
    M llvm/tools/llvm-lto2/llvm-lto2.cpp

  Log Message:
  -----------
  Make WriteIndexesThinBackend multi threaded (#109847)

We've noticed that for large builds executing thin-link can take on the
order of 10s of minutes. We are only using a single thread to write the
sharded indices and import files for each input bitcode file. While we
need to ensure the index file produced lists modules in a deterministic
order, that doesn't prevent us from executing the rest of the work in
parallel.

In this change we use a thread pool to execute as much of the backend's
work as possible in parallel. In local testing on a machine with 80
cores, this change makes a thin-link for ~100,000 input files run in ~2
minutes. Without this change it takes upwards of 10 minutes.

---------

Co-authored-by: Nuri Amari <nuriamari at fb.com>



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications