[PATCH] D130229: [ELF] Add --thinlto-index= and --remapping-file=

[Fangrui Song via Phabricator via llvm-commits]

MaskRay created this revision.
Herald added subscribers: ormris, StephenFan, arphaman, steven_wu, hiraditya, arichardson, inglorion, emaste.
Herald added a project: All.
MaskRay requested review of this revision.
Herald added a project: LLVM.
Herald added a subscriber: llvm-commits.

--thinlto-index= is designed to replace --thinlto-index-only= for distributed
ThinLTO. Example:

  echo 'int g() { return 0; }' > a.c
  echo 'int f(); int main() { return f(); }' > b.c
  echo 'int g(); int f() { return g(); }' > c.c
  mkdir -p out
  
  # ThinLTO pre-link compilation
  clang -c -flto=thin -O2 b.c -o out/b.o -fthin-link-bitcode=out/b.indexing.o
  clang -c -flto=thin -O2 c.c -o out/c.o -fthin-link-bitcode=out/c.indexing.o
  clang -c -O2 a.c -o out/a.o
  
  # ThinLTO indexing
  clang -flto=thin -fuse-ld=lld -Wl,--thinlto-index=out/exe.map -Wl,--thinlto-prefix-replace='out;lto' \
    -Wl,--thinlto-object-suffix-replace='.indexing.o;.o' out/a.o out/b.indexing.o out/c.indexing.o
  
  # Backend compilation
  clang -c -O2 -fthinlto-index=lto/b.o.thinlto.bc out/b.o -o lto/b.o
  clang -c -O2 -fthinlto-index=lto/c.o.thinlto.bc out/c.o -o lto/c.o
  
  # Final link
  clang -fuse-ld=lld -Wl,--remapping-file=out/exe.map out/a.o out/b.indexing.o out/c.indexing.o -o out/exe

The indexing action writes a remapping file out/exe.map:

  out/b.indexing.o	lto/b.o
  out/c.indexing.o	lto/c.o

For the final link, the file redirects input bitcode files to backend
compilation produced native object files.

---

The old way does ThinLTO indexing and the final link differently.

  # ThinLTO indexing
  clang -flto=thin -fuse-ld=lld -Wl,--thinlto-index-only=out/exe.params -Wl,--thinlto-prefix-replace='out;lto' \
    -Wl,--thinlto-object-suffix-replace='.indexing.o;.o' out/a.o out/b.indexing.o out/c.indexing.o
  
  # Final link
  clang -fuse-ld=lld @out/a.params out/a.o -o out/exe

--thinlto-index-only= specifies a response file containing all the backend
compilation produced native object files. The final link command line drops all
bitcode files (the build system recognize these files) and inserts `@exe.params`
at the beginning. This reordering may cause different symbol resolution results
for ThinLTO indexing and the final link, which may lead to

- a different --start-lib native object file is picked
- spurious "undefined symbol": say, the different native object file may call a function defined in a bitcode file. The backend compilation for the bitcode file may not export the definition since it doesn't know it is referenced by this native object file.

Link: https://discourse.llvm.org/t/distributed-thinlto-final-linking-order/63804


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D130229

Files:
  lld/ELF/Config.h
  lld/ELF/Driver.cpp
  lld/ELF/InputFiles.cpp
  lld/ELF/LTO.cpp
  lld/ELF/Options.td
  lld/test/ELF/lto/thinlto-index-file.ll
  lld/test/ELF/lto/thinlto-index.ll
  lld/test/ELF/lto/thinlto-single-module.ll
  lld/test/ELF/remapping-file.test

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D130229.446384.patch
Type: text/x-patch
Size: 12116 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20220721/f32437c3/attachment.bin>