[PATCH] D130229: [ELF] Add --thinlto-index= and --remapping-file=

[Fangrui Song via Phabricator via llvm-commits]

MaskRay added a comment.

In D130229#3673200 <https://reviews.llvm.org/D130229#3673200>, @weiwang wrote:

> An example I made to show the different results among local thinlto, thinlto with `--thinlto-index-only` and thinlto with `--thinlto-index`:
>
>   > echo 'int g() { return 0; }' > a.cpp
>   > echo 'int g() { return 1; } void f() { return; }' > b.cpp
>   > echo 'void f(); int g(); int main() { [[clang::always_inline]] f(); [[clang::noinline]] return g(); }' > c.cpp
>   
>   # compile
>   > clang++ -c -flto=thin -O2 c.cpp -o c.o
>   > clang++ -c -flto=thin -O2 b.cpp -o b.o
>   > clang++ -c -O2 a.cpp -o a.o
>   
>   # local thinlto
>   > clang++ -flto=thin -fuse-ld=lld -Wl,--start-lib a.o -Wl,--end-lib -Wl,--start-lib b.o -Wl,--end-lib -Wl,--start-lib c.o -Wl,--end-lib -o local
>   > ./local
>   > echo $?
>   1
>   
>   # dist_thinlto with --thinlto-index-only
>   > clang++ -flto=thin -fuse-ld=lld -Wl,--thinlto-index-only=thinlto.index,--thinlto-emit-imports-files -Wl,--start-lib a.o -Wl,--end-lib -Wl,--start-lib b.o -Wl,--end-lib -Wl,--start-lib c.o -Wl,--end-lib
>   
>   > cat thinlto.index
>   c.o
>   b.o
>   
>   > clang++ -O2 -c -x ir b.o -fthinlto-index=b.o.thinlto.bc -o b.native.o
>   > clang++ -O2 -c -x ir c.o -fthinlto-index=c.o.thinlto.bc -o c.native.o
>   > clang++ -fuse-ld=lld -Wl,--start-lib a.o -Wl,--end-lib c.native.o b.native.o -o dist1
>   ld.lld: error: duplicate symbol: g()
>   >>> defined at a.cpp
>   >>>            a.o:(g())
>   >>> defined at b.cpp
>   >>>            b.native.o:(.text+0x0)
>   
>   # dist_thinlto with --thinlto-index
>   > clang++ -flto=thin -fuse-ld=lld -Wl,--thinlto-index=thinlto.map,--thinlto-emit-imports-files,--thinlto-object-suffix-replace='.o;.native.o' -Wl,--start-lib a.o -Wl,--end-lib -Wl,--start-lib b.o -Wl,--end-lib -Wl,--start-lib c.o -Wl,--end-lib
>   
>   > cat thinlto.map
>   c.o c.native.o
>   b.o b.native.o
>   
>   > clang++ -O2 -c -x ir b.o -fthinlto-index=b.o.thinlto.bc -o b.native.o
>   > clang++ -O2 -c -x ir c.o -fthinlto-index=c.o.thinlto.bc -o c.native.o
>   
>   > clang++ -fuse-ld=lld -Wl,--remapping-file=thinlto.map -Wl,--start-lib a.o -Wl,--end-lib -Wl,--start-lib b.o -Wl,--end-lib -Wl,--start-lib c.o -Wl,--end-lib -o dist2
>   > ./dist2
>   > echo $?
>   0

Thanks for sharing the example. However, I don't consider such an example a reasonable case. I think the --thinlto-index= behavior is ok.

In-process ThinLTO places lto.tmp native object files at the end. This causes sections from bitcode files to be placed after native object files' sections. For certain linker script usage this can be seen as an issue (https://github.com/llvm/llvm-project/issues/38738).
If distributed ThinLTO uses a different order and has a different behavior from in-process ThinLTO, I'll not consider it an issue in distributed ThinLTO.
E.g. --thinlto-index-only= places the backend compiled object files before explicit native input (but after implicit files like `crt1.o`). This will inevitably have differences in some duplicate definition cases.

In the example, a.o is before b.o but a.o doesn't properly shadow b.o. If we consider a.o an interceptor, it fails the requirement "The defined symbols of the interceptor should be a superset of the archive member being shadowed." for reliable archive member extraction: https://maskray.me/blog/2021-06-20-symbol-processing#relocatable-object-file-suppressing-archive-member-extraction


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D130229/new/

https://reviews.llvm.org/D130229