[PATCH] D144596: Add extra parameter to thinlto-prefix-replace

[Ivan Tadeu Ferreira Antunes Filho via Phabricator via llvm-commits]

itf added a comment.

In D144596#4151498 <https://reviews.llvm.org/D144596#4151498>, @MaskRay wrote:

> In D144596#4151481 <https://reviews.llvm.org/D144596#4151481>, @itf wrote:
>
>>> Can Bazel rewrite `a.rsp` with content `xxx/a.o` to `xxx/random_hash/a.o`? It will create a new file but I don't think that's a bottleneck.
>>
>> Currently there is no code in bazel, that reads and modifies a parameter file like `a.rsp`. It is possible to do so by, for example, calling a shell script / binary executable that modifies the file and saves a new one as one of the actions (like so: https://github.com/bazelbuild/examples/tree/main/rules/actions_run).
>>
>> However, I'd argue that it is better for `lld` to have this extra argument. The link objects `.o` are outputs of actions that can be run in parallel, while the `thinlto.bc` and the `.import` files are a result of the thinlto indexing action which run in series, i.e. all the `thinlto.bc` are generated simultaneously.  I find it unexpected that `lld` would force those files to be in the same folders, given that `thinlto-index-only=` is particularly useful for distributed thinLTO, where there could be restrictions on what can be saved in each location
>
> I am not convinced. For `--thinlto-prefix-replace=';lto/;unique_prefix_for_rsp/'`, the output bitcode files and import files are in `lto/` while the response file uses a different prefix.
> This doesn't look like a reasonable build system request for the linker. Why can't Bazel place output bitcode files and import files under `unique_prefix_for_rsp/` as well?

Bazel could output bitcode files and import files under `unique_prefix_for_rsp/` as well, the issue is the linker enforcing that the object file generated in the next step must also be in this same folder.

> I prefer a generic solution for more build systems, not just a feature tuned for some strange aspects of a build system.

The original `-thinlto-prefix-replace` D64542 <https://reviews.llvm.org/D64542>  is generic because it allows the source files and the index files to be stored in different folders. `thinlto-index-only=` forces index files and the object files to be stored in the same folder.  This would allow index-files and object files to be stored in different folders.

I believed being able to store the results of different compilation steps in different folders was a common aspect of build systems, and often required when running a build in a distributed way.

Modifying your example, currently:

  clang -flto=thin -fuse-ld=lld -Wl,--thinlto-index-only=a.rsp -Wl,--thinlto-prefix-replace=';lto/' elf0.o a.o -Wl,--start-lib b.o c.o -Wl,--end-lib elf1.o
  clang -c -O2 -fthinlto-index=lto/a.o.thinlto.bc a.o -o otherfolder/a.o
  ...

Is invalid (would require an intermediate step `mv otherfolder/a.o lto/`), but

  clang -flto=thin -fuse-ld=lld -Wl,--thinlto-index-only=a.rsp -Wl,--thinlto-prefix-replace=';lto/;otherfolder/' elf0.o a.o -Wl,--start-lib b.o c.o -Wl,--end-lib elf1.o
  clang -c -O2 -fthinlto-index=lto/a.o.thinlto.bc a.o -o otherfolder/a.o
  ...

would become valid.


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D144596/new/

https://reviews.llvm.org/D144596