[clang] [llvm] [ThinLTO][Split] Split module for parallel compilation in backend (1/N) (PR #198702)

[via cfe-commits]

mmjjpp wrote:

> > This change has no description and no tests - it needs both, and possibly an RFC for a change as big as this. At first glance, a couple of high level comments: LTO should not be directly calling lld. Also, regular LTO already has support for split codegen, can that be leveraged? In particular, look at how that adds the split modules to the link, which is via the AddStream callback.
> 
> Looking at the existing Regular LTO split-codegen path, I think the ThinLTO handling has two distinct cases.
> 
> ### 1. Linker-driven In-process ThinLTO
> Since `lld` is already the LTO client, I agree this should follow the Regular LTO model:
> 
> * The ThinLTO backend should emit each split partition through `AddStream`.
> * `lld` should receive those object buffers from `LTO::run()`, create `InputFiles` from them, and add them to the final link.
> * **Key constraint:** There should be no recursive linker invocation from `LTOBackend` in this path.
> * **Note:** The main extra piece is task numbering / `getMaxTasks()`, since one ThinLTO backend task can produce multiple partition objects.
> 
> ### 2. Distributed ThinLTO Backend Compilation (`clang -fthinlto-index -c`)
> Here, there is no in-process `lld` client receiving `AddStream` buffers. `cc1` is just producing backend objects, while the clang driver still has the normal `-c -o foo.o` contract to produce a single relocatable object.
> 
> **Proposed direction:**
> 
> 1. `cc1` uses `AddStream` to emit the split partition objects and writes an output list.
> 2. The `clang` driver performs a follow-up merge: `ld.lld -r @list -o foo.o`.
> 
> _Benefit:_ This keeps the linker invocation out of the LTO library/backend, while preserving the existing distributed backend single-output contract.
> 
> ### Summary of Responsibilities
> Does that split of responsibilities sound reasonable to you? In other words:
> 
> * **In-process ThinLTO:** `LTO` -> `AddStream` -> `lld` consumes partition objects directly.
> * **Distributed `-fthinlto-index -c`:** `LTO`/`cc1` -> `AddStream` writes partition objects -> `clang` driver merges them with `ld.lld -r`.

pr： https://github.com/mmjjpp/llvm-project/pull/16 [ThinLTO][Driver] Move split-codegen partition merge out of LTOBackend

https://github.com/llvm/llvm-project/pull/198702