[llvm] [DTLTO][LLVM] Integrated Distributed ThinLTO (DTLTO) (PR #127749)

[via llvm-commits]

kbelochapka wrote:

> This looks reasonable to me. The LTO part might need @teresajohnson 's approval.

@teresajohnson , @MaskRay , @cachemeifyoucan , 

Could you please look at the alternative implementation of DTLTO (with no DTLTO backend and with plugins) and compare it with the DTLTO implementation described in this PR (with out-of-process DTLTO backend and external processes/python scripts) before approving this PR?

Here is the link to this branch:
https://github.com/romanova-ekaterina/llvm-project/pull/new/kromanova/main/integrated-DTLTO-no-backend

It's a fully working DTLTO implementation, although it has been minimized to make it suitable for a code review. We spent considerable time working on this project, but we haven't received any comments/feedback about it.  The link to this branch was posted as a comment in this PR, but might have been lost between hundreds of other comments.


Katya Romanova (@[romanova-ekaterina](https://github.com/romanova-ekaterina)) already described the main ideas behind the alternative implementation and the difference between both implementations in one of the comments in the PR, but I will re-iterate it again: 

(1)	Very little footprint on lld and llvm sources.
(2)	Easy to use, only one command line option to control –distribute=<dbs-name>
(3)	Both regular and thin archive members are fully supported. This contrasts with the DTLTO backend solution, since archive support for DTLTO cannot be implemented within a LTO backend.
(4)	Only COFF and ELF formats are supported in the initial commit. Support for all other formats is implemented, but it will be submitted later.
(5)	Plugins (shared libraries) are used in this implementation rather than external processes/python scripts. Plugin that supports external scripts was implemented but was not included in the initial review. Plugins can be statically linked into a lld executable. We have several plugins implemented that support most popular distributed build systems – DistCC, IceCream, FastBuild, Icredibuild, Re-Client, HomeCC
(6)	Plugins allow direct calls and two-way communication between LLVM and DBS, while external processes require parameter serialization and allow only one-way communication. For example, one if the advantages of using plugins is the ability to pass memory buffers to DBS client and get output back from DBS in memory (which saves 2 writes and 2 reads from the file). Note: We have a plugin prototype implemented for DistCC distribution system which implements direct memory to memory interface with DistCC client. (we can share it on request). Direct memory to memory interface could not be implemented in one-way external process interface.
(7)	Throughput is one of the two main factors that contribute to distribution build system efficiency. Load balancing is the other factor. The Plugin interface could be integrated more closely with a DBS, allowing do more advanced compilation jobs scheduling based on current distributed build system state, which in turn would improve the  load balancing.

https://github.com/llvm/llvm-project/pull/127749