[llvm-dev] RFC: Revisiting LLD-as-a-library design

[Andrew Kelley via llvm-dev]

Throwing flowers on behalf of the Zig project 🌻🥀 :)

Currently Zig embeds LLD with a renamed main() and then exposes `zig 
ld.lld`, `zig ld64.lld`, `zig lld-link`, and `zig wasm-ld` which call 
into this renamed main() function. Then when Zig wants to invoke LLD, it 
executes itself as a child process, using one of these sub-commands.

LLD-as-a-library would remove the need for this trick, improving 
performance especially on systems such as Windows which have a high cost 
of spawning child processes.

It also has the possibility to improve error reporting, avoiding the 
need for parsing the stderr of LLD. Errors could be communicated in a 
more semantic way, with less room for bugs.

With our upcoming self-hosted backend, compile times are on order of 
milliseconds, and so child process spawning times in order to invoke LLD 
end up being a substantial portion of total compilation time. That said, 
we are also entering the linking space to provide an alternative to LLD, 
but it will be some time before any possibility of removing a dependency 
on LLD, so this RFC would be greatly beneficial to the Zig project.

Cheers,
Andrew

On 6/10/21 11:14 AM, Reid Kleckner via llvm-dev wrote:
> Hey all,
> 
> Long ago, the LLD project contributors decided that they weren't going 
> to design LLD as a library, which stands in opposition to the way that 
> the rest of LLVM strives to be a reusable library. Part of the reasoning 
> was that, at the time, LLD wasn't done yet, and the top priority was to 
> finish making LLD a fast, useful, usable product. If sacrificing 
> reusability helped LLD achieve its project goals, the contributors at 
> the time felt that was the right tradeoff, and that carried the day.
> 
> However, it is now ${YEAR} 2021, and I think we ought to reconsider this 
> design decision. LLD was a great success: it works, it is fast, it is 
> simple, many users have adopted it, it has many ports 
> (COFF/ELF/mingw/wasm/new MachO). Today, we have actual users who want to 
> run the linker as a library, and they aren't satisfied with the option 
> of launching a child process. Some users are interested in process reuse 
> as a performance optimization, some are including the linker in the 
> frontend. Who knows. I try not to pre-judge any of these efforts, I 
> think we should do what we can to enable experimentation.
> 
> So, concretely, what could change? The main points of reusability are:
> - Fatal errors and warnings exit the process without returning control 
> to the caller
> - Conflicts over global variables between threads
> 
> Error recovery is the big imposition here. To avoid a giant rewrite of 
> all error handling code in LLD, I think we should *avoid* returning 
> failure via the llvm::Error class or std::error_code. We should instead 
> use an approach more like clang, where diagnostics are delivered to a 
> diagnostic consumer on the side. The success of the link is determined 
> by whether any errors were reported. Functions may return a simple 
> success boolean in cases where higher level functions need to exit 
> early. This has worked reasonably well for clang. The main failure mode 
> here is that we miss an error check, and crash or report useless 
> follow-on errors after an error that would normally have been fatal.
> 
> Another motivation for all of this is increasing the use of parallelism 
> in LLD. Emitting errors in parallel from threads and then exiting the 
> process is risky business. A new diagnostic context or consumer could 
> make this more reliable. MLIR has this issue as well, and I believe they 
> use this pattern. They use some kind of thread shard index to order the 
> diagnostics, LLD could do the same.
> 
> Finally, we'd work to eliminate globals. I think this is mainly a small 
> matter of programming (SMOP) and doesn't need much discussion, although 
> the `make` template presents interesting challenges.
> 
> Thoughts? Tomatoes? Flowers? I apologize for the lack of context links 
> to the original discussions. It takes more time than I have to dig those up.
> 
> Reid
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210610/cb716c98/attachment.sig>