[llvm-dev] RFC: Revisiting LLD-as-a-library design

[Petr Hosek via llvm-dev]

Sure, I'm thinking about porting LLVM to Fuchsia and our process creation
is not (and might never be) as fast as on *NIX, it's actually closer to
Windows, so I'm interested in integrating LLD into Clang to reduce the cost
(similarly to what Zig does).

More generally, I'd also like to explore alternative models like
llvm-buildozer <https://reviews.llvm.org/D86351> or a client-server model
similar to what Daniel described in his talk
<https://www.youtube.com/watch?v=b_T-eCToX1I>, and having LLD-as-a-library
makes that easier.

On Thu, Jun 10, 2021 at 12:00 PM David Blaikie <dblaikie at gmail.com> wrote:

> +1 generally.
>
> Petr - if you have particular context for your use case, might be handy to
> have as reference/motivation.
> Reid - if you have any particular use case of your own in mind, or links
> to other discussions/users who are having friction with the current state
> of affairs, would be hand to have.
>
> (though I don't generally want the thread to become about picking apart
> those use cases - library-based design and flexibility is a fairly core
> tenant of the LLVM project overall regardless of the validity of specific
> use cases)
>
> On Thu, Jun 10, 2021 at 11:27 AM Petr Hosek via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> A big +1 from our side since we have a potential use case for
>> LLD-as-a-library (I was going to write a similar RFC but you beat me to it).
>>
>> On Thu, Jun 10, 2021 at 11:15 AM Reid Kleckner via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Hey all,
>>>
>>> Long ago, the LLD project contributors decided that they weren't going
>>> to design LLD as a library, which stands in opposition to the way that the
>>> rest of LLVM strives to be a reusable library. Part of the reasoning was
>>> that, at the time, LLD wasn't done yet, and the top priority was to finish
>>> making LLD a fast, useful, usable product. If sacrificing reusability
>>> helped LLD achieve its project goals, the contributors at the time felt
>>> that was the right tradeoff, and that carried the day.
>>>
>>> However, it is now ${YEAR} 2021, and I think we ought to reconsider this
>>> design decision. LLD was a great success: it works, it is fast, it is
>>> simple, many users have adopted it, it has many ports
>>> (COFF/ELF/mingw/wasm/new MachO). Today, we have actual users who want to
>>> run the linker as a library, and they aren't satisfied with the option of
>>> launching a child process. Some users are interested in process reuse as a
>>> performance optimization, some are including the linker in the frontend.
>>> Who knows. I try not to pre-judge any of these efforts, I think we should
>>> do what we can to enable experimentation.
>>>
>>> So, concretely, what could change? The main points of reusability are:
>>> - Fatal errors and warnings exit the process without returning control
>>> to the caller
>>> - Conflicts over global variables between threads
>>>
>>> Error recovery is the big imposition here. To avoid a giant rewrite of
>>> all error handling code in LLD, I think we should *avoid* returning failure
>>> via the llvm::Error class or std::error_code. We should instead use an
>>> approach more like clang, where diagnostics are delivered to a diagnostic
>>> consumer on the side. The success of the link is determined by whether any
>>> errors were reported. Functions may return a simple success boolean in
>>> cases where higher level functions need to exit early. This has worked
>>> reasonably well for clang. The main failure mode here is that we miss an
>>> error check, and crash or report useless follow-on errors after an error
>>> that would normally have been fatal.
>>>
>>> Another motivation for all of this is increasing the use of parallelism
>>> in LLD. Emitting errors in parallel from threads and then exiting the
>>> process is risky business. A new diagnostic context or consumer could make
>>> this more reliable. MLIR has this issue as well, and I believe they use
>>> this pattern. They use some kind of thread shard index to order the
>>> diagnostics, LLD could do the same.
>>>
>>> Finally, we'd work to eliminate globals. I think this is mainly a small
>>> matter of programming (SMOP) and doesn't need much discussion, although the
>>> `make` template presents interesting challenges.
>>>
>>> Thoughts? Tomatoes? Flowers? I apologize for the lack of context links
>>> to the original discussions. It takes more time than I have to dig those up.
>>>
>>> Reid
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210611/dfb4c257/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3996 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210611/dfb4c257/attachment.bin>