[llvm-dev] RFC: Revisiting LLD-as-a-library design

Thu Jun 10 21:21:06 PDT 2021

> thoughts? Tomatoes? Flowers? I apologize for the lack of context links to the original discussions. It takes more time than I have to dig those up.

+1

> - Fatal errors and warnings exit the process without returning control to the caller

This means every single fatal() call needs scrutiny. The function is
noreturn and there are 147 references.
In many places returning from fatal() can indeed crash.

> Another motivation for all of this is increasing the use of parallelism in LLD. Emitting errors in parallel from threads and then exiting the process is risky business. A new diagnostic context or consumer could make this more reliable. MLIR has this issue as well, and I believe they use this pattern. They use some kind of thread shard index to order the diagnostics, LLD could do the same.

Yes, I remember that I refrained from warn() in some parallel*,
because warn() becomes error() in --fatal-warnings mode and error() is
similar to fatal() after --error-limit is reached.

On Thu, Jun 10, 2021 at 8:57 PM Tom Stellard <tstellar at redhat.com> wrote:
>
> On 6/10/21 3:28 PM, David Blaikie wrote:
> > On Thu, Jun 10, 2021 at 3:16 PM Tom Stellard via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> >
> >     On 6/10/21 11:14 AM, Reid Kleckner via llvm-dev wrote:
> >      > Hey all,
> >      >
> >      > Long ago, the LLD project contributors decided that they weren't going to design LLD as a library, which stands in opposition to the way that the rest of LLVM strives to be a reusable library. Part of the reasoning was that, at the time, LLD wasn't done yet, and the top priority was to finish making LLD a fast, useful, usable product. If sacrificing reusability helped LLD achieve its project goals, the contributors at the time felt that was the right tradeoff, and that carried the day.
> >      >
> >
> >     I think it would be great to move in this directions.  It's a little bit unclear
> >     at the moment whether or not the library use case is supported, because we
> >     do include headers and libraries in the install targets.
> >
> >     As a package maintainer my wish list for an LLD library is:
> >
> >     1. Single shared object: https://reviews.llvm.org/D85278 <https://reviews.llvm.org/D85278>
> >     2. All symbols are assumed private unless explicitly given library visibility.
> >
> >
> > Is this ^ consistent with how other parts of LLVM are handled? My understanding was generally LLVM's API is wide/unbounded and not stable. I'd hesitate to restrict future libraries in some way due to some benefits that provides - ease of refactoring (LLVM's ability to be changed frequently is very valuable).
> >
>
> The single shared object is consistent with clang and llvm, but
> not the private symbols by default.  We have discussed changing
> this in clang and llvm, though, and for a library like lld that is
> smaller than the clang and llvm libraries, it seems like it would
> be an easier task and something that would be useful to do right
> from the start.

I support that we compile lld source files with -fvisibility=hidden on
ELF systems.

lld::*::link may be the only API which need LLVM_EXTERNAL_VISIBILITY
(i.e. default visibility for ELF)
(LLVM_EXTERNAL_VISIBILITY is defined in llvm/include/llvm/Support/Compiler.h
Windows doesn't customize it currently.)

> -Tom
>
> >     3. Some subset of the API that is stable across major releases.
> >
> >
> > A limited stable C API seems plausible to me, if there's need.
> >
> > - Dave
> >
> >
> >     -Tom
> >
> >
> >      > However, it is now ${YEAR} 2021, and I think we ought to reconsider this design decision. LLD was a great success: it works, it is fast, it is simple, many users have adopted it, it has many ports (COFF/ELF/mingw/wasm/new MachO). Today, we have actual users who want to run the linker as a library, and they aren't satisfied with the option of launching a child process. Some users are interested in process reuse as a performance optimization, some are including the linker in the frontend. Who knows. I try not to pre-judge any of these efforts, I think we should do what we can to enable experimentation.
> >      >
> >      > So, concretely, what could change? The main points of reusability are:
> >      > - Fatal errors and warnings exit the process without returning control to the caller
> >      > - Conflicts over global variables between threads
> >      >
> >      > Error recovery is the big imposition here. To avoid a giant rewrite of all error handling code in LLD, I think we should *avoid* returning failure via the llvm::Error class or std::error_code. We should instead use an approach more like clang, where diagnostics are delivered to a diagnostic consumer on the side. The success of the link is determined by whether any errors were reported. Functions may return a simple success boolean in cases where higher level functions need to exit early. This has worked reasonably well for clang. The main failure mode here is that we miss an error check, and crash or report useless follow-on errors after an error that would normally have been fatal.
> >      >
> >      > Another motivation for all of this is increasing the use of parallelism in LLD. Emitting errors in parallel from threads and then exiting the process is risky business. A new diagnostic context or consumer could make this more reliable. MLIR has this issue as well, and I believe they use this pattern. They use some kind of thread shard index to order the diagnostics, LLD could do the same.
> >      >
> >      > Finally, we'd work to eliminate globals. I think this is mainly a small matter of programming (SMOP) and doesn't need much discussion, although the `make` template presents interesting challenges.
> >      >
> >      > Thoughts? Tomatoes? Flowers? I apologize for the lack of context links to the original discussions. It takes more time than I have to dig those up.
> >      >
> >      > Reid
> >      >
> >      > _______________________________________________
> >      > LLVM Developers mailing list
> >      > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> >      > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> >      >
> >
> >     _______________________________________________
> >     LLVM Developers mailing list
> >     llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> >     https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> >
>

-- 
宋方睿