[llvm-dev] [RFC] Modernize CMake LLVM "Components"/libLLVM Facility

Sun Jan 3 22:58:22 PST 2021

On Sun, Jan 3, 2021 at 8:46 PM Mehdi AMINI <joker.eph at gmail.com> wrote:

>
>
> On Sun, Jan 3, 2021 at 1:50 PM Stella Laurenzo via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi folks, happy new year!
>>
>> *Proposal:*
>>
>>    - See comments at the top of LLVMComponents.cmake
>>    <https://github.com/stellaraccident/llvm-project/blob/newcomponents/llvm/cmake/modules/LLVMComponents.cmake>
>>    in my fork
>>    <https://github.com/stellaraccident/llvm-project/tree/newcomponents>.
>>    - Draft phab: https://reviews.llvm.org/D94000
>>
>>
>> *Background:*
>> As I've been working on NPCOMP <https://github.com/llvm/mlir-npcomp> trying
>> to come up with a release flow for MLIR derived Python projects (see
>> py-mlir-release <https://github.com/stellaraccident/mlir-py-release>),
>> I've repeatedly run into issues with how the LLVM build system generates
>> shared libraries. While the problems have been varied, I pattern match most
>> of them to a certain "pragmatic" nature to how components/libLLVM/libMLIR
>> have come to be: in my experience, you can fix most individual dynamic
>> linkage issues with another work-around, but the need for this tends to be
>> rooted in a lack of definition and structure to the libraries themselves,
>> causing various kinds of problems and scenarios that don't arise if
>> developed to stricter standards. (This isn't a knock on anyone -- I know
>> how these things tend to grow. My main observation is that I think we have
>> outgrown the ad-hoc nature of shared libraries in the LLVM build now).
>>
>> I think I'm hitting this because reasonable Python projects and releases
>> pre-supposes a robust dynamic linkage story. Also, I use Windows and am
>> very aware that LLVM basically does not support dynamic linking on Windows
>> -- and cannot without more structure (and in my experience, this structure
>> would also benefit the robustness of dynamic linking on the others).
>>
>> Several of us got together to discuss this in November
>> <https://llvm.discourse.group/t/meeting-notes-mlir-build-install-and-shared-libraries/2257>.
>> We generally agreed that BUILD_SHARED_LIBS was closer to what we wanted vs
>> libLLVM/libMLIR, but the result is really only factored for development
>> (i.e. not every add_library should result in a shared object -- the shared
>> library surface should mirror public interface boundaries and add_library
>> mirrors private boundaries). The primary difference between the two is:
>>
>>    - BUILD_SHARED_LIBS preserves the invariant that every translation
>>    unit will be "homed" in one library at link time (either .so/.dll or .a)
>>    and the system will never try to link together shared and static
>>    dependencies of the same thing (which is what libLLVM/libMLIR do today). It
>>    turns out that this is merely a good idea on most platforms but is the core
>>    requirement on native Windows (leaving out mingw, which uses some clever
>>    and dirty tricks to try to blend the worlds).
>>    - LLVM_BUILD_LLVM_DYLIB treats libLLVM.so as a "bucket" to throw
>>    things that might benefit from shared linkage, but end binaries end up also
>>    needing to link against the static libraries in case if what you want isn't
>>    in libLLVM.so. When this is done just right, it can work (on Unix) but it
>>    is very fragile and prone to multiple definition and other linkage issues
>>    that can be extremely hard to track down.
>>
>> *What I did:*
>>
>>    1. Well, first, I tried looking the other way for a few months and
>>    hoping someone else would fix it :)
>>    2. When I started trying to generalize some of the shared library
>>    handling for MLIR and NPCOMP, I noted that the LLVM_LINK_COMPONENTS (as in
>>    named groups of things) are in the right direction of having a structure to
>>    the libraries, and I found that I could actually rebase all of what the
>>    LLVM_LINK_COMPONENTS was trying to do on the same facility, relegating the
>>    existing LLVM_LINK_COMPONENTS to a name normalization layer on top of a
>>    more generic "LLVM Components" facility that enforces stricter layering and
>>    more control than the old libLLVM.so facility did.
>>    3. I rewrote it twice to progressively more modern CMake and was able
>>    to eliminate all of the ad-hoc dependency tracking in favor of
>>    straight-forward use of INTERFACE libraries and $<TARGET_PROPERTY>
>>    generator expressions for selecting static or dynamic component trees based
>>    on global flags and the presence (or absence) of per-executable
>>    LLVM_LINK_STATIC properties
>>       1. Note that since this is rooted only in CMake features and not
>>       LLVM macros, out of tree, non-LLVM projects should be able to depend on
>>       LLVM components in their own targets.
>>    4. I hacked up AddLLVM/LLVM-Build/LLVM-Config to (mostly) use the new
>>    facility (leaving out a few things that can be fixed but aren't conceptual
>>    issues), applied a bunch of fixes to the tree that were revealed by
>>    stricter checks and got all related tests passing for LLVM and MLIR (on X86
>>    -- some mechanical changes need to be made to other targets) for both
>>    dynamic and static builds.
>>
>> *What I'd like to do:*
>>
>>    - Get some consensus that we'd like to improve things in this area
>>    and that the approach I'm taking makes sense. I can do a lot of the work,
>>    but I don't want to waste my time, and this stuff is fragile if we keep it
>>    in an intermediate state for too long (I'm already paying this price
>>    downstream).
>>    - Land LLVMComponents.cmake
>>    <https://github.com/stellaraccident/llvm-project/blob/newcomponents/llvm/cmake/modules/LLVMComponents.cmake>
>>    as the basis of the new facility.
>>    - Finish implementing the "Redirection" feature that would allow us
>>    to emulate an aggregate libLLVM as it is today.
>>    - Start pre-staging the various stricter constraints to the build
>>    tree that will be needed to swap AddLLVM to use the new facility.
>>    - Rewrite component-related AddLLVM/LLVM-Build/LLVM-Config bits in a
>>    more principled way to use the new facility (or remove features entirely
>>    that are no longer needed) -- what I did in the above patch was just a
>>    minimal amount of working around for a POC.
>>    - Agree on whether we should try to have the two co-exist for a time
>>    or do a more clean break with the old.
>>    - Start applying the facility to downstream projects like MLIR and
>>    NPCOMP.
>>
>> *What I would need:*
>>
>>    - Help, testing and expertise. I am reasonably confident in my
>>    understanding of how to make shared libraries work and how to use CMake,
>>    but the legacy in LLVM here is deep -- I likely pattern matched some old
>>    features as no longer needed when they actually are (I am not clear at all
>>    on how much of LLVM-Config is still relevant).
>>    - Pointers to who the stakeholders are that I should be coordinating
>>    with.
>>
>> Comments?
>>
>
> Looks great! In particular it is interesting to see how more modern CMake
> features could replace some of the custom-LLVM CMake macros that are likely
> almost a decade old now.
>
> One thing I wonder about trying to see BUILD_SHARED_LIBS as some desirable
> for a production environment: I seem to remember that there were
> non-trivial performance regression when using many .so instead of a single
> libLLVM.so (even a single libLLVM.so was showing a measurable performance
> impact for clang IIRC).
>

It's a good question, and one for which any hard data I can recall is
hopelessly out of date. I know that back in x86 32bit days, there were
non-trivial costs related to PIC and various data access indirections that
were induced, but my mental model has these costs as having been very
reduced/eliminated on x64 (and not a factor for others). On Windows, I
know, it is very easy to end up with extra call and data access
indirections unless if export/import are paired properly so that the
compiler can eliminate them.

In the modern era, aside from fixed startup costs (which are obviously
higher for shared libraries), I suspect that the biggest performance
impacts will come from missed optimizations and export bloat. It is my
understanding that a primary offender on that front is exporting everything
with default visibility causing too many should-have-been-internal entry
points to be fully materialized as black boxes and the corresponding
decreased scope of any cross module optimizations. As an example, the X86
component aggregates 5 different libraries. In a BUILD_SHARED_LIBS setup,
this would be 5 shared objects that expose a lot of internal symbols and
potential for indirection. Since Targets are compiled with
visibility=hidden, I did measure the impact of that. If IIRC, for a
stripped libX86.so with visibility=hidden, the size was about 10MiB, and it
was >11MiB for visibility=default. I did not measure the performance
impact, but considering even just dynamic-link-time, an estimated 10%
increase in the exports of one large target is going to have a measurable
impact on a short lived process like clang. The finer granularity of your
shared objects, the more unavoidable exported symbol bloat you are going to
have (in fact, there is a special carve-out in the Targets to export
everything for BUILD_SHARED_LIBS mode because it doesn't work otherwise).
The coarser your shared objects, the more fixed startup overheads you will
incur.

To be clear, what I am proposing is a facility that will let us break the
shared-library granularity down to the component level, but I expect most
distributions will elect some coarser granularity (up to what libLLVM.so is
today). For the Python MLIR distribution, I probably want something like
(Support, Core, Per-Target, and OrcJit) libraries. Aside from making for
potentially smaller packages that can omit components, lazy loading some of
that stands to avoid startup costs of massive shared libraries. I suspect
that the sweet spot, performance wise is more fine-grained than libLLVM is
today and more coarse grained than BUILD_SHARED_LIBS allows (and it will
vary depending on whether the user biases towards modularity over raw
size/performance).

>
> --
> Mehdi
>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210103/79696c12/attachment.html>