[llvm-dev] [RFC] Modernize CMake LLVM "Components"/libLLVM Facility

[Stella Laurenzo via llvm-dev]

On Mon, Jan 4, 2021 at 6:06 PM Michael Kruse <llvmdev at meinersbur.de> wrote:

> Thank you for the proposal. Would you also consider unifying the
> library handling of LLVM and clang (and potentially other subprojects)
> as well? I found the differences always annoying, e.g. there is no
> equivalent to llvm-config in clang, or clangXYZ/MLIRxyz  libraries
> creating object libraries under some conditions (e.g. to create static
> as well as shared libraries in the same build), but LLVM itself not
> doing that.
>
> Michael
>

It would be great if some of the approaches we use in the core llvm/
project help unify things in the others! While I don't have the bandwidth
to go down all of those rabbit holes (I'm responsible for three downstream
projects myself), I would be happy to work with folks on the clang side to
help improve the situation once we get the core a bit more upgraded in this
fashion.


>
> Am So., 3. Jan. 2021 um 15:50 Uhr schrieb Stella Laurenzo via llvm-dev
> <llvm-dev at lists.llvm.org>:
> >
> > Hi folks, happy new year!
> >
> > Proposal:
> >
> > See comments at the top of LLVMComponents.cmake in my fork.
> > Draft phab: https://reviews.llvm.org/D94000
> >
> >
> > Background:
> > As I've been working on NPCOMP trying to come up with a release flow for
> MLIR derived Python projects (see py-mlir-release), I've repeatedly run
> into issues with how the LLVM build system generates shared libraries.
> While the problems have been varied, I pattern match most of them to a
> certain "pragmatic" nature to how components/libLLVM/libMLIR have come to
> be: in my experience, you can fix most individual dynamic linkage issues
> with another work-around, but the need for this tends to be rooted in a
> lack of definition and structure to the libraries themselves, causing
> various kinds of problems and scenarios that don't arise if developed to
> stricter standards. (This isn't a knock on anyone -- I know how these
> things tend to grow. My main observation is that I think we have outgrown
> the ad-hoc nature of shared libraries in the LLVM build now).
> >
> > I think I'm hitting this because reasonable Python projects and releases
> pre-supposes a robust dynamic linkage story. Also, I use Windows and am
> very aware that LLVM basically does not support dynamic linking on Windows
> -- and cannot without more structure (and in my experience, this structure
> would also benefit the robustness of dynamic linking on the others).
> >
> > Several of us got together to discuss this in November. We generally
> agreed that BUILD_SHARED_LIBS was closer to what we wanted vs
> libLLVM/libMLIR, but the result is really only factored for development
> (i.e. not every add_library should result in a shared object -- the shared
> library surface should mirror public interface boundaries and add_library
> mirrors private boundaries). The primary difference between the two is:
> >
> > BUILD_SHARED_LIBS preserves the invariant that every translation unit
> will be "homed" in one library at link time (either .so/.dll or .a) and the
> system will never try to link together shared and static dependencies of
> the same thing (which is what libLLVM/libMLIR do today). It turns out that
> this is merely a good idea on most platforms but is the core requirement on
> native Windows (leaving out mingw, which uses some clever and dirty tricks
> to try to blend the worlds).
> > LLVM_BUILD_LLVM_DYLIB treats libLLVM.so as a "bucket" to throw things
> that might benefit from shared linkage, but end binaries end up also
> needing to link against the static libraries in case if what you want isn't
> in libLLVM.so. When this is done just right, it can work (on Unix) but it
> is very fragile and prone to multiple definition and other linkage issues
> that can be extremely hard to track down.
> >
> > What I did:
> >
> > Well, first, I tried looking the other way for a few months and hoping
> someone else would fix it :)
> > When I started trying to generalize some of the shared library handling
> for MLIR and NPCOMP, I noted that the LLVM_LINK_COMPONENTS (as in named
> groups of things) are in the right direction of having a structure to the
> libraries, and I found that I could actually rebase all of what the
> LLVM_LINK_COMPONENTS was trying to do on the same facility, relegating the
> existing LLVM_LINK_COMPONENTS to a name normalization layer on top of a
> more generic "LLVM Components" facility that enforces stricter layering and
> more control than the old libLLVM.so facility did.
> > I rewrote it twice to progressively more modern CMake and was able to
> eliminate all of the ad-hoc dependency tracking in favor of
> straight-forward use of INTERFACE libraries and $<TARGET_PROPERTY>
> generator expressions for selecting static or dynamic component trees based
> on global flags and the presence (or absence) of per-executable
> LLVM_LINK_STATIC properties
> >
> > Note that since this is rooted only in CMake features and not LLVM
> macros, out of tree, non-LLVM projects should be able to depend on LLVM
> components in their own targets.
> >
> > I hacked up AddLLVM/LLVM-Build/LLVM-Config to (mostly) use the new
> facility (leaving out a few things that can be fixed but aren't conceptual
> issues), applied a bunch of fixes to the tree that were revealed by
> stricter checks and got all related tests passing for LLVM and MLIR (on X86
> -- some mechanical changes need to be made to other targets) for both
> dynamic and static builds.
> >
> > What I'd like to do:
> >
> > Get some consensus that we'd like to improve things in this area and
> that the approach I'm taking makes sense. I can do a lot of the work, but I
> don't want to waste my time, and this stuff is fragile if we keep it in an
> intermediate state for too long (I'm already paying this price downstream).
> > Land LLVMComponents.cmake as the basis of the new facility.
> > Finish implementing the "Redirection" feature that would allow us to
> emulate an aggregate libLLVM as it is today.
> > Start pre-staging the various stricter constraints to the build tree
> that will be needed to swap AddLLVM to use the new facility.
> > Rewrite component-related AddLLVM/LLVM-Build/LLVM-Config bits in a more
> principled way to use the new facility (or remove features entirely that
> are no longer needed) -- what I did in the above patch was just a minimal
> amount of working around for a POC.
> > Agree on whether we should try to have the two co-exist for a time or do
> a more clean break with the old.
> > Start applying the facility to downstream projects like MLIR and NPCOMP.
> >
> > What I would need:
> >
> > Help, testing and expertise. I am reasonably confident in my
> understanding of how to make shared libraries work and how to use CMake,
> but the legacy in LLVM here is deep -- I likely pattern matched some old
> features as no longer needed when they actually are (I am not clear at all
> on how much of LLVM-Config is still relevant).
> > Pointers to who the stakeholders are that I should be coordinating with.
> >
> > Comments?
> >
> > Thanks!
> > - Stella
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210104/f84bf6c3/attachment.html>