[llvm-dev] [RFC] LLVM Directory Structure Changes (was Re: [PATCH] D20992: [CMake] Add LLVM runtimes directory)

[Renato Golin via llvm-dev]

On 14 June 2016 at 17:40, Chris Bieneman <beanz at apple.com> wrote:
> The problem comes from that fact that generally builds are structured as configure -> build. What we actually need is "configure some" -> "build some" -> "configure some more" -> "build some more" -> repeat until done.

Well, GCC has been doing this for years, and I have just setup a
simpler version of it here:

http://buildmaster.tcwglab.linaro.org/builders/clang-cmake-aarch64-prototype

To avoid the partial configuration problem, you can build in multiple
stages and increment the level of support in each one until you have a
complete toolchain in the end.


> For our projects the first step of a bootstrap is building a clang that runs on host supporting the target. After building that clang you need to build the runtime libraries. For sake of simplicity let's assume we're only building compiler-rt's current libraries. After building clang the logical step would be CMake + Ninja for compiler-rt then you're done right? Except, that doesn't work. You can't *configure* the sanitizer build until after you've fully configured and built the builtin libraries.

Another reason to move the sanitizers to their own repo.


> If you configure a project before the tools and libraries are available your configuration system may make the wrong decisions. To put it in more normal compiler terminology, it is undefined behavior.

So, IIUC, the GCC guys have worked around that problem by defining the
environment with more rigour that we do. It *is* different to build
LLVM with glibc X instead of X+1, and the same is true for all
dependencies.

There isn't a strong reason why we can't do the same, we just don't do
it because people want to build LLVM in very different architectures.

So, a simpler proposal would be to have a long matrix of tools we need
versus OS we build on and mark the accepted range (or single version)
that you need to have to build it on that particular platform.

Different people might have different toolsets, and that's fine. We
can easily have an entry for Ubuntu and another for Suse, or even one
to Ubuntu X and another for Ubuntu X+1.

Other than doing that, you can't rely on your first stage Clang+LLVM
only build if you have no idea which system compiler was used anyway.
Differences will dilute over the later stages, and ultimately, if you
build enough of them, they'll end up virtually identical. But
validation on those terms become very costly.


> We have viable workarounds for this in our build today. We can extend these workarounds to make bootstrapping simpler. There are some rough edges that are caused by limitations in CMake, but there is no fundamental limitation in CMake that makes this not work.

This is good news! :)


> At Apple we workaround this by having some out-of-tree script goop that configures and builds clang, then configures and builds the builtins, then configures and builds the runtimes. Due to complications interfacing with our internal build system, we'll likely always have that script goop, but I'd really like to be able to run clean bootstrap builds against open source without the crazy complications.

That'd probably help other downstream users as well.


> I believe that splitting compiler-rt's sanitizer libraries and builtin archives into separate projects will allow the project structure to more clearly represent the way it fits into the project build graph. It is not strictly necessary for anything, so if people generally don't agree I won't fight for it.

I think my general feeling of this thread is that most people agree
with the idea. I personally do.

+1 from me.

cheers,
--renato