[llvm-dev] [RFC] LLVM Directory Structure Changes (was Re: [PATCH] D20992: [CMake] Add LLVM runtimes directory)

[Chris Bieneman via llvm-dev]

> On Jun 14, 2016, at 10:19 AM, Renato Golin <renato.golin at linaro.org> wrote:
> 
>> On 14 June 2016 at 17:40, Chris Bieneman <beanz at apple.com> wrote:
>> The problem comes from that fact that generally builds are structured as configure -> build. What we actually need is "configure some" -> "build some" -> "configure some more" -> "build some more" -> repeat until done.
> 
> Well, GCC has been doing this for years, and I have just setup a
> simpler version of it here:
> 
> http://buildmaster.tcwglab.linaro.org/builders/clang-cmake-aarch64-prototype
> 
> To avoid the partial configuration problem, you can build in multiple
> stages and increment the level of support in each one until you have a
> complete toolchain in the end.

I'm not really familiar with GCC's build, but I assume it isn't too conceptually different from what we need to be doing. The biggest difference is that we're using CMake.

> 
> 
>> For our projects the first step of a bootstrap is building a clang that runs on host supporting the target. After building that clang you need to build the runtime libraries. For sake of simplicity let's assume we're only building compiler-rt's current libraries. After building clang the logical step would be CMake + Ninja for compiler-rt then you're done right? Except, that doesn't work. You can't *configure* the sanitizer build until after you've fully configured and built the builtin libraries.
> 
> Another reason to move the sanitizers to their own repo.
> 
> 
>> If you configure a project before the tools and libraries are available your configuration system may make the wrong decisions. To put it in more normal compiler terminology, it is undefined behavior.
> 
> So, IIUC, the GCC guys have worked around that problem by defining the
> environment with more rigour that we do. It *is* different to build
> LLVM with glibc X instead of X+1, and the same is true for all
> dependencies.
> 
> There isn't a strong reason why we can't do the same, we just don't do
> it because people want to build LLVM in very different architectures.
> 
> So, a simpler proposal would be to have a long matrix of tools we need
> versus OS we build on and mark the accepted range (or single version)
> that you need to have to build it on that particular platform.
> 
> Different people might have different toolsets, and that's fine. We
> can easily have an entry for Ubuntu and another for Suse, or even one
> to Ubuntu X and another for Ubuntu X+1.
> 
> Other than doing that, you can't rely on your first stage Clang+LLVM
> only build if you have no idea which system compiler was used anyway.
> Differences will dilute over the later stages, and ultimately, if you
> build enough of them, they'll end up virtually identical. But
> validation on those terms become very costly.

We actually have a validation that the second and third clang builds are identical running on a bot (lab.llvm.org:8011/builders/clang-3stage-ubuntu).

Mike Edwards set that up a while back, and it is really great.

In Apple's release process for clang we always build clang twice to ensure that the compiler we're shipping is effectively built with itself.

-Chris

> 
> 
>> We have viable workarounds for this in our build today. We can extend these workarounds to make bootstrapping simpler. There are some rough edges that are caused by limitations in CMake, but there is no fundamental limitation in CMake that makes this not work.
> 
> This is good news! :)
> 
> 
>> At Apple we workaround this by having some out-of-tree script goop that configures and builds clang, then configures and builds the builtins, then configures and builds the runtimes. Due to complications interfacing with our internal build system, we'll likely always have that script goop, but I'd really like to be able to run clean bootstrap builds against open source without the crazy complications.
> 
> That'd probably help other downstream users as well.
> 
> 
>> I believe that splitting compiler-rt's sanitizer libraries and builtin archives into separate projects will allow the project structure to more clearly represent the way it fits into the project build graph. It is not strictly necessary for anything, so if people generally don't agree I won't fight for it.
> 
> I think my general feeling of this thread is that most people agree
> with the idea. I personally do.
> 
> +1 from me.
> 
> cheers,
> --renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160614/972a2783/attachment.html>