[cfe-dev] CMake builld system for compiler-rt rework

[Alexey Samsonov]

On Tue, Oct 30, 2012 at 3:58 PM, Evgeniy Stepanov <eugenis at google.com>wrote:

>
>
>
> On Tue, Oct 30, 2012 at 2:03 PM, Alexey Samsonov <samsonov at google.com>wrote:
>
>> +cfe-dev
>>
>>
>> On Tue, Oct 30, 2012 at 1:50 PM, Alexey Samsonov <samsonov at google.com>wrote:
>>
>>> Hi there!
>>>
>>> tl;dr we should build compiler-rt with just-built clang
>>>
>>> A number of issues with CMake build of compiler-rt libraries and tests
>>> are caused by the fact that we build compiler-rt with host compiler
>>> instead of
>>> just-built Clang (examples: http://llvm.org/bugs/show_bug.cgi?id=13864,
>>> http://llvm.org/bugs/show_bug.cgi?id=14109)
>>>
>>> Using Clang looks like the Right thing to do, and it's probably time to
>>> make this happen:
>>> 1) configure+make build system uses fresh Clang
>>> 2) using just one fresh Clang will allow us to drop support for all
>>> kinds of host compilers users
>>> are building compiler-rt with:
>>>   * we will be able to build compiler-rt with -Werror
>>>   * we will be able to build versions of compiler-rt libraries for all
>>> target triples supported by Clang (not by host compiler)
>>>   * we will be able to run compiler-rt tests, regardless of host
>>> compiler. For example, my gcc-4.4 build of compiler-rt
>>>     fails on "check-ubsan" tests because of lacking 128-bit int support.
>>>
>>> OK, how can we build with fresh Clang? (One can't simply tell CMake to
>>> build some targets by a compiler produced
>>> while building another target).
>>>
>>> Chandler (and other developers) frequently mention the bootstrap process
>>> (external to CMake), that (IIUC) build Clang
>>> and then uses it to re-build LLVM[+Clang?]+compiler-rt.
>>> Can we:
>>> 1) *always* build compiler-rt as a part of bootstrap process (run CMake
>>> to build clang, then run it again with Clang as a C compiler to build
>>> compiler-rt).
>>> 2) document the usage.
>>> 3) make a script that would run bootstrap process in (1), so that the
>>> users can invoke a single command instead of running a sequence of
>>> instructions?
>>>
>>> So, when CMake is called with Clang compiler, it can run a set of
>>> try_compile() tests to determine possible target triples and build
>>> necessary compiler-rt
>>> libraries for all these triples. But I'm afraid it's *not* that easy:
>>> 1) if a compiler-rt library ever needs to use LLVM libraries, it needs
>>> to have these LLVM libs built for the same target. That is, in general we
>>> may require
>>> a bunch of LLVM libraries compiled for different targets, which is not
>>> supported by LLVM build system at all.
>>> 2) part (1) is already true for compiler-rt unit tests: there is no way
>>> we can build 64-bit and 32-bit version of the same unit test in a single
>>> build tree, as we'd
>>> need two different versions of googletest and LLVMSupport.
>>> 3) Notable case is Android (eugenis@ may comment more on this) -
>>> building for Android requires many changes and is hardly possible in the
>>> same
>>> build tree we use for regular (say, x86_64-linux) build tree.
>>>
>>
> The difference is you can not get away with simply changing CFLAGS (-m32
> vs -m64) in this case. You need to add quite a few other flags, and, more
> often than not, host compiler can not target Android at all. So, while we
> can do some hacks to build both x86_32 and x86_64 runtime libraries in the
> same build tree (with the host compiler), we really need a separate tree,
> and a different compiler, for Android targets.
>
> Recursive trees (below) seem like the only option.
>
>
>>
>>> We can try to resolve this by using recursive build trees:
>>> * for each target create its own "recursive" LLVM build tree (e.g.
>>> projects/compiler-rt/Linux-x86_64) and invoke CMake
>>> there with some additional compile/link flags.
>>> * describe a single target (say, "clang_rt") in a CMakeLists.txt in each
>>> target-specific recursive build tree.
>>> * Build rule for "clang_rt.linux-x86_64" in main build tree in  will
>>> simply invoke building "clang_rt" in necessary
>>> target-specific subtree and copy result from there.
>>>
>>> Overall, I believe this is doable (but pretty ugly) and have a raw
>>> working prototype for this. If we decide to go this way,
>>> I'm ready to complete this if anyone signs up for a careful review (as
>>> my CMake skills are far from good).
>>>
>>> Problems:
>>> 1) this adds significant complexity to CMake build system. Rules for
>>> building specific libraries may become easier and shorter, but all the
>>> infrastructure
>>> (building dependencies between separate build trees is... challenging)
>>> would be messy and contain a lot of implicit details. Moreover, each
>>> CMakeLists.txt
>>> would have to behave in two different modes (whether it's included from
>>> target-specific build tree or from main one).
>>>
>>
> Why two modes? Sounds like it just needs to build whatever it is building
> with the current CC/CXX/CFLAGS/LDFLAGS/etc (which, of course, depend on
> each tree configuration).
>

Nah, I mean your CMakeLists.txt would probably look like this:

if(IN_TARGET_SPECIFIC_BUILD_TREE)
  add_library(clang_rt.asan ${ASAN_SOURCES} ...)
  ..
else()
  AddRuntimeForTarget(Linux-x86_64 clang_rt clang_rt.x86_64)
  AddRuntimeForTarget(Android clang_rt clang_rt.arm-android)
endif()

Alternatively, you can move all calls of AddRuntimeForTarget
(AddTestForTarget etc...) to some other file.


>
>
>> 2) building with debug Clang is slow (default for many developers).
>>> Though, compiler-rt libraries are not that large for now, so we've observed
>>> this problem
>>> with ASan unit tests (we may split it and make use of parallelism).
>>>
>>
> And, for correctness, any change that affects the compiler binary would
> trigger full rebuild of the runtime libraries.
>

Good point.


> As we bring more llvm code into compiler_rt (like we have done with
> LLVMSupport), this will slow down development tremendously.
>

This would slow down the development for Clang guys :) If you touch
compiler-rt parts of code, the compiler is not changed, and you
don't have to rebuild everything.


> We need to have some kind of "fast" mode that would not rebuild runtime
> every time.
>

Yes, smth like "make clang-only" that wouldn't use fresh Clang to rebuild
compiler-rt libraries.


>
>
>>
>>> WDYT?
>>>
>>> --
>>> Alexey Samsonov, MSK
>>>
>>
>>
>>
>> --
>> Alexey Samsonov, MSK
>>
>>
>


-- 
Alexey Samsonov, MSK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20121030/6e2e9065/attachment.html>