[cfe-dev] CMake builld system for compiler-rt rework

[Alexey Samsonov]

On Tue, Oct 30, 2012 at 4:27 PM, Evgeniy Stepanov <eugenis at google.com>wrote:

>
>
>
> On Tue, Oct 30, 2012 at 4:09 PM, Alexey Samsonov <samsonov at google.com>wrote:
>
>>
>>
>> On Tue, Oct 30, 2012 at 3:58 PM, Evgeniy Stepanov <eugenis at google.com>wrote:
>>
>>>
>>>
>>>
>>> On Tue, Oct 30, 2012 at 2:03 PM, Alexey Samsonov <samsonov at google.com>wrote:
>>>
>>>> +cfe-dev
>>>>
>>>>
>>>> On Tue, Oct 30, 2012 at 1:50 PM, Alexey Samsonov <samsonov at google.com>wrote:
>>>>
>>>>> Hi there!
>>>>>
>>>>> tl;dr we should build compiler-rt with just-built clang
>>>>>
>>>>> A number of issues with CMake build of compiler-rt libraries and tests
>>>>> are caused by the fact that we build compiler-rt with host compiler
>>>>> instead of
>>>>> just-built Clang (examples: http://llvm.org/bugs/show_bug.cgi?id=13864
>>>>> , http://llvm.org/bugs/show_bug.cgi?id=14109)
>>>>>
>>>>> Using Clang looks like the Right thing to do, and it's probably time
>>>>> to make this happen:
>>>>> 1) configure+make build system uses fresh Clang
>>>>> 2) using just one fresh Clang will allow us to drop support for all
>>>>> kinds of host compilers users
>>>>> are building compiler-rt with:
>>>>>   * we will be able to build compiler-rt with -Werror
>>>>>   * we will be able to build versions of compiler-rt libraries for all
>>>>> target triples supported by Clang (not by host compiler)
>>>>>   * we will be able to run compiler-rt tests, regardless of host
>>>>> compiler. For example, my gcc-4.4 build of compiler-rt
>>>>>     fails on "check-ubsan" tests because of lacking 128-bit int
>>>>> support.
>>>>>
>>>>> OK, how can we build with fresh Clang? (One can't simply tell CMake to
>>>>> build some targets by a compiler produced
>>>>> while building another target).
>>>>>
>>>>> Chandler (and other developers) frequently mention the bootstrap
>>>>> process (external to CMake), that (IIUC) build Clang
>>>>> and then uses it to re-build LLVM[+Clang?]+compiler-rt.
>>>>> Can we:
>>>>> 1) *always* build compiler-rt as a part of bootstrap process (run
>>>>> CMake to build clang, then run it again with Clang as a C compiler to build
>>>>> compiler-rt).
>>>>> 2) document the usage.
>>>>> 3) make a script that would run bootstrap process in (1), so that the
>>>>> users can invoke a single command instead of running a sequence of
>>>>> instructions?
>>>>>
>>>>> So, when CMake is called with Clang compiler, it can run a set of
>>>>> try_compile() tests to determine possible target triples and build
>>>>> necessary compiler-rt
>>>>> libraries for all these triples. But I'm afraid it's *not* that easy:
>>>>> 1) if a compiler-rt library ever needs to use LLVM libraries, it needs
>>>>> to have these LLVM libs built for the same target. That is, in general we
>>>>> may require
>>>>> a bunch of LLVM libraries compiled for different targets, which is not
>>>>> supported by LLVM build system at all.
>>>>> 2) part (1) is already true for compiler-rt unit tests: there is no
>>>>> way we can build 64-bit and 32-bit version of the same unit test in a
>>>>> single build tree, as we'd
>>>>> need two different versions of googletest and LLVMSupport.
>>>>> 3) Notable case is Android (eugenis@ may comment more on this) -
>>>>> building for Android requires many changes and is hardly possible in the
>>>>> same
>>>>> build tree we use for regular (say, x86_64-linux) build tree.
>>>>>
>>>>
>>> The difference is you can not get away with simply changing CFLAGS (-m32
>>> vs -m64) in this case. You need to add quite a few other flags, and, more
>>> often than not, host compiler can not target Android at all. So, while we
>>> can do some hacks to build both x86_32 and x86_64 runtime libraries in the
>>> same build tree (with the host compiler), we really need a separate tree,
>>> and a different compiler, for Android targets.
>>>
>>> Recursive trees (below) seem like the only option.
>>>
>>>
>>>>
>>>>> We can try to resolve this by using recursive build trees:
>>>>> * for each target create its own "recursive" LLVM build tree (e.g.
>>>>> projects/compiler-rt/Linux-x86_64) and invoke CMake
>>>>> there with some additional compile/link flags.
>>>>> * describe a single target (say, "clang_rt") in a CMakeLists.txt in
>>>>> each target-specific recursive build tree.
>>>>> * Build rule for "clang_rt.linux-x86_64" in main build tree in  will
>>>>> simply invoke building "clang_rt" in necessary
>>>>> target-specific subtree and copy result from there.
>>>>>
>>>>> Overall, I believe this is doable (but pretty ugly) and have a raw
>>>>> working prototype for this. If we decide to go this way,
>>>>> I'm ready to complete this if anyone signs up for a careful review (as
>>>>> my CMake skills are far from good).
>>>>>
>>>>> Problems:
>>>>> 1) this adds significant complexity to CMake build system. Rules for
>>>>> building specific libraries may become easier and shorter, but all the
>>>>> infrastructure
>>>>> (building dependencies between separate build trees is... challenging)
>>>>> would be messy and contain a lot of implicit details. Moreover, each
>>>>> CMakeLists.txt
>>>>> would have to behave in two different modes (whether it's included
>>>>> from target-specific build tree or from main one).
>>>>>
>>>>
>>> Why two modes? Sounds like it just needs to build whatever it is
>>> building with the current CC/CXX/CFLAGS/LDFLAGS/etc (which, of course,
>>> depend on each tree configuration).
>>>
>>
>> Nah, I mean your CMakeLists.txt would probably look like this:
>>
>> if(IN_TARGET_SPECIFIC_BUILD_TREE)
>>   add_library(clang_rt.asan ${ASAN_SOURCES} ...)
>>   ..
>> else()
>>   AddRuntimeForTarget(Linux-x86_64 clang_rt clang_rt.x86_64)
>>   AddRuntimeForTarget(Android clang_rt clang_rt.arm-android)
>> endif()
>>
>> Alternatively, you can move all calls of AddRuntimeForTarget
>> (AddTestForTarget etc...) to some other file.
>>
>>
>
> But we don't need this in every CMakeLists.txt, right?
>

Probably.


> This switch goes under tools/clang/runtime/. Every other CMakeLists.txt
> simply knows how to build stuff for the current configuration - we just
> don't execute compiler-rt's rules in the main tree (or do we?).
>

Nah, currently CMake goes into //projects/compiler-rt directly from root
directory (it's different from configure+make...)


>
>
>>
>>>
>>>> 2) building with debug Clang is slow (default for many developers).
>>>>> Though, compiler-rt libraries are not that large for now, so we've observed
>>>>> this problem
>>>>> with ASan unit tests (we may split it and make use of parallelism).
>>>>>
>>>>
>>> And, for correctness, any change that affects the compiler binary would
>>> trigger full rebuild of the runtime libraries.
>>>
>>
>> Good point.
>>
>>
>>> As we bring more llvm code into compiler_rt (like we have done with
>>> LLVMSupport), this will slow down development tremendously.
>>>
>>
>> This would slow down the development for Clang guys :) If you touch
>> compiler-rt parts of code, the compiler is not changed, and you
>> don't have to rebuild everything.
>>
>>
>>> We need to have some kind of "fast" mode that would not rebuild runtime
>>> every time.
>>>
>>
>> Yes, smth like "make clang-only" that wouldn't use fresh Clang to rebuild
>> compiler-rt libraries.
>>
>>
>>>
>>>
>>>>
>>>>> WDYT?
>>>>>
>>>>> --
>>>>> Alexey Samsonov, MSK
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Alexey Samsonov, MSK
>>>>
>>>>
>>>
>>
>>
>> --
>> Alexey Samsonov, MSK
>>
>>
>


-- 
Alexey Samsonov, MSK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20121030/736b4122/attachment.html>