[cfe-dev] [LLVMdev] Sanitizers libs in Compiler-RT

Fri Jan 31 00:50:49 PST 2014

Chandler,

Thanks for the great overview.

On Fri, Jan 31, 2014 at 12:12 PM, Chandler Carruth <chandlerc at google.com>wrote:

>
> On Thu, Jan 30, 2014 at 1:54 PM, Renato Golin <renato.golin at linaro.org>wrote:
>
>> On 30 January 2014 21:50, Reid Kleckner <rnk at google.com> wrote:
>>>
>>> I don't see any compelling reason to split the sanitizers out today.
>>>
>>
>> Clear, next.
>>
>
> Just as a side note, I had some thoughts about the organization of this
> stuff a while back that I wanted to replay here.
>
> Fundamentally, I feel like we're in particular getting a few things
> conflated. Maybe separating them out helps:
>
> - There is the repository "compiler-rt" that holds all of the runtime
> libraries that (if desired) need to be shipped along side the compiler.
> They're separated so that they can be omitted when they aren't needed, and
> potentially to isolate an unusual desired property: we would really like to
> (eventually) build them with the just-built compiler rather than the host
> compiler (where possible, clearly this requires the target compiler to be
> executable on the host).
>
> - There is the core runtime library. Historically this was called
> 'compiler-rt' informally, but perhaps better called 'libclang_rt', which
> provides the core necessary runtime library facilities to compile C or C++
> applications. It's analogous to libgcc but without some of the unwinding
> code (as I understand it, there may be details I'm wrong about here or
> glossing over, but it's not relevant to the organization of things).
>
> - There are several optional runtime libraries to support specific
> compiler / toolchain features. These include the sanitizers and profiling
> libraries.
>
> I think all of the libraries here make sense in the same repository
> because of the shared concerns of building runtime libraries. For example,
> it would be useful to compile them with the just-built-clang when target is
> executable on the host, and it would be useful to automatically cross
> compile versions of them for as many targets as are possible on the given
> host and supported by the host compiler (if used) and the just-built-clang.
>
> However, the organization of the tree is ... very hard to understand.
> originally, there was only the one 'libclang_rt' library that had its
> generic C99 implementation in 'lib', and architecture-specific assembly
> routines when desirable in architecture subdirectories of lib. When we
> added new libraries, we put them in subdirectories of lib, making the whole
> thing kind of a mess. My suggestion to fix this was to create a "core"
> subdirectory of lib to contain the code used for 'libclang_rt'. "core" is a
> terrible name, but i've no better. suggestions welcome there. Then we would
> have a more sensible organization of the 'lib' tree.
>

I can't suggest a better name than "core" (maybe, "runtime"?), but would
*love* to put 'libclang_rt' into a separate directory under
/compiler-rt/lib :) We should've done it long ago, IMO.

>
> I also think it might be useful to have a single large test tree (much
> like with llvm or clang) that has subdirectories for the various tests
> rather than test directories under lib/asan/ and friends, but maybe the
> sanitizer folks have objections to that. consistency seems a compelling
> reason here, but there might be other compelling concerns.
>

Yes, I don't like the way testing is organized either. Originally it made
sense to put sanitizer lit tests under lib/xsan/, but now we've got a lot
of sanitizers, and a lot of duplicating configuration code. As Renato
mentioned, browsing
through lit configs is a bit painful. I would like to proceed with
re-organizing the tests for sanitizers, unless anyone objects. Kostya?

> Now, the build system has always been problematic because the build system
> for this tree is *hard* and no one who has worked on it has really had the
> time to do an extremely thorough job and finish all aspects. It's a huge
> project. Right now, the makefile system does a good job of using the
> just-built-clang and can do a limited amount of cross-building runtimes for
> other targets. But the makefile system of compiler-rt is also terribly,
> terribly complex, doesn't follow the conventions of LLVM's makefiles, and
> is generally painful to maintain and update, so folks have been reluctant
> to flesh out its support for new libraries and other new things.
>

Note that it would be really hard to add support for running test suites
into Makefile build system. It's also good for the one-time build only, not
for continuous development - I don't think it respects dependencies
correctly.

>
> The CMake system is much cleaner in some respects (a bit less opaque to
> the folks trying to maintain it), but CMake makes it much harder to use the
> just-built-clang, especially for the C++ runtime code. The consequence is
> that we've never finished either the cross-building or
> just-built-clang-hosting features that are desirable.
>

I once tried to implement a pseudo-build-system for compiler-rt on top of
CMake, so that we use "just-built" Clang instead of a host compiler, but
failed miserably. Maybe I was doing wrong things, but I got the impression
that CMake isn't suited for switching the compiler on the fly :)

> Any work toward these would be really awesome to see, but is a huge pile
> of work. Finally, when I was originally doing the CMake build for this I
> didn't understand what really needed to be done to build and use the core
> 'libclang_rt' library, and so I don't think I got it right. Some folks have
> sent patches to improve it, but I suspect it still really needs more work
> to be a solid system to use instead of libgcc. So I'm really excited about
> your emails. =] Having a more self-contained stack would be a significant
> improvement.
>
>
> I think the obvious incremental steps are to disable building any parts of
> compiler-rt that don't build cleanly.
>

As I mentioned above, currently in CMake build system we try to avoid
building anything if we're not sure we can produce a working and correct
library on the host platform. That is, I still don't see what the problem
is - it's relatively easy to enable building just the compiler-rt library
on ARM and not enable building sanitizers on ARM.

> There should never be a requirement for you to port an optional runtime
> library unless you need it. =] I think having good ports is important, but
> that should never block progress getting other thinsg ported and working
> well.
>
> If it helps to reorganize things, I'm happy to even help there as much as
> I can. I agree that the organization isn't great, but I *really* didn't
> want to fight the makefile build system to do the reorganization myself, so
> its something that has lingered too long.
>
>
> Sorry for the long ramble, but hopefully this gives you some of tho
> context.
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
>

-- 
Alexey Samsonov, MSK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20140131/c987d752/attachment.html>