[LLVMdev] [cfe-dev] Sanitizers libs in Compiler-RT

Fri Jan 31 01:22:29 PST 2014

On Fri, Jan 31, 2014 at 1:04 PM, Iain Sandoe <iain at codesourcery.com> wrote:

> Hi Alexey,
>
> On 31 Jan 2014, at 08:50, Alexey Samsonov wrote:
>
> > Thanks for the great overview.
> +1
> ( and +1 for David's comment about moving the unwind stuff into this area)
>
> >
> > On Fri, Jan 31, 2014 at 12:12 PM, Chandler Carruth <chandlerc at google.com>
> wrote:
> >
> > On Thu, Jan 30, 2014 at 1:54 PM, Renato Golin <renato.golin at linaro.org>
> wrote:
> > On 30 January 2014 21:50, Reid Kleckner <rnk at google.com> wrote:
> > I don't see any compelling reason to split the sanitizers out today.
> >
> > Clear, next.
> >
> > Just as a side note, I had some thoughts about the organization of this
> stuff a while back that I wanted to replay here.
> >
> > Fundamentally, I feel like we're in particular getting a few things
> conflated. Maybe separating them out helps:
> >
> > - There is the repository "compiler-rt" that holds all of the runtime
> libraries that (if desired) need to be shipped along side the compiler.
> They're separated so that they can be omitted when they aren't needed, and
> potentially to isolate an unusual desired property: we would really like to
> (eventually) build them with the just-built compiler rather than the host
> compiler (where possible, clearly this requires the target compiler to be
> executable on the host).
> >
> > - There is the core runtime library. Historically this was called
> 'compiler-rt' informally, but perhaps better called 'libclang_rt', which
> provides the core necessary runtime library facilities to compile C or C++
> applications. It's analogous to libgcc but without some of the unwinding
> code (as I understand it, there may be details I'm wrong about here or
> glossing over, but it's not relevant to the organization of things).
> >
> > - There are several optional runtime libraries to support specific
> compiler / toolchain features. These include the sanitizers and profiling
> libraries.
> >
> > I think all of the libraries here make sense in the same repository
> because of the shared concerns of building runtime libraries. For example,
> it would be useful to compile them with the just-built-clang when target is
> executable on the host, and it would be useful to automatically cross
> compile versions of them for as many targets as are possible on the given
> host and supported by the host compiler (if used) and the just-built-clang.
> >
> > However, the organization of the tree is ... very hard to understand.
> originally, there was only the one 'libclang_rt' library that had its
> generic C99 implementation in 'lib', and architecture-specific assembly
> routines when desirable in architecture subdirectories of lib. When we
> added new libraries, we put them in subdirectories of lib, making the whole
> thing kind of a mess. My suggestion to fix this was to create a "core"
> subdirectory of lib to contain the code used for 'libclang_rt'. "core" is a
> terrible name, but i've no better. suggestions welcome there. Then we would
> have a more sensible organization of the 'lib' tree.
> >
> > I can't suggest a better name than "core" (maybe, "runtime"?), but would
> *love* to put 'libclang_rt' into a separate directory under
> /compiler-rt/lib :) We should've done it long ago, IMO.
> >
> > I also think it might be useful to have a single large test tree (much
> like with llvm or clang) that has subdirectories for the various tests
> rather than test directories under lib/asan/ and friends, but maybe the
> sanitizer folks have objections to that. consistency seems a compelling
> reason here, but there might be other compelling concerns.
> >
> > Yes, I don't like the way testing is organized either. Originally it
> made sense to put sanitizer lit tests under lib/xsan/, but now we've got a
> lot of sanitizers, and a lot of duplicating configuration code. As Renato
> mentioned, browsing
> > through lit configs is a bit painful. I would like to proceed with
> re-organizing the tests for sanitizers, unless anyone objects. Kostya?
> >
> >
> > Now, the build system has always been problematic because the build
> system for this tree is *hard* and no one who has worked on it has really
> had the time to do an extremely thorough job and finish all aspects. It's a
> huge project. Right now, the makefile system does a good job of using the
> just-built-clang and can do a limited amount of cross-building runtimes for
> other targets. But the makefile system of compiler-rt is also terribly,
> terribly complex, doesn't follow the conventions of LLVM's makefiles, and
> is generally painful to maintain and update, so folks have been reluctant
> to flesh out its support for new libraries and other new things.
> >
> > Note that it would be really hard to add support for running test suites
> into Makefile build system. It's also good for the one-time build only, not
> for continuous development - I don't think it respects dependencies
> correctly.
>
> Please could you expand upon this?
>

Well, LLVM/Clang's configure+make and compiler-rt's make are disjoint -
when you run "make" in Clang build tree, at one point it simply invokes,
the Makefile in compiler-rt directory. But if we want more functionality,
like running the tests, we ought to have a single build system, single set
of "targets" (binaries, libraries, test suites) with dependencies between
them. For example, I want to have "make check-asan" command which I can run
from the root of the LLVM/Clang's build tree. More, if I run "make
check-asan", I need to scan the dependencies of that test suite, and:
1) If ASan runtime has changed, re-build it, and re-run tests.
2) If FileCheck or lit sources have changed, I should re-build them (but
not ASan runtime) and re-run tests.
3) If Clang sources have changed, I should re-build Clang, re-build ASan
runtime (*) and re-run tests.
With CMake we're able to do all that (except for (*), as we build runtimes
with host compiler) at the moment.

> Certainly, one can expect to test the runtime on the host?
> likewise, any pointers to where you see wrong dependencies would be
> appreciated.
>

IIRC, "make check-all" in configure+make build tree doesn't re-build the
necessary libraries automatically.

>
> At present, config & make appears to build a sensible lib using the
> just-built compiler [and in the general case, that's the only sensible
> solution, since the host might not have cross-tools for the set of archs
> you want to support with llvm/clang]
>

Yes, that's what configure+make is good at.

> In some future ideal world, we might cook up a cross-testing environment -
> of course, it's a given that that would require testers to have access to
> the target hardware they wanted to test on.
>
> Iain
>
> >
> >
> > The CMake system is much cleaner in some respects (a bit less opaque to
> the folks trying to maintain it), but CMake makes it much harder to use the
> just-built-clang, especially for the C++ runtime code. The consequence is
> that we've never finished either the cross-building or
> just-built-clang-hosting features that are desirable.
> >
> > I once tried to implement a pseudo-build-system for compiler-rt on top
> of CMake, so that we use "just-built" Clang instead of a host compiler, but
> failed miserably. Maybe I was doing wrong things, but I got the impression
> > that CMake isn't suited for switching the compiler on the fly :)
> >
> > Any work toward these would be really awesome to see, but is a huge pile
> of work. Finally, when I was originally doing the CMake build for this I
> didn't understand what really needed to be done to build and use the core
> 'libclang_rt' library, and so I don't think I got it right. Some folks have
> sent patches to improve it, but I suspect it still really needs more work
> to be a solid system to use instead of libgcc. So I'm really excited about
> your emails. =] Having a more self-contained stack would be a significant
> improvement.
> >
> >
> > I think the obvious incremental steps are to disable building any parts
> of compiler-rt that don't build cleanly.
> >
> > As I mentioned above, currently in CMake build system we try to avoid
> building anything if we're not sure we can produce a working and correct
> library on the host platform. That is, I still don't see what the problem
> is - it's relatively easy to enable building just the compiler-rt library
> on ARM and not enable building sanitizers on ARM.
> >
> > There should never be a requirement for you to port an optional runtime
> library unless you need it. =] I think having good ports is important, but
> that should never block progress getting other thinsg ported and working
> well.
> >
> > If it helps to reorganize things, I'm happy to even help there as much
> as I can. I agree that the organization isn't great, but I *really* didn't
> want to fight the makefile build system to do the reorganization myself, so
> its something that has lingered too long.
> >
> >
> > Sorry for the long ramble, but hopefully this gives you some of tho
> context.
> >
> > _______________________________________________
> > cfe-dev mailing list
> > cfe-dev at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >
> >
> >
> >
> > --
> > Alexey Samsonov, MSK
> > _______________________________________________
> > LLVM Developers mailing list
> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
>

-- 
Alexey Samsonov, MSK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140131/c75185d8/attachment.html>