[llvm-dev] Policy for compiler-rt ABI stability and external dependencies?

[Dean Michael Berris via llvm-dev]

> On 12 Oct 2017, at 03:30, Kostya Serebryany <kcc at google.com> wrote:
> 
> Hi Dean. 
> 
> The sanitizers don't use any of the C++ std library. 
> More than that, they don't include *any* system headers in most of the sources (exception is some OS-dependent .cc files). 
> 
> This rule is *somewhat* documented, e.g. 
> tsan/rtl/tsan_rtl.h:
> 
> //   - No system headers included in header files (*).
> //   - Platform specific headres included only into platform-specific files (*).
> 
> It's enforced only by code review, although if one attempts to use stl in e.g. asan some tests will most likely break. 
> If you can suggest a simple way to enforce this at build time -- let's do it. 
> 

Thanks for confirming Kostya, this really helps!

I'm not sure how to prevent this, but we can look at the generated archives/shared-objects and find references to known symbols from the C++ standard library (or even if there are references to specific files/.so's). Let me think about it a bit more, but it might be good to have this conversation in person, perhaps this coming week.

Documentation would probably also help, if this is in some .rst document that's available from llvm.org/docs <http://llvm.org/docs>.

> libFuzzer on the contrary uses STL, including the containers, and it causes all sorts of problems. 
> (I've anticipated it from the beginning, but just couldn't force myself to not use STL).
> 
> My recommendation is to ban all of C++ std lib in xray unless you find a way to "privatize" the bits of STL you are using. 
> Ideally, those bits will be header-only and you'll be able to change the namespace for STL. 
> See e.g. a long discussion in https://reviews.llvm.org/D37631 <https://reviews.llvm.org/D37631> (trying to privatize STL in libFuzzer)
> 

I think this may be the best option for now as well.

I'll go ahead and spend some time to remove the std lib references in the XRay runtime, and implement analogues for the things we actually really need in the implementation.

> 
> 
> On Wed, Oct 11, 2017 at 4:07 AM, Dean Michael Berris <dean.berris at gmail.com <mailto:dean.berris at gmail.com>> wrote:
> Hi Kostya, Evgenii, and David,
> 
> Recently I've been making some incremental changes to the XRay runtime implementation in compiler-rt to reduce the reliance on the C++ standard library components that might have external linkage dependencies. This involves not using containers from the STL and not using non-trivially destructible C++11 thread_local objects.
> 
> I was wondering whether the sanitisers follow an explicit policy of not depending on the C++ standard library or avoiding linkage/ABI issues that come up when building the sanitisers with C++ standard library dependencies. If it does, do we document these policies somewhere? If we don't document it somewhere, does it make sense for us to do so and maybe potentially enforce it at build time?
> 
> One of the policies I can think of is not to use *anything* from the C++ standard library even if they are templates anyway (e.g. std::unique_ptr<...>, std::shared_ptr<...>, std::tuple<...>, std::aligned_storage<...>, etc.). The downside to this would be myriad, where we'd need to implement analogues of these ourselves or resort to only things we can use from libc. I'm not sure how this could be automatically enforced, maybe potentially through a blacklist on `std::` types.
> 
> A slightly more permissive policy might be to allow using some of these types but not in any of the exposed interfaces and not require any runtime dependencies from the C++ standard library. This might be possible to enforce by checking the external/dymamic symbols in the runtime archives. I'm not sure though whether it's even safe to assume that the templates wouldn't introduce some weird ABI constraint even if the types don't cross the API boundaries.
> 
> I'm curious what the sanitisers currently do and what techniques/conventions we can follow on the XRay implementation that would be helpful to emulate as well.
> 
> In a similar vein, how do the sanitisers handle making ABI breaking changes? For the published types, is there a symbol naming policy and explicit deprecation/support horizon? How about API stability, do we intend to keep the API stable across N stable releases?
> 
> We generally don't promise compatibility between the compiler and rt from different revisions. 
> However, just in case, we do a version mismatch check (e.g. see lib/asan/asan_init_version.h)
> 

Cool stuff, thanks!

It probably also helps to be explicit here, so that we can communicate it better to users.

Right now, it's been implicit that XRay might change APIs.

Would be nice to have in some central documentation too. :)

Cheers

-- Dean

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171014/5b758a7f/attachment.html>