[LLVMdev] Building sanitizers for Android

Greg Fitzgerald garious at gmail.com
Wed Apr 2 18:04:48 PDT 2014 

> we would still want to use compiler-rt test-suite in a standalone mode, to test fully built/installed toolchains,
and even GCC.

Sounds good.


> Clang driver links the static xsan runtimes from a hardcoded
> paths in Clang resource directory, and doesn't add flags like
> "-lasan -L/path/to/clang/resource/dir". I find this behavior reasonable.

Can we change the flags to "-lasan -L/path/to/clang/resource/dir"?  If
we make that change, I can stop suggesting that the clang build should
invoke the compiler-rt test suite directly.  :-)


> Do you want to give it a try?

I'll take a crack it, sure.


By the way, locally, I now have just over half the ASan test suite
passing ARM-Linux via QEMU.  It's been really convenient to build
compiler-rt in standalone mode.  The edit-compile-run cycle is much
shorter this way.  Thanks for your help in making this work!

-Greg



On Wed, Apr 2, 2014 at 7:56 AM, Alexey Samsonov <samsonov at google.com> wrote:
> Hi Greg,
>
> On Wed, Apr 2, 2014 at 2:50 AM, Greg Fitzgerald <garious at gmail.com> wrote:
>>
>> Alexey, Evgeniy,
>>
>> I propose the following steps to unify multi-arch support in compiler-rt:
>>
>> 1) The compiler-rt test suite adds "-L${CMAKE_CURRENT_BINARY_DIR}/lib"
>> to its 'clang' variables.  This way we can test the sanitizers without
>> installing any libs to the just-built-clang install directory.
>
>
> This is possible, but please keep in mind that:
> 1) we would still want to use compiler-rt test-suite in a standalone mode,
> to test fully built/installed toolchains,
> and even GCC.
> 2) Adding -L... wouldn't work: Clang driver links the static xsan runtimes
> from a hardcoded paths in Clang resource directory,
> and doesn't add flags like "-lasan -L/path/to/clang/resource/dir". I find
> this behavior reasonable.
>
> I don't see a problem with the current approach - we can make "run sanitizer
> test suite" command in the
> top-level build tree depend on "build/install compiler-rt ExternalProject"
> steps (see how it's done currently).
>
>>
>>
>> 2) The "clang/runtime" build calls ExternalProject once for each arch
>> it needs of compiler-rt.  So once to create x86_64 libs and once for
>> i386 libs.
>>
>> 3) The compiler-rt build drops the ${arch} suffix from its libs, and
>> the "clang/runtime" build uses CMAKE_INSTALL_PREFIX to control where
>> the compiler-rt libs go.
>>
>> "CMAKE_INSTALL_PREFIX=${CMAKE_CURRENT_BINARY_DIR}/lib/clang/${CLANG_VERSION}/x86_64-linux"
>>
>>
>> 4) Remove multi-arch support from the compiler-rt build.  Instead,
>> declare compiler-rt as an "any-arch" build, configured with:
>>     CMAKE_CXX_COMPILER (defaults to just-built-clang)
>>     CMAKE_CXX_FLAGS (defaults to llvm's default target)
>>     COMPILER_RT_TEST_COMPILER (defaults to CMAKE_CXX_COMPILER)
>>     COMPILER_RT_TEST_FLAGS (defaults to CMAKE_CXX_FLAGS)
>
>
> Unfortunately, this could be problematic for Mac - we use "universal
> binaries" there - a single static/dynamic runtime
> which contains runtimes for all the possible architectures. We might work
> around this, however, by adding a custom
> Mac-specific step that would merge all single-arch binaries into a
> multi-arch one - I think this is fine as long as we
> greatly simplify the rest of the ugly build system.
>
> Do you want to give it a try?
>
>>
>>
>> Thoughts?
>>
>> Thanks,
>> Greg
>>
>> On Tue, Apr 1, 2014 at 12:58 AM, Evgeniy Stepanov
>> <eugeni.stepanov at gmail.com> wrote:
>> > It does sound like Android is better suited for "honest"
>> > cross-compilation, rather than "build compiler-rt for all targets we
>> > can find" model.
>> >
>> > I'm still not convinced that we must require the "ninja install" step.
>> > Could we just "ninja clang" and then build the second stage against
>> > the first stage build directory? Will this "find_package" thing not
>> > work that way, or do you mean it as a way to copy target asan runtime
>> > where the first stage compiler will find it (or probably both)?
>> >
>> > On Mon, Mar 31, 2014 at 6:46 PM, Alexey Samsonov <samsonov at google.com>
>> > wrote:
>> >>
>> >> On Fri, Mar 28, 2014 at 10:56 PM, Greg Fitzgerald <garious at gmail.com>
>> >> wrote:
>> >>>
>> >>> > Note that ASan tests on Android require llvm-symbolizer binary.
>> >>>
>> >>> That's a really good point.  And I see that llvm-symbolizer can't just
>> >>> be pulled into compiler-rt because it has dependencies on DebugInfo,
>> >>> Object, and Support libraries.
>> >>>
>> >>>
>> >>> This throws a big wrench in Alexey's plan to have the native
>> >>> compiler-rt build generate the cross-compiled binaries for all
>> >>> supported targets.  We would need to do the same in the llvm repo.
>> >>> The alternative is to run a separate cross-compiled build for each
>> >>> architecture.  :-/
>> >>
>> >>
>> >> Yes, this llvm-symbolizer part is frustrating... I though we can get
>> >> away
>> >> with it at first, as llvm-symbolizer is optional, and we generally
>> >> won't be
>> >> able
>> >> to produce working *binaries* for architectures different from the host
>> >> one
>> >> (we might not have a linker, for instance).
>> >> Android with its NDK looks like an exception, though.
>> >>
>> >>>
>> >>>
>> >>>
>> >>> > Yes, so "ninja install" is optional.
>> >>>
>> >>> But the 'install' depends on the 'all' target.
>> >>>
>> >>> -Greg
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Fri, Mar 28, 2014 at 12:39 AM, Evgeniy Stepanov
>> >>> <eugeni.stepanov at gmail.com> wrote:
>> >>> > On Thu, Mar 27, 2014 at 10:18 PM, Greg Fitzgerald
>> >>> > <garious at gmail.com>
>> >>> > wrote:
>> >>> >>> Alexey's approach with CMake sub-projects.
>> >>> >>
>> >>> >> I prefer that direction as well, but what I've proposed is a
>> >>> >> solution
>> >>> >> that works today.  To support cross-compilation, we'll need to loop
>> >>> >> over each supported arch (llvm-config --targets-built), then loop
>> >>> >> over
>> >>> >> each supported triple for each arch (hard-coded map?), and then
>> >>> >> pair
>> >>> >> up each triple with a sysroot (system paths provided by the user).
>> >>> >>
>> >>> >>
>> >>> >>> I think it already copies
>> >>> >>> compiler-rt build products to the parent build directory, right?
>> >>> >>
>> >>> >> The Android compiler-rt build does this, but in an unnecessarily
>> >>> >> complex way.  My proposal is a simplification of that process.
>> >>> >> What
>> >>> >> is described in "llvm/cmake/platforms/Android.cmake" is to build
>> >>> >> llvm+clang for the host architecture and then run a second build to
>> >>> >> cross-compile llvm+compiler-rt to get a the ASan shared object and
>> >>> >> test suite.  Instead, I propose building only llvm for the host
>> >>> >> architecture (for the purpose of building llvm-config) and then
>> >>> >> only
>> >>> >> compile compiler-rt for Android.  No need to build clang or to
>> >>> >> cross-compile llvm.
>> >>> >>
>> >>> >>
>> >>> >>> Requiring that llvm is installed complicates things.
>> >>> >>
>> >>> >> Sorry for the confusion.  By "llvm install directory", I don't mean
>> >>> >> to
>> >>> >> install llvm to the root directory.  I'm referring to the
>> >>> >> intermediary
>> >>> >> directory pointed to by CMAKE_INSTALL_PREFIX.  When you run "ninja
>> >>> >> install", the build populates this directory with only the libs,
>> >>> >> headers, docs, and executables that the LLVM build intends
>> >>> >> customers
>> >>> >> to use directly.  In the case of compiler-rt, we need LLVM's
>> >>> >> "bin/llvm-config" to determine the build mode, flags, and targets
>> >>> >> available.
>> >>> >
>> >>> > Yes, so "ninja install" is optional. This should work with LLVM
>> >>> > build
>> >>> > directory just as well.
>> >>> >
>> >>> >>
>> >>> >>
>> >>> >>> This should not be too hard to fix.
>> >>> >>
>> >>> >> Porting the LLVM build to Android?  That seems like a more
>> >>> >> challenging
>> >>> >> solution than to cut out the unused cross-compiled llvm build.
>> >>> >
>> >>> > Note that ASan tests on Android require llvm-symbolizer binary.
>> >>> > Building just compiler-rt for the target would not be enough.
>> >>> >
>> >>> >>
>> >>> >> -Greg
>> >>> >>
>> >>> >> On Thu, Mar 27, 2014 at 1:14 AM, Evgeniy Stepanov
>> >>> >> <eugeni.stepanov at gmail.com> wrote:
>> >>> >>> I'd prefer something based on Alexey's approach with CMake
>> >>> >>> sub-projects. I think it already copies compiler-rt build products
>> >>> >>> to
>> >>> >>> the parent build directory, right?
>> >>> >>>
>> >>> >>> Requiring that llvm is installed complicates things.
>> >>> >>>
>> >>> >>> On Thu, Mar 27, 2014 at 5:00 AM, Greg Fitzgerald
>> >>> >>> <garious at gmail.com>
>> >>> >>> wrote:
>> >>> >>>> The build for the Android sanitizers is unique in that it needs
>> >>> >>>> to
>> >>> >>>> link against the Android system libraries to create a shared
>> >>> >>>> object
>> >>> >>>> and its test suites.  The current solution to build ASan is to
>> >>> >>>> drop
>> >>> >>>> the compiler-rt repo into the llvm source tree and cross-compile
>> >>> >>>> the
>> >>> >>>> llvm build for Android.  This is a bit awkward for few reasons:
>> >>> >>>>
>> >>> >>>> 1) Not all of llvm can be cross-compiled for Android.  "ninja
>> >>> >>>> all"
>> >>> >>>> causes expected build errors.
>> >>> >>>
>> >>> >>> This should not be too hard to fix.
>> >>> >>>
>> >>> >>>> 2) The sanitizers don't depend on any LLVM libraries, only
>> >>> >>>> llvm-config
>> >>> >>>> for its build configuration.
>> >>> >>>> 3) No "install" rule.  Instead, you cherry-pick files from the
>> >>> >>>> build
>> >>> >>>> directory.
>> >>> >>>>
>> >>> >>>> Building against the LLVM install directory cleans this up nicely
>> >>> >>>> and
>> >>> >>>> as it turns out, this mostly works today (see CMake configuration
>> >>> >>>> below).  The only missing pieces are that the shared object is
>> >>> >>>> not
>> >>> >>>> added to the install directory and the test suites are not built.
>> >>> >>>> Is
>> >>> >>>> this a build configuration you'd consider using?
>> >>> >>>>
>> >>> >>>> $ cmake -G Ninja .. \
>> >>> >>>>     -DCMAKE_INSTALL_PREFIX=ship \
>> >>> >>>>     -DCMAKE_C_COMPILER=arm-linux-androideabi-gcc \
>> >>> >>>>     -DCMAKE_CXX_COMPILER=arm-linux-androideabi-g++ \
>> >>> >>>>     -DCMAKE_PREFIX_PATH=`pwd`/../../llvm/out/ship \
>> >>> >>>>     -DANDROID=1 \
>> >>> >>>>     -DCMAKE_SYSTEM_NAME=Linux \
>> >>> >>>>
>> >>> >>>> -DCMAKE_C_FLAGS=--sysroot=$(ndkDir)/platforms/android-19/arch-arm
>> >>> >>>> \
>> >>> >>>>
>> >>> >>>>
>> >>> >>>> -DCMAKE_CXX_FLAGS=--sysroot=$(ndkDir)/platforms/android-19/arch-arm
>> >>> >>>> ...
>> >>> >>>> $ ninja install
>> >>> >>>> ...
>> >>> >>>> [68/68]
>> >>> >>>> -- Install configuration: "Release"
>> >>> >>>> -- Installing: ship/include/sanitizer/asan_interface.h
>> >>> >>>> -- Installing: ship/include/sanitizer/common_interface_defs.h
>> >>> >>>> -- Installing: ship/include/sanitizer/dfsan_interface.h
>> >>> >>>> -- Installing: ship/include/sanitizer/linux_syscall_hooks.h
>> >>> >>>> -- Installing: ship/include/sanitizer/lsan_interface.h
>> >>> >>>> -- Installing: ship/include/sanitizer/msan_interface.h
>> >>> >>>> -- Installing: ship/include/sanitizer/tsan_interface_atomic.h
>> >>> >>>> -- Installing: ship/asan_blacklist.txt
>> >>> >>>> -- Installing: ship/bin/asan_device_setup
>> >>> >>>>
>> >>> >>>> Thanks,
>> >>> >>>> Greg
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Alexey Samsonov, MSK
>
>
>
>
> --
> Alexey Samsonov, MSK

More information about the llvm-dev mailing list