[LLVMdev] Building sanitizers for Android

Tue Apr 1 15:50:50 PDT 2014

Alexey, Evgeniy,

I propose the following steps to unify multi-arch support in compiler-rt:

1) The compiler-rt test suite adds "-L${CMAKE_CURRENT_BINARY_DIR}/lib"
to its 'clang' variables.  This way we can test the sanitizers without
installing any libs to the just-built-clang install directory.

2) The "clang/runtime" build calls ExternalProject once for each arch
it needs of compiler-rt.  So once to create x86_64 libs and once for
i386 libs.

3) The compiler-rt build drops the ${arch} suffix from its libs, and
the "clang/runtime" build uses CMAKE_INSTALL_PREFIX to control where
the compiler-rt libs go.
"CMAKE_INSTALL_PREFIX=${CMAKE_CURRENT_BINARY_DIR}/lib/clang/${CLANG_VERSION}/x86_64-linux"

4) Remove multi-arch support from the compiler-rt build.  Instead,
declare compiler-rt as an "any-arch" build, configured with:
    CMAKE_CXX_COMPILER (defaults to just-built-clang)
    CMAKE_CXX_FLAGS (defaults to llvm's default target)
    COMPILER_RT_TEST_COMPILER (defaults to CMAKE_CXX_COMPILER)
    COMPILER_RT_TEST_FLAGS (defaults to CMAKE_CXX_FLAGS)

Thoughts?

Thanks,
Greg

On Tue, Apr 1, 2014 at 12:58 AM, Evgeniy Stepanov
<eugeni.stepanov at gmail.com> wrote:
> It does sound like Android is better suited for "honest"
> cross-compilation, rather than "build compiler-rt for all targets we
> can find" model.
>
> I'm still not convinced that we must require the "ninja install" step.
> Could we just "ninja clang" and then build the second stage against
> the first stage build directory? Will this "find_package" thing not
> work that way, or do you mean it as a way to copy target asan runtime
> where the first stage compiler will find it (or probably both)?
>
> On Mon, Mar 31, 2014 at 6:46 PM, Alexey Samsonov <samsonov at google.com> wrote:
>>
>> On Fri, Mar 28, 2014 at 10:56 PM, Greg Fitzgerald <garious at gmail.com> wrote:
>>>
>>> > Note that ASan tests on Android require llvm-symbolizer binary.
>>>
>>> That's a really good point.  And I see that llvm-symbolizer can't just
>>> be pulled into compiler-rt because it has dependencies on DebugInfo,
>>> Object, and Support libraries.
>>>
>>>
>>> This throws a big wrench in Alexey's plan to have the native
>>> compiler-rt build generate the cross-compiled binaries for all
>>> supported targets.  We would need to do the same in the llvm repo.
>>> The alternative is to run a separate cross-compiled build for each
>>> architecture.  :-/
>>
>>
>> Yes, this llvm-symbolizer part is frustrating... I though we can get away
>> with it at first, as llvm-symbolizer is optional, and we generally won't be
>> able
>> to produce working *binaries* for architectures different from the host one
>> (we might not have a linker, for instance).
>> Android with its NDK looks like an exception, though.
>>
>>>
>>>
>>>
>>> > Yes, so "ninja install" is optional.
>>>
>>> But the 'install' depends on the 'all' target.
>>>
>>> -Greg
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Mar 28, 2014 at 12:39 AM, Evgeniy Stepanov
>>> <eugeni.stepanov at gmail.com> wrote:
>>> > On Thu, Mar 27, 2014 at 10:18 PM, Greg Fitzgerald <garious at gmail.com>
>>> > wrote:
>>> >>> Alexey's approach with CMake sub-projects.
>>> >>
>>> >> I prefer that direction as well, but what I've proposed is a solution
>>> >> that works today.  To support cross-compilation, we'll need to loop
>>> >> over each supported arch (llvm-config --targets-built), then loop over
>>> >> each supported triple for each arch (hard-coded map?), and then pair
>>> >> up each triple with a sysroot (system paths provided by the user).
>>> >>
>>> >>
>>> >>> I think it already copies
>>> >>> compiler-rt build products to the parent build directory, right?
>>> >>
>>> >> The Android compiler-rt build does this, but in an unnecessarily
>>> >> complex way.  My proposal is a simplification of that process.  What
>>> >> is described in "llvm/cmake/platforms/Android.cmake" is to build
>>> >> llvm+clang for the host architecture and then run a second build to
>>> >> cross-compile llvm+compiler-rt to get a the ASan shared object and
>>> >> test suite.  Instead, I propose building only llvm for the host
>>> >> architecture (for the purpose of building llvm-config) and then only
>>> >> compile compiler-rt for Android.  No need to build clang or to
>>> >> cross-compile llvm.
>>> >>
>>> >>
>>> >>> Requiring that llvm is installed complicates things.
>>> >>
>>> >> Sorry for the confusion.  By "llvm install directory", I don't mean to
>>> >> install llvm to the root directory.  I'm referring to the intermediary
>>> >> directory pointed to by CMAKE_INSTALL_PREFIX.  When you run "ninja
>>> >> install", the build populates this directory with only the libs,
>>> >> headers, docs, and executables that the LLVM build intends customers
>>> >> to use directly.  In the case of compiler-rt, we need LLVM's
>>> >> "bin/llvm-config" to determine the build mode, flags, and targets
>>> >> available.
>>> >
>>> > Yes, so "ninja install" is optional. This should work with LLVM build
>>> > directory just as well.
>>> >
>>> >>
>>> >>
>>> >>> This should not be too hard to fix.
>>> >>
>>> >> Porting the LLVM build to Android?  That seems like a more challenging
>>> >> solution than to cut out the unused cross-compiled llvm build.
>>> >
>>> > Note that ASan tests on Android require llvm-symbolizer binary.
>>> > Building just compiler-rt for the target would not be enough.
>>> >
>>> >>
>>> >> -Greg
>>> >>
>>> >> On Thu, Mar 27, 2014 at 1:14 AM, Evgeniy Stepanov
>>> >> <eugeni.stepanov at gmail.com> wrote:
>>> >>> I'd prefer something based on Alexey's approach with CMake
>>> >>> sub-projects. I think it already copies compiler-rt build products to
>>> >>> the parent build directory, right?
>>> >>>
>>> >>> Requiring that llvm is installed complicates things.
>>> >>>
>>> >>> On Thu, Mar 27, 2014 at 5:00 AM, Greg Fitzgerald <garious at gmail.com>
>>> >>> wrote:
>>> >>>> The build for the Android sanitizers is unique in that it needs to
>>> >>>> link against the Android system libraries to create a shared object
>>> >>>> and its test suites.  The current solution to build ASan is to drop
>>> >>>> the compiler-rt repo into the llvm source tree and cross-compile the
>>> >>>> llvm build for Android.  This is a bit awkward for few reasons:
>>> >>>>
>>> >>>> 1) Not all of llvm can be cross-compiled for Android.  "ninja all"
>>> >>>> causes expected build errors.
>>> >>>
>>> >>> This should not be too hard to fix.
>>> >>>
>>> >>>> 2) The sanitizers don't depend on any LLVM libraries, only
>>> >>>> llvm-config
>>> >>>> for its build configuration.
>>> >>>> 3) No "install" rule.  Instead, you cherry-pick files from the build
>>> >>>> directory.
>>> >>>>
>>> >>>> Building against the LLVM install directory cleans this up nicely and
>>> >>>> as it turns out, this mostly works today (see CMake configuration
>>> >>>> below).  The only missing pieces are that the shared object is not
>>> >>>> added to the install directory and the test suites are not built.  Is
>>> >>>> this a build configuration you'd consider using?
>>> >>>>
>>> >>>> $ cmake -G Ninja .. \
>>> >>>>     -DCMAKE_INSTALL_PREFIX=ship \
>>> >>>>     -DCMAKE_C_COMPILER=arm-linux-androideabi-gcc \
>>> >>>>     -DCMAKE_CXX_COMPILER=arm-linux-androideabi-g++ \
>>> >>>>     -DCMAKE_PREFIX_PATH=`pwd`/../../llvm/out/ship \
>>> >>>>     -DANDROID=1 \
>>> >>>>     -DCMAKE_SYSTEM_NAME=Linux \
>>> >>>>     -DCMAKE_C_FLAGS=--sysroot=$(ndkDir)/platforms/android-19/arch-arm
>>> >>>> \
>>> >>>>
>>> >>>> -DCMAKE_CXX_FLAGS=--sysroot=$(ndkDir)/platforms/android-19/arch-arm
>>> >>>> ...
>>> >>>> $ ninja install
>>> >>>> ...
>>> >>>> [68/68]
>>> >>>> -- Install configuration: "Release"
>>> >>>> -- Installing: ship/include/sanitizer/asan_interface.h
>>> >>>> -- Installing: ship/include/sanitizer/common_interface_defs.h
>>> >>>> -- Installing: ship/include/sanitizer/dfsan_interface.h
>>> >>>> -- Installing: ship/include/sanitizer/linux_syscall_hooks.h
>>> >>>> -- Installing: ship/include/sanitizer/lsan_interface.h
>>> >>>> -- Installing: ship/include/sanitizer/msan_interface.h
>>> >>>> -- Installing: ship/include/sanitizer/tsan_interface_atomic.h
>>> >>>> -- Installing: ship/asan_blacklist.txt
>>> >>>> -- Installing: ship/bin/asan_device_setup
>>> >>>>
>>> >>>> Thanks,
>>> >>>> Greg
>>
>>
>>
>>
>> --
>> Alexey Samsonov, MSK