[LLVMdev] [RFC] Improving the testing of exported LLVM CMake targets

[Dan Liew]

Hi,

Following on from another thread (Long-Term Support for LLVM Projects
Extension to Build System) I'd like to discuss the testing of the
exported LLVM CMake targets which can be used by consumers of LLVM as
documented in [1].

Right now we don't test this feature **at all** and as a result it has
been or is

* broken in trunk
* broken by those packaging LLVM
* broken in the official release binaries

I'd like to propose a few ideas I have to improve this situation.

I've created a small toy project [2] that uses the exported LLVM CMake
targets. What I'd like to happen is for this project to become part of
the LLVM source tree (or as a sub project) so that it can be tested in
various scenarios. I do not want this toy project to be a maintenance
burden so it makes very minimal use of LLVM's C++ API.

## Testing trunk

We could teach some of the existing build bots to try building this
toy project after building LLVM. The scenarios we should test are
building the toy project

- Against the LLVM build tree when LLVM is built with CMake
- Against the installed LLVM when LLVM is built with CMake
- Against the installed LLVM when LLVM is built with the
Autoconf/Makefile build system

## Testing packaged LLVM

For those producing LLVM packages for Linux/*BSD, OSX and Windows it
would be good if we could encourage them to test that the toy project
actually builds against their package. Building the toy projects only
takes a few seconds so this shouldn't be much of a burden but can
greatly increase a packagers confidence that LLVM has been properly
packaged.

## Testing official LLVM release binaries

This situation is currently broken. I've observed that

- The LLVM 3.6.* binary tarballs have the LLVM CMake files completely
missing, how did this happen?
- The LLVM3.5 binary tarballs have the CMake files but they're
completely broken.

When making official releases I think it would be a good idea to test
that building the toy project against an extracted binary tarball
actually works.

There are two problems currently AFAICT:

First, the current test-release.sh script does not package an install
properly it uses --prefix to set a temporary directory for the tarball
install when instead it should be using the DESTDIR makefile variable.
I have sent a patch to llvm-commits to try to fix this [3]. The result
of this mistake is that the CMake files end up containing absolute
paths to the temporary tarball directory which means they cannot be
imported by another project.

Second, the LLVM CMake files (LLVMConfig.cmake and LLVMExports.cmake)
are not relocatable (they contain absolute paths which are based on
the install prefix). So even if the first issue is fixed we still have
the problem that a user cannot simply extract the binary tarball to an
arbitrary location their system and build the toy project against it.

I think the LLVMExport.cmake file and LLVM-Config.cmake files need to
be loaded from the same directory containing the found
LLVMConfig.cmake file. The automatically generated LLVMExports.cmake
file would also need to contain relative paths to the libraries (it
currently has absolute paths). I'm not sure how to do this for both of
LLVM's build systems. Any ideas?

Alternatively we could explicitly state in our docs that the LLVM
release binaries can only be used from CMake when it is installed into
the right prefix. This simpler but not very useful because it means
those who want to use the official LLVM binary release builds on their
machine can't unless they have root access (so they can write to
/usr/local/).

Thoughts?

[1] http://llvm.org/docs/CMake.html#embedding-llvm-in-your-project
[2] https://github.com/delcypher/llvmCMakeImportDemo
[3] http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20150622/284436.html

Thanks,
Dan.