[PATCH] D14046: [test-suite] Introduce a new CMake+LIT based harness

[James Molloy via llvm-commits]

jmolloy created this revision.
jmolloy added reviewers: beanz, cmatthews.
jmolloy added a subscriber: llvm-commits.
jmolloy set the repository for this revision to rL LLVM.

This introduces a replacement for the Makefile based build system. This replacement build system uses CMake for configuring and compiling the tests, and LIT for running them. LIT will report both compile and execution times as metrics, available by using the '-o' option to output JSON or '-v' (without '-s').

The advantages of this setup are:
  * Use of the same technologies used in LLVM itself means there's more expertise available to debug, fix and extend, and fewer disparate command line flags to remember.
  * A common-off-the-shelf solution means there's much less custom code to maintain, only glue.
  * More readable output by default, more debuggable by far.
  * The ability to run just one test, quickly and easily.
  * The ability to perform test repeats without cleaning and recompiling first.
  * Because the command line is simple, LNT is not required to run a test.

The disadvantages are:
  * Not all of the features in the Makefile system are supported or are going to be. For example, llvm-gcc support, compiling with opt & llc. Clang is stable and expected now - we don't need these workarounds.

This commit introduces the CMake and LIT glue and a python script that can parse all the Makefiles in SingleSource & MultiSource and can generate equivalent CMakeLists.txt. The intention is that this is run once and the resulting files checked in then never used again.

With this script run, the entire test-suite builds and runs. Example command lines:

  $ mkdir build; cd build
  $ cmake -DCMAKE_C_COMPILER=/path/to/my/clang .. # CMake finds clang++ and introspects the target OS/arch/endianness
  $ make -j12
  $ llvm-lit -sv .

Ninja also works, but the way the ninja makefiles are generated means there is no way to build all tests in an arbitrary subtree (as LNT's --only-test does). With ninja you can build all tests or a single test.

All feedback welcome! Please also see the discussion on llvm-commits.

Repository:
  rL LLVM

http://reviews.llvm.org/D14046

Files:
  CMakeLists.txt
  cmake/Makefile2CMakeLists.py
  cmake/lit-test-template.in
  cmake/modules/DetectArchitecture.c
  cmake/modules/DetectArchitecture.cmake
  cmake/modules/FindYACC.cmake
  cmake/modules/MakefileFunctions.cmake
  cmake/modules/SingleMultiSource.cmake
  lit.cfg
  tools/CMakeLists.txt

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D14046.38344.patch
Type: text/x-patch
Size: 31754 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20151025/f334c339/attachment-0001.bin>