[PATCH] D14046: [test-suite] Introduce a new CMake+LIT based harness

[Renato Golin via llvm-commits]

On 25 October 2015 at 13:29, James Molloy via llvm-commits
<llvm-commits at lists.llvm.org> wrote:
> The advantages of this setup are:
>   * Use of the same technologies used in LLVM itself means there's more expertise available to debug, fix and extend, and fewer disparate command line flags to remember.
>   * A common-off-the-shelf solution means there's much less custom code to maintain, only glue.

LNT is pretty stable, but I agree, less is more.


>   * More readable output by default, more debuggable by far.

This is one of the main problems of the current setup, for me. Finding
anything in the test.log takes ages. This this follows the same
pattern as check-all, then the buildbots will also give us much better
answers than "foo.test execute time".


>   * The ability to run just one test, quickly and easily.

We can do that already, what's different?


>   * The ability to perform test repeats without cleaning and recompiling first.

That'd be great for benchmark mode.


>   * Because the command line is simple, LNT is not required to run a test.

Does that support all features of LNT? The ones I use are:

 --build-mode
 --optimize-option
 --benchmarking-only
 --build-threads=N
 --threads=1
 --multisample=3
 --use-perf
 --no-timestamp
 --small
 --only-test

And also CFLAGS, CXXFLAGS and LDFLAGS environment variables.


> This commit introduces the CMake and LIT glue and a python script that can parse all the Makefiles in SingleSource & MultiSource and can generate equivalent CMakeLists.txt. The intention is that this is run once and the resulting files checked in then never used again.

What's the benefit over committing the CMakeLists.txt files?


>   $ mkdir build; cd build

I'm assuming this is inside the test-suite source directory.

>   $ cmake -DCMAKE_C_COMPILER=/path/to/my/clang .. # CMake finds clang++ and introspects the target OS/arch/endianness
>   $ make -j12
>   $ llvm-lit -sv .

This sounds very simple, and could easily be integrated into the main
CMake files to do a "make test-suite" from the root build dir, if you
check-out the test-suite inside src/projects.


> Ninja also works, but the way the ninja makefiles are generated means there is no way to build all tests in an arbitrary subtree (as LNT's --only-test does). With ninja you can build all tests or a single test.

TBH, I never needed to do that, anyway. :)

Overall, I think this is a great idea. Unfortunately, I'm not
comfortable enough with either Python or CMake to review the code. But
I'm very supportive of moving the test-suite to LIT, as that would
simplify my test-suite bots as well as encourage more people to run
the test-suite when committing big patches.

cheers,
--renato