[PATCH] D14046: [test-suite] Introduce a new CMake+LIT based harness

[Matthias Braun via llvm-commits]

MatzeB added a subscriber: MatzeB.
MatzeB added a comment.

First: I really like where this is going! Thanks for working on this!

Some early comments after playing around with it for a while:

- I could not get lit to show the commandlines of tests that do not fail. This would be very nice to have for the benchmarks.
- I think we need cmake to generate a lit.cfg in the build directory or it will fail when the builddirectory is not a subdirectory of the test-suite.

I'd like to go further in simplifying the test commandlines. Right now the .test files still contain all the RunSafely.sh/timeit/... stuff. IMO this should be part of the benchmark driver (=lit.cfg). I think we should allow "RUN: " lines for the benchmark and "VERIFY:" lines for checking the benchmark results. With a custom lit test we should be able to prefix the RunSafely stuff automatically for all RUN: lines and run the VERIFY: lines as is. This is probably also useful if we are going to make the benchmark driver smarter and allow it for example to run a benchmark multiple times and aggregate the times :) In fact I started hacking on this and may have a proof of concept for this in the next days...


Repository:
  rL LLVM

http://reviews.llvm.org/D14046