LIT-ify the test-suite

[Chris Matthews]

James, this is a great idea.  LIT is a much better way to be doing this.  

I wonder if it might be better to actually add the performance timing part of the code into LIT proper as a new feature?

I will try your patch next week!

We might have to change some stuff in LNT.  I am fine with that.
  
> On Oct 24, 2014, at 7:20 AM, James Molloy <james at jamesmolloy.co.uk> wrote:
> 
> Hi all,
> 
> We've been trying to improve the LNT test driver to be a bit more clever with reruns, and have run into the problem that it is very tightly coupled to the Makefile system in the test-suite.
> 
> The Makefile system was obviously designed some time ago before Clang was production ready, and it has a bunch of features to support running through LLC and suchlike that add serious complexity. It's nontrivial to just run a test and get a result!
> 
> So I got to thinking, why don't we just use LIT, the really well written test driver that our regression tests use? That would allow us to simplify the interface, reduce the number of test drivers to 1, have pretty progress bars, export the results in JSON, shuffle the test order...
> 
> So I attach an alpha version of a lit.cfg for the test-suite. Some caveats with this that make it not quite production-ready:
>   * It doesn't use perf for timing yet, it just uses python's time() interface.
>   * It doesn't yet report compile time, but that's actually a one line change.
>   * The biggest problem: A bunch of the test suites require lit.local.cfg files in them to tell lit how to build and run them. However, many of the test suite directories are under blanket non-LLVM licenses which makes for problems with our legal permissions.
> 
> So attached are two patches: 0001 adds the lit.cfg - just apply it to the test-suite directory, and 0002 adds a selection of lit.local.cfgs - for all the subfolders that aren't covered by blanket licenses.
> 
> Because it excludes some lit.local.cfgs, some tests won't run. However, all the unit tests do, so that's an easy proof of concept:
> 
> $ cd test-suite
> $ CC=clang llvm-lit -sv ./MultiSource/UnitTests
> 
> Please give it a try and let us know your thoughts, so I know how hard I should push this.
> 
> Cheers,
> 
> James
> <0001-Add-lit.cfg.patch><0002-Add-lit.local.cfgs.patch>_______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits