LIT-ify the test-suite

[Hal Finkel]

----- Original Message -----
> From: "James Molloy" <james at jamesmolloy.co.uk>
> To: "LLVM Commits" <llvm-commits at cs.uiuc.edu>
> Sent: Friday, October 24, 2014 9:20:50 AM
> Subject: LIT-ify the test-suite
> 
> Hi all,
> 
> 
> We've been trying to improve the LNT test driver to be a bit more
> clever with reruns, and have run into the problem that it is very
> tightly coupled to the Makefile system in the test-suite.
> 
> 
> The Makefile system was obviously designed some time ago before Clang
> was production ready, and it has a bunch of features to support
> running through LLC and suchlike that add serious complexity. It's
> nontrivial to just run a test and get a result!
> 
> 
> So I got to thinking, why don't we just use LIT, the really well
> written test driver that our regression tests use? That would allow
> us to simplify the interface, reduce the number of test drivers to
> 1, have pretty progress bars, export the results in JSON, shuffle
> the test order...
> 
> 
> So I attach an alpha version of a lit.cfg for the test-suite. Some
> caveats with this that make it not quite production-ready:
> * It doesn't use perf for timing yet, it just uses python's time()
> interface.
> * It doesn't yet report compile time, but that's actually a one line
> change.
> * The biggest problem: A bunch of the test suites require
> lit.local.cfg files in them to tell lit how to build and run them.
> However, many of the test suite directories are under blanket
> non-LLVM licenses which makes for problems with our legal
> permissions.

I don't understand this last point; are you concerned that the licenses preclude adding an alternative build description? cc'ing Danny.

 -Hal

> 
> 
> So attached are two patches: 0001 adds the lit.cfg - just apply it to
> the test-suite directory, and 0002 adds a selection of
> lit.local.cfgs - for all the subfolders that aren't covered by
> blanket licenses.
> 
> 
> Because it excludes some lit.local.cfgs, some tests won't run.
> However, all the unit tests do, so that's an easy proof of concept:
> 
> 
> $ cd test-suite
> $ CC=clang llvm-lit -sv ./MultiSource/UnitTests
> 
> 
> Please give it a try and let us know your thoughts, so I know how
> hard I should push this.
> 
> 
> Cheers,
> 
> 
> James
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory