LIT-ify the test-suite

[James Molloy]

Hi Hal,

That's right. Specifically, we have permission to upstream patches that
modify or add files under the LLVM license. Because some of the directories
have blanket licenses, it is unclear whether we can add additional build
descriptions and license those descriptions under the LLVM license.

I believe the only way we can do this currently is have our legal
department take a look, which would likely mean them examining every
license in the test-suite for virality clauses. IANAL, and everyone I've
asked has agreed that this case is sufficiently not clear cut that we would
have to involve legal, which may take some time.

It can be done though, it just requires effort and time so I am trying to
sound out the community to see if it would be effort well spent. The
feedback so far suggests so!

I usually try very hard to conceal our internal procedures as the community
rightly doesn't care, but in this case I've failed. Sorry! :(

Cheers,

James

On Sunday, 26 October 2014, Hal Finkel <hfinkel at anl.gov> wrote:

> ----- Original Message -----
> > From: "James Molloy" <james at jamesmolloy.co.uk <javascript:;>>
> > To: "LLVM Commits" <llvm-commits at cs.uiuc.edu <javascript:;>>
> > Sent: Friday, October 24, 2014 9:20:50 AM
> > Subject: LIT-ify the test-suite
> >
> > Hi all,
> >
> >
> > We've been trying to improve the LNT test driver to be a bit more
> > clever with reruns, and have run into the problem that it is very
> > tightly coupled to the Makefile system in the test-suite.
> >
> >
> > The Makefile system was obviously designed some time ago before Clang
> > was production ready, and it has a bunch of features to support
> > running through LLC and suchlike that add serious complexity. It's
> > nontrivial to just run a test and get a result!
> >
> >
> > So I got to thinking, why don't we just use LIT, the really well
> > written test driver that our regression tests use? That would allow
> > us to simplify the interface, reduce the number of test drivers to
> > 1, have pretty progress bars, export the results in JSON, shuffle
> > the test order...
> >
> >
> > So I attach an alpha version of a lit.cfg for the test-suite. Some
> > caveats with this that make it not quite production-ready:
> > * It doesn't use perf for timing yet, it just uses python's time()
> > interface.
> > * It doesn't yet report compile time, but that's actually a one line
> > change.
> > * The biggest problem: A bunch of the test suites require
> > lit.local.cfg files in them to tell lit how to build and run them.
> > However, many of the test suite directories are under blanket
> > non-LLVM licenses which makes for problems with our legal
> > permissions.
>
> I don't understand this last point; are you concerned that the licenses
> preclude adding an alternative build description? cc'ing Danny.
>
>  -Hal
>
> >
> >
> > So attached are two patches: 0001 adds the lit.cfg - just apply it to
> > the test-suite directory, and 0002 adds a selection of
> > lit.local.cfgs - for all the subfolders that aren't covered by
> > blanket licenses.
> >
> >
> > Because it excludes some lit.local.cfgs, some tests won't run.
> > However, all the unit tests do, so that's an easy proof of concept:
> >
> >
> > $ cd test-suite
> > $ CC=clang llvm-lit -sv ./MultiSource/UnitTests
> >
> >
> > Please give it a try and let us know your thoughts, so I know how
> > hard I should push this.
> >
> >
> > Cheers,
> >
> >
> > James
> > _______________________________________________
> > llvm-commits mailing list
> > llvm-commits at cs.uiuc.edu <javascript:;>
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> >
>
> --
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20141026/8ebf950c/attachment.html>