LIT-ify the test-suite

[Hal Finkel]

Hi James,

I think that you should contribute all of the new lit files under the LLVM license for all directories. We can either:

 1. Add a license note in the files and place a note in the top-level LICENSE.TXT that the alternative licenses do not apply to the build-system files (I imagine that they're not intended to apply to the existing Makefiles either, so perhaps we can clarify that as a separate issue).

 2. You can make peer foo-build directories and place the new files in there. This avoids the issue for you all together.

While I think that some clarification here is certainly in order, overall I'm fairly certain this will be a non-issue.

Thanks again,
Hal

----- Original Message -----
> From: "James Molloy" <james at jamesmolloy.co.uk>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "LLVM Commits" <llvm-commits at cs.uiuc.edu>, "Daniel Berlin" <dberlin at dberlin.org>
> Sent: Sunday, October 26, 2014 6:16:01 AM
> Subject: Re: LIT-ify the test-suite
> 
> Hi Hal,
> 
> 
> That's right. Specifically, we have permission to upstream patches
> that modify or add files under the LLVM license. Because some of the
> directories have blanket licenses, it is unclear whether we can add
> additional build descriptions and license those descriptions under
> the LLVM license.
> 
> 
> I believe the only way we can do this currently is have our legal
> department take a look, which would likely mean them examining every
> license in the test-suite for virality clauses. IANAL, and everyone
> I've asked has agreed that this case is sufficiently not clear cut
> that we would have to involve legal, which may take some time.
> 
> It can be done though, it just requires effort and time so I am
> trying to sound out the community to see if it would be effort well
> spent. The feedback so far suggests so!
> 
> 
> I usually try very hard to conceal our internal procedures as the
> community rightly doesn't care, but in this case I've failed. Sorry!
> :(
> 
> 
> Cheers,
> 
> 
> James
> 
> 
> On Sunday, 26 October 2014, Hal Finkel < hfinkel at anl.gov > wrote:
> 
> 
> ----- Original Message -----
> > From: "James Molloy" < james at jamesmolloy.co.uk >
> > To: "LLVM Commits" < llvm-commits at cs.uiuc.edu >
> > Sent: Friday, October 24, 2014 9:20:50 AM
> > Subject: LIT-ify the test-suite
> > 
> > Hi all,
> > 
> > 
> > We've been trying to improve the LNT test driver to be a bit more
> > clever with reruns, and have run into the problem that it is very
> > tightly coupled to the Makefile system in the test-suite.
> > 
> > 
> > The Makefile system was obviously designed some time ago before
> > Clang
> > was production ready, and it has a bunch of features to support
> > running through LLC and suchlike that add serious complexity. It's
> > nontrivial to just run a test and get a result!
> > 
> > 
> > So I got to thinking, why don't we just use LIT, the really well
> > written test driver that our regression tests use? That would allow
> > us to simplify the interface, reduce the number of test drivers to
> > 1, have pretty progress bars, export the results in JSON, shuffle
> > the test order...
> > 
> > 
> > So I attach an alpha version of a lit.cfg for the test-suite. Some
> > caveats with this that make it not quite production-ready:
> > * It doesn't use perf for timing yet, it just uses python's time()
> > interface.
> > * It doesn't yet report compile time, but that's actually a one
> > line
> > change.
> > * The biggest problem: A bunch of the test suites require
> > lit.local.cfg files in them to tell lit how to build and run them.
> > However, many of the test suite directories are under blanket
> > non-LLVM licenses which makes for problems with our legal
> > permissions.
> 
> I don't understand this last point; are you concerned that the
> licenses preclude adding an alternative build description? cc'ing
> Danny.
> 
> -Hal
> 
> > 
> > 
> > So attached are two patches: 0001 adds the lit.cfg - just apply it
> > to
> > the test-suite directory, and 0002 adds a selection of
> > lit.local.cfgs - for all the subfolders that aren't covered by
> > blanket licenses.
> > 
> > 
> > Because it excludes some lit.local.cfgs, some tests won't run.
> > However, all the unit tests do, so that's an easy proof of concept:
> > 
> > 
> > $ cd test-suite
> > $ CC=clang llvm-lit -sv ./MultiSource/UnitTests
> > 
> > 
> > Please give it a try and let us know your thoughts, so I know how
> > hard I should push this.
> > 
> > 
> > Cheers,
> > 
> > 
> > James
> > _______________________________________________
> > llvm-commits mailing list
> > llvm-commits at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> > 
> 
> --
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory