LIT-ify the test-suite

[Steven Wu]

Hi James

I haven’t tried your patch but I don’t think it is quite the same. When I talked about dependency, I mean dependencies in a multisrc testcase. One multi source testcase can sometimes take quite a long time, especially trying to run pgo which involves build-run-build-run. Makefile comes in handy when you can control what to rebuild in the project. I don’t think it is that important to the majority of the user, just a feature nice to have.
I will let you know after I try the patch.

Steven
 
> On Oct 24, 2014, at 1:33 PM, James Molloy <james at jamesmolloy.co.uk> wrote:
> 
> Hi Steven,
> 
> Thanks for your feedback!
> 
> Would lit's --incremental (-i) option fill this requirement? It runs previously failed tests first.
> 
> Cheers,
> 
> James
> 
> On 24 October 2014 21:20, Steven Wu <stevenwu at apple.com <mailto:stevenwu at apple.com>> wrote:
> Hi James
> 
> Thank you for suggesting fixes for build system in test-suite. I spent quite lots of time previously trying to read through the makefiles and adding new features. I found the makefile we have are extremely hard to understand and they are also impossible to modify without breaking something else. I like the lit approach because it certainly makes debugging the system become a lot easier.
> 
> I think local.cfg are fine. We already have plenty of Makefile that specifies the rules for single project. It is ok as long as we are not making it worse (and I don’t have better idea to deal with this problem). Ideally, we don’t need to go back modify these files once they are set up.
> 
> However, I really want something similar to the dependency check in Makefile. test-suite is not small regression test and many of them takes long time to build and run. The ability to continue from where it stopped is valuable to someone who is working on the test-suite or someone who is debugging the compiler.
> 
> Thanks
> 
> Steven
> 
> > On Oct 24, 2014, at 7:20 AM, James Molloy <james at jamesmolloy.co.uk <mailto:james at jamesmolloy.co.uk>> wrote:
> >
> > Hi all,
> >
> > We've been trying to improve the LNT test driver to be a bit more clever with reruns, and have run into the problem that it is very tightly coupled to the Makefile system in the test-suite.
> >
> > The Makefile system was obviously designed some time ago before Clang was production ready, and it has a bunch of features to support running through LLC and suchlike that add serious complexity. It's nontrivial to just run a test and get a result!
> >
> > So I got to thinking, why don't we just use LIT, the really well written test driver that our regression tests use? That would allow us to simplify the interface, reduce the number of test drivers to 1, have pretty progress bars, export the results in JSON, shuffle the test order...
> >
> > So I attach an alpha version of a lit.cfg for the test-suite. Some caveats with this that make it not quite production-ready:
> >   * It doesn't use perf for timing yet, it just uses python's time() interface.
> >   * It doesn't yet report compile time, but that's actually a one line change.
> >   * The biggest problem: A bunch of the test suites require lit.local.cfg files in them to tell lit how to build and run them. However, many of the test suite directories are under blanket non-LLVM licenses which makes for problems with our legal permissions.
> >
> > So attached are two patches: 0001 adds the lit.cfg - just apply it to the test-suite directory, and 0002 adds a selection of lit.local.cfgs - for all the subfolders that aren't covered by blanket licenses.
> >
> > Because it excludes some lit.local.cfgs, some tests won't run. However, all the unit tests do, so that's an easy proof of concept:
> >
> > $ cd test-suite
> > $ CC=clang llvm-lit -sv ./MultiSource/UnitTests
> >
> > Please give it a try and let us know your thoughts, so I know how hard I should push this.
> >
> > Cheers,
> >
> > James
> > <0001-Add-lit.cfg.patch><0002-Add-lit.local.cfgs.patch>_______________________________________________
> > llvm-commits mailing list
> > llvm-commits at cs.uiuc.edu <mailto:llvm-commits at cs.uiuc.edu>
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits <http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits>
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20141024/a111edc0/attachment.html>