LIT-ify the test-suite

[James Molloy]

Hi Steven,

Thanks for your feedback!

Would lit's --incremental (-i) option fill this requirement? It runs
previously failed tests first.

Cheers,

James

On 24 October 2014 21:20, Steven Wu <stevenwu at apple.com> wrote:

> Hi James
>
> Thank you for suggesting fixes for build system in test-suite. I spent
> quite lots of time previously trying to read through the makefiles and
> adding new features. I found the makefile we have are extremely hard to
> understand and they are also impossible to modify without breaking
> something else. I like the lit approach because it certainly makes
> debugging the system become a lot easier.
>
> I think local.cfg are fine. We already have plenty of Makefile that
> specifies the rules for single project. It is ok as long as we are not
> making it worse (and I don’t have better idea to deal with this problem).
> Ideally, we don’t need to go back modify these files once they are set up.
>
> However, I really want something similar to the dependency check in
> Makefile. test-suite is not small regression test and many of them takes
> long time to build and run. The ability to continue from where it stopped
> is valuable to someone who is working on the test-suite or someone who is
> debugging the compiler.
>
> Thanks
>
> Steven
>
> > On Oct 24, 2014, at 7:20 AM, James Molloy <james at jamesmolloy.co.uk>
> wrote:
> >
> > Hi all,
> >
> > We've been trying to improve the LNT test driver to be a bit more clever
> with reruns, and have run into the problem that it is very tightly coupled
> to the Makefile system in the test-suite.
> >
> > The Makefile system was obviously designed some time ago before Clang
> was production ready, and it has a bunch of features to support running
> through LLC and suchlike that add serious complexity. It's nontrivial to
> just run a test and get a result!
> >
> > So I got to thinking, why don't we just use LIT, the really well written
> test driver that our regression tests use? That would allow us to simplify
> the interface, reduce the number of test drivers to 1, have pretty progress
> bars, export the results in JSON, shuffle the test order...
> >
> > So I attach an alpha version of a lit.cfg for the test-suite. Some
> caveats with this that make it not quite production-ready:
> >   * It doesn't use perf for timing yet, it just uses python's time()
> interface.
> >   * It doesn't yet report compile time, but that's actually a one line
> change.
> >   * The biggest problem: A bunch of the test suites require
> lit.local.cfg files in them to tell lit how to build and run them. However,
> many of the test suite directories are under blanket non-LLVM licenses
> which makes for problems with our legal permissions.
> >
> > So attached are two patches: 0001 adds the lit.cfg - just apply it to
> the test-suite directory, and 0002 adds a selection of lit.local.cfgs - for
> all the subfolders that aren't covered by blanket licenses.
> >
> > Because it excludes some lit.local.cfgs, some tests won't run. However,
> all the unit tests do, so that's an easy proof of concept:
> >
> > $ cd test-suite
> > $ CC=clang llvm-lit -sv ./MultiSource/UnitTests
> >
> > Please give it a try and let us know your thoughts, so I know how hard I
> should push this.
> >
> > Cheers,
> >
> > James
> >
> <0001-Add-lit.cfg.patch><0002-Add-lit.local.cfgs.patch>_______________________________________________
> > llvm-commits mailing list
> > llvm-commits at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20141024/dc71c824/attachment.html>