LIT-ify the test-suite

[Daniel Dunbar]

Hi James,

Thanks for working on this!

Comments on the specific patch:

--

1. For tidyness, I think it would be good to move the test format
implementation into its own module (and just change lit.cfg to update
sys.path relative to __file__) instead of having it in the lit.cfg.

2. Instead of having the diamond inheritance in the test format, I would
just define two different test formats the SingleSrc format and the
MultiSrc format, and in the lit.local.cfg file for the top-level
directories, set the appropriate format. They can of course inherit from a
common base in the same module (see #1).

3. Re "FIXME: Push this into lit.formats" I disagree and think it should
just be local to the test-suite, it isn't a configuration any other project
is likely to use. There should be no harm in having it be independent
though, this is a use case lit should support.

4. I'm skeptical about moving fpcmp into the Python implementation -- why
not just use a subprocess here? The fpcmp implementation has had bugs over
the years, and actually its runtime performance does have an impact on the
test suite time (IIRC).

5. As for the lit.local.cfg issue -- one way to deal with this is to move
all of the per-test configuration information (hash outputs, command line
flags, etc.) into a well defined format. This could either be a JSON file
in the local directory, or it could be we just say the Makefile needs to
follow a very strict format for now, and have the Lit test format scrape
the information out of the local Makefile (and barf if it can't parse it).

6. Things like SMALL/LARGE problem size should probably always be pushed
into the C/C++ flags, not something individual lit.local.cfg files have to
check. Similarly, maybe "-I%S" should also be a default every test gets.

--

As for the general approach, I have also worked on this a couple times over
the years, and the blocker that has prevented me from ever checking
something in is:

1. How to manage the incremental development part of things? This approach
is great for automation and approachability, but for people who are very
familiar with the way the Makefiles work there are a lot of useful
workflows that aren't supported anymore (e.g., the ability to manually
rebuild different parts, the different ways to tweak the config, etc.).

2. Where does this leave the Makefile based config?

Any thoughts on those?

 - Daniel

On Fri, Oct 24, 2014 at 7:20 AM, James Molloy <james at jamesmolloy.co.uk>
wrote:

> Hi all,
>
> We've been trying to improve the LNT test driver to be a bit more clever
> with reruns, and have run into the problem that it is very tightly coupled
> to the Makefile system in the test-suite.
>
> The Makefile system was obviously designed some time ago before Clang was
> production ready, and it has a bunch of features to support running through
> LLC and suchlike that add serious complexity. It's nontrivial to just run a
> test and get a result!
>
> So I got to thinking, why don't we just use LIT, the really well written
> test driver that our regression tests use? That would allow us to simplify
> the interface, reduce the number of test drivers to 1, have pretty progress
> bars, export the results in JSON, shuffle the test order...
>
> So I attach an alpha version of a lit.cfg for the test-suite. Some caveats
> with this that make it not quite production-ready:
>   * It doesn't use perf for timing yet, it just uses python's time()
> interface.
>   * It doesn't yet report compile time, but that's actually a one line
> change.
>   * The biggest problem: A bunch of the test suites require lit.local.cfg
> files in them to tell lit how to build and run them. However, many of the
> test suite directories are under blanket non-LLVM licenses which makes for
> problems with our legal permissions.
>
> So attached are two patches: 0001 adds the lit.cfg - just apply it to the
> test-suite directory, and 0002 adds a selection of lit.local.cfgs - for all
> the subfolders that aren't covered by blanket licenses.
>
> Because it excludes some lit.local.cfgs, some tests won't run. However,
> all the unit tests do, so that's an easy proof of concept:
>
> $ cd test-suite
> $ CC=clang llvm-lit -sv ./MultiSource/UnitTests
>
> Please give it a try and let us know your thoughts, so I know how hard I
> should push this.
>
> Cheers,
>
> James
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20141026/82c62abb/attachment.html>