LIT-ify the test-suite

Sun Oct 25 06:46:39 PDT 2015

----- Original Message -----
> From: "James Molloy via llvm-commits" <llvm-commits at lists.llvm.org>
> To: "LLVM Commits" <llvm-commits at lists.llvm.org>
> Sent: Sunday, October 25, 2015 8:33:32 AM
> Subject: LIT-ify the test-suite
> 
> Hi all,
> 
> It's about a year later and I've finally found time to work on this
> and get it up to scratch.

Great, thanks!

 -Hal

> 
> 
> Because there's been a mailing list change in the meantime, the
> original proposal and conversation was here:
> http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20141020/241422.html
> 
> 
> The major objection from the version I posted in March was the lack
> of support for restarting failed builds where they left off. I agree
> now that is a showstopper. So I've changed tack slightly, and
> instead of making everything run through LIT, now LIT only runs the
> tests. The tests are now built using CMake+Make/Ninja. So now the
> flow would be:
> 
> 
> $ cd build
> $ cmake -DCMAKE_C_COMPILER=/path/to/clang ..
> $ make -j12
> $ llvm-lit -sv .
> 
> 
> timeit is still used for timing, and compile times + execution times
> appear as metrics in the overall LIT report.
> 
> 
> I think this is a useful change, as it reduces by far the amount of
> custom code required. Now we just need some small amount of CMake
> glue and a small lit.cfg. Also, everyone (should) already know how
> to use both of these tools, so there is no new technology here.
> 
> 
> The patch at http://reviews.llvm.org/D14046 has all of the glue
> within it, as well as a python script that can parse the current
> Makefiles in SingleSource/MultiSource and produce equivalent
> CMakeLists.txt. The intention would be that this script would get
> run once and the resulting CMakeLists.txt committed to the
> repository.
> 
> 
> With this patch applied and the script run, the entire test-suite can
> be built and tested successfully.
> 
> 
> Features that are not implemented yet:
> * Cross-compilation
> * The --use-perf option (using timeit.sh instead of timeit.exe).
> * I can't think of any others, but I'm sure there are some.
> 
> 
> What do people think? is this a good direction to go down? What
> features are people using in their buildbots that the new system
> would miss?
> 
> 
> Cheers,
> 
> 
> James
> 
> 
> ---------- Forwarded message ---------
> From: James Molloy < james at jamesmolloy.co.uk >
> Date: Tue, 31 Mar 2015 at 16:21
> Subject: Re: LIT-ify the test-suite
> To: Daniel Dunbar < daniel at zuster.org >
> Cc: LLVM Commits < llvm-commits at cs.uiuc.edu >
> 
> 
> 
> Hi Chris, Daniel,
> 
> 
> 
> Let's get this going again :)
> 
> 
> I've taken on board your points and have put a work-in-progress patch
> up at http://reviews.llvm.org/D8721 . The autogenerated
> lit.local.cfg's still aren't perfect, and there are still some
> missing (legal issues have not subsided I'm afraid!) but we're
> getting there.
> 
> 
> Regarding Makefiles - after discussion at the LLVM conference in
> November I'm still not really sure what the Makefiles provide over
> LIT in terms of incremental development. They allow one particular
> source file to fail, and then to remake and everything picks up at
> that source file, but I really don't think that adds much value on
> top of LIT, which can do one test at a time.
> 
> 
> Granted, the multisource tests consist of multiple files so this is
> where Makefiles win, but is that really a big enough pull? can we
> not just deal with the test granularity?
> 
> 
> Cheers,
> 
> 
> James
> 
> On Mon, 27 Oct 2014 at 00:41 James Molloy < james at jamesmolloy.co.uk >
> wrote:
> 
> 
> 
> Hi Daniel,
> 
> 
> Thanks for the comments!
> 
> 
> 1. I had trouble getting modules loadable from lit.local.cfgs. I'll
> try again, because I agree separating it out is a good thing.
> 
> 
> 2. I didn't do this because I had trouble getting the common module
> loaded. I also had trouble using a different format in different
> subdirectories - I didn't know if this was supposed to work. I can
> go back to this.
> 
> 
> 3. Sure.
> 
> 
> 4. The rationale for this was the lack of a need to compile a helper
> tool. It's also a fairly small python function. I didn't see any
> particular slowdown, I just felt it was a nicer design unless there
> is a performance reason otherwise.
> 
> 
> 5. Hmm. I think scraping Makefiles is pretty nasty, I'd tend more
> towards something like an equivalent of LLVMBuild.txt, which could
> be used to generate both lit.local.cfgs and Makefiles. What do you
> think about that?
> 
> 
> ----
> 
> 
> 1/2. My intention on sending my original email was to try and work
> out what people use the Makefiles for. They have a *lot* of
> features, and I really didn't know which of them people were using.
> It's been fairly clear from the responses that people still require
> incremental development, which really needs a dependency-based
> system like Make. I don't think putting all that complexity in LIT
> is a good idea at all, so at the moment I'm erring towards keeping
> both systems, for the different use cases. That way complexity can
> continue to be piled into the Makefiles without ruining the "Just
> Run It!" use case.
> 
> 
> What do you think?
> 
> 
> Cheers,
> 
> 
> James
> 
> 
> On 26 October 2014 18:39, Daniel Dunbar < daniel at zuster.org > wrote:
> 
> 
> 
> Hi James,
> 
> 
> Thanks for working on this!
> 
> 
> Comments on the specific patch:
> 
> 
> --
> 
> 
> 1. For tidyness, I think it would be good to move the test format
> implementation into its own module (and just change lit.cfg to
> update sys.path relative to __file__) instead of having it in the
> lit.cfg.
> 
> 
> 2. Instead of having the diamond inheritance in the test format, I
> would just define two different test formats the SingleSrc format
> and the MultiSrc format, and in the lit.local.cfg file for the
> top-level directories, set the appropriate format. They can of
> course inherit from a common base in the same module (see #1).
> 
> 
> 3. Re "FIXME: Push this into lit.formats" I disagree and think it
> should just be local to the test-suite, it isn't a configuration any
> other project is likely to use. There should be no harm in having it
> be independent though, this is a use case lit should support.
> 
> 
> 4. I'm skeptical about moving fpcmp into the Python implementation --
> why not just use a subprocess here? The fpcmp implementation has had
> bugs over the years, and actually its runtime performance does have
> an impact on the test suite time (IIRC).
> 
> 
> 5. As for the lit.local.cfg issue -- one way to deal with this is to
> move all of the per-test configuration information (hash outputs,
> command line flags, etc.) into a well defined format. This could
> either be a JSON file in the local directory, or it could be we just
> say the Makefile needs to follow a very strict format for now, and
> have the Lit test format scrape the information out of the local
> Makefile (and barf if it can't parse it).
> 
> 
> 6. Things like SMALL/LARGE problem size should probably always be
> pushed into the C/C++ flags, not something individual lit.local.cfg
> files have to check. Similarly, maybe "-I%S" should also be a
> default every test gets.
> 
> 
> --
> 
> 
> As for the general approach, I have also worked on this a couple
> times over the years, and the blocker that has prevented me from
> ever checking something in is:
> 
> 
> 1. How to manage the incremental development part of things? This
> approach is great for automation and approachability, but for people
> who are very familiar with the way the Makefiles work there are a
> lot of useful workflows that aren't supported anymore (e.g., the
> ability to manually rebuild different parts, the different ways to
> tweak the config, etc.).
> 
> 
> 2. Where does this leave the Makefile based config?
> 
> 
> Any thoughts on those?
> 
> 
> - Daniel
> 
> 
> 
> 
> On Fri, Oct 24, 2014 at 7:20 AM, James Molloy <
> james at jamesmolloy.co.uk > wrote:
> 
> 
> 
> 
> 
> Hi all,
> 
> 
> We've been trying to improve the LNT test driver to be a bit more
> clever with reruns, and have run into the problem that it is very
> tightly coupled to the Makefile system in the test-suite.
> 
> 
> The Makefile system was obviously designed some time ago before Clang
> was production ready, and it has a bunch of features to support
> running through LLC and suchlike that add serious complexity. It's
> nontrivial to just run a test and get a result!
> 
> 
> So I got to thinking, why don't we just use LIT, the really well
> written test driver that our regression tests use? That would allow
> us to simplify the interface, reduce the number of test drivers to
> 1, have pretty progress bars, export the results in JSON, shuffle
> the test order...
> 
> 
> So I attach an alpha version of a lit.cfg for the test-suite. Some
> caveats with this that make it not quite production-ready:
> * It doesn't use perf for timing yet, it just uses python's time()
> interface.
> * It doesn't yet report compile time, but that's actually a one line
> change.
> * The biggest problem: A bunch of the test suites require
> lit.local.cfg files in them to tell lit how to build and run them.
> However, many of the test suite directories are under blanket
> non-LLVM licenses which makes for problems with our legal
> permissions.
> 
> 
> So attached are two patches: 0001 adds the lit.cfg - just apply it to
> the test-suite directory, and 0002 adds a selection of
> lit.local.cfgs - for all the subfolders that aren't covered by
> blanket licenses.
> 
> 
> Because it excludes some lit.local.cfgs, some tests won't run.
> However, all the unit tests do, so that's an easy proof of concept:
> 
> 
> $ cd test-suite
> $ CC=clang llvm-lit -sv ./MultiSource/UnitTests
> 
> 
> Please give it a try and let us know your thoughts, so I know how
> hard I should push this.
> 
> 
> Cheers,
> 
> 
> James
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> 
> 
> 
> 
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory