[llvm-dev] RFC: Add bitcode tests to test-suite

Hal Finkel via llvm-dev llvm-dev at lists.llvm.org
Thu Feb 18 10:52:38 PST 2016 

----- Original Message -----

> From: "Hal Finkel via llvm-dev" <llvm-dev at lists.llvm.org>
> To: "David Blaikie" <dblaikie at gmail.com>
> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>
> Sent: Thursday, February 18, 2016 12:35:39 PM
> Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite

> ----- Original Message -----

> > From: "David Blaikie via llvm-dev" <llvm-dev at lists.llvm.org>
> 
> > To: "Mehdi Amini" <mehdi.amini at apple.com>
> 
> > Cc: "llvm-dev" <llvm-dev at lists.llvm.org>
> 
> > Sent: Thursday, February 18, 2016 12:15:52 PM
> 
> > Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite
> 

> > On Thu, Feb 18, 2016 at 10:03 AM, Mehdi Amini via llvm-dev <
> > llvm-dev at lists.llvm.org > wrote:
> 

> > > > On Feb 18, 2016, at 8:42 AM, Philip Reames via llvm-dev <
> > > > llvm-dev at lists.llvm.org > wrote:
> > 
> 
> > > >
> > 
> 
> > > > On 02/18/2016 06:54 AM, Hal Finkel via llvm-dev wrote:
> > 
> 
> > > >> Hi Chandler, et al.,
> > 
> 
> > > >>
> > 
> 
> > > >> While this proposal to put IR into the test suite technically
> > > >> non-problematic, I've convinced myself that this is a
> > > >> suboptimal
> > > >> direction for the LLVM project. Here's what I think would be
> > > >> better:
> > 
> 
> > > >>
> > 
> 
> > > >> - We create a test-suite/Frontends directory, and open this
> > > >> directory to actively-maintained external frontends, subject
> > > >> to
> > > >> the following restrictions:
> > 
> 
> > > >>
> > 
> 
> > > >> - The frontend must be actively maintained, and the project
> > > >> must
> > > >> agree to actively maintain the test-suite version
> > 
> 
> > > >> - The frontend must use the LLVM API (either C or C++) - no
> > > >> printing textual IR
> > 
> 
> > > >> - The frontend must have no significant (non-optional)
> > > >> dependencies outside of LLVM itself, or things on which LLVM
> > > >> itself depends
> > 
> 
> > > >> - The frontend must have regression tests and
> > > >> benchmarks/correctness tests providing significant coverage of
> > > >> the frontend and its associated code generation
> > 
> 
> > > >>
> > 
> 
> > > >> Here's the quid pro quo:
> > 
> 
> > > >>
> > 
> 
> > > >> - The LLVM community gains additional testing coverage (which
> > > >> we
> > > >> definitely need)
> > 
> 
> > > >> - The LLVM community gains extra insight into how its APIs are
> > > >> being used (hopefully allowing us to make more-informed
> > > >> decisions
> > > >> about how to update them)
> > 
> 
> > > >>
> > 
> 
> > > >> - The frontend gains free API updates
> > 
> 
> > > >> - The frontend's use of LLVM will be more stable
> > 
> 
> > > >>
> > 
> 
> > > >> This involves extra work for everybody, but will help us all
> > > >> deliver higher-quality products. Plus, given the constant
> > > >> discussions about the difficulty for external projects to
> > > >> follow
> > > >> API updates, etc., this is a good way to help address those
> > > >> difficulties.
> > 
> 
> > > >>
> > 
> 
> > > >> The fact that Halide will provide extra coverage of our vector
> > > >> code generation (aside from whatever we happen to produce from
> > > >> our autovectorizers), and our JIT infrastructure, makes it a
> > > >> good
> > > >> candidate for this. Intel's ispc, POCL, (maybe whatever bit of
> > > >> Mesa uses LLVM), etc. would also be natural candidates should
> > > >> the
> > > >> projects be interested.
> > 
> 
> > > > I think this is a really bad tradeoff and am strongly opposed
> > > > to
> > > > this proposal.
> > 
> 
> > > >
> > 
> 
> > > > If we want to focus on improving test coverage, that's
> > > > reasonable,
> > > > but doing so at the cost of requiring LLVM contributors to
> > > > maintain everyone's frontend is not a reasonable approach.
> > 
> 
> > > >
> > 
> 
> > > > A couple of alternate approaches which might be worth
> > > > considering:
> > 
> 
> > > > 1) The IR corpus approach mentioned previously. So long as
> > > > external
> > > > teams are willing to update the corpus regularly (weekly), this
> > > > gives most of the backend coverage with none of the maintenance
> > > > burden.
> > 
> 
> > > > 2) Use coverage information to determine which code paths
> > > > Halide
> > > > covers which are not covered by existing unit tests. Work to
> > > > improve those unit tests. Using something along the lines with
> > > > a
> > > > mutation testing (i.e. change the source code and see what
> > > > breaks), combined with test reduction (bugpoint), could greatly
> > > > improve our test coverage in tree fairly quickly. This would
> > > > require a lot of work from a single contributor, but that's
> > > > much
> > > > better than requiring a lot of work from all contributors.
> > 
> 

> > > I support the view of Philip on this topic.
> > 
> 
> > > I think that having the bitcode without the burden of the API and
> > > linking is the best trade-off for LLVM.
> > 
> 

> > > About the bitcode, I'm not convince that it needs to be updated
> > > that
> > > often: we're interested in coverage for LLVM, not trying to
> > > validate
> > > that we support correctly frontend X or Y in a particular
> > > version.
> > > I'd even argue that if Halide version 13 generates a very
> > > different
> > > IR than Halide 12, then we should keep the Halide 12 generated
> > > bitcode in a separate directory because it is likely to stress
> > > LLVM
> > > differently.
> > 
> 

> > Potentially, though I hesitate to make test-suite too much of a
> > dumping ground (it is already, to an extent). Quantity isn't
> > quality
> > and all that
> 
> I agree with this.

Also, the relative value of compile-only tests seems unclear to me. Not that we don't need better testing, because we do, but this kind of testing can also be done with random input generators (llvm-stress, csmith, etc.), and it seems like investing in configurations/improvements for that kind of technology would give us a much better coverage than a few pre-defined bitcode files. Plus, crashes are much easier to debug without a full setup capable of running the code. A lot of the value in the test suite is that the details of how to compile and run the various tests are all uniformly specified, so that, when a miscompile happens, we don't need to worry about how to download/configure/build/run the code to reproduce the problem. 

-Hal 

> > - making sure bots cycle quickly (in a relative sense - of course
> > running the test-suite is going to be a more involved process than
> > our high priority regression 'unit'-y tests) is important for them
> > to be meaningful/actionable by developers & not just noise.
> 

> However, our test suite is already much too small. It is still quite
> common that correctness bugs show up in self hosting and not in the
> test suite, and that's not good. Fast turn around is important, but
> that can be done by having more extensive 'nightly' test sets and
> smaller 'hourly' ones, etc.

> -Hal

> > > I have more questions for Alina. What kind of tests do you have:
> > 
> 

> > > - "the compiler takes the bitcode and generates code without
> > > crashing"
> > 
> 
> > > - "the compiled test runs without crashing"
> > 
> 
> > > - "the compiled test will produce an output that be checked
> > > against
> > > a
> > > reference"
> > 
> 
> > > - "the compiled test is meaningful as a benchmarks"
> > 
> 

> > > All these different aspects of testing can be interesting, but
> > > knowing what we're talking may influence the way forward.
> > 
> 

> > > As mentioned before, the test-suite has a mechanism of "external"
> > > suites, it may be limited right now but could probably be
> > > expanded.
> > > Ideally there could be multiple repositories that would just be
> > > checked out independently (think about the way we build LLVM
> > > alone
> > > or LLVM+clang+libcxx+compiler-rt+...).
> > 
> 

> > > --
> > 
> 
> > > Mehdi
> > 
> 

> > > >>
> > 
> 
> > > >> Thanks again,
> > 
> 
> > > >> Hal
> > 
> 
> > > >>
> > 
> 
> > > >> ----- Original Message -----
> > 
> 

> > > >>> From: "Chandler Carruth" < chandlerc at google.com >
> > 
> 
> > > >>> To: "Hal Finkel" < hfinkel at anl.gov >, "Alina Sbirlea" <
> > > >>> alina.sbirlea at gmail.com >
> > 
> 
> > > >>> Cc: "llvm-dev" < llvm-dev at lists.llvm.org >
> > 
> 
> > > >>> Sent: Wednesday, February 17, 2016 9:34:24 PM
> > 
> 
> > > >>> Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> Some perhaps relevant aspects that make testing users of LLVM
> > > >>> like
> > 
> 
> > > >>> Halide challenging:
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> Halide uses the LLVM C++ APIs, but there isn't a good way to
> > 
> 
> > > >>> lock-step update it. So if we were to directly test Halide,
> > > >>> it
> > 
> 
> > > >>> wouldn't link against the new LLVM.
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> Practically speaking though, the LLVM IR generated by Halide
> > > >>> should
> > 
> 
> > > >>> continue to work with newer LLVM optimizations and code
> > > >>> generation.
> > 
> 
> > > >>> So the idea would be to snapshot the IR in bitcode (which is
> > > >>> at
> > 
> 
> > > >>> least reasonably stable) so that we could replay the tests as
> > > >>> LLVM
> > 
> 
> > > >>> changes. We can freshen the bitcode by re-generating it
> > > >>> periodically
> > 
> 
> > > >>> so it doesn't drift too far from what Halide actually uses.
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> The interesting questions IMO are:
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> 1) Are folks happy using bitcode as the format here? I agree
> > > >>> with
> > > >>> Hal
> > 
> 
> > > >>> that it should be easy since Clang will actually Do The Right
> > > >>> Thing
> > 
> 
> > > >>> if given a bitcode input.
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> 2) Are folks happy with non-execution tests in some cases? I
> > > >>> think
> > 
> 
> > > >>> Alina is looking at whether we can get a runtime library that
> > > >>> will
> > 
> 
> > > >>> allow some of these to actually execute, but at least some of
> > > >>> the
> > 
> 
> > > >>> tests are just snap-shots of a JIT, and would need the full
> > > >>> Halide
> > 
> 
> > > >>> libraries (and introspection) to execute usefully.
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> -Chandler
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> On Wed, Feb 17, 2016 at 7:25 PM Hal Finkel via llvm-dev <
> > 
> 
> > > >>> llvm-dev at lists.llvm.org > wrote:
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> From: "Alina Sbirlea via llvm-dev" < llvm-dev at lists.llvm.org
> > > >>> >
> > 
> 
> > > >>> To: "llvm-dev" < llvm-dev at lists.llvm.org >
> > 
> 
> > > >>> Sent: Wednesday, February 17, 2016 7:25:17 PM
> > 
> 
> > > >>> Subject: [llvm-dev] RFC: Add bitcode tests to test-suite
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> Hi all,
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> TL;DR: Add *.bc to test-suite; llc *.bc; run some.
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> We would like to propose adding bitcode tests to the llvm
> > > >>> test-suite.
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> Recent LLVM bugs [2-4] prompted us to look into upstreaming a
> > > >>> subset
> > 
> 
> > > >>> of the tests the Halide library [1] is running and we'd like
> > > >>> the
> > 
> 
> > > >>> community's feedback on moving forward with this.
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> Halide uses LLVM and can generate bitcode, but we cannot add
> > > >>> C++
> > 
> 
> > > >>> tests to test-suite without including the library itself.
> > 
> 
> > > >>> This proposal is also potentially useful for other cases
> > > >>> where
> > > >>> there
> > 
> 
> > > >>> is no C++ front-end.
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> As a first step we are interested in adding a set of
> > > >>> correctness
> > 
> 
> > > >>> tests, for testing the IR without running the tests. Since
> > > >>> these
> > 
> 
> > > >>> tests are generated, they are not instrumented like the .ll
> > > >>> files
> > > >>> in
> > 
> 
> > > >>> trunk, however we believe checking that llc runs without
> > > >>> errors
> > > >>> is
> > 
> 
> > > >>> still useful.
> > 
> 
> > > >>> The bitcode files for Halide may also be large, so including
> > > >>> them
> > > >>> as
> > 
> 
> > > >>> regression tests is not an option. If the smaller tests are
> > > >>> found
> > > >>> to
> > 
> 
> > > >>> be valuable or covering cases no other tests cover, we can
> > 
> 
> > > >>> instrument them and move them into the llvm trunk further
> > > >>> along,
> > > >>> but
> > 
> 
> > > >>> that is not the goal of this proposal.
> > 
> 
> > > >>> In addition, we're not sure whether the format for the tests
> > > >>> should
> > 
> 
> > > >>> be .ll or .bc, we're open to either.
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> After this first step, we're interested in upstreaming
> > > >>> bitcode
> > > >>> tests
> > 
> 
> > > >>> and also running them.
> > 
> 
> > > >>> We are very interested in tests for multiple architectures,
> > > >>> aarch64
> > 
> 
> > > >>> in particular, since this is where we have seen things break.
> > > >>> This
> > 
> 
> > > >>> may motivate adding .ll files rather than .bc in order to
> > > >>> include
> > 
> 
> > > >>> the "RUN:" target.
> > 
> 
> > > >>> Where would these tests reside and with what directory
> > > >>> structure?
> > 
> 
> > > >>> (similar to test/CodeGen?)
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> Suggestion on what's the best approach for extending the
> > > >>> test-suite
> > 
> 
> > > >>> framework for this proposal are more than welcome.
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> We already have architecture-specific tests in the test suite
> > > >>> (e.g.
> > 
> 
> > > >>> SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and Clang
> > > >>> can
> > > >>> deal
> > 
> 
> > > >>> with IR inputs. I suppose you need to compile some
> > > >>> corresponding
> > 
> 
> > > >>> runtime library, but this does not seem like a big deal
> > > >>> either.
> > 
> 
> > > >>> Mechanically, I don't see this as particularly complicated. I
> > > >>> think
> > 
> 
> > > >>> the real question is: Is this the best way to have a kind of
> > > >>> 'halide
> > 
> 
> > > >>> buildbot' that can inform the LLVM developer community?
> > 
> 
> > > >>>
> > 
> 
> > > >>> -Hal
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> This is just the high-level overview to start off the
> > > >>> discussion,
> > > >>> I'm
> > 
> 
> > > >>> sure there are many more aspects to touch on. Looking forward
> > > >>> to
> > 
> 
> > > >>> your feedback!
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> Thanks,
> > 
> 
> > > >>> Alina
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> [1] http:// halide - lang.org/
> > 
> 
> > > >>> [2] Broken: r259800 => Fixed: r260131
> > 
> 
> > > >>> [3] Broken: r260569 => Fixed: r260701
> > 
> 
> > > >>>
> > 
> 
> > > >>> [4] https://llvm.org/bugs/show_bug.cgi?id=26642
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> _______________________________________________
> > 
> 
> > > >>> LLVM Developers mailing list
> > 
> 
> > > >>> llvm-dev at lists.llvm.org
> > 
> 
> > > >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>>
> > 
> 
> > > >>> --
> > 
> 
> > > >>>
> > 
> 
> > > >>> Hal Finkel
> > 
> 
> > > >>> Assistant Computational Scientist
> > 
> 
> > > >>> Leadership Computing Facility
> > 
> 
> > > >>> Argonne National Laboratory
> > 
> 
> > > >>> _______________________________________________
> > 
> 
> > > >>> LLVM Developers mailing list
> > 
> 
> > > >>> llvm-dev at lists.llvm.org
> > 
> 
> > > >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > 
> 
> > > >>>
> > 
> 
> > > >
> > 
> 
> > > > _______________________________________________
> > 
> 
> > > > LLVM Developers mailing list
> > 
> 
> > > > llvm-dev at lists.llvm.org
> > 
> 
> > > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > 
> 

> > > _______________________________________________
> > 
> 
> > > LLVM Developers mailing list
> > 
> 
> > > llvm-dev at lists.llvm.org
> > 
> 
> > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > 
> 

> > _______________________________________________
> 
> > LLVM Developers mailing list
> 
> > llvm-dev at lists.llvm.org
> 
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 

> --

> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory

> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-- 

Hal Finkel 
Assistant Computational Scientist 
Leadership Computing Facility 
Argonne National Laboratory 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160218/625d6432/attachment.html>

More information about the llvm-dev mailing list