[llvm-dev] RFC: Add bitcode tests to test-suite

Thu Feb 18 10:15:52 PST 2016

On Thu, Feb 18, 2016 at 10:03 AM, Mehdi Amini via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

>
> > On Feb 18, 2016, at 8:42 AM, Philip Reames via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >
> > On 02/18/2016 06:54 AM, Hal Finkel via llvm-dev wrote:
> >> Hi Chandler, et al.,
> >>
> >> While this proposal to put IR into the test suite technically
> non-problematic, I've convinced myself that this is a suboptimal direction
> for the LLVM project. Here's what I think would be better:
> >>
> >>  - We create a test-suite/Frontends directory, and open this directory
> to actively-maintained external frontends, subject to the following
> restrictions:
> >>
> >>    - The frontend must be actively maintained, and the project must
> agree to actively maintain the test-suite version
> >>    - The frontend must use the LLVM API (either C or C++) - no printing
> textual IR
> >>    - The frontend must have no significant (non-optional) dependencies
> outside of LLVM itself, or things on which LLVM itself depends
> >>    - The frontend must have regression tests and benchmarks/correctness
> tests providing significant coverage of the frontend and its associated
> code generation
> >>
> >> Here's the quid pro quo:
> >>
> >>    - The LLVM community gains additional testing coverage (which we
> definitely need)
> >>    - The LLVM community gains extra insight into how its APIs are being
> used (hopefully allowing us to make more-informed decisions about how to
> update them)
> >>
> >>    - The frontend gains free API updates
> >>    - The frontend's use of LLVM will be more stable
> >>
> >> This involves extra work for everybody, but will help us all deliver
> higher-quality products. Plus, given the constant discussions about the
> difficulty for external projects to follow API updates, etc., this is a
> good way to help address those difficulties.
> >>
> >> The fact that Halide will provide extra coverage of our vector code
> generation (aside from whatever we happen to produce from our
> autovectorizers), and our JIT infrastructure, makes it a good candidate for
> this. Intel's ispc, POCL, (maybe whatever bit of Mesa uses LLVM), etc.
> would also be natural candidates should the projects be interested.
> > I think this is a really bad tradeoff and am strongly opposed to this
> proposal.
> >
> > If we want to focus on improving test coverage, that's reasonable, but
> doing so at the cost of requiring LLVM contributors to maintain everyone's
> frontend is not a reasonable approach.
> >
> > A couple of alternate approaches which might be worth considering:
> > 1) The IR corpus approach mentioned previously.  So long as external
> teams are willing to update the corpus regularly (weekly), this gives most
> of the backend coverage with none of the maintenance burden.
> > 2) Use coverage information to determine which code paths Halide covers
> which are not covered by existing unit tests.  Work to improve those unit
> tests.  Using something along the lines with a mutation testing (i.e.
> change the source code and see what breaks), combined with test reduction
> (bugpoint), could greatly improve our test coverage in tree fairly
> quickly.  This would require a lot of work from a single contributor, but
> that's much better than requiring a lot of work from all contributors.
>
> I support the view of Philip on this topic.
> I think that having the bitcode without the burden of the API and linking
> is the best trade-off for LLVM.
>
> About the bitcode, I'm not convince that it needs to be updated that
> often: we're interested in coverage for LLVM, not trying to validate that
> we support correctly frontend X or Y in a particular version. I'd even
> argue that if Halide version 13 generates a very different IR than Halide
> 12, then we should keep the Halide 12 generated bitcode in a separate
> directory because it is likely to stress LLVM differently.
>

Potentially, though I hesitate to make test-suite too much of a dumping
ground (it is already, to an extent). Quantity isn't quality and all that -
making sure bots cycle quickly (in a relative sense - of course running the
test-suite is going to be a more involved process than our high priority
regression 'unit'-y tests) is important for them to be
meaningful/actionable by developers & not just noise.

>
> I have more questions for Alina. What kind of tests do you have:
>
> - "the compiler takes the bitcode and generates code without crashing"
> - "the compiled test runs without crashing"
> - "the compiled test will produce an output that be checked against a
> reference"
> - "the compiled test is meaningful as a benchmarks"
>
> All these different aspects of testing can be interesting, but knowing
> what we're talking may influence the way forward.
>
> As mentioned before, the test-suite has a mechanism of "external" suites,
> it may be limited right now but could probably be expanded. Ideally there
> could be multiple repositories that would just be checked out independently
> (think about the way we build LLVM alone or
> LLVM+clang+libcxx+compiler-rt+...).
>
> --
> Mehdi
>
>
> >>
> >> Thanks again,
> >> Hal
> >>
> >> ----- Original Message -----
> >>> From: "Chandler Carruth" <chandlerc at google.com>
> >>> To: "Hal Finkel" <hfinkel at anl.gov>, "Alina Sbirlea" <
> alina.sbirlea at gmail.com>
> >>> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>
> >>> Sent: Wednesday, February 17, 2016 9:34:24 PM
> >>> Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite
> >>>
> >>>
> >>> Some perhaps relevant aspects that make testing users of LLVM like
> >>> Halide challenging:
> >>>
> >>>
> >>> Halide uses the LLVM C++ APIs, but there isn't a good way to
> >>> lock-step update it. So if we were to directly test Halide, it
> >>> wouldn't link against the new LLVM.
> >>>
> >>>
> >>> Practically speaking though, the LLVM IR generated by Halide should
> >>> continue to work with newer LLVM optimizations and code generation.
> >>> So the idea would be to snapshot the IR in bitcode (which is at
> >>> least reasonably stable) so that we could replay the tests as LLVM
> >>> changes. We can freshen the bitcode by re-generating it periodically
> >>> so it doesn't drift too far from what Halide actually uses.
> >>>
> >>>
> >>> The interesting questions IMO are:
> >>>
> >>>
> >>> 1) Are folks happy using bitcode as the format here? I agree with Hal
> >>> that it should be easy since Clang will actually Do The Right Thing
> >>> if given a bitcode input.
> >>>
> >>>
> >>> 2) Are folks happy with non-execution tests in some cases? I think
> >>> Alina is looking at whether we can get a runtime library that will
> >>> allow some of these to actually execute, but at least some of the
> >>> tests are just snap-shots of a JIT, and would need the full Halide
> >>> libraries (and introspection) to execute usefully.
> >>>
> >>>
> >>> -Chandler
> >>>
> >>>
> >>> On Wed, Feb 17, 2016 at 7:25 PM Hal Finkel via llvm-dev <
> >>> llvm-dev at lists.llvm.org > wrote:
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> From: "Alina Sbirlea via llvm-dev" < llvm-dev at lists.llvm.org >
> >>> To: "llvm-dev" < llvm-dev at lists.llvm.org >
> >>> Sent: Wednesday, February 17, 2016 7:25:17 PM
> >>> Subject: [llvm-dev] RFC: Add bitcode tests to test-suite
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Hi all,
> >>>
> >>>
> >>> TL;DR: Add *.bc to test-suite; llc *.bc; run some.
> >>>
> >>>
> >>>
> >>> We would like to propose adding bitcode tests to the llvm test-suite.
> >>>
> >>>
> >>> Recent LLVM bugs [2-4] prompted us to look into upstreaming a subset
> >>> of the tests the Halide library [1] is running and we'd like the
> >>> community's feedback on moving forward with this.
> >>>
> >>>
> >>>
> >>> Halide uses LLVM and can generate bitcode, but we cannot add C++
> >>> tests to test-suite without including the library itself.
> >>> This proposal is also potentially useful for other cases where there
> >>> is no C++ front-end.
> >>>
> >>>
> >>> As a first step we are interested in adding a set of correctness
> >>> tests, for testing the IR without running the tests. Since these
> >>> tests are generated, they are not instrumented like the .ll files in
> >>> trunk, however we believe checking that llc runs without errors is
> >>> still useful.
> >>> The bitcode files for Halide may also be large, so including them as
> >>> regression tests is not an option. If the smaller tests are found to
> >>> be valuable or covering cases no other tests cover, we can
> >>> instrument them and move them into the llvm trunk further along, but
> >>> that is not the goal of this proposal.
> >>> In addition, we're not sure whether the format for the tests should
> >>> be .ll or .bc, we're open to either.
> >>>
> >>>
> >>> After this first step, we're interested in upstreaming bitcode tests
> >>> and also running them.
> >>> We are very interested in tests for multiple architectures, aarch64
> >>> in particular, since this is where we have seen things break. This
> >>> may motivate adding .ll files rather than .bc in order to include
> >>> the "RUN:" target.
> >>> Where would these tests reside and with what directory structure?
> >>> (similar to test/CodeGen?)
> >>>
> >>>
> >>> Suggestion on what's the best approach for extending the test-suite
> >>> framework for this proposal are more than welcome.
> >>>
> >>>
> >>>
> >>> We already have architecture-specific tests in the test suite (e.g.
> >>> SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and Clang can deal
> >>> with IR inputs. I suppose you need to compile some corresponding
> >>> runtime library, but this does not seem like a big deal either.
> >>> Mechanically, I don't see this as particularly complicated. I think
> >>> the real question is: Is this the best way to have a kind of 'halide
> >>> buildbot' that can inform the LLVM developer community?
> >>>
> >>> -Hal
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> This is just the high-level overview to start off the discussion, I'm
> >>> sure there are many more aspects to touch on. Looking forward to
> >>> your feedback!
> >>>
> >>>
> >>>
> >>> Thanks,
> >>> Alina
> >>>
> >>>
> >>> [1] http:// halide -lang.org/
> >>> [2] Broken: r259800 => Fixed: r260131
> >>> [3] Broken: r260569 => Fixed: r260701
> >>>
> >>> [4] https://llvm.org/bugs/show_bug.cgi?id=26642
> >>>
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> LLVM Developers mailing list
> >>> llvm-dev at lists.llvm.org
> >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>> Hal Finkel
> >>> Assistant Computational Scientist
> >>> Leadership Computing Facility
> >>> Argonne National Laboratory
> >>> _______________________________________________
> >>> LLVM Developers mailing list
> >>> llvm-dev at lists.llvm.org
> >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >>>
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160218/8fe9204d/attachment.html>