[llvm-dev] RFC: Add bitcode tests to test-suite
Philip Reames via llvm-dev
llvm-dev at lists.llvm.org
Thu Feb 18 09:27:33 PST 2016
On 02/18/2016 09:08 AM, David Blaikie wrote:
>
>
> On Thu, Feb 18, 2016 at 8:42 AM, Philip Reames via llvm-dev
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>
>
>
> On 02/18/2016 06:54 AM, Hal Finkel via llvm-dev wrote:
>
> Hi Chandler, et al.,
>
> While this proposal to put IR into the test suite technically
> non-problematic, I've convinced myself that this is a
> suboptimal direction for the LLVM project. Here's what I think
> would be better:
>
> - We create a test-suite/Frontends directory, and open this
> directory to actively-maintained external frontends, subject
> to the following restrictions:
>
> - The frontend must be actively maintained, and the
> project must agree to actively maintain the test-suite version
> - The frontend must use the LLVM API (either C or C++) -
> no printing textual IR
> - The frontend must have no significant (non-optional)
> dependencies outside of LLVM itself, or things on which LLVM
> itself depends
> - The frontend must have regression tests and
> benchmarks/correctness tests providing significant coverage of
> the frontend and its associated code generation
>
> Here's the quid pro quo:
>
> - The LLVM community gains additional testing coverage
> (which we definitely need)
> - The LLVM community gains extra insight into how its APIs
> are being used (hopefully allowing us to make more-informed
> decisions about how to update them)
>
> - The frontend gains free API updates
> - The frontend's use of LLVM will be more stable
>
> This involves extra work for everybody, but will help us all
> deliver higher-quality products. Plus, given the constant
> discussions about the difficulty for external projects to
> follow API updates, etc., this is a good way to help address
> those difficulties.
>
> The fact that Halide will provide extra coverage of our vector
> code generation (aside from whatever we happen to produce from
> our autovectorizers), and our JIT infrastructure, makes it a
> good candidate for this. Intel's ispc, POCL, (maybe whatever
> bit of Mesa uses LLVM), etc. would also be natural candidates
> should the projects be interested.
>
> I think this is a really bad tradeoff and am strongly opposed to
> this proposal.
>
> If we want to focus on improving test coverage, that's reasonable,
> but doing so at the cost of requiring LLVM contributors to
> maintain everyone's frontend is not a reasonable approach.
>
> A couple of alternate approaches which might be worth considering:
> 1) The IR corpus approach mentioned previously. So long as
> external teams are willing to update the corpus regularly
> (weekly), this gives most of the backend coverage with none of the
> maintenance burden.
>
>
> Why weekly? & why not bitcode, that would be long lasting? (still,
> updating it regularly would be helpful, but in theory we should keep
> working on the same bitcode for a fairly long timeframe & means when I
> go and make breaking IR changes I don't have to add the test-suite to
> the list of things I need to fix :))
I should have written bitcode to start with. :) All of your points are
sound.
I said "weekly" mostly as a placeholder for requiring active involvement
from the frontend and as a means to keep the two projects roughly in
sync. If Halide started generating radically different IR all of a
sudden, we want the bitcode tests to reflect that.
>
> 2) Use coverage information to determine which code paths Halide
> covers which are not covered by existing unit tests. Work to
> improve those unit tests. Using something along the lines with a
> mutation testing (i.e. change the source code and see what
> breaks), combined with test reduction (bugpoint), could greatly
> improve our test coverage in tree fairly quickly. This would
> require a lot of work from a single contributor, but that's much
> better than requiring a lot of work from all contributors.
>
>
> While this would be awesome (& I'd love to see some LLVM/Clang-based
> mutation testing tools, and to improve our test coverage using them)
> that seems like a pretty big investment that I'm not sure anyone is
> signing up for just now.
Fair point. However, before we ask the entire project to sign up for a
lot of work, asking some particular motivated person to do so seems
reasonable. :)
I'll also note that I was thinking of a very simple version initially.
Something on the order of "replace all untested lines with
llvm_unreachable, reduce one test, rerun coverage, repeat". This could
be done mostly manually and would yield a lot of improvement.
>
>
>
> Thanks again,
> Hal
>
> ----- Original Message -----
>
> From: "Chandler Carruth" <chandlerc at google.com
> <mailto:chandlerc at google.com>>
> To: "Hal Finkel" <hfinkel at anl.gov
> <mailto:hfinkel at anl.gov>>, "Alina Sbirlea"
> <alina.sbirlea at gmail.com <mailto:alina.sbirlea at gmail.com>>
> Cc: "llvm-dev" <llvm-dev at lists.llvm.org
> <mailto:llvm-dev at lists.llvm.org>>
> Sent: Wednesday, February 17, 2016 9:34:24 PM
> Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite
>
>
> Some perhaps relevant aspects that make testing users of
> LLVM like
> Halide challenging:
>
>
> Halide uses the LLVM C++ APIs, but there isn't a good way to
> lock-step update it. So if we were to directly test Halide, it
> wouldn't link against the new LLVM.
>
>
> Practically speaking though, the LLVM IR generated by
> Halide should
> continue to work with newer LLVM optimizations and code
> generation.
> So the idea would be to snapshot the IR in bitcode (which
> is at
> least reasonably stable) so that we could replay the tests
> as LLVM
> changes. We can freshen the bitcode by re-generating it
> periodically
> so it doesn't drift too far from what Halide actually uses.
>
>
> The interesting questions IMO are:
>
>
> 1) Are folks happy using bitcode as the format here? I
> agree with Hal
> that it should be easy since Clang will actually Do The
> Right Thing
> if given a bitcode input.
>
>
> 2) Are folks happy with non-execution tests in some cases?
> I think
> Alina is looking at whether we can get a runtime library
> that will
> allow some of these to actually execute, but at least some
> of the
> tests are just snap-shots of a JIT, and would need the
> full Halide
> libraries (and introspection) to execute usefully.
>
>
> -Chandler
>
>
> On Wed, Feb 17, 2016 at 7:25 PM Hal Finkel via llvm-dev <
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >
> wrote:
>
>
>
>
>
>
>
>
> From: "Alina Sbirlea via llvm-dev" <
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >
> To: "llvm-dev" < llvm-dev at lists.llvm.org
> <mailto:llvm-dev at lists.llvm.org> >
> Sent: Wednesday, February 17, 2016 7:25:17 PM
> Subject: [llvm-dev] RFC: Add bitcode tests to test-suite
>
>
>
>
>
>
> Hi all,
>
>
> TL;DR: Add *.bc to test-suite; llc *.bc; run some.
>
>
>
> We would like to propose adding bitcode tests to the llvm
> test-suite.
>
>
> Recent LLVM bugs [2-4] prompted us to look into
> upstreaming a subset
> of the tests the Halide library [1] is running and we'd
> like the
> community's feedback on moving forward with this.
>
>
>
> Halide uses LLVM and can generate bitcode, but we cannot
> add C++
> tests to test-suite without including the library itself.
> This proposal is also potentially useful for other cases
> where there
> is no C++ front-end.
>
>
> As a first step we are interested in adding a set of
> correctness
> tests, for testing the IR without running the tests. Since
> these
> tests are generated, they are not instrumented like the
> .ll files in
> trunk, however we believe checking that llc runs without
> errors is
> still useful.
> The bitcode files for Halide may also be large, so
> including them as
> regression tests is not an option. If the smaller tests
> are found to
> be valuable or covering cases no other tests cover, we can
> instrument them and move them into the llvm trunk further
> along, but
> that is not the goal of this proposal.
> In addition, we're not sure whether the format for the
> tests should
> be .ll or .bc, we're open to either.
>
>
> After this first step, we're interested in upstreaming
> bitcode tests
> and also running them.
> We are very interested in tests for multiple
> architectures, aarch64
> in particular, since this is where we have seen things
> break. This
> may motivate adding .ll files rather than .bc in order to
> include
> the "RUN:" target.
> Where would these tests reside and with what directory
> structure?
> (similar to test/CodeGen?)
>
>
> Suggestion on what's the best approach for extending the
> test-suite
> framework for this proposal are more than welcome.
>
>
>
> We already have architecture-specific tests in the test
> suite (e.g.
> SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and
> Clang can deal
> with IR inputs. I suppose you need to compile some
> corresponding
> runtime library, but this does not seem like a big deal
> either.
> Mechanically, I don't see this as particularly
> complicated. I think
> the real question is: Is this the best way to have a kind
> of 'halide
> buildbot' that can inform the LLVM developer community?
>
> -Hal
>
>
>
>
>
>
>
>
>
>
> This is just the high-level overview to start off the
> discussion, I'm
> sure there are many more aspects to touch on. Looking
> forward to
> your feedback!
>
>
>
> Thanks,
> Alina
>
>
> [1] http:// halide -lang.org/ <http://lang.org/>
> [2] Broken: r259800 => Fixed: r260131
> [3] Broken: r260569 => Fixed: r260701
>
> [4] https://llvm.org/bugs/show_bug.cgi?id=26642
>
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
>
>
> --
>
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160218/ee36acb5/attachment.html>
More information about the llvm-dev
mailing list