[llvm-dev] RFC: Add bitcode tests to test-suite

[Alina Sbirlea via llvm-dev]

On Wed, Feb 17, 2016 at 8:34 PM, Philip Reames <listmail at philipreames.com>
wrote:

> I'm a bit confused as to what's being proposed for immediate action.  Is
> the proposal essentially to add a set of binary bitcode files and ensure
> that running each of them through LLC does not trigger any assertions?  If
> so, arranging that as an external build bot would seem entirely
> reasonable.  If on the other hand, we were testing for equivalence of the
> output assembly, that would probably NOT be okay, just because of false
> positive rate.
>

No, we are not planning for equivalence of output assembly.

>
> (Running with the assumption I rephrased the proposal correctly... if not,
> this will be totally off topic.)
>
> Beyond Halide, this sounds like a potentially useful general mechanism for
> integration testing (not unit testing).  We (upstream llvm) have something
> sorta similar today in the "self host clang, see what breaks" workflow.  We
> (my downstream team) have a similar mechanism where we've collected a
> corpus of IR files from key benchmarks (regenerated weekly), that we use to
> stress test tricky commits before submission.
>
> Standardizing such a mechanism and policy around it's usage seems useful.
> My major concerns are a) keeping the corpus small enough to be useful, b)
> not having them intermixed with unit tests and c) making sure that the
> *frontend* authors pay the primary cost to maintain them.
>

>From my side I would like to have this standardized. As I said, I think
this proposal's benefits are not just for Halide.
a) I believe the frontend authors need to ensure this. For Halide we would
like to include a larger set of unit tests and a small set of large
applications. The unit tests are meant to make sure no assertions are
triggered. The applications are meant to run and compare with reference
results.
Do you consider such a corpus of IR files reasonable?
b) I agree. How/where should these be integrated into test-suite?
c) Primarily the tests would be maintained by the frontend authors, but may
be updated by llvm devs. For example, in the event of a change in llvm IR
the easiest would be to have the tests as bitcode files.


>
> Philip
>
>
> On 02/17/2016 05:25 PM, Alina Sbirlea via llvm-dev wrote:
>
> Hi all,
>
> TL;DR: Add *.bc to test-suite; llc *.bc; run some.
>
> We would like to propose adding bitcode tests to the llvm test-suite.
>
> Recent LLVM bugs [2-4] prompted us to look into upstreaming a subset of
> the tests the Halide library [1] is running and we'd like the community's
> feedback on moving forward with this.
>
> Halide uses LLVM and can generate bitcode, but we cannot add C++ tests to
> test-suite without including the library itself.
> This proposal is also potentially useful for other cases where there is no
> C++ front-end.
>
> As a first step we are interested in adding a set of correctness tests,
> for testing the IR without running the tests. Since these tests are
> generated, they are not instrumented like the .ll files in trunk, however
> we believe checking that llc runs without errors is still useful.
> The bitcode files for Halide may also be large, so including them as
> regression tests is not an option. If the smaller tests are found to be
> valuable or covering cases no other tests cover, we can instrument them and
> move them into the llvm trunk further along, but that is not the goal of
> this proposal.
> In addition, we're not sure whether the format for the tests should be .ll
> or .bc, we're open to either.
>
> After this first step, we're interested in upstreaming bitcode tests and
> also running them.
> We are very interested in tests for multiple architectures, aarch64 in
> particular, since this is where we have seen things break. This may
> motivate adding .ll files rather than .bc in order to include the "RUN:"
> target.
> Where would these tests reside and with what directory structure? (similar
> to test/CodeGen?)
>
> Suggestion on what's the best approach for extending the test-suite
> framework for this proposal are more than welcome.
>
> This is just the high-level overview to start off the discussion, I'm sure
> there are many more aspects to touch on. Looking forward to your feedback!
>
> Thanks,
> Alina
>
> [1]  <http://halide-lang.org/>http://halide-lang.org/
> [2] Broken: r259800 => Fixed: r260131
> [3] Broken: r260569 => Fixed: r260701
> [4] https://llvm.org/bugs/show_bug.cgi?id=26642
>
>
>
>
> _______________________________________________
> LLVM Developers mailing listllvm-dev at lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160217/123a791b/attachment.html>