[llvm-dev] RFC: Add bitcode tests to test-suite

[Philip Reames via llvm-dev]

I'm a bit confused as to what's being proposed for immediate action.  Is 
the proposal essentially to add a set of binary bitcode files and ensure 
that running each of them through LLC does not trigger any assertions?  
If so, arranging that as an external build bot would seem entirely 
reasonable.  If on the other hand, we were testing for equivalence of 
the output assembly, that would probably NOT be okay, just because of 
false positive rate.

(Running with the assumption I rephrased the proposal correctly... if 
not, this will be totally off topic.)

Beyond Halide, this sounds like a potentially useful general mechanism 
for integration testing (not unit testing).  We (upstream llvm) have 
something sorta similar today in the "self host clang, see what breaks" 
workflow.  We (my downstream team) have a similar mechanism where we've 
collected a corpus of IR files from key benchmarks (regenerated weekly), 
that we use to stress test tricky commits before submission.

Standardizing such a mechanism and policy around it's usage seems 
useful.  My major concerns are a) keeping the corpus small enough to be 
useful, b) not having them intermixed with unit tests and c) making sure 
that the *frontend* authors pay the primary cost to maintain them.

Philip

On 02/17/2016 05:25 PM, Alina Sbirlea via llvm-dev wrote:
> Hi all,
>
> TL;DR: Add *.bc to test-suite; llc *.bc; run some.
>
> We would like to propose adding bitcode tests to the llvm test-suite.
>
> Recent LLVM bugs [2-4] prompted us to look into upstreaming a subset 
> of the tests the Halide library [1] is running and we'd like the 
> community's feedback on moving forward with this.
>
> Halide uses LLVM and can generate bitcode, but we cannot add C++ tests 
> to test-suite without including the library itself.
> This proposal is also potentially useful for other cases where there 
> is no C++ front-end.
>
> As a first step we are interested in adding a set of correctness 
> tests, for testing the IR without running the tests. Since these tests 
> are generated, they are not instrumented like the .ll files in trunk, 
> however we believe checking that llc runs without errors is still useful.
> The bitcode files for Halide may also be large, so including them as 
> regression tests is not an option. If the smaller tests are found to 
> be valuable or covering cases no other tests cover, we can instrument 
> them and move them into the llvm trunk further along, but that is not 
> the goal of this proposal.
> In addition, we're not sure whether the format for the tests should be 
> .ll or .bc, we're open to either.
>
> After this first step, we're interested in upstreaming bitcode tests 
> and also running them.
> We are very interested in tests for multiple architectures, aarch64 in 
> particular, since this is where we have seen things break. This may 
> motivate adding .ll files rather than .bc in order to include the 
> "RUN:" target.
> Where would these tests reside and with what directory structure? 
> (similar to test/CodeGen?)
>
> Suggestion on what's the best approach for extending the test-suite 
> framework for this proposal are more than welcome.
>
> This is just the high-level overview to start off the discussion, I'm 
> sure there are many more aspects to touch on. Looking forward to your 
> feedback!
>
> Thanks,
> Alina
>
> [1] http://halide-lang.org/ <http://halide-lang.org/>
> [2] Broken: r259800 => Fixed: r260131
> [3] Broken: r260569 => Fixed: r260701
> [4] https://llvm.org/bugs/show_bug.cgi?id=26642
>
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160217/f08c378e/attachment.html>