[llvm-dev] RFC: Add bitcode tests to test-suite

[Hal Finkel via llvm-dev]

----- Original Message -----

> From: "Alina Sbirlea via llvm-dev" <llvm-dev at lists.llvm.org>
> To: "llvm-dev" <llvm-dev at lists.llvm.org>
> Sent: Wednesday, February 17, 2016 7:25:17 PM
> Subject: [llvm-dev] RFC: Add bitcode tests to test-suite

> Hi all,

> TL;DR: Add *.bc to test-suite; llc *.bc; run some.

> We would like to propose adding bitcode tests to the llvm test-suite.

> Recent LLVM bugs [2-4] prompted us to look into upstreaming a subset
> of the tests the Halide library [1] is running and we'd like the
> community's feedback on moving forward with this.

> Halide uses LLVM and can generate bitcode, but we cannot add C++
> tests to test-suite without including the library itself.
> This proposal is also potentially useful for other cases where there
> is no C++ front-end.

> As a first step we are interested in adding a set of correctness
> tests, for testing the IR without running the tests. Since these
> tests are generated, they are not instrumented like the .ll files in
> trunk, however we believe checking that llc runs without errors is
> still useful.
> The bitcode files for Halide may also be large, so including them as
> regression tests is not an option. If the smaller tests are found to
> be valuable or covering cases no other tests cover, we can
> instrument them and move them into the llvm trunk further along, but
> that is not the goal of this proposal.
> In addition, we're not sure whether the format for the tests should
> be .ll or .bc, we're open to either.

> After this first step, we're interested in upstreaming bitcode tests
> and also running them.
> We are very interested in tests for multiple architectures, aarch64
> in particular, since this is where we have seen things break. This
> may motivate adding .ll files rather than .bc in order to include
> the "RUN:" target.
> Where would these tests reside and with what directory structure?
> (similar to test/CodeGen?)

> Suggestion on what's the best approach for extending the test-suite
> framework for this proposal are more than welcome.
We already have architecture-specific tests in the test suite (e.g. SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and Clang can deal with IR inputs. I suppose you need to compile some corresponding runtime library, but this does not seem like a big deal either. Mechanically, I don't see this as particularly complicated. I think the real question is: Is this the best way to have a kind of 'halide buildbot' that can inform the LLVM developer community? 

-Hal 

> This is just the high-level overview to start off the discussion, I'm
> sure there are many more aspects to touch on. Looking forward to
> your feedback!

> Thanks,
> Alina

> [1] http:// halide -lang.org/
> [2] Broken: r259800 => Fixed: r260131
> [3] Broken: r260569 => Fixed: r260701

> [4] https://llvm.org/bugs/show_bug.cgi?id=26642

> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-- 

Hal Finkel 
Assistant Computational Scientist 
Leadership Computing Facility 
Argonne National Laboratory 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160217/2296bd36/attachment.html>