[llvm-dev] RFC: Add bitcode tests to test-suite

Wed Feb 17 20:53:05 PST 2016

----- Original Message -----
> From: "Chandler Carruth" <chandlerc at google.com>
> To: "Hal Finkel" <hfinkel at anl.gov>, "Alina Sbirlea" <alina.sbirlea at gmail.com>
> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>
> Sent: Wednesday, February 17, 2016 9:34:24 PM
> Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite
> 
> 
> Some perhaps relevant aspects that make testing users of LLVM like
> Halide challenging:
> 
> 
> Halide uses the LLVM C++ APIs, but there isn't a good way to
> lock-step update it. So if we were to directly test Halide, it
> wouldn't link against the new LLVM.
> 
> 
> Practically speaking though, the LLVM IR generated by Halide should
> continue to work with newer LLVM optimizations and code generation.
> So the idea would be to snapshot the IR in bitcode (which is at
> least reasonably stable) so that we could replay the tests as LLVM
> changes. We can freshen the bitcode by re-generating it periodically
> so it doesn't drift too far from what Halide actually uses.
> 
> 
> The interesting questions IMO are:
> 
> 
> 1) Are folks happy using bitcode as the format here? I agree with Hal
> that it should be easy since Clang will actually Do The Right Thing
> if given a bitcode input.
> 
> 
> 2) Are folks happy with non-execution tests in some cases? I think
> Alina is looking at whether we can get a runtime library that will
> allow some of these to actually execute, but at least some of the
> tests are just snap-shots of a JIT, and would need the full Halide
> libraries (and introspection) to execute usefully.
> 

As far as I can tell, Halide is < 100K LOC and has no external dependencies other than LLVM itself. I think we should just add it to the test suite. I realize that means the community updating it for API changes, but if the additional test coverage is as significant as I suspect, and the project authors will help and are responsive, that seems worthwhile. It is a JIT and a heavy generator of vector code, two areas in which our story on regular upstream testing coverage is not great.

 -Hal

> 
> -Chandler
> 
> 
> On Wed, Feb 17, 2016 at 7:25 PM Hal Finkel via llvm-dev <
> llvm-dev at lists.llvm.org > wrote:
> 
> 
> 
> 
> 
> 
> 
> 
> From: "Alina Sbirlea via llvm-dev" < llvm-dev at lists.llvm.org >
> To: "llvm-dev" < llvm-dev at lists.llvm.org >
> Sent: Wednesday, February 17, 2016 7:25:17 PM
> Subject: [llvm-dev] RFC: Add bitcode tests to test-suite
> 
> 
> 
> 
> 
> 
> Hi all,
> 
> 
> TL;DR: Add *.bc to test-suite; llc *.bc; run some.
> 
> 
> 
> We would like to propose adding bitcode tests to the llvm test-suite.
> 
> 
> Recent LLVM bugs [2-4] prompted us to look into upstreaming a subset
> of the tests the Halide library [1] is running and we'd like the
> community's feedback on moving forward with this.
> 
> 
> 
> Halide uses LLVM and can generate bitcode, but we cannot add C++
> tests to test-suite without including the library itself.
> This proposal is also potentially useful for other cases where there
> is no C++ front-end.
> 
> 
> As a first step we are interested in adding a set of correctness
> tests, for testing the IR without running the tests. Since these
> tests are generated, they are not instrumented like the .ll files in
> trunk, however we believe checking that llc runs without errors is
> still useful.
> The bitcode files for Halide may also be large, so including them as
> regression tests is not an option. If the smaller tests are found to
> be valuable or covering cases no other tests cover, we can
> instrument them and move them into the llvm trunk further along, but
> that is not the goal of this proposal.
> In addition, we're not sure whether the format for the tests should
> be .ll or .bc, we're open to either.
> 
> 
> After this first step, we're interested in upstreaming bitcode tests
> and also running them.
> We are very interested in tests for multiple architectures, aarch64
> in particular, since this is where we have seen things break. This
> may motivate adding .ll files rather than .bc in order to include
> the "RUN:" target.
> Where would these tests reside and with what directory structure?
> (similar to test/CodeGen?)
> 
> 
> Suggestion on what's the best approach for extending the test-suite
> framework for this proposal are more than welcome.
> 
> 
> 
> We already have architecture-specific tests in the test suite (e.g.
> SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and Clang can deal
> with IR inputs. I suppose you need to compile some corresponding
> runtime library, but this does not seem like a big deal either.
> Mechanically, I don't see this as particularly complicated. I think
> the real question is: Is this the best way to have a kind of 'halide
> buildbot' that can inform the LLVM developer community?
> 
> -Hal
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> This is just the high-level overview to start off the discussion, I'm
> sure there are many more aspects to touch on. Looking forward to
> your feedback!
> 
> 
> 
> Thanks,
> Alina
> 
> 
> [1] http:// halide -lang.org/
> [2] Broken: r259800 => Fixed: r260131
> [3] Broken: r260569 => Fixed: r260701
> 
> [4] https://llvm.org/bugs/show_bug.cgi?id=26642
> 
> 
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
> 
> 
> 
> 
> --
> 
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory