[llvm-dev] RFC: Add bitcode tests to test-suite

Wed Feb 17 21:55:46 PST 2016

> On Feb 17, 2016, at 9:27 PM, Alina Sbirlea via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> >I think the real question is: Is this the best way to have a kind of 'halide buildbot' that can inform the LLVM developer community?
> 
> Halide already has a buildbot running every few hours which is being used to inform LLVM developer community when something breaks. It would be a lot more useful however to have the tests in an LLVM repository to inform LLVM devs which test broke right away. You're right that the underlying reason is the fact that Halide has test coverage of areas currently not covered.

It is not clear to me why your "Halide buildbot" is not enough?

> > As far as I can tell, Halide is < 100K LOC and has no external dependencies other than LLVM itself. I think we should just add it to the test suite.

Which suite are we talking about? Is it https://llvm.org/svn/llvm-project/test-suite/trunk/ ?

> I realize that means the community updating it for API changes,

The only project we are maintaining on top of LLVM is clang. 
There is nothing in the test-suite repository that link to LLVM (AFAIK), changing this means to add a requirement to build the the test-suite when changing some API in LLVM (even for "NFC" changes).

> but if the additional test coverage is as significant as I suspect, and the project authors will help and are responsive, that seems worthwhile. It is a JIT and a heavy generator of vector code, two areas in which our story on regular upstream testing coverage is not great.

Keeping Halide, and the Halide test-suite in a totally separate repository, with a continuous integration with LLVM trunk, provides the same coverage in practice.

Also the test-suite has a mechanism for "External" suites, which could be used here.

-- 
Mehdi

> Halide can do both JIT and AOT compilation. Would the community be happy to have non-execution tests for the JITted tests and execution tests for the AOT ones? This would in theory use a small set of Halide and not need the entire library, which is what we are trying to avoid here.
> The approach is meant to not clutter test-suite with a sizable amount of code but still get the test coverage offered by Halide.
> 
> 
> 
> On Wed, Feb 17, 2016 at 8:53 PM, Hal Finkel <hfinkel at anl.gov <mailto:hfinkel at anl.gov>> wrote:
> ----- Original Message -----
> > From: "Chandler Carruth" <chandlerc at google.com <mailto:chandlerc at google.com>>
> > To: "Hal Finkel" <hfinkel at anl.gov <mailto:hfinkel at anl.gov>>, "Alina Sbirlea" <alina.sbirlea at gmail.com <mailto:alina.sbirlea at gmail.com>>
> > Cc: "llvm-dev" <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
> > Sent: Wednesday, February 17, 2016 9:34:24 PM
> > Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite
> >
> >
> > Some perhaps relevant aspects that make testing users of LLVM like
> > Halide challenging:
> >
> >
> > Halide uses the LLVM C++ APIs, but there isn't a good way to
> > lock-step update it. So if we were to directly test Halide, it
> > wouldn't link against the new LLVM.
> >
> >
> > Practically speaking though, the LLVM IR generated by Halide should
> > continue to work with newer LLVM optimizations and code generation.
> > So the idea would be to snapshot the IR in bitcode (which is at
> > least reasonably stable) so that we could replay the tests as LLVM
> > changes. We can freshen the bitcode by re-generating it periodically
> > so it doesn't drift too far from what Halide actually uses.
> >
> >
> > The interesting questions IMO are:
> >
> >
> > 1) Are folks happy using bitcode as the format here? I agree with Hal
> > that it should be easy since Clang will actually Do The Right Thing
> > if given a bitcode input.
> >
> >
> > 2) Are folks happy with non-execution tests in some cases? I think
> > Alina is looking at whether we can get a runtime library that will
> > allow some of these to actually execute, but at least some of the
> > tests are just snap-shots of a JIT, and would need the full Halide
> > libraries (and introspection) to execute usefully.
> >
> 
> As far as I can tell, Halide is < 100K LOC and has no external dependencies other than LLVM itself. I think we should just add it to the test suite. I realize that means the community updating it for API changes, but if the additional test coverage is as significant as I suspect, and the project authors will help and are responsive, that seems worthwhile. It is a JIT and a heavy generator of vector code, two areas in which our story on regular upstream testing coverage is not great.
> 
>  -Hal
> 
> >
> > -Chandler
> >
> >
> > On Wed, Feb 17, 2016 at 7:25 PM Hal Finkel via llvm-dev <
> > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> > wrote:
> >
> >
> >
> >
> >
> >
> >
> >
> > From: "Alina Sbirlea via llvm-dev" < llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >
> > To: "llvm-dev" < llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> >
> > Sent: Wednesday, February 17, 2016 7:25:17 PM
> > Subject: [llvm-dev] RFC: Add bitcode tests to test-suite
> >
> >
> >
> >
> >
> >
> > Hi all,
> >
> >
> > TL;DR: Add *.bc to test-suite; llc *.bc; run some.
> >
> >
> >
> > We would like to propose adding bitcode tests to the llvm test-suite.
> >
> >
> > Recent LLVM bugs [2-4] prompted us to look into upstreaming a subset
> > of the tests the Halide library [1] is running and we'd like the
> > community's feedback on moving forward with this.
> >
> >
> >
> > Halide uses LLVM and can generate bitcode, but we cannot add C++
> > tests to test-suite without including the library itself.
> > This proposal is also potentially useful for other cases where there
> > is no C++ front-end.
> >
> >
> > As a first step we are interested in adding a set of correctness
> > tests, for testing the IR without running the tests. Since these
> > tests are generated, they are not instrumented like the .ll files in
> > trunk, however we believe checking that llc runs without errors is
> > still useful.
> > The bitcode files for Halide may also be large, so including them as
> > regression tests is not an option. If the smaller tests are found to
> > be valuable or covering cases no other tests cover, we can
> > instrument them and move them into the llvm trunk further along, but
> > that is not the goal of this proposal.
> > In addition, we're not sure whether the format for the tests should
> > be .ll or .bc, we're open to either.
> >
> >
> > After this first step, we're interested in upstreaming bitcode tests
> > and also running them.
> > We are very interested in tests for multiple architectures, aarch64
> > in particular, since this is where we have seen things break. This
> > may motivate adding .ll files rather than .bc in order to include
> > the "RUN:" target.
> > Where would these tests reside and with what directory structure?
> > (similar to test/CodeGen?)
> >
> >
> > Suggestion on what's the best approach for extending the test-suite
> > framework for this proposal are more than welcome.
> >
> >
> >
> > We already have architecture-specific tests in the test suite (e.g.
> > SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and Clang can deal
> > with IR inputs. I suppose you need to compile some corresponding
> > runtime library, but this does not seem like a big deal either.
> > Mechanically, I don't see this as particularly complicated. I think
> > the real question is: Is this the best way to have a kind of 'halide
> > buildbot' that can inform the LLVM developer community?
> >
> > -Hal
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > This is just the high-level overview to start off the discussion, I'm
> > sure there are many more aspects to touch on. Looking forward to
> > your feedback!
> >
> >
> >
> > Thanks,
> > Alina
> >
> >
> > [1] http:// halide -lang.org/ <http://lang.org/>
> > [2] Broken: r259800 => Fixed: r260131
> > [3] Broken: r260569 => Fixed: r260701
> >
> > [4] https://llvm.org/bugs/show_bug.cgi?id=26642 <https://llvm.org/bugs/show_bug.cgi?id=26642>
> >
> >
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> >
> >
> >
> >
> >
> > --
> >
> > Hal Finkel
> > Assistant Computational Scientist
> > Leadership Computing Facility
> > Argonne National Laboratory
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> >
> 
> --
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160217/6f03716d/attachment.html>