[llvm-dev] RFC: Add bitcode tests to test-suite

Wed Feb 17 22:13:06 PST 2016

On Wed, Feb 17, 2016 at 9:55 PM, Mehdi Amini via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

>
> On Feb 17, 2016, at 9:27 PM, Alina Sbirlea via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> >I think the real question is: Is this the best way to have a kind of
> 'halide buildbot' that can inform the LLVM developer community?
>
> Halide already has a buildbot running every few hours which is being used
> to inform LLVM developer community when something breaks. It would be a lot
> more useful however to have the tests in an LLVM repository to inform LLVM
> devs which test broke right away. You're right that the underlying reason
> is the fact that Halide has test coverage of areas currently not covered.
>
>
> It is not clear to me why your "Halide buildbot" is not enough?
>
> > As far as I can tell, Halide is < 100K LOC and has no external
> dependencies other than LLVM itself. I think we should just add it to the
> test suite.
>
>
> Which suite are we talking about? Is it
> https://llvm.org/svn/llvm-project/test-suite/trunk/ ?
>
> I realize that means the community updating it for API changes,
>
>
> The only project we are maintaining on top of LLVM is clang.
> There is nothing in the test-suite repository that link to LLVM (AFAIK),
> changing this means to add a requirement to build the the test-suite when
> changing some API in LLVM (even for "NFC" changes).
>

Yeah, it is out of the question for us to wholesale import Halide and for
us to take on any kind of maintenance burden for updating it.

-- Sean Silva

>
> but if the additional test coverage is as significant as I suspect, and
> the project authors will help and are responsive, that seems worthwhile. It
> is a JIT and a heavy generator of vector code, two areas in which our story
> on regular upstream testing coverage is not great.
>
>
> Keeping Halide, and the Halide test-suite in a totally separate
> repository, with a continuous integration with LLVM trunk, provides the
> same coverage in practice.
>
> Also the test-suite has a mechanism for "External" suites, which could be
> used here.
>
> --
> Mehdi
>
>
> Halide can do both JIT and AOT compilation. Would the community be happy
> to have non-execution tests for the JITted tests and execution tests for
> the AOT ones? This would in theory use a small set of Halide and not need
> the entire library, which is what we are trying to avoid here.
> The approach is meant to not clutter test-suite with a sizable amount of
> code but still get the test coverage offered by Halide.
>
>
>
> On Wed, Feb 17, 2016 at 8:53 PM, Hal Finkel <hfinkel at anl.gov> wrote:
>
>> ----- Original Message -----
>> > From: "Chandler Carruth" <chandlerc at google.com>
>> > To: "Hal Finkel" <hfinkel at anl.gov>, "Alina Sbirlea" <
>> alina.sbirlea at gmail.com>
>> > Cc: "llvm-dev" <llvm-dev at lists.llvm.org>
>> > Sent: Wednesday, February 17, 2016 9:34:24 PM
>> > Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite
>> >
>> >
>> > Some perhaps relevant aspects that make testing users of LLVM like
>> > Halide challenging:
>> >
>> >
>> > Halide uses the LLVM C++ APIs, but there isn't a good way to
>> > lock-step update it. So if we were to directly test Halide, it
>> > wouldn't link against the new LLVM.
>> >
>> >
>> > Practically speaking though, the LLVM IR generated by Halide should
>> > continue to work with newer LLVM optimizations and code generation.
>> > So the idea would be to snapshot the IR in bitcode (which is at
>> > least reasonably stable) so that we could replay the tests as LLVM
>> > changes. We can freshen the bitcode by re-generating it periodically
>> > so it doesn't drift too far from what Halide actually uses.
>> >
>> >
>> > The interesting questions IMO are:
>> >
>> >
>> > 1) Are folks happy using bitcode as the format here? I agree with Hal
>> > that it should be easy since Clang will actually Do The Right Thing
>> > if given a bitcode input.
>> >
>> >
>> > 2) Are folks happy with non-execution tests in some cases? I think
>> > Alina is looking at whether we can get a runtime library that will
>> > allow some of these to actually execute, but at least some of the
>> > tests are just snap-shots of a JIT, and would need the full Halide
>> > libraries (and introspection) to execute usefully.
>> >
>>
>> As far as I can tell, Halide is < 100K LOC and has no external
>> dependencies other than LLVM itself. I think we should just add it to the
>> test suite. I realize that means the community updating it for API changes,
>> but if the additional test coverage is as significant as I suspect, and the
>> project authors will help and are responsive, that seems worthwhile. It is
>> a JIT and a heavy generator of vector code, two areas in which our story on
>> regular upstream testing coverage is not great.
>>
>>  -Hal
>>
>> >
>> > -Chandler
>> >
>> >
>> > On Wed, Feb 17, 2016 at 7:25 PM Hal Finkel via llvm-dev <
>> > llvm-dev at lists.llvm.org > wrote:
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > From: "Alina Sbirlea via llvm-dev" < llvm-dev at lists.llvm.org >
>> > To: "llvm-dev" < llvm-dev at lists.llvm.org >
>> > Sent: Wednesday, February 17, 2016 7:25:17 PM
>> > Subject: [llvm-dev] RFC: Add bitcode tests to test-suite
>> >
>> >
>> >
>> >
>> >
>> >
>> > Hi all,
>> >
>> >
>> > TL;DR: Add *.bc to test-suite; llc *.bc; run some.
>> >
>> >
>> >
>> > We would like to propose adding bitcode tests to the llvm test-suite.
>> >
>> >
>> > Recent LLVM bugs [2-4] prompted us to look into upstreaming a subset
>> > of the tests the Halide library [1] is running and we'd like the
>> > community's feedback on moving forward with this.
>> >
>> >
>> >
>> > Halide uses LLVM and can generate bitcode, but we cannot add C++
>> > tests to test-suite without including the library itself.
>> > This proposal is also potentially useful for other cases where there
>> > is no C++ front-end.
>> >
>> >
>> > As a first step we are interested in adding a set of correctness
>> > tests, for testing the IR without running the tests. Since these
>> > tests are generated, they are not instrumented like the .ll files in
>> > trunk, however we believe checking that llc runs without errors is
>> > still useful.
>> > The bitcode files for Halide may also be large, so including them as
>> > regression tests is not an option. If the smaller tests are found to
>> > be valuable or covering cases no other tests cover, we can
>> > instrument them and move them into the llvm trunk further along, but
>> > that is not the goal of this proposal.
>> > In addition, we're not sure whether the format for the tests should
>> > be .ll or .bc, we're open to either.
>> >
>> >
>> > After this first step, we're interested in upstreaming bitcode tests
>> > and also running them.
>> > We are very interested in tests for multiple architectures, aarch64
>> > in particular, since this is where we have seen things break. This
>> > may motivate adding .ll files rather than .bc in order to include
>> > the "RUN:" target.
>> > Where would these tests reside and with what directory structure?
>> > (similar to test/CodeGen?)
>> >
>> >
>> > Suggestion on what's the best approach for extending the test-suite
>> > framework for this proposal are more than welcome.
>> >
>> >
>> >
>> > We already have architecture-specific tests in the test suite (e.g.
>> > SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and Clang can deal
>> > with IR inputs. I suppose you need to compile some corresponding
>> > runtime library, but this does not seem like a big deal either.
>> > Mechanically, I don't see this as particularly complicated. I think
>> > the real question is: Is this the best way to have a kind of 'halide
>> > buildbot' that can inform the LLVM developer community?
>> >
>> > -Hal
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > This is just the high-level overview to start off the discussion, I'm
>> > sure there are many more aspects to touch on. Looking forward to
>> > your feedback!
>> >
>> >
>> >
>> > Thanks,
>> > Alina
>> >
>> >
>> > [1] http:// halide -lang.org/
>> > [2] Broken: r259800 => Fixed: r260131
>> > [3] Broken: r260569 => Fixed: r260701
>> >
>> > [4] https://llvm.org/bugs/show_bug.cgi?id=26642
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > LLVM Developers mailing list
>> > llvm-dev at lists.llvm.org
>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> >
>> >
>> >
>> >
>> >
>> > --
>> >
>> > Hal Finkel
>> > Assistant Computational Scientist
>> > Leadership Computing Facility
>> > Argonne National Laboratory
>> > _______________________________________________
>> > LLVM Developers mailing list
>> > llvm-dev at lists.llvm.org
>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> >
>>
>> --
>> Hal Finkel
>> Assistant Computational Scientist
>> Leadership Computing Facility
>> Argonne National Laboratory
>>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160217/d7ead4a8/attachment.html>