[llvm-dev] RFC: Add bitcode tests to test-suite

[Hal Finkel via llvm-dev]

----- Original Message ----- 

> From: "Mehdi Amini" <mehdi.amini at apple.com>
> To: "Alina Sbirlea" <alina.sbirlea at gmail.com>
> Cc: "Hal Finkel" <hfinkel at anl.gov>, "llvm-dev"
> <llvm-dev at lists.llvm.org>
> Sent: Wednesday, February 17, 2016 11:55:46 PM
> Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite

> > On Feb 17, 2016, at 9:27 PM, Alina Sbirlea via llvm-dev <
> > llvm-dev at lists.llvm.org > wrote:
> 

> > >I think the real question is: Is this the best way to have a kind
> > >of
> > >'halide buildbot' that can inform the LLVM developer community?
> 

> > Halide already has a buildbot running every few hours which is
> > being
> > used to inform LLVM developer community when something breaks. It
> > would be a lot more useful however to have the tests in an LLVM
> > repository to inform LLVM devs which test broke right away. You're
> > right that the underlying reason is the fact that Halide has test
> > coverage of areas currently not covered.
> 
> It is not clear to me why your "Halide buildbot" is not enough?

> > > As far as I can tell, Halide is < 100K LOC and has no external
> > > dependencies other than LLVM itself. I think we should just add
> > > it
> > > to the test suite.
> 
> Which suite are we talking about? Is it
> https://llvm.org/svn/llvm-project/test-suite/trunk/ ?

> > I realize that means the community updating it for API changes,
> 
> The only project we are maintaining on top of LLVM is clang.
> There is nothing in the test-suite repository that link to LLVM
> (AFAIK), changing this means to add a requirement to build the the
> test-suite when changing some API in LLVM (even for "NFC" changes).

> > but if the additional test coverage is as significant as I suspect,
> > and the project authors will help and are responsive, that seems
> > worthwhile. It is a JIT and a heavy generator of vector code, two
> > areas in which our story on regular upstream testing coverage is
> > not
> > great.
> 

> Keeping Halide, and the Halide test-suite in a totally separate
> repository, with a continuous integration with LLVM trunk, provides
> the same coverage in practice.

> Also the test-suite has a mechanism for "External" suites, which
> could be used here.

I think this is a good idea.

 -Hal

> --
> Mehdi

> > Halide can do both JIT and AOT compilation. Would the community be
> > happy to have non-execution tests for the JITted tests and
> > execution
> > tests for the AOT ones? This would in theory use a small set of
> > Halide and not need the entire library, which is what we are trying
> > to avoid here.
> 
> > The approach is meant to not clutter test-suite with a sizable
> > amount
> > of code but still get the test coverage offered by Halide.
> 

> > On Wed, Feb 17, 2016 at 8:53 PM, Hal Finkel < hfinkel at anl.gov >
> > wrote:
> 

> > > ----- Original Message -----
> > 
> 

> > > > From: "Chandler Carruth" < chandlerc at google.com >
> > 
> 
> > > > To: "Hal Finkel" < hfinkel at anl.gov >, "Alina Sbirlea" <
> > > > alina.sbirlea at gmail.com >
> > 
> 
> > > > Cc: "llvm-dev" < llvm-dev at lists.llvm.org >
> > 
> 
> > > > Sent: Wednesday, February 17, 2016 9:34:24 PM
> > 
> 
> > > > Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > Some perhaps relevant aspects that make testing users of LLVM
> > > > like
> > 
> 
> > > > Halide challenging:
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > Halide uses the LLVM C++ APIs, but there isn't a good way to
> > 
> 
> > > > lock-step update it. So if we were to directly test Halide, it
> > 
> 
> > > > wouldn't link against the new LLVM.
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > Practically speaking though, the LLVM IR generated by Halide
> > > > should
> > 
> 
> > > > continue to work with newer LLVM optimizations and code
> > > > generation.
> > 
> 
> > > > So the idea would be to snapshot the IR in bitcode (which is at
> > 
> 
> > > > least reasonably stable) so that we could replay the tests as
> > > > LLVM
> > 
> 
> > > > changes. We can freshen the bitcode by re-generating it
> > > > periodically
> > 
> 
> > > > so it doesn't drift too far from what Halide actually uses.
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > The interesting questions IMO are:
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > 1) Are folks happy using bitcode as the format here? I agree
> > > > with
> > > > Hal
> > 
> 
> > > > that it should be easy since Clang will actually Do The Right
> > > > Thing
> > 
> 
> > > > if given a bitcode input.
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > 2) Are folks happy with non-execution tests in some cases? I
> > > > think
> > 
> 
> > > > Alina is looking at whether we can get a runtime library that
> > > > will
> > 
> 
> > > > allow some of these to actually execute, but at least some of
> > > > the
> > 
> 
> > > > tests are just snap-shots of a JIT, and would need the full
> > > > Halide
> > 
> 
> > > > libraries (and introspection) to execute usefully.
> > 
> 
> > > >
> > 
> 

> > > As far as I can tell, Halide is < 100K LOC and has no external
> > > dependencies other than LLVM itself. I think we should just add
> > > it
> > > to the test suite. I realize that means the community updating it
> > > for API changes, but if the additional test coverage is as
> > > significant as I suspect, and the project authors will help and
> > > are
> > > responsive, that seems worthwhile. It is a JIT and a heavy
> > > generator
> > > of vector code, two areas in which our story on regular upstream
> > > testing coverage is not great.
> > 
> 

> > > -Hal
> > 
> 

> > > >
> > 
> 
> > > > -Chandler
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > On Wed, Feb 17, 2016 at 7:25 PM Hal Finkel via llvm-dev <
> > 
> 
> > > > llvm-dev at lists.llvm.org > wrote:
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 

> > > > From: "Alina Sbirlea via llvm-dev" < llvm-dev at lists.llvm.org >
> > 
> 
> > > > To: "llvm-dev" < llvm-dev at lists.llvm.org >
> > 
> 
> > > > Sent: Wednesday, February 17, 2016 7:25:17 PM
> > 
> 
> > > > Subject: [llvm-dev] RFC: Add bitcode tests to test-suite
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > Hi all,
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > TL;DR: Add *.bc to test-suite; llc *.bc; run some.
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > We would like to propose adding bitcode tests to the llvm
> > > > test-suite.
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > Recent LLVM bugs [2-4] prompted us to look into upstreaming a
> > > > subset
> > 
> 
> > > > of the tests the Halide library [1] is running and we'd like
> > > > the
> > 
> 
> > > > community's feedback on moving forward with this.
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > Halide uses LLVM and can generate bitcode, but we cannot add
> > > > C++
> > 
> 
> > > > tests to test-suite without including the library itself.
> > 
> 
> > > > This proposal is also potentially useful for other cases where
> > > > there
> > 
> 
> > > > is no C++ front-end.
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > As a first step we are interested in adding a set of
> > > > correctness
> > 
> 
> > > > tests, for testing the IR without running the tests. Since
> > > > these
> > 
> 
> > > > tests are generated, they are not instrumented like the .ll
> > > > files
> > > > in
> > 
> 
> > > > trunk, however we believe checking that llc runs without errors
> > > > is
> > 
> 
> > > > still useful.
> > 
> 
> > > > The bitcode files for Halide may also be large, so including
> > > > them
> > > > as
> > 
> 
> > > > regression tests is not an option. If the smaller tests are
> > > > found
> > > > to
> > 
> 
> > > > be valuable or covering cases no other tests cover, we can
> > 
> 
> > > > instrument them and move them into the llvm trunk further
> > > > along,
> > > > but
> > 
> 
> > > > that is not the goal of this proposal.
> > 
> 
> > > > In addition, we're not sure whether the format for the tests
> > > > should
> > 
> 
> > > > be .ll or .bc, we're open to either.
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > After this first step, we're interested in upstreaming bitcode
> > > > tests
> > 
> 
> > > > and also running them.
> > 
> 
> > > > We are very interested in tests for multiple architectures,
> > > > aarch64
> > 
> 
> > > > in particular, since this is where we have seen things break.
> > > > This
> > 
> 
> > > > may motivate adding .ll files rather than .bc in order to
> > > > include
> > 
> 
> > > > the "RUN:" target.
> > 
> 
> > > > Where would these tests reside and with what directory
> > > > structure?
> > 
> 
> > > > (similar to test/CodeGen?)
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > Suggestion on what's the best approach for extending the
> > > > test-suite
> > 
> 
> > > > framework for this proposal are more than welcome.
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > We already have architecture-specific tests in the test suite
> > > > (e.g.
> > 
> 
> > > > SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and Clang can
> > > > deal
> > 
> 
> > > > with IR inputs. I suppose you need to compile some
> > > > corresponding
> > 
> 
> > > > runtime library, but this does not seem like a big deal either.
> > 
> 
> > > > Mechanically, I don't see this as particularly complicated. I
> > > > think
> > 
> 
> > > > the real question is: Is this the best way to have a kind of
> > > > 'halide
> > 
> 
> > > > buildbot' that can inform the LLVM developer community?
> > 
> 
> > > >
> > 
> 
> > > > -Hal
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > This is just the high-level overview to start off the
> > > > discussion,
> > > > I'm
> > 
> 
> > > > sure there are many more aspects to touch on. Looking forward
> > > > to
> > 
> 
> > > > your feedback!
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > Thanks,
> > 
> 
> > > > Alina
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > [1] http:// halide - lang.org/
> > 
> 
> > > > [2] Broken: r259800 => Fixed: r260131
> > 
> 
> > > > [3] Broken: r260569 => Fixed: r260701
> > 
> 
> > > >
> > 
> 
> > > > [4] https://llvm.org/bugs/show_bug.cgi?id=26642
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > _______________________________________________
> > 
> 
> > > > LLVM Developers mailing list
> > 
> 
> > > > llvm-dev at lists.llvm.org
> > 
> 
> > > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > --
> > 
> 
> > > >
> > 
> 
> > > > Hal Finkel
> > 
> 
> > > > Assistant Computational Scientist
> > 
> 
> > > > Leadership Computing Facility
> > 
> 
> > > > Argonne National Laboratory
> > 
> 
> > > > _______________________________________________
> > 
> 
> > > > LLVM Developers mailing list
> > 
> 
> > > > llvm-dev at lists.llvm.org
> > 
> 
> > > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > 
> 
> > > >
> > 
> 

> > > --
> > 
> 
> > > Hal Finkel
> > 
> 
> > > Assistant Computational Scientist
> > 
> 
> > > Leadership Computing Facility
> > 
> 
> > > Argonne National Laboratory
> > 
> 

> > _______________________________________________
> 
> > LLVM Developers mailing list
> 
> > llvm-dev at lists.llvm.org
> 
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 

-- 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory