[llvm-dev] RFC: Add bitcode tests to test-suite
Hal Finkel via llvm-dev
llvm-dev at lists.llvm.org
Thu Feb 18 06:28:14 PST 2016
----- Original Message -----
> From: "Hal Finkel via llvm-dev" <llvm-dev at lists.llvm.org>
> To: "Mehdi Amini" <mehdi.amini at apple.com>
> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>
> Sent: Thursday, February 18, 2016 6:51:33 AM
> Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite
>
> ----- Original Message -----
>
> > From: "Mehdi Amini" <mehdi.amini at apple.com>
> > To: "Alina Sbirlea" <alina.sbirlea at gmail.com>
> > Cc: "Hal Finkel" <hfinkel at anl.gov>, "llvm-dev"
> > <llvm-dev at lists.llvm.org>
> > Sent: Wednesday, February 17, 2016 11:55:46 PM
> > Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite
>
> > > On Feb 17, 2016, at 9:27 PM, Alina Sbirlea via llvm-dev <
> > > llvm-dev at lists.llvm.org > wrote:
> >
>
> > > >I think the real question is: Is this the best way to have a
> > > >kind
> > > >of
> > > >'halide buildbot' that can inform the LLVM developer community?
> >
>
> > > Halide already has a buildbot running every few hours which is
> > > being
> > > used to inform LLVM developer community when something breaks. It
> > > would be a lot more useful however to have the tests in an LLVM
> > > repository to inform LLVM devs which test broke right away.
> > > You're
> > > right that the underlying reason is the fact that Halide has test
> > > coverage of areas currently not covered.
> >
> > It is not clear to me why your "Halide buildbot" is not enough?
>
> > > > As far as I can tell, Halide is < 100K LOC and has no external
> > > > dependencies other than LLVM itself. I think we should just add
> > > > it
> > > > to the test suite.
> >
> > Which suite are we talking about? Is it
> > https://llvm.org/svn/llvm-project/test-suite/trunk/ ?
>
> > > I realize that means the community updating it for API changes,
> >
> > The only project we are maintaining on top of LLVM is clang.
> > There is nothing in the test-suite repository that link to LLVM
> > (AFAIK), changing this means to add a requirement to build the the
> > test-suite when changing some API in LLVM (even for "NFC" changes).
>
> > > but if the additional test coverage is as significant as I
> > > suspect,
> > > and the project authors will help and are responsive, that seems
> > > worthwhile. It is a JIT and a heavy generator of vector code, two
> > > areas in which our story on regular upstream testing coverage is
> > > not
> > > great.
> >
>
> > Keeping Halide, and the Halide test-suite in a totally separate
> > repository, with a continuous integration with LLVM trunk, provides
> > the same coverage in practice.
>
> > Also the test-suite has a mechanism for "External" suites, which
> > could be used here.
>
> I think this is a good idea.
However, we'd need to ignore build failures (instead of treating them as actual problems) because we can't block in-tree API updates on out-of-tree projects.
-Hal
>
> -Hal
>
> > --
> > Mehdi
>
> > > Halide can do both JIT and AOT compilation. Would the community
> > > be
> > > happy to have non-execution tests for the JITted tests and
> > > execution
> > > tests for the AOT ones? This would in theory use a small set of
> > > Halide and not need the entire library, which is what we are
> > > trying
> > > to avoid here.
> >
> > > The approach is meant to not clutter test-suite with a sizable
> > > amount
> > > of code but still get the test coverage offered by Halide.
> >
>
> > > On Wed, Feb 17, 2016 at 8:53 PM, Hal Finkel < hfinkel at anl.gov >
> > > wrote:
> >
>
> > > > ----- Original Message -----
> > >
> >
>
> > > > > From: "Chandler Carruth" < chandlerc at google.com >
> > >
> >
> > > > > To: "Hal Finkel" < hfinkel at anl.gov >, "Alina Sbirlea" <
> > > > > alina.sbirlea at gmail.com >
> > >
> >
> > > > > Cc: "llvm-dev" < llvm-dev at lists.llvm.org >
> > >
> >
> > > > > Sent: Wednesday, February 17, 2016 9:34:24 PM
> > >
> >
> > > > > Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > Some perhaps relevant aspects that make testing users of LLVM
> > > > > like
> > >
> >
> > > > > Halide challenging:
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > Halide uses the LLVM C++ APIs, but there isn't a good way to
> > >
> >
> > > > > lock-step update it. So if we were to directly test Halide,
> > > > > it
> > >
> >
> > > > > wouldn't link against the new LLVM.
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > Practically speaking though, the LLVM IR generated by Halide
> > > > > should
> > >
> >
> > > > > continue to work with newer LLVM optimizations and code
> > > > > generation.
> > >
> >
> > > > > So the idea would be to snapshot the IR in bitcode (which is
> > > > > at
> > >
> >
> > > > > least reasonably stable) so that we could replay the tests as
> > > > > LLVM
> > >
> >
> > > > > changes. We can freshen the bitcode by re-generating it
> > > > > periodically
> > >
> >
> > > > > so it doesn't drift too far from what Halide actually uses.
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > The interesting questions IMO are:
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > 1) Are folks happy using bitcode as the format here? I agree
> > > > > with
> > > > > Hal
> > >
> >
> > > > > that it should be easy since Clang will actually Do The Right
> > > > > Thing
> > >
> >
> > > > > if given a bitcode input.
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > 2) Are folks happy with non-execution tests in some cases? I
> > > > > think
> > >
> >
> > > > > Alina is looking at whether we can get a runtime library that
> > > > > will
> > >
> >
> > > > > allow some of these to actually execute, but at least some of
> > > > > the
> > >
> >
> > > > > tests are just snap-shots of a JIT, and would need the full
> > > > > Halide
> > >
> >
> > > > > libraries (and introspection) to execute usefully.
> > >
> >
> > > > >
> > >
> >
>
> > > > As far as I can tell, Halide is < 100K LOC and has no external
> > > > dependencies other than LLVM itself. I think we should just add
> > > > it
> > > > to the test suite. I realize that means the community updating
> > > > it
> > > > for API changes, but if the additional test coverage is as
> > > > significant as I suspect, and the project authors will help and
> > > > are
> > > > responsive, that seems worthwhile. It is a JIT and a heavy
> > > > generator
> > > > of vector code, two areas in which our story on regular
> > > > upstream
> > > > testing coverage is not great.
> > >
> >
>
> > > > -Hal
> > >
> >
>
> > > > >
> > >
> >
> > > > > -Chandler
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > On Wed, Feb 17, 2016 at 7:25 PM Hal Finkel via llvm-dev <
> > >
> >
> > > > > llvm-dev at lists.llvm.org > wrote:
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
>
> > > > > From: "Alina Sbirlea via llvm-dev" < llvm-dev at lists.llvm.org
> > > > > >
> > >
> >
> > > > > To: "llvm-dev" < llvm-dev at lists.llvm.org >
> > >
> >
> > > > > Sent: Wednesday, February 17, 2016 7:25:17 PM
> > >
> >
> > > > > Subject: [llvm-dev] RFC: Add bitcode tests to test-suite
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > Hi all,
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > TL;DR: Add *.bc to test-suite; llc *.bc; run some.
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > We would like to propose adding bitcode tests to the llvm
> > > > > test-suite.
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > Recent LLVM bugs [2-4] prompted us to look into upstreaming a
> > > > > subset
> > >
> >
> > > > > of the tests the Halide library [1] is running and we'd like
> > > > > the
> > >
> >
> > > > > community's feedback on moving forward with this.
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > Halide uses LLVM and can generate bitcode, but we cannot add
> > > > > C++
> > >
> >
> > > > > tests to test-suite without including the library itself.
> > >
> >
> > > > > This proposal is also potentially useful for other cases
> > > > > where
> > > > > there
> > >
> >
> > > > > is no C++ front-end.
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > As a first step we are interested in adding a set of
> > > > > correctness
> > >
> >
> > > > > tests, for testing the IR without running the tests. Since
> > > > > these
> > >
> >
> > > > > tests are generated, they are not instrumented like the .ll
> > > > > files
> > > > > in
> > >
> >
> > > > > trunk, however we believe checking that llc runs without
> > > > > errors
> > > > > is
> > >
> >
> > > > > still useful.
> > >
> >
> > > > > The bitcode files for Halide may also be large, so including
> > > > > them
> > > > > as
> > >
> >
> > > > > regression tests is not an option. If the smaller tests are
> > > > > found
> > > > > to
> > >
> >
> > > > > be valuable or covering cases no other tests cover, we can
> > >
> >
> > > > > instrument them and move them into the llvm trunk further
> > > > > along,
> > > > > but
> > >
> >
> > > > > that is not the goal of this proposal.
> > >
> >
> > > > > In addition, we're not sure whether the format for the tests
> > > > > should
> > >
> >
> > > > > be .ll or .bc, we're open to either.
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > After this first step, we're interested in upstreaming
> > > > > bitcode
> > > > > tests
> > >
> >
> > > > > and also running them.
> > >
> >
> > > > > We are very interested in tests for multiple architectures,
> > > > > aarch64
> > >
> >
> > > > > in particular, since this is where we have seen things break.
> > > > > This
> > >
> >
> > > > > may motivate adding .ll files rather than .bc in order to
> > > > > include
> > >
> >
> > > > > the "RUN:" target.
> > >
> >
> > > > > Where would these tests reside and with what directory
> > > > > structure?
> > >
> >
> > > > > (similar to test/CodeGen?)
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > Suggestion on what's the best approach for extending the
> > > > > test-suite
> > >
> >
> > > > > framework for this proposal are more than welcome.
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > We already have architecture-specific tests in the test suite
> > > > > (e.g.
> > >
> >
> > > > > SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and Clang
> > > > > can
> > > > > deal
> > >
> >
> > > > > with IR inputs. I suppose you need to compile some
> > > > > corresponding
> > >
> >
> > > > > runtime library, but this does not seem like a big deal
> > > > > either.
> > >
> >
> > > > > Mechanically, I don't see this as particularly complicated. I
> > > > > think
> > >
> >
> > > > > the real question is: Is this the best way to have a kind of
> > > > > 'halide
> > >
> >
> > > > > buildbot' that can inform the LLVM developer community?
> > >
> >
> > > > >
> > >
> >
> > > > > -Hal
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > This is just the high-level overview to start off the
> > > > > discussion,
> > > > > I'm
> > >
> >
> > > > > sure there are many more aspects to touch on. Looking forward
> > > > > to
> > >
> >
> > > > > your feedback!
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > Thanks,
> > >
> >
> > > > > Alina
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > [1] http:// halide - lang.org/
> > >
> >
> > > > > [2] Broken: r259800 => Fixed: r260131
> > >
> >
> > > > > [3] Broken: r260569 => Fixed: r260701
> > >
> >
> > > > >
> > >
> >
> > > > > [4] https://llvm.org/bugs/show_bug.cgi?id=26642
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > _______________________________________________
> > >
> >
> > > > > LLVM Developers mailing list
> > >
> >
> > > > > llvm-dev at lists.llvm.org
> > >
> >
> > > > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > >
> > >
> >
> > > > > --
> > >
> >
> > > > >
> > >
> >
> > > > > Hal Finkel
> > >
> >
> > > > > Assistant Computational Scientist
> > >
> >
> > > > > Leadership Computing Facility
> > >
> >
> > > > > Argonne National Laboratory
> > >
> >
> > > > > _______________________________________________
> > >
> >
> > > > > LLVM Developers mailing list
> > >
> >
> > > > > llvm-dev at lists.llvm.org
> > >
> >
> > > > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > >
> >
> > > > >
> > >
> >
>
> > > > --
> > >
> >
> > > > Hal Finkel
> > >
> >
> > > > Assistant Computational Scientist
> > >
> >
> > > > Leadership Computing Facility
> > >
> >
> > > > Argonne National Laboratory
> > >
> >
>
> > > _______________________________________________
> >
> > > LLVM Developers mailing list
> >
> > > llvm-dev at lists.llvm.org
> >
> > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
>
> --
>
> --
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
--
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory
More information about the llvm-dev
mailing list