[llvm-dev] RFC: Add bitcode tests to test-suite

Thu Feb 18 09:08:33 PST 2016

On Thu, Feb 18, 2016 at 8:42 AM, Philip Reames via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

>
>
> On 02/18/2016 06:54 AM, Hal Finkel via llvm-dev wrote:
>
>> Hi Chandler, et al.,
>>
>> While this proposal to put IR into the test suite technically
>> non-problematic, I've convinced myself that this is a suboptimal direction
>> for the LLVM project. Here's what I think would be better:
>>
>>   - We create a test-suite/Frontends directory, and open this directory
>> to actively-maintained external frontends, subject to the following
>> restrictions:
>>
>>     - The frontend must be actively maintained, and the project must
>> agree to actively maintain the test-suite version
>>     - The frontend must use the LLVM API (either C or C++) - no printing
>> textual IR
>>     - The frontend must have no significant (non-optional) dependencies
>> outside of LLVM itself, or things on which LLVM itself depends
>>     - The frontend must have regression tests and benchmarks/correctness
>> tests providing significant coverage of the frontend and its associated
>> code generation
>>
>> Here's the quid pro quo:
>>
>>     - The LLVM community gains additional testing coverage (which we
>> definitely need)
>>     - The LLVM community gains extra insight into how its APIs are being
>> used (hopefully allowing us to make more-informed decisions about how to
>> update them)
>>
>>     - The frontend gains free API updates
>>     - The frontend's use of LLVM will be more stable
>>
>> This involves extra work for everybody, but will help us all deliver
>> higher-quality products. Plus, given the constant discussions about the
>> difficulty for external projects to follow API updates, etc., this is a
>> good way to help address those difficulties.
>>
>> The fact that Halide will provide extra coverage of our vector code
>> generation (aside from whatever we happen to produce from our
>> autovectorizers), and our JIT infrastructure, makes it a good candidate for
>> this. Intel's ispc, POCL, (maybe whatever bit of Mesa uses LLVM), etc.
>> would also be natural candidates should the projects be interested.
>>
> I think this is a really bad tradeoff and am strongly opposed to this
> proposal.
>
> If we want to focus on improving test coverage, that's reasonable, but
> doing so at the cost of requiring LLVM contributors to maintain everyone's
> frontend is not a reasonable approach.
>
> A couple of alternate approaches which might be worth considering:
> 1) The IR corpus approach mentioned previously.  So long as external teams
> are willing to update the corpus regularly (weekly), this gives most of the
> backend coverage with none of the maintenance burden.
>

Why weekly? & why not bitcode, that would be long lasting? (still, updating
it regularly would be helpful, but in theory we should keep working on the
same bitcode for a fairly long timeframe & means when I go and make
breaking IR changes I don't have to add the test-suite to the list of
things I need to fix :))

> 2) Use coverage information to determine which code paths Halide covers
> which are not covered by existing unit tests.  Work to improve those unit
> tests.  Using something along the lines with a mutation testing (i.e.
> change the source code and see what breaks), combined with test reduction
> (bugpoint), could greatly improve our test coverage in tree fairly
> quickly.  This would require a lot of work from a single contributor, but
> that's much better than requiring a lot of work from all contributors.

While this would be awesome (& I'd love to see some LLVM/Clang-based
mutation testing tools, and to improve our test coverage using them) that
seems like a pretty big investment that I'm not sure anyone is signing up
for just now.

>
>
>> Thanks again,
>> Hal
>>
>> ----- Original Message -----
>>
>>> From: "Chandler Carruth" <chandlerc at google.com>
>>> To: "Hal Finkel" <hfinkel at anl.gov>, "Alina Sbirlea" <
>>> alina.sbirlea at gmail.com>
>>> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>
>>> Sent: Wednesday, February 17, 2016 9:34:24 PM
>>> Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite
>>>
>>>
>>> Some perhaps relevant aspects that make testing users of LLVM like
>>> Halide challenging:
>>>
>>>
>>> Halide uses the LLVM C++ APIs, but there isn't a good way to
>>> lock-step update it. So if we were to directly test Halide, it
>>> wouldn't link against the new LLVM.
>>>
>>>
>>> Practically speaking though, the LLVM IR generated by Halide should
>>> continue to work with newer LLVM optimizations and code generation.
>>> So the idea would be to snapshot the IR in bitcode (which is at
>>> least reasonably stable) so that we could replay the tests as LLVM
>>> changes. We can freshen the bitcode by re-generating it periodically
>>> so it doesn't drift too far from what Halide actually uses.
>>>
>>>
>>> The interesting questions IMO are:
>>>
>>>
>>> 1) Are folks happy using bitcode as the format here? I agree with Hal
>>> that it should be easy since Clang will actually Do The Right Thing
>>> if given a bitcode input.
>>>
>>>
>>> 2) Are folks happy with non-execution tests in some cases? I think
>>> Alina is looking at whether we can get a runtime library that will
>>> allow some of these to actually execute, but at least some of the
>>> tests are just snap-shots of a JIT, and would need the full Halide
>>> libraries (and introspection) to execute usefully.
>>>
>>>
>>> -Chandler
>>>
>>>
>>> On Wed, Feb 17, 2016 at 7:25 PM Hal Finkel via llvm-dev <
>>> llvm-dev at lists.llvm.org > wrote:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> From: "Alina Sbirlea via llvm-dev" < llvm-dev at lists.llvm.org >
>>> To: "llvm-dev" < llvm-dev at lists.llvm.org >
>>> Sent: Wednesday, February 17, 2016 7:25:17 PM
>>> Subject: [llvm-dev] RFC: Add bitcode tests to test-suite
>>>
>>>
>>>
>>>
>>>
>>>
>>> Hi all,
>>>
>>>
>>> TL;DR: Add *.bc to test-suite; llc *.bc; run some.
>>>
>>>
>>>
>>> We would like to propose adding bitcode tests to the llvm test-suite.
>>>
>>>
>>> Recent LLVM bugs [2-4] prompted us to look into upstreaming a subset
>>> of the tests the Halide library [1] is running and we'd like the
>>> community's feedback on moving forward with this.
>>>
>>>
>>>
>>> Halide uses LLVM and can generate bitcode, but we cannot add C++
>>> tests to test-suite without including the library itself.
>>> This proposal is also potentially useful for other cases where there
>>> is no C++ front-end.
>>>
>>>
>>> As a first step we are interested in adding a set of correctness
>>> tests, for testing the IR without running the tests. Since these
>>> tests are generated, they are not instrumented like the .ll files in
>>> trunk, however we believe checking that llc runs without errors is
>>> still useful.
>>> The bitcode files for Halide may also be large, so including them as
>>> regression tests is not an option. If the smaller tests are found to
>>> be valuable or covering cases no other tests cover, we can
>>> instrument them and move them into the llvm trunk further along, but
>>> that is not the goal of this proposal.
>>> In addition, we're not sure whether the format for the tests should
>>> be .ll or .bc, we're open to either.
>>>
>>>
>>> After this first step, we're interested in upstreaming bitcode tests
>>> and also running them.
>>> We are very interested in tests for multiple architectures, aarch64
>>> in particular, since this is where we have seen things break. This
>>> may motivate adding .ll files rather than .bc in order to include
>>> the "RUN:" target.
>>> Where would these tests reside and with what directory structure?
>>> (similar to test/CodeGen?)
>>>
>>>
>>> Suggestion on what's the best approach for extending the test-suite
>>> framework for this proposal are more than welcome.
>>>
>>>
>>>
>>> We already have architecture-specific tests in the test suite (e.g.
>>> SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and Clang can deal
>>> with IR inputs. I suppose you need to compile some corresponding
>>> runtime library, but this does not seem like a big deal either.
>>> Mechanically, I don't see this as particularly complicated. I think
>>> the real question is: Is this the best way to have a kind of 'halide
>>> buildbot' that can inform the LLVM developer community?
>>>
>>> -Hal
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> This is just the high-level overview to start off the discussion, I'm
>>> sure there are many more aspects to touch on. Looking forward to
>>> your feedback!
>>>
>>>
>>>
>>> Thanks,
>>> Alina
>>>
>>>
>>> [1] http:// halide -lang.org/
>>> [2] Broken: r259800 => Fixed: r260131
>>> [3] Broken: r260569 => Fixed: r260701
>>>
>>> [4] https://llvm.org/bugs/show_bug.cgi?id=26642
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Hal Finkel
>>> Assistant Computational Scientist
>>> Leadership Computing Facility
>>> Argonne National Laboratory
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160218/4f818b5e/attachment.html>