[llvm-dev] RFC: Add bitcode tests to test-suite

Thu Feb 18 09:37:59 PST 2016

On Thu, Feb 18, 2016 at 9:27 AM, Philip Reames <listmail at philipreames.com>
wrote:

>
>
> On 02/18/2016 09:08 AM, David Blaikie wrote:
>
>
>
> On Thu, Feb 18, 2016 at 8:42 AM, Philip Reames via llvm-dev <
> <llvm-dev at lists.llvm.org>llvm-dev at lists.llvm.org> wrote:
>
>>
>>
>> On 02/18/2016 06:54 AM, Hal Finkel via llvm-dev wrote:
>>
>>> Hi Chandler, et al.,
>>>
>>> While this proposal to put IR into the test suite technically
>>> non-problematic, I've convinced myself that this is a suboptimal direction
>>> for the LLVM project. Here's what I think would be better:
>>>
>>>   - We create a test-suite/Frontends directory, and open this directory
>>> to actively-maintained external frontends, subject to the following
>>> restrictions:
>>>
>>>     - The frontend must be actively maintained, and the project must
>>> agree to actively maintain the test-suite version
>>>     - The frontend must use the LLVM API (either C or C++) - no printing
>>> textual IR
>>>     - The frontend must have no significant (non-optional) dependencies
>>> outside of LLVM itself, or things on which LLVM itself depends
>>>     - The frontend must have regression tests and benchmarks/correctness
>>> tests providing significant coverage of the frontend and its associated
>>> code generation
>>>
>>> Here's the quid pro quo:
>>>
>>>     - The LLVM community gains additional testing coverage (which we
>>> definitely need)
>>>     - The LLVM community gains extra insight into how its APIs are being
>>> used (hopefully allowing us to make more-informed decisions about how to
>>> update them)
>>>
>>>     - The frontend gains free API updates
>>>     - The frontend's use of LLVM will be more stable
>>>
>>> This involves extra work for everybody, but will help us all deliver
>>> higher-quality products. Plus, given the constant discussions about the
>>> difficulty for external projects to follow API updates, etc., this is a
>>> good way to help address those difficulties.
>>>
>>> The fact that Halide will provide extra coverage of our vector code
>>> generation (aside from whatever we happen to produce from our
>>> autovectorizers), and our JIT infrastructure, makes it a good candidate for
>>> this. Intel's ispc, POCL, (maybe whatever bit of Mesa uses LLVM), etc.
>>> would also be natural candidates should the projects be interested.
>>>
>> I think this is a really bad tradeoff and am strongly opposed to this
>> proposal.
>>
>> If we want to focus on improving test coverage, that's reasonable, but
>> doing so at the cost of requiring LLVM contributors to maintain everyone's
>> frontend is not a reasonable approach.
>>
>> A couple of alternate approaches which might be worth considering:
>> 1) The IR corpus approach mentioned previously.  So long as external
>> teams are willing to update the corpus regularly (weekly), this gives most
>> of the backend coverage with none of the maintenance burden.
>>
>
> Why weekly? & why not bitcode, that would be long lasting? (still,
> updating it regularly would be helpful, but in theory we should keep
> working on the same bitcode for a fairly long timeframe & means when I go
> and make breaking IR changes I don't have to add the test-suite to the list
> of things I need to fix :))
>
> I should have written bitcode to start with.  :)  All of your points are
> sound.
>
> I said "weekly" mostly as a placeholder for requiring active involvement
> from the frontend and as a means to keep the two projects roughly in sync.
> If Halide started generating radically different IR all of a sudden, we
> want the bitcode tests to reflect that.
>

Fair enough - I imagine this'd look a lot like retiring old backends. If
someone's not updating it it's mostly their loss, but once it's enough of a
burden on the LLVM project, we just remove it.

>
>
>> 2) Use coverage information to determine which code paths Halide covers
>> which are not covered by existing unit tests.  Work to improve those unit
>> tests.  Using something along the lines with a mutation testing (i.e.
>> change the source code and see what breaks), combined with test reduction
>> (bugpoint), could greatly improve our test coverage in tree fairly
>> quickly.  This would require a lot of work from a single contributor, but
>> that's much better than requiring a lot of work from all contributors.
>
>
> While this would be awesome (& I'd love to see some LLVM/Clang-based
> mutation testing tools, and to improve our test coverage using them) that
> seems like a pretty big investment that I'm not sure anyone is signing up
> for just now.
>
> Fair point.  However, before we ask the entire project to sign up for a
> lot of work, asking some particular motivated person to do so seems
> reasonable.  :)
>

Sure - I suspect, realistically, that neither of the expensive options is
really the way to go, though.

>
> I'll also note that I was thinking of a very simple version initially.
> Something on the order of "replace all untested lines with
> llvm_unreachable, reduce one test, rerun coverage, repeat".  This could be
> done mostly manually and would yield a lot of improvement.
>

That sounds more or less like coverage based fuzzing, which we have (in
asan/libFuzzer). Mutation testing's a bit more involved, but would be fun
to have.

- Dave

>
>
>>
>>
>>> Thanks again,
>>> Hal
>>>
>>> ----- Original Message -----
>>>
>>>> From: "Chandler Carruth" < <chandlerc at google.com>chandlerc at google.com>
>>>> To: "Hal Finkel" <hfinkel at anl.gov>, "Alina Sbirlea" <
>>>> alina.sbirlea at gmail.com>
>>>> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>
>>>> Sent: Wednesday, February 17, 2016 9:34:24 PM
>>>> Subject: Re: [llvm-dev] RFC: Add bitcode tests to test-suite
>>>>
>>>>
>>>> Some perhaps relevant aspects that make testing users of LLVM like
>>>> Halide challenging:
>>>>
>>>>
>>>> Halide uses the LLVM C++ APIs, but there isn't a good way to
>>>> lock-step update it. So if we were to directly test Halide, it
>>>> wouldn't link against the new LLVM.
>>>>
>>>>
>>>> Practically speaking though, the LLVM IR generated by Halide should
>>>> continue to work with newer LLVM optimizations and code generation.
>>>> So the idea would be to snapshot the IR in bitcode (which is at
>>>> least reasonably stable) so that we could replay the tests as LLVM
>>>> changes. We can freshen the bitcode by re-generating it periodically
>>>> so it doesn't drift too far from what Halide actually uses.
>>>>
>>>>
>>>> The interesting questions IMO are:
>>>>
>>>>
>>>> 1) Are folks happy using bitcode as the format here? I agree with Hal
>>>> that it should be easy since Clang will actually Do The Right Thing
>>>> if given a bitcode input.
>>>>
>>>>
>>>> 2) Are folks happy with non-execution tests in some cases? I think
>>>> Alina is looking at whether we can get a runtime library that will
>>>> allow some of these to actually execute, but at least some of the
>>>> tests are just snap-shots of a JIT, and would need the full Halide
>>>> libraries (and introspection) to execute usefully.
>>>>
>>>>
>>>> -Chandler
>>>>
>>>>
>>>> On Wed, Feb 17, 2016 at 7:25 PM Hal Finkel via llvm-dev <
>>>> llvm-dev at lists.llvm.org > wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> From: "Alina Sbirlea via llvm-dev" < <llvm-dev at lists.llvm.org>
>>>> llvm-dev at lists.llvm.org >
>>>> To: "llvm-dev" < llvm-dev at lists.llvm.org >
>>>> Sent: Wednesday, February 17, 2016 7:25:17 PM
>>>> Subject: [llvm-dev] RFC: Add bitcode tests to test-suite
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Hi all,
>>>>
>>>>
>>>> TL;DR: Add *.bc to test-suite; llc *.bc; run some.
>>>>
>>>>
>>>>
>>>> We would like to propose adding bitcode tests to the llvm test-suite.
>>>>
>>>>
>>>> Recent LLVM bugs [2-4] prompted us to look into upstreaming a subset
>>>> of the tests the Halide library [1] is running and we'd like the
>>>> community's feedback on moving forward with this.
>>>>
>>>>
>>>>
>>>> Halide uses LLVM and can generate bitcode, but we cannot add C++
>>>> tests to test-suite without including the library itself.
>>>> This proposal is also potentially useful for other cases where there
>>>> is no C++ front-end.
>>>>
>>>>
>>>> As a first step we are interested in adding a set of correctness
>>>> tests, for testing the IR without running the tests. Since these
>>>> tests are generated, they are not instrumented like the .ll files in
>>>> trunk, however we believe checking that llc runs without errors is
>>>> still useful.
>>>> The bitcode files for Halide may also be large, so including them as
>>>> regression tests is not an option. If the smaller tests are found to
>>>> be valuable or covering cases no other tests cover, we can
>>>> instrument them and move them into the llvm trunk further along, but
>>>> that is not the goal of this proposal.
>>>> In addition, we're not sure whether the format for the tests should
>>>> be .ll or .bc, we're open to either.
>>>>
>>>>
>>>> After this first step, we're interested in upstreaming bitcode tests
>>>> and also running them.
>>>> We are very interested in tests for multiple architectures, aarch64
>>>> in particular, since this is where we have seen things break. This
>>>> may motivate adding .ll files rather than .bc in order to include
>>>> the "RUN:" target.
>>>> Where would these tests reside and with what directory structure?
>>>> (similar to test/CodeGen?)
>>>>
>>>>
>>>> Suggestion on what's the best approach for extending the test-suite
>>>> framework for this proposal are more than welcome.
>>>>
>>>>
>>>>
>>>> We already have architecture-specific tests in the test suite (e.g.
>>>> SingleSource/UnitTests/Vector/{SSE,Altivec,etc.}, and Clang can deal
>>>> with IR inputs. I suppose you need to compile some corresponding
>>>> runtime library, but this does not seem like a big deal either.
>>>> Mechanically, I don't see this as particularly complicated. I think
>>>> the real question is: Is this the best way to have a kind of 'halide
>>>> buildbot' that can inform the LLVM developer community?
>>>>
>>>> -Hal
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> This is just the high-level overview to start off the discussion, I'm
>>>> sure there are many more aspects to touch on. Looking forward to
>>>> your feedback!
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Alina
>>>>
>>>>
>>>> [1] http:// halide -lang.org/
>>>> [2] Broken: r259800 => Fixed: r260131
>>>> [3] Broken: r260569 => Fixed: r260701
>>>>
>>>> [4] https://llvm.org/bugs/show_bug.cgi?id=26642
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Hal Finkel
>>>> Assistant Computational Scientist
>>>> Leadership Computing Facility
>>>> Argonne National Laboratory
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160218/a3beccbf/attachment.html>