<div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 18, 2016 at 9:37 AM, David Blaikie via llvm-dev <span dir="ltr"><<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Thu, Feb 18, 2016 at 9:27 AM, Philip Reames <span dir="ltr"><<a href="mailto:listmail@philipreames.com" target="_blank">listmail@philipreames.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span>
<br>
<br>
<div>On 02/18/2016 09:08 AM, David Blaikie
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Feb 18, 2016 at 8:42 AM,
Philip Reames via llvm-dev <span dir="ltr"><<a href="mailto:llvm-dev@lists.llvm.org" target="_blank"></a><a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><br>
<br>
On 02/18/2016 06:54 AM, Hal Finkel via llvm-dev wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Chandler, et al.,<br>
<br>
While this proposal to put IR into the test suite
technically non-problematic, I've convinced myself
that this is a suboptimal direction for the LLVM
project. Here's what I think would be better:<br>
<br>
- We create a test-suite/Frontends directory, and
open this directory to actively-maintained external
frontends, subject to the following restrictions:<br>
<br>
- The frontend must be actively maintained, and
the project must agree to actively maintain the
test-suite version<br>
- The frontend must use the LLVM API (either C or
C++) - no printing textual IR<br>
- The frontend must have no significant
(non-optional) dependencies outside of LLVM itself, or
things on which LLVM itself depends<br>
- The frontend must have regression tests and
benchmarks/correctness tests providing significant
coverage of the frontend and its associated code
generation<br>
<br>
Here's the quid pro quo:<br>
<br>
- The LLVM community gains additional testing
coverage (which we definitely need)<br>
- The LLVM community gains extra insight into how
its APIs are being used (hopefully allowing us to make
more-informed decisions about how to update them)<br>
<br>
- The frontend gains free API updates<br>
- The frontend's use of LLVM will be more stable<br>
<br>
This involves extra work for everybody, but will help
us all deliver higher-quality products. Plus, given
the constant discussions about the difficulty for
external projects to follow API updates, etc., this is
a good way to help address those difficulties.<br>
<br>
The fact that Halide will provide extra coverage of
our vector code generation (aside from whatever we
happen to produce from our autovectorizers), and our
JIT infrastructure, makes it a good candidate for
this. Intel's ispc, POCL, (maybe whatever bit of Mesa
uses LLVM), etc. would also be natural candidates
should the projects be interested.<br>
</blockquote>
</span>
I think this is a really bad tradeoff and am strongly
opposed to this proposal.<br>
<br>
If we want to focus on improving test coverage, that's
reasonable, but doing so at the cost of requiring LLVM
contributors to maintain everyone's frontend is not a
reasonable approach.<br>
<br>
A couple of alternate approaches which might be worth
considering:<br>
1) The IR corpus approach mentioned previously. So long
as external teams are willing to update the corpus
regularly (weekly), this gives most of the backend
coverage with none of the maintenance burden.<br>
</blockquote>
<div><br>
</div>
<div>Why weekly? & why not bitcode, that would be long
lasting? (still, updating it regularly would be helpful,
but in theory we should keep working on the same bitcode
for a fairly long timeframe & means when I go and make
breaking IR changes I don't have to add the test-suite to
the list of things I need to fix :))</div>
</div>
</div>
</div>
</blockquote></span>
I should have written bitcode to start with. :) All of your points
are sound.<br>
<br>
I said "weekly" mostly as a placeholder for requiring active
involvement from the frontend and as a means to keep the two
projects roughly in sync. If Halide started generating radically
different IR all of a sudden, we want the bitcode tests to reflect
that. <br></div></blockquote><div><br></div></div></div><div>Fair enough - I imagine this'd look a lot like retiring old backends. If someone's not updating it it's mostly their loss, but once it's enough of a burden on the LLVM project, we just remove it.<br></div></div></div></div></blockquote><div><br></div><div>Sure, that's more than reasonable!</div><div><br></div><div>To clarify, I don't want the discussion to drift in the direction of whether to add Halide or other front-ends to LLVM. That's a very involved decision that would affect all the community and it's not a burden we want to add. Halide is maintained separately.</div><div><br></div><div>The proposal at hand is what is the best way to get test coverage when the C++ front-end is missing, independent of where the tests come from.</div><div>i.e. What is the best test format? Is it ok to have non-runnable tests checking just the IR?</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><span>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2) Use coverage information to determine which code paths
Halide covers which are not covered by existing unit
tests. Work to improve those unit tests. Using something
along the lines with a mutation testing (i.e. change the
source code and see what breaks), combined with test
reduction (bugpoint), could greatly improve our test
coverage in tree fairly quickly. This would require a lot
of work from a single contributor, but that's much better
than requiring a lot of work from all contributors.</blockquote>
<div><br>
</div>
<div>While this would be awesome (& I'd love to see some
LLVM/Clang-based mutation testing tools, and to improve
our test coverage using them) that seems like a pretty big
investment that I'm not sure anyone is signing up for just
now.</div>
</div>
</div>
</div>
</blockquote></span>
Fair point. However, before we ask the entire project to sign up
for a lot of work, asking some particular motivated person to do so
seems reasonable. :)<br></div></blockquote><div><br></div></span><div>Sure - I suspect, realistically, that neither of the expensive options is really the way to go, though.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
<br>
I'll also note that I was thinking of a very simple version
initially. Something on the order of "replace all untested lines
with llvm_unreachable, reduce one test, rerun coverage, repeat".
This could be done mostly manually and would yield a lot of
improvement. <br></div></blockquote><div><br></div></span><div>That sounds more or less like coverage based fuzzing, which we have (in asan/libFuzzer). Mutation testing's a bit more involved, but would be fun to have.<br></div></div></div></div></blockquote><div><br></div><div>Agreed on both your points. While these options would be nice, they are not the immediate solution.</div><div>If at some point someone signs up to make this happen, we can revisit the decision of including the corpus of IR tests.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br>- Dave</div><div><div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><div><div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Thanks again,<br>
Hal<br>
<br>
----- Original Message -----<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
From: "Chandler Carruth" <<a href="mailto:chandlerc@google.com" target="_blank"></a><a href="mailto:chandlerc@google.com" target="_blank">chandlerc@google.com</a>><br>
To: "Hal Finkel" <<a href="mailto:hfinkel@anl.gov" target="_blank">hfinkel@anl.gov</a>>,
"Alina Sbirlea" <<a href="mailto:alina.sbirlea@gmail.com" target="_blank">alina.sbirlea@gmail.com</a>><br>
Cc: "llvm-dev" <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br>
Sent: Wednesday, February 17, 2016 9:34:24 PM<br>
Subject: Re: [llvm-dev] RFC: Add bitcode tests to
test-suite<br>
<br>
<br>
Some perhaps relevant aspects that make testing
users of LLVM like<br>
Halide challenging:<br>
<br>
<br>
Halide uses the LLVM C++ APIs, but there isn't a
good way to<br>
lock-step update it. So if we were to directly
test Halide, it<br>
wouldn't link against the new LLVM.<br>
<br>
<br>
Practically speaking though, the LLVM IR generated
by Halide should<br>
continue to work with newer LLVM optimizations and
code generation.<br>
So the idea would be to snapshot the IR in bitcode
(which is at<br>
least reasonably stable) so that we could replay
the tests as LLVM<br>
changes. We can freshen the bitcode by
re-generating it periodically<br>
so it doesn't drift too far from what Halide
actually uses.<br>
<br>
<br>
The interesting questions IMO are:<br>
<br>
<br>
1) Are folks happy using bitcode as the format
here? I agree with Hal<br>
that it should be easy since Clang will actually
Do The Right Thing<br>
if given a bitcode input.<br>
<br>
<br>
2) Are folks happy with non-execution tests in
some cases? I think<br>
Alina is looking at whether we can get a runtime
library that will<br>
allow some of these to actually execute, but at
least some of the<br>
tests are just snap-shots of a JIT, and would need
the full Halide<br>
libraries (and introspection) to execute usefully.<br>
<br>
<br>
-Chandler<br>
<br>
<br>
On Wed, Feb 17, 2016 at 7:25 PM Hal Finkel via
llvm-dev <<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a> >
wrote:<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
From: "Alina Sbirlea via llvm-dev" < <a href="mailto:llvm-dev@lists.llvm.org" target="_blank"></a><a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a> ><br>
To: "llvm-dev" < <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a> ><br>
Sent: Wednesday, February 17, 2016 7:25:17 PM<br>
Subject: [llvm-dev] RFC: Add bitcode tests to
test-suite<br>
<br>
<br>
<br>
<br>
<br>
<br>
Hi all,<br>
<br>
<br>
TL;DR: Add *.bc to test-suite; llc *.bc; run some.<br>
<br>
<br>
<br>
We would like to propose adding bitcode tests to
the llvm test-suite.<br>
<br>
<br>
Recent LLVM bugs [2-4] prompted us to look into
upstreaming a subset<br>
of the tests the Halide library [1] is running and
we'd like the<br>
community's feedback on moving forward with this.<br>
<br>
<br>
<br>
Halide uses LLVM and can generate bitcode, but we
cannot add C++<br>
tests to test-suite without including the library
itself.<br>
This proposal is also potentially useful for other
cases where there<br>
is no C++ front-end.<br>
<br>
<br>
As a first step we are interested in adding a set
of correctness<br>
tests, for testing the IR without running the
tests. Since these<br>
tests are generated, they are not instrumented
like the .ll files in<br>
trunk, however we believe checking that llc runs
without errors is<br>
still useful.<br>
The bitcode files for Halide may also be large, so
including them as<br>
regression tests is not an option. If the smaller
tests are found to<br>
be valuable or covering cases no other tests
cover, we can<br>
instrument them and move them into the llvm trunk
further along, but<br>
that is not the goal of this proposal.<br>
In addition, we're not sure whether the format for
the tests should<br>
be .ll or .bc, we're open to either.<br>
<br>
<br>
After this first step, we're interested in
upstreaming bitcode tests<br>
and also running them.<br>
We are very interested in tests for multiple
architectures, aarch64<br>
in particular, since this is where we have seen
things break. This<br>
may motivate adding .ll files rather than .bc in
order to include<br>
the "RUN:" target.<br>
Where would these tests reside and with what
directory structure?<br>
(similar to test/CodeGen?)<br>
<br>
<br>
Suggestion on what's the best approach for
extending the test-suite<br>
framework for this proposal are more than welcome.<br>
<br>
<br>
<br>
We already have architecture-specific tests in the
test suite (e.g.<br>
SingleSource/UnitTests/Vector/{SSE,Altivec,etc.},
and Clang can deal<br>
with IR inputs. I suppose you need to compile some
corresponding<br>
runtime library, but this does not seem like a big
deal either.<br>
Mechanically, I don't see this as particularly
complicated. I think<br>
the real question is: Is this the best way to have
a kind of 'halide<br>
buildbot' that can inform the LLVM developer
community?<br>
<br>
-Hal<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
This is just the high-level overview to start off
the discussion, I'm<br>
sure there are many more aspects to touch on.
Looking forward to<br>
your feedback!<br>
<br>
<br>
<br>
Thanks,<br>
Alina<br>
<br>
<br>
[1] <a>http://</a> halide -<a href="http://lang.org/" rel="noreferrer" target="_blank">lang.org/</a><br>
[2] Broken: r259800 => Fixed: r260131<br>
[3] Broken: r260569 => Fixed: r260701<br>
<br>
[4] <a href="https://llvm.org/bugs/show_bug.cgi?id=26642" rel="noreferrer" target="_blank">https://llvm.org/bugs/show_bug.cgi?id=26642</a><br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
<br>
<br>
<br>
<br>
<br>
--<br>
<br>
Hal Finkel<br>
Assistant Computational Scientist<br>
Leadership Computing Facility<br>
Argonne National Laboratory<br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
<br>
</blockquote>
</blockquote>
<br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div></div></div><br></div></div>
<br>_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
<br></blockquote></div><br></div></div>