<div dir="ltr"><br><div class="gmail_extra"><div class="gmail_quote">On Wed, Feb 17, 2016 at 8:34 PM, Philip Reames <span dir="ltr"><<a href="mailto:listmail@philipreames.com" target="_blank">listmail@philipreames.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
I'm a bit confused as to what's being proposed for immediate
action. Is the proposal essentially to add a set of binary bitcode
files and ensure that running each of them through LLC does not
trigger any assertions? If so, arranging that as an external build
bot would seem entirely reasonable. If on the other hand, we were
testing for equivalence of the output assembly, that would probably
NOT be okay, just because of false positive rate. <br></div></blockquote><div><br></div><div>No, we are not planning for equivalence of output assembly. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
<br>
(Running with the assumption I rephrased the proposal correctly...
if not, this will be totally off topic.)<br>
<br>
Beyond Halide, this sounds like a potentially useful general
mechanism for integration testing (not unit testing). We (upstream
llvm) have something sorta similar today in the "self host clang,
see what breaks" workflow. We (my downstream team) have a similar
mechanism where we've collected a corpus of IR files from key
benchmarks (regenerated weekly), that we use to stress test tricky
commits before submission. <br>
<br>
Standardizing such a mechanism and policy around it's usage seems
useful. My major concerns are a) keeping the corpus small enough to
be useful, b) not having them intermixed with unit tests and c)
making sure that the *frontend* authors pay the primary cost to
maintain them. <br></div></blockquote><div><br></div><div>From my side I would like to have this standardized. As I said, I think this proposal's benefits are not just for Halide.</div><div>a) I believe the frontend authors need to ensure this. For Halide we would like to include a larger set of unit tests and a small set of large applications. The unit tests are meant to make sure no assertions are triggered. The applications are meant to run and compare with reference results.</div><div>Do you consider such a corpus of IR files reasonable?</div><div>b) I agree. How/where should these be integrated into test-suite?</div><div>c) Primarily the tests would be maintained by the frontend authors, but may be updated by llvm devs. For example, in the event of a change in llvm IR the easiest would be to have the tests as bitcode files. </div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><span class="HOEnZb"><font color="#888888">
<br>
Philip</font></span><div><div class="h5"><br>
<br>
<div>On 02/17/2016 05:25 PM, Alina Sbirlea
via llvm-dev wrote:<br>
</div>
</div></div><blockquote type="cite"><div><div class="h5">
<div dir="ltr">Hi all,
<div><br>
</div>
<div>TL;DR: Add *.bc to test-suite; llc *.bc; run some.</div>
<div><br>
</div>
<div>
<div>We would like to propose adding bitcode tests to the llvm
test-suite.</div>
</div>
<div><br>
</div>
<div>Recent LLVM bugs [2-4] prompted us to look into upstreaming
a subset of the tests the Halide library [1] is running and
we'd like the community's feedback on moving forward with
this.<br>
</div>
<div><br>
</div>
<div>Halide uses LLVM and can generate bitcode, but we cannot
add C++ tests to test-suite without including the library
itself.</div>
<div>This proposal is also potentially useful for other cases
where there is no C++ front-end.</div>
<div><br>
</div>
<div>As a first step we are interested in adding a set of
correctness tests, for testing the IR without running the
tests. Since these tests are generated, they are not
instrumented like the .ll files in trunk, however we believe
checking that llc runs without errors is still useful.</div>
<div>The bitcode files for Halide may also be large, so
including them as regression tests is not an option. If the
smaller tests are found to be valuable or covering cases no
other tests cover, we can instrument them and move them into
the llvm trunk further along, but that is not the goal of this
proposal.</div>
<div>In addition, we're not sure whether the format for the
tests should be .ll or .bc, we're open to either.</div>
<div><br>
</div>
<div>After this first step, we're interested in upstreaming
bitcode tests and also running them.</div>
<div>We are very interested in tests for multiple architectures,
aarch64 in particular, since this is where we have seen things
break. This may motivate adding .ll files rather than .bc in
order to include the "RUN:" target. </div>
<div>Where would these tests reside and with what directory
structure? (similar to test/CodeGen?)</div>
<div><br>
</div>
<div>Suggestion on what's the best approach for extending the
test-suite framework for this proposal are more than welcome.</div>
<div><br>
</div>
<div>This is just the high-level overview to start off the
discussion, I'm sure there are many more aspects to touch on.
Looking forward to your feedback!</div>
<div><br>
</div>
<div>
<div>Thanks,</div>
<div>Alina</div>
<div><br>
</div>
<div><span style="font-size:12.8px">[1] </span><a href="http://halide-lang.org/" style="font-size:12.8px" target="_blank"></a><a>http://</a><span>halide</span>-<a href="http://lang.org/">lang.org/</a></div>
<div>[2] Broken: r259800 => Fixed: r260131</div>
<div>[3] Broken: <span style="font-size:12.8px">r260569 => </span>Fixed: <span style="font-size:12.8px">r260701</span><br>
</div>
<div>[4] <a href="https://llvm.org/bugs/show_bug.cgi?id=26642" target="_blank">https://llvm.org/bugs/show_bug.cgi?id=26642</a></div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div></div><span class=""><pre>_______________________________________________
LLVM Developers mailing list
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a>
</pre>
</span></blockquote>
<br>
</div>
</blockquote></div><br></div></div>