Hi David,<div><br></div><div>I think this is a valuable topic, but I also think it is somewhat off topic from my original post.</div><div><br></div><div>What you are primarily focusing on here, is, I believe, how we right tests that are more target independent (i.e. can focus more on "test the thing being tested" and less on other stuff). That is a great topic, but can we separate it out to a different discussion? I think it can be treated orthogonally to my goals which is just to have the test suite behave uniformly for all developers.</div>
<div><br></div><div> - Daniel</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Dec 1, 2012 at 5:06 AM, David Tweed <span dir="ltr"><<a href="mailto:david.tweed@gmail.com" target="_blank">david.tweed@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_quote">My thoughts:</div>
<div class="gmail_quote"> </div>
<div class="gmail_quote"><div class="im">On Sat, Dec 1, 2012 at 9:06 AM, Chandler Carruth <span dir="ltr"><<a href="mailto:chandlerc@google.com" target="_blank">chandlerc@google.com</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div style="FONT-FAMILY:arial,helvetica,sans-serif;FONT-SIZE:10pt">
<div dir="ltr">
<div>On Fri, Nov 30, 2012 at 8:04 PM, Chris Lattner <span dir="ltr"><<a href="mailto:clattner@apple.com" target="_blank">clattner@apple.com</a>></span> wrote:<br></div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">I'm ok with this in principle, but how about with the nuance that some tests (eg test/codegen) explicitly opt into march=native?<br>
</blockquote>
<div><br></div></div>
<div>I'd really like the default behavior to be something that forces the test to either be independent of the targeted triple, or explicitly set a target. I like the default being unknown.<br></div></div></div></div>
</div></blockquote>
</div><div>To state the obvious, there's two different things that go on in tests: the specific thing being tested and things that aren't being tested but just need to be done to provide enough "context" for what's being tested. I was involved in a long slog trying to fix up a lot of the ARM regression test failues (using my work email address). Here's some "roughly right" statistics:</div>
<div> </div>
<div>There were probably about 25--30 bugs where the issue was behaviour that FileCheck regexps didn't account for (mostly due to ABI issues). There the prevailing opinion seemed to be keep simple FileCheck tests but that tests should be run with a specific triple; however that triple shouldn't always be x86_64 (because that's a bit special). There've been about 5-10 tests where the test was testing something that was architecture specific without secifying they needed it (eg, testing for specific x86_64 machine optimizations without doing that); again the upshot has been to have these requirements specified explicitly. There have been 5-10 JIT code tests where "support" code pasted from, eg, lli into a test hadn't been updated when the JIT core was changed. The one definite bug that was there was in devirtualisation (which probably lowered ok but failed the module verifier). This bug was definitely not visible due to the general set of ARM failures that were basically issues with the tests.</div>
<div> </div>
<div>So on the one hand, I'd love it if the tests were constructed in such a way that the "fuzz" of ABI differences didn't need to be considered. On the other hand, if the devirtualization test had been run only using an x86_64 triple the issue wouldn't have come to light as quickly. That seems to me to be the crux of the problem: LLVM (and especially Clang) is only _mostly_ target independent, and getting the smaller set of target dependent elements wrong breaks compilation just as much as a generic bug so finding these things as early as possible seems desirable.</div>
<div class="im">
<div> </div>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div style="FONT-FAMILY:arial,helvetica,sans-serif;FONT-SIZE:10pt">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>I wonder, would the ability to run the entire test suite with all of the 'default' triples (that lit sets to unknown in normal runs) instead set to the host, or to a specific triple maybe be a useful extra form of checking? This would let both humans and build bots find bugs and discrepancies specific to a particular target.<br>
</div></div></div></div></div></blockquote>
</div><div>Dreaming here: I wonder if one could come up with some set of meta-regexps that describe the annoying stuff like ABI differences so that if the above were done, one could try t separate the test failures into "test regreps written implicitly assuming different system, failures look due to correct system dependent stuff being generated" and "test is failing on this system in a non-understood way". That sounds too tricky to be reliable, but I don't know...</div>
<div class="im">
<div> </div>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div style="FONT-FAMILY:arial,helvetica,sans-serif;FONT-SIZE:10pt">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>We could even have a common test target that build bots use which runs all the tests both in the default, and in the host-triple mode so that we force people to converge on target independent tests or explicit triples.</div>
</div></div></div></div></blockquote>
</div><div>A reasonable idea, except I'll reiterate: it assumes there _is_ completely target independent behaviour in non-trivial test code. If it's the case that there's not really it might be a situation where biting the bullet and trying to put a wide ranging set of triples randomly throughout the test suite and hoping to catch stuff that way is the best idea.</div>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div style="FONT-FAMILY:arial,helvetica,sans-serif;FONT-SIZE:10pt">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div>-- <br></div></div></div></div></div></div></blockquote></div>
<div>cheers, dave tweed__________________________</div>
<div>high-performance computing and machine vision expert: <a href="mailto:david.tweed@gmail.com" target="_blank">david.tweed@gmail.com</a></div>
<div>"while having code so boring anyone can maintain it, use Python." -- attempted insult seen on slashdot</div>
<div> </div><br>
<br>_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@cs.uiuc.edu">cfe-dev@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev</a><br>
<br></blockquote></div><br></div>