[LLVMdev] [cfe-dev] RFC: Change tests to run with fixed (not-host dependent) triple

[Daniel Dunbar]

Hi David,

I think this is a valuable topic, but I also think it is somewhat off topic
from my original post.

What you are primarily focusing on here, is, I believe, how we right tests
that are more target independent (i.e. can focus more on "test the thing
being tested" and less on other stuff). That is a great topic, but can we
separate it out to a different discussion? I think it can be treated
orthogonally to my goals which is just to have the test suite behave
uniformly for all developers.

 - Daniel


On Sat, Dec 1, 2012 at 5:06 AM, David Tweed <david.tweed at gmail.com> wrote:

> My thoughts:
>
> On Sat, Dec 1, 2012 at 9:06 AM, Chandler Carruth <chandlerc at google.com>wrote:
>
>>  On Fri, Nov 30, 2012 at 8:04 PM, Chris Lattner <clattner at apple.com>wrote:
>>
>>> I'm ok with this in principle, but how about with the nuance that some
>>> tests (eg test/codegen) explicitly opt into march=native?
>>>
>>
>> I'd really like the default behavior to be something that forces the test
>> to either be independent of the targeted triple, or explicitly set a
>> target. I like the default being unknown.
>>
> To state the obvious, there's two different things that go on in tests:
> the specific thing being tested and things that aren't being tested but
> just need to be done to provide enough "context" for what's being tested. I
> was involved in a long slog trying to fix up a lot of the ARM regression
> test failues (using my work email address). Here's some "roughly right"
> statistics:
>
> There were probably about 25--30 bugs where the issue was behaviour that
> FileCheck regexps didn't account for (mostly due to ABI issues). There the
> prevailing opinion seemed to be keep simple FileCheck tests but that tests
> should be run with a specific triple; however that triple shouldn't always
> be x86_64 (because that's a bit special). There've been about 5-10 tests
> where the test was testing something that was architecture specific without
> secifying they needed it (eg, testing for specific x86_64 machine
> optimizations without doing that); again the upshot has been to have these
> requirements specified explicitly. There have been 5-10 JIT code tests
> where "support" code pasted from, eg, lli into a test hadn't been updated
> when the JIT core was changed. The one definite bug that was there was in
> devirtualisation (which probably lowered ok but failed the module
> verifier). This bug was definitely not visible due to the general set of
> ARM failures that were basically issues with the tests.
>
> So on the one hand, I'd love it if the tests were constructed in such a
> way that the "fuzz" of ABI differences didn't need to be considered. On the
> other hand, if the devirtualization test had been run only using an x86_64
> triple the issue wouldn't have come to light as quickly. That seems to me
> to be the crux of the problem: LLVM (and especially Clang) is only _mostly_
> target independent, and getting the smaller set of target dependent
> elements wrong breaks compilation just as much as a generic bug so finding
> these things as early as possible seems desirable.
>
>
>>   I wonder, would the ability to run the entire test suite with all of
>> the 'default' triples (that lit sets to unknown in normal runs) instead set
>> to the host, or to a specific triple maybe be a useful extra form of
>> checking? This would let both humans and build bots find bugs and
>> discrepancies specific to a particular target.
>>
> Dreaming here: I wonder if one could come up with some set of meta-regexps
> that describe the annoying stuff like ABI differences so that if the above
> were done, one could try t separate the test failures into "test regreps
> written implicitly assuming different system, failures look due to correct
> system dependent stuff being generated" and "test is failing on this system
> in a non-understood way". That sounds too tricky to be reliable, but I
> don't know...
>
>
>>   We could even have a common test target that build bots use which runs
>> all the tests both in the default, and in the host-triple mode so that we
>> force people to converge on target independent tests or explicit triples.
>>
> A reasonable idea, except I'll reiterate: it assumes there _is_ completely
> target independent behaviour in non-trivial test code. If it's the case
> that there's not really it might be a situation where biting the bullet and
> trying to put a wide ranging set of triples randomly throughout the test
> suite and hoping to catch stuff that way is the best idea.
>
>>    --
>>
> cheers, dave tweed__________________________
> high-performance computing and machine vision expert:
> david.tweed at gmail.com
> "while having code so boring anyone can maintain it, use Python." --
> attempted insult seen on slashdot
>
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20121203/57816e1f/attachment.html>