[LLVMdev] [cfe-dev] RFC: Change tests to run with fixed (not-host dependent) triple

[David Blaikie]

On Mon, Dec 3, 2012 at 10:47 AM, Daniel Dunbar <daniel at zuster.org> wrote:
> On Sat, Dec 1, 2012 at 1:06 AM, Chandler Carruth <chandlerc at google.com>
> wrote:
>>
>> On Fri, Nov 30, 2012 at 8:04 PM, Chris Lattner <clattner at apple.com> wrote:
>>>
>>> I'm ok with this in principle, but how about with the nuance that some
>>> tests (eg test/codegen) explicitly opt into march=native?
>>
>>
>> I'd really like the default behavior to be something that forces the test
>> to either be independent of the targeted triple, or explicitly set a target.
>> I like the default being unknown.
>>
>> I wonder, would the ability to run the entire test suite with all of the
>> 'default' triples (that lit sets to unknown in normal runs) instead set to
>> the host, or to a specific triple maybe be a useful extra form of checking?
>> This would let both humans and build bots find bugs and discrepancies
>> specific to a particular target.
>
>
> I'd prefer not to do this universally (i.e., run the whole test suite that
> way),

I'm not entirely sure which distinction you're drawing here when you
say "universally". I assume we're specifically talking about only
those tests in the lit-based regression suite that don't already have
an explicit triple. Only these tests would get the default ("unknown")
triple and be affected by some lit-parameter to override the default
with any other triple or possibly a list of triples.

Icing would be to have that list be autodetected from the supported
targets in the current build.

This would not be the default (the default would just be to run them
once, with the "unknown" triple) but would be an easily accessible
target for developers and would be the target that any respectable
buildbot would use. (without enforcement by bots the feature would be
useless to users since the results would not be clean).

> but what I do think would be useful is to add enough test suite
> support to be able to easily run the same test on multiple triples (or even
> all configured ones).
>
> My primary goal is to have the tests that individual developers be
> equivalent (independent of the target they are running on).

Did you a word? ("have the tests that individual developers <write?
run?> be equivalent")

- David

>
>  - Daniel
>
>> We could even have a common test target that build bots use which runs all
>> the tests both in the default, and in the host-triple mode so that we force
>> people to converge on target independent tests or explicit triples.
>>
>>>
>>>
>>> -Chris
>>>
>>> On Nov 30, 2012, at 12:16 PM, Daniel Dunbar <daniel at zuster.org> wrote:
>>>
>>> > Hi all,
>>> >
>>> > We consistently see test failures arising because by default many of
>>> > our tests run in a mode where the tool (clang or llc) pick host-dependent
>>> > behavior. This makes it easy for developers to write tests that pass on
>>> > their system, but fail for other developers.
>>> >
>>> > There is some utility in this behavior, as it gives us (unintended)
>>> > testing coverage of some things, but overall I think it is a net loss for
>>> > productivity.
>>> >
>>> > I propose:
>>> >
>>> > a. We change the test suite to run in such a way that all tools default
>>> > to an "unknown" host triple.
>>> >
>>> > b. If someone feels there is missing coverage in some area, we add
>>> > increased tests for that area (which get run on all platforms).
>>> >
>>> > c. If there is some reason that running with an "unknown" host triple
>>> > is undesirable, I propose that we set the default test triple to be
>>> > "x86_64-pc-linux-gnu", and require deviations to be specified.
>>> >
>>> > Comments?
>>> >
>>> >  - Daniel
>>> >
>>> > _______________________________________________
>>> > LLVM Developers mailing list
>>> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>>> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>> _______________________________________________
>>> cfe-dev mailing list
>>> cfe-dev at cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>
>>
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>