[LLVMdev] [cfe-dev] RFC: Change tests to run with fixed (not-host dependent) triple

[David Blaikie]

On Mon, Dec 3, 2012 at 11:51 AM, Daniel Dunbar <daniel at zuster.org> wrote:
>
>
>
> On Mon, Dec 3, 2012 at 10:56 AM, David Blaikie <dblaikie at gmail.com> wrote:
>>
>> On Mon, Dec 3, 2012 at 10:47 AM, Daniel Dunbar <daniel at zuster.org> wrote:
>> > On Sat, Dec 1, 2012 at 1:06 AM, Chandler Carruth <chandlerc at google.com>
>> > wrote:
>> >>
>> >> On Fri, Nov 30, 2012 at 8:04 PM, Chris Lattner <clattner at apple.com>
>> >> wrote:
>> >>>
>> >>> I'm ok with this in principle, but how about with the nuance that some
>> >>> tests (eg test/codegen) explicitly opt into march=native?
>> >>
>> >>
>> >> I'd really like the default behavior to be something that forces the
>> >> test
>> >> to either be independent of the targeted triple, or explicitly set a
>> >> target.
>> >> I like the default being unknown.
>> >>
>> >> I wonder, would the ability to run the entire test suite with all of
>> >> the
>> >> 'default' triples (that lit sets to unknown in normal runs) instead set
>> >> to
>> >> the host, or to a specific triple maybe be a useful extra form of
>> >> checking?
>> >> This would let both humans and build bots find bugs and discrepancies
>> >> specific to a particular target.
>> >
>> >
>> > I'd prefer not to do this universally (i.e., run the whole test suite
>> > that
>> > way),
>>
>> I'm not entirely sure which distinction you're drawing here when you
>> say "universally". I assume we're specifically talking about only
>> those tests in the lit-based regression suite that don't already have
>> an explicit triple. Only these tests would get the default ("unknown")
>> triple and be affected by some lit-parameter to override the default
>> with any other triple or possibly a list of triples.
>>
>> Icing would be to have that list be autodetected from the supported
>> targets in the current build.
>
>
> What I want is for this to not be "implicit" behavior. Many tests have no
> value being run with multiple triples.
>
> What I want is for the author of the test to explicitly declare what they
> are trying to do. If a test is useful on multiple triples, then they should
> write something that says so, and the test suite will take the extra time to
> do it.
>
> My opinion on tests is that they are significantly better when they are
> written with an explicit idea of what coverage they are trying to get (and
> hopefully a comment explaining that too).

I agree wholeheartedly, I perhaps got a little carried away trying to
express other (Chandler) people's ideas in this thread on the
assumption that my personal position wasn't necessarily
correct/strong.

I'll leave it to Chandler to champion his idea of multi-target tests
if he so desires.

>> This would not be the default (the default would just be to run them
>> once, with the "unknown" triple) but would be an easily accessible
>> target for developers and would be the target that any respectable
>> buildbot would use. (without enforcement by bots the feature would be
>> useless to users since the results would not be clean).
>>
>> > but what I do think would be useful is to add enough test suite
>> > support to be able to easily run the same test on multiple triples (or
>> > even
>> > all configured ones).
>> >
>> > My primary goal is to have the tests that individual developers be
>> > equivalent (independent of the target they are running on).
>>
>> Did you a word? ("have the tests that individual developers <write?
>> run?> be equivalent")
>
>
> Hah. Yes, "tests that individual developers write".
>
>  - Daniel
>
>>
>> - David
>>
>> >
>> >  - Daniel
>> >
>> >> We could even have a common test target that build bots use which runs
>> >> all
>> >> the tests both in the default, and in the host-triple mode so that we
>> >> force
>> >> people to converge on target independent tests or explicit triples.
>> >>
>> >>>
>> >>>
>> >>> -Chris
>> >>>
>> >>> On Nov 30, 2012, at 12:16 PM, Daniel Dunbar <daniel at zuster.org> wrote:
>> >>>
>> >>> > Hi all,
>> >>> >
>> >>> > We consistently see test failures arising because by default many of
>> >>> > our tests run in a mode where the tool (clang or llc) pick
>> >>> > host-dependent
>> >>> > behavior. This makes it easy for developers to write tests that pass
>> >>> > on
>> >>> > their system, but fail for other developers.
>> >>> >
>> >>> > There is some utility in this behavior, as it gives us (unintended)
>> >>> > testing coverage of some things, but overall I think it is a net
>> >>> > loss for
>> >>> > productivity.
>> >>> >
>> >>> > I propose:
>> >>> >
>> >>> > a. We change the test suite to run in such a way that all tools
>> >>> > default
>> >>> > to an "unknown" host triple.
>> >>> >
>> >>> > b. If someone feels there is missing coverage in some area, we add
>> >>> > increased tests for that area (which get run on all platforms).
>> >>> >
>> >>> > c. If there is some reason that running with an "unknown" host
>> >>> > triple
>> >>> > is undesirable, I propose that we set the default test triple to be
>> >>> > "x86_64-pc-linux-gnu", and require deviations to be specified.
>> >>> >
>> >>> > Comments?
>> >>> >
>> >>> >  - Daniel
>> >>> >
>> >>> > _______________________________________________
>> >>> > LLVM Developers mailing list
>> >>> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> >>> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>> >>> _______________________________________________
>> >>> cfe-dev mailing list
>> >>> cfe-dev at cs.uiuc.edu
>> >>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> >>
>> >>
>> >
>> >
>> > _______________________________________________
>> > cfe-dev mailing list
>> > cfe-dev at cs.uiuc.edu
>> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>> >
>
>