[LLVMdev] [cfe-dev] RFC: Change tests to run with fixed (not-host dependent) triple

[Daniel Dunbar]

On Mon, Dec 3, 2012 at 10:56 AM, David Blaikie <dblaikie at gmail.com> wrote:

> On Mon, Dec 3, 2012 at 10:47 AM, Daniel Dunbar <daniel at zuster.org> wrote:
> > On Sat, Dec 1, 2012 at 1:06 AM, Chandler Carruth <chandlerc at google.com>
> > wrote:
> >>
> >> On Fri, Nov 30, 2012 at 8:04 PM, Chris Lattner <clattner at apple.com>
> wrote:
> >>>
> >>> I'm ok with this in principle, but how about with the nuance that some
> >>> tests (eg test/codegen) explicitly opt into march=native?
> >>
> >>
> >> I'd really like the default behavior to be something that forces the
> test
> >> to either be independent of the targeted triple, or explicitly set a
> target.
> >> I like the default being unknown.
> >>
> >> I wonder, would the ability to run the entire test suite with all of the
> >> 'default' triples (that lit sets to unknown in normal runs) instead set
> to
> >> the host, or to a specific triple maybe be a useful extra form of
> checking?
> >> This would let both humans and build bots find bugs and discrepancies
> >> specific to a particular target.
> >
> >
> > I'd prefer not to do this universally (i.e., run the whole test suite
> that
> > way),
>
> I'm not entirely sure which distinction you're drawing here when you
> say "universally". I assume we're specifically talking about only
> those tests in the lit-based regression suite that don't already have
> an explicit triple. Only these tests would get the default ("unknown")
> triple and be affected by some lit-parameter to override the default
> with any other triple or possibly a list of triples.
>
> Icing would be to have that list be autodetected from the supported
> targets in the current build.
>

What I want is for this to not be "implicit" behavior. Many tests have no
value being run with multiple triples.

What I want is for the author of the test to explicitly declare what they
are trying to do. If a test is useful on multiple triples, then they should
write something that says so, and the test suite will take the extra time
to do it.

My opinion on tests is that they are significantly better when they are
written with an explicit idea of what coverage they are trying to get (and
hopefully a comment explaining that too).

This would not be the default (the default would just be to run them
> once, with the "unknown" triple) but would be an easily accessible
> target for developers and would be the target that any respectable
> buildbot would use. (without enforcement by bots the feature would be
> useless to users since the results would not be clean).
>
> > but what I do think would be useful is to add enough test suite
> > support to be able to easily run the same test on multiple triples (or
> even
> > all configured ones).
> >
> > My primary goal is to have the tests that individual developers be
> > equivalent (independent of the target they are running on).
>
> Did you a word? ("have the tests that individual developers <write?
> run?> be equivalent")
>

Hah. Yes, "tests that individual developers write".

 - Daniel


> - David
>
> >
> >  - Daniel
> >
> >> We could even have a common test target that build bots use which runs
> all
> >> the tests both in the default, and in the host-triple mode so that we
> force
> >> people to converge on target independent tests or explicit triples.
> >>
> >>>
> >>>
> >>> -Chris
> >>>
> >>> On Nov 30, 2012, at 12:16 PM, Daniel Dunbar <daniel at zuster.org> wrote:
> >>>
> >>> > Hi all,
> >>> >
> >>> > We consistently see test failures arising because by default many of
> >>> > our tests run in a mode where the tool (clang or llc) pick
> host-dependent
> >>> > behavior. This makes it easy for developers to write tests that pass
> on
> >>> > their system, but fail for other developers.
> >>> >
> >>> > There is some utility in this behavior, as it gives us (unintended)
> >>> > testing coverage of some things, but overall I think it is a net
> loss for
> >>> > productivity.
> >>> >
> >>> > I propose:
> >>> >
> >>> > a. We change the test suite to run in such a way that all tools
> default
> >>> > to an "unknown" host triple.
> >>> >
> >>> > b. If someone feels there is missing coverage in some area, we add
> >>> > increased tests for that area (which get run on all platforms).
> >>> >
> >>> > c. If there is some reason that running with an "unknown" host triple
> >>> > is undesirable, I propose that we set the default test triple to be
> >>> > "x86_64-pc-linux-gnu", and require deviations to be specified.
> >>> >
> >>> > Comments?
> >>> >
> >>> >  - Daniel
> >>> >
> >>> > _______________________________________________
> >>> > LLVM Developers mailing list
> >>> > LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> >>> > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
> >>> _______________________________________________
> >>> cfe-dev mailing list
> >>> cfe-dev at cs.uiuc.edu
> >>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >>
> >>
> >
> >
> > _______________________________________________
> > cfe-dev mailing list
> > cfe-dev at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20121203/65605da5/attachment.html>