[llvm-dev] RFC: Adding GCC C Torture Suite to External Test Suites

Kristof Beyls via llvm-dev llvm-dev at lists.llvm.org
Tue Sep 3 10:36:50 PDT 2019


Op di 3 sep. 2019 om 18:36 schreef Finkel, Hal J. via llvm-dev <
llvm-dev at lists.llvm.org>:

> On 9/3/19 7:19 AM, Sam Elliott wrote:
> > There are 1500 tests total, and about 100 on the platform-agnostic
> blacklist. Alex and I do not think this is an onerous burden for
> maintenance, either as an external test suite or if the test suite is
> imported.
> >
> > In the long term, if we import the tests, we know we will have to do
> updates when the Embecosm work lands, and beyond that updates can be more
> sporadic. It’s not clear to me how much harder these updates will be than
> if the test suite remains external.
> >
> > We would welcome more views as to whether this suite should be imported
> or should be an external test suite.
>
>
> I lean toward importing - I suspect we'll get better coverage on
> buildbots, and just in general more people will end up using the tests,
> than if it is external. I'm also curious what other people think.
>
>   -Hal
>

I also thought that importing the tests will result in them being run far
more regularly.
I wonder in how far regressions happen in these tests after a backend has
been brought up with it. I.e. once all these tests are made to pass, do
they later still capture regressions from time to time, or do they pretty
much always keep on passing after?
If they do catch regressions later on, the value of running them more
frequently (e.g. on buildbots) goes up quite a bit.

Maybe the only reason I could think of to not import them is if they would
take a long time to run - making buildbots slower. Is there any data on how
long it takes to run these tests?

Thanks,

Kristof
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190903/722aec9b/attachment.html>


More information about the llvm-dev mailing list