[llvm-dev] RFC: Adding GCC C Torture Suite to External Test Suites

[Alex Bradbury via llvm-dev]

On Tue, 3 Sep 2019 at 18:37, Kristof Beyls <kristof.beyls at gmail.com> wrote:
> Op di 3 sep. 2019 om 18:36 schreef Finkel, Hal J. via llvm-dev <llvm-dev at lists.llvm.org>:
>>
>> On 9/3/19 7:19 AM, Sam Elliott wrote:
>> > There are 1500 tests total, and about 100 on the platform-agnostic blacklist. Alex and I do not think this is an onerous burden for maintenance, either as an external test suite or if the test suite is imported.
>> >
>> > In the long term, if we import the tests, we know we will have to do updates when the Embecosm work lands, and beyond that updates can be more sporadic. It’s not clear to me how much harder these updates will be than if the test suite remains external.
>> >
>> > We would welcome more views as to whether this suite should be imported or should be an external test suite.
>>
>>
>> I lean toward importing - I suspect we'll get better coverage on
>> buildbots, and just in general more people will end up using the tests,
>> than if it is external. I'm also curious what other people think.
>>
>>   -Hal
>
>
> I also thought that importing the tests will result in them being run far more regularly.
> I wonder in how far regressions happen in these tests after a backend has been brought up with it. I.e. once all these tests are made to pass, do they later still capture regressions from time to time, or do they pretty much always keep on passing after?
> If they do catch regressions later on, the value of running them more frequently (e.g. on buildbots) goes up quite a bit.
>
> Maybe the only reason I could think of to not import them is if they would take a long time to run - making buildbots slower. Is there any data on how long it takes to run these tests?

They're fast to build+run. Compiling and executing all ~1400
non-masked tests on an i9-9900k takes less than a minute (~40s):
* That's using a release build of Clang (will be slower for Debug+Asserts)
* Compiling tests with O2 targeting RISC-V
* Running tests using qemu-user

I typically run a whole bunch of ISA variant + opt level + ABI
variants. You'd obviously hope that regressions don't happen, but it's
a useful sanity check to complement the in-tree unit tests. It's
helped me catch things when reviewing patches from others and
developing my own.

I also tend to generate .s, check them into a git repo and use git
diff to check for unexpected changes in output.

My general feeling is that if a regression or unexpected code change
(e.g. code size regression) can be found in one of the torture suite
tests then it's great news (vs seeing tthe same problem in a larger
program) - you've already got a fairly small and easy to understand
input so it's usually pretty easy to minimise and get to the root of
the problem.

Best,

Alex