[llvm-dev] RFC: Adding GCC C Torture Suite to External Test Suites

[Finkel, Hal J. via llvm-dev]

On 8/30/19 2:21 PM, Alex Bradbury wrote:
> On Fri, 30 Aug 2019 at 17:34, Finkel, Hal J. via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>>
>> On 8/30/19 10:18 AM, Sam Elliott via llvm-dev wrote:
>>> TL;DR: I am proposing to add the GCC C Torture suite [1], as an additional external source of tests for the “nightly” test suite. If you are willing to review the patch, it is here: https://reviews.llvm.org/D66887
>>>
>>> Background:
>>>
>>> While working on the RISC-V backend, we have found it useful to use additional test suites beyond the in-tree Clang and LLVM tests and the LLVM nightly tests, in order to ensure the compiler is correct. Internally at lowRISC, we have been running the GCC C Torture suite [1] using custom scripts in order to ensure the backend can handle these tests as well.
>>>
>>> The main advantage is that the torture suite provide a corpus of simple executable tests that can relatively easily be minimised to identify the sources of bugs or regressions. During the bringup of the RISC-V backend, the executable torture tests were the main litmus test for backend correctness.
>>>
>>> We are not the only backend to have found this test suite helpful, it is noted in the WebAssembly backend that they have found the GCC Torture suite helpful during their backend bring-up as well [2].
>>>
>>> The GCC C Torture Suite is composed of four sets of tests. The set we have found most useful is the “execute” tests, which are single-source-file test cases that should be able to be compiled, linked, and run without issues. The next most useful group are the “compile” tests, which should be able to be compiled without issues. There is more documentation about the test suites here, under “gcc.c-torture” [3].
>>>
>>> Implementation:
>>>
>>> The LLVM Nightly test suite has support for external test suites like SPEC, where there are licencing difficulties with the tests,
>>
>> Given the issues with blacklisting tests, etc. I wonder whether it would
>> be more useful, and acceptable, to import the tests themselves. Are the
>> tests all, to the best of your knowledge, GPLv3?
> Yes, the tests are, by our understanding, GPLv3. Importing could be a
> reasonable option - but how do you see that interacting with the need
> for blacklisting tests? Are you imagining that we just wouldn't import
> the GCC-only tests into the repo?


That was, indeed, what I had in mind, but...


>   Obviously there'd still be potential
> architecture-specific blacklisting, and hassle around tests that need
> custom options.


In general, it seems like keeping any blacklists and special options in 
sync with the tests will be easiest if the configurations and the tests 
are together in the repository.

I'm just curious what your thoughts are on which will be better in the 
long term.

Thanks again,

Hal


>
> Best,
>
> Alex

-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory