[llvm-dev] RFC: Adding GCC C Torture Suite to External Test Suites

[Finkel, Hal J. via llvm-dev]

On 9/3/19 7:19 AM, Sam Elliott wrote:
> There are 1500 tests total, and about 100 on the platform-agnostic blacklist. Alex and I do not think this is an onerous burden for maintenance, either as an external test suite or if the test suite is imported.
>
> In the long term, if we import the tests, we know we will have to do updates when the Embecosm work lands, and beyond that updates can be more sporadic. It’s not clear to me how much harder these updates will be than if the test suite remains external.
>
> We would welcome more views as to whether this suite should be imported or should be an external test suite.


I lean toward importing - I suspect we'll get better coverage on 
buildbots, and just in general more people will end up using the tests, 
than if it is external. I'm also curious what other people think.

  -Hal


>
> It would also be useful to understand the status of the CMake vs Makefile build system in the nightly test suite, to understand whether we need to support Makefiles, or whether Makefiles will be disappearing from the nightly tests soon (as they have already from the main project repo).
>
> Sam
>
>
>> On 30 Aug 2019, at 9:22PM, Finkel, Hal J. <hfinkel at anl.gov> wrote:
>>
>>
>> On 8/30/19 2:21 PM, Alex Bradbury wrote:
>>> On Fri, 30 Aug 2019 at 17:34, Finkel, Hal J. via llvm-dev
>>> <llvm-dev at lists.llvm.org> wrote:
>>>> On 8/30/19 10:18 AM, Sam Elliott via llvm-dev wrote:
>>>>> TL;DR: I am proposing to add the GCC C Torture suite [1], as an additional external source of tests for the “nightly” test suite. If you are willing to review the patch, it is here: https://reviews.llvm.org/D66887
>>>>>
>>>>> Background:
>>>>>
>>>>> While working on the RISC-V backend, we have found it useful to use additional test suites beyond the in-tree Clang and LLVM tests and the LLVM nightly tests, in order to ensure the compiler is correct. Internally at lowRISC, we have been running the GCC C Torture suite [1] using custom scripts in order to ensure the backend can handle these tests as well.
>>>>>
>>>>> The main advantage is that the torture suite provide a corpus of simple executable tests that can relatively easily be minimised to identify the sources of bugs or regressions. During the bringup of the RISC-V backend, the executable torture tests were the main litmus test for backend correctness.
>>>>>
>>>>> We are not the only backend to have found this test suite helpful, it is noted in the WebAssembly backend that they have found the GCC Torture suite helpful during their backend bring-up as well [2].
>>>>>
>>>>> The GCC C Torture Suite is composed of four sets of tests. The set we have found most useful is the “execute” tests, which are single-source-file test cases that should be able to be compiled, linked, and run without issues. The next most useful group are the “compile” tests, which should be able to be compiled without issues. There is more documentation about the test suites here, under “gcc.c-torture” [3].
>>>>>
>>>>> Implementation:
>>>>>
>>>>> The LLVM Nightly test suite has support for external test suites like SPEC, where there are licencing difficulties with the tests,
>>>> Given the issues with blacklisting tests, etc. I wonder whether it would
>>>> be more useful, and acceptable, to import the tests themselves. Are the
>>>> tests all, to the best of your knowledge, GPLv3?
>>> Yes, the tests are, by our understanding, GPLv3. Importing could be a
>>> reasonable option - but how do you see that interacting with the need
>>> for blacklisting tests? Are you imagining that we just wouldn't import
>>> the GCC-only tests into the repo?
>>
>> That was, indeed, what I had in mind, but...
>>
>>
>>>   Obviously there'd still be potential
>>> architecture-specific blacklisting, and hassle around tests that need
>>> custom options.
>>
>> In general, it seems like keeping any blacklists and special options in
>> sync with the tests will be easiest if the configurations and the tests
>> are together in the repository.
>>
>> I'm just curious what your thoughts are on which will be better in the
>> long term.
>>
>> Thanks again,
>>
>> Hal
>>
>>
>>> Best,
>>>
>>> Alex
>> -- 
>> Hal Finkel
>> Lead, Compiler Technology and Programming Languages
>> Leadership Computing Facility
>> Argonne National Laboratory
> --
> Sam Elliott
> Software Developer - LLVM
> lowRISC CIC
> selliott at lowrisc.org
> --
>
-- 
Hal Finkel
Lead, Compiler Technology and Programming Languages
Leadership Computing Facility
Argonne National Laboratory