[llvm-dev] RFC: Adding GCC C Torture Suite to External Test Suites

[Sam Elliott via llvm-dev]

TL;DR: I am proposing to add the GCC C Torture suite [1], as an additional external source of tests for the “nightly” test suite. If you are willing to review the patch, it is here: https://reviews.llvm.org/D66887

Background:

While working on the RISC-V backend, we have found it useful to use additional test suites beyond the in-tree Clang and LLVM tests and the LLVM nightly tests, in order to ensure the compiler is correct. Internally at lowRISC, we have been running the GCC C Torture suite [1] using custom scripts in order to ensure the backend can handle these tests as well. 

The main advantage is that the torture suite provide a corpus of simple executable tests that can relatively easily be minimised to identify the sources of bugs or regressions. During the bringup of the RISC-V backend, the executable torture tests were the main litmus test for backend correctness.

We are not the only backend to have found this test suite helpful, it is noted in the WebAssembly backend that they have found the GCC Torture suite helpful during their backend bring-up as well [2].

The GCC C Torture Suite is composed of four sets of tests. The set we have found most useful is the “execute” tests, which are single-source-file test cases that should be able to be compiled, linked, and run without issues. The next most useful group are the “compile” tests, which should be able to be compiled without issues. There is more documentation about the test suites here, under “gcc.c-torture” [3].

Implementation:

The LLVM Nightly test suite has support for external test suites like SPEC, where there are licencing difficulties with the tests, but where the llvm project would still like to be able to run these tests as part of our general compiler testing and benchmarking. 

I have prepared a patch to llvm-test-suite [0], which adds support for the “execute” tests in the GCC C Torture Suite. At the moment, this patch only covers the CMake build system, which I am more familiar with.

Limitations:

The main limitation at the moment (beyond lack of Makefile support) is that the GCC torture suite has a lot of tests that rely on internal features of GCC, or GNU extensions that clang has not implemented (like nested functions). At the moment the patch includes a long blacklist of test cases that do not run with clang. This is sufficient for the moment, but could of course be improved - the DejaGnu comments in the test cases allow them to add target-specific C Flags, and to skip tests that rely on internal features. 

A promising direction on this point is the work by some Embecosm developers on a version of the GCC Test Suite that is explicit about which test cases need GCC internal features [4]. This has included both modernising tests, and adding better DejaGnu filters and options. 

In the future, it would be nice to be able to generate the list of compiler-compatible target-specific test cases (and their cflags) from the DejaGnu information, and use this information within the CMake test suite. Most optimal would be the ability to generate this info, and save the generated info into the test suite so that test runners don’t need to run an additional tool during configuration.

Summary:

I am proposing to add the GCC C Torture suite as an additional external test suite. This will allow us to run these extra test cases regularly and within existing LLVM test suite infrastructure. I welcome volunteers to review [0], and input from anyone else who has been using the torture suite in compiler development or has relevant out-of-tree work.

[0]: https://reviews.llvm.org/D66887 
[1]: https://gcc.gnu.org/viewcvs/gcc/trunk/gcc/testsuite/gcc.c-torture/
[2]: https://github.com/llvm/llvm-project/blob/master/llvm/lib/Target/WebAssembly/README.txt 
[3]: https://gcc.gnu.org/onlinedocs/gccint/C-Tests.html 
[4]: https://github.com/embecosm/gcc-for-llvm-testing 

--
Sam Elliott
Software Developer - LLVM
lowRISC CIC
selliott at lowrisc.org
--