[cfe-dev] Positive tests in the clang test suite?

[David Chisnall via cfe-dev]

> On 18 Jan 2018, at 15:46, Joel Jacobson via cfe-dev <cfe-dev at lists.llvm.org> wrote:
> 
> Dear Clang hackers,
> 
> I'm trying to understand how Clang is tested,
> I've looked at lots of .c and .cpp files in ./test/,
> and I see a lot of "// expected-warning" and "// expected-error" annotations,
> but I don't see any positive tests checking for expected values?

These are the expected values.  You are looking at Sema checks, which check that the semantic analyser is correctly diagnosing errors in programs.  

> 
> Other compilers such as rustc uses macros such as assert_eq, e.g.:
> #[test]
> fn test_extend_specialization() {
>    let mut a = BinaryHeap::from(vec![-10, 1, 2, 3, 3]);
>    let b = BinaryHeap::from(vec![-20, 5, 43]);
> 
>    a.extend(b);
> 
>    assert_eq!(a.into_sorted_vec(), [-20, -10, 1, 2, 3, 3, 5, 43]);
> }

This looks like an executable test (i.e. one where the output of the compiler must be run).  The clang test suite does not contain these, all of the CodeGen tests check that the correct IR is generated, but it’s then up to LLVM to correctly generate code from this.  In LLVM’s test suite, you will find tests that check that the expected assembly is generated from IR and that the expected binary sequences are generated from assembly.

The advantage of a test suite of this structure is that it can run on any platform (there is no requirement, for example, that the platform that is currently running clang can run code using Windows or DWARF exceptions to test both of those).  The tests are all unit tests and so you can quickly (typically in under 30 seconds) run just the relevant part of the test suite while debugging a new feature.

The down side is that they are not integration tests and so don’t check that the code is correct, only that it is the same as last time.  The LLVM Nightly Test suite contains a number of executable tests, but these typically take much longer to run and so are run by the CI infrastructure on a variety of different platforms.

David