r254251 - ARM v8.1a adds Advanced SIMD instructions for Rounding Double Multiply

[Renato Golin via cfe-commits]

On 1 December 2015 at 18:29, David Blaikie <dblaikie at gmail.com> wrote:
> Are they things the test-suite couldn't (either technically or
> philosophically) cover, or only that it doesn't cover it at the moment, but
> could do so?

IMO, it's a philosophical issue. The test-suite is a whole-program
execution, and all that matters is the end result, the output of the
program, not the program itself.

Those tests are mostly large (> 3 lines of code) so, if we were trying
to objdump and match for instruction patterns, any optimisation
variance could destroy them, as well as you may find the pattern
elsewhere in the same object, not belonging to the function you want
(inlining, etc). All in all, all the problems with reducing C code
would be pertinent here, so a big hammer to kill small issues. On the
other hand, the Clang based asm/obj examples are all small, and
hand-crafted to suit the problem being tested.

We could, however, add a new directory in the test-suite for that kind
of tests, like we already have for regressions in C/C++. That would
dissociate the front-end from the back-end, and essentially any
compiler used would have to abide by the rules of the test (which may
not be true for some compilers).

I don't mind where they end up, really, but we need that kind of test
somewhere. Clang is an easy place because we know we have it built
when the Clang repo is checked out.


> That would sort of defeat the point of having the testing and projects
> separated though - it would tie the tests together and produce most of the
> undesirable outcomes of having single end-to-end tests.

Right, I agree, but since the relationship between Clang and LLVM is
non-trivial, and there are lots of changes that need to be done on
both sides, I can't see why we shouldn't have tests that span across
projects.

Clearly, the sanitizer tests need Clang and Compiler-RT. Some Libc++
tests need Compiler-RT, others need libunwind, on ARM I need Clang to
build it. There are a lot of dependencies that are not there by
accident, but by design.

The more you move tests away from the big targets (LLVM, Clang), the
less they're ran by people committing patches, and the harder will be
to pick up the failures. Having said that, a decent buildbot coverage
would account for most of those problems, but it would also put a
stress on slow/expensive/experimental hardware support. It's a
balance, I think.


> (it would mean that at least the pure (or at least non-Clang) LLVM developer
> would have a test to run where they would not if the test remained in Clang
> only)

That is a good point. Right now, RT/libc++ tests in that category are
controlled by enabling/disabling support via CMake flags. We may be
able to do the same with Clang (if LLVM is not built with it, disable
asm/obj tests). Moving them all to a regression package in the
test-suite would be another option, but one that would take a lot
longer...

cheers,
--renato