[llvm-dev] How LLVM guarantee the qualify of the product from the limited test suit?

[Renato Golin via llvm-dev]

On 9 November 2015 at 09:04, via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> After searching the whole project, I only find about ~10000 cases from
> "llvm/test" for each commit, and a separate testsuit wrote with high level
> language(i.e. C/C++) to verify the quality and performance. As a general
> Backend, you know, it must be strong enough to cope with all the IR
> generated by Frontend. I cannot believe what I see. Did I miss something ?

Hi,

You're missing all the Clang, compiler-rt, libc++ tests on their own
repositories. That'll add a few more tens of thousands of tests into
the bundle. But that's just the regression / base validation, the
release has a second stage which is less visible than, I believe, for
GCC. We also run the test-suite and benchmarks on trunk validation,
which is something I believe GCC doesn't do, so releases have a lot
less bugs than GCC to begin with. GCC focus on fixing all bugs at
release time, while LLVM focus on fixing them as they happen, which is
much easier and more stable for all developers.


> Further, I notice that, under llvm project repo, there is also a clang-tests
> that using gcc test
> suits(http://llvm.org/svn/llvm-project/clang-tests/trunk/gcc-4_2-testsuite/).
> Is that test suit used by llvm to guarantee the quality before the release ?

The GCC test-suite, AFAIK, has very poor quality on what's considered
a pass or a failure, and it's common to release GCC with thousands of
failures on those tests. Some people may run it, but I honestly don't
trust it myself, nor have the time to sift through every single test
to make sure my errors are compiler errors or test errors. You can't
assume that just because GCC runs *more* tests, that what they're
testing is more *thorough*. There are also lots of tests that have
erratic behaviour, which only adds noise to the process.

The release process also involves passing standard compiler benchmarks
from the part of the base testers, and higher level applications (like
Chromium) from the community. Different targets may get different
community interest, but most targets have an additional phase inside
companies like, ARM, Mips, Intel, Apple, Google, Sony, Qualcomm, etc.
They all have internal work-loads that represent a larger piece of
real world code that the test-suite can offer. Whenever those work
loads fail, we get bug reports. It's also good practice to add a
snippet to the test-suite or the regression tests in these cases.

As a separate quality control, there are a few efforts tracking
trunk/releases to build the Linux kernel, Debian, FreeBSD, Mandriva,
OpenEmbedded and other large scale projects. Whenever something breaks
on those projects, bugs are reported and fixed on the next stable
release possible.

I think it's a pretty solid validation process for both trunk and releases.

cheers,
--renato