[llvm-dev] How LLVM guarantee the qualify of the product from the limited test suit?

[Robinson, Paul via llvm-dev]

> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
> Renato Golin via llvm-dev
> Sent: Monday, November 09, 2015 1:45 AM
> To: bluedream_zqs at sina.com
> Cc: llvm-dev
> Subject: Re: [llvm-dev] How LLVM guarantee the qualify of the product from
> the limited test suit?
> 
> On 9 November 2015 at 09:04, via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> > After searching the whole project, I only find about ~10000 cases from
> > "llvm/test" for each commit, and a separate testsuit wrote with high
> level
> > language(i.e. C/C++) to verify the quality and performance. As a general
> > Backend, you know, it must be strong enough to cope with all the IR
> > generated by Frontend. I cannot believe what I see. Did I miss something
> ?
> 
> Hi,
> 
> You're missing all the Clang, compiler-rt, libc++ tests on their own
> repositories. That'll add a few more tens of thousands of tests into
> the bundle. But that's just the regression / base validation, the
> release has a second stage which is less visible than, I believe, for
> GCC. We also run the test-suite and benchmarks on trunk validation,
> which is something I believe GCC doesn't do, so releases have a lot
> less bugs than GCC to begin with. GCC focus on fixing all bugs at
> release time, while LLVM focus on fixing them as they happen, which is
> much easier and more stable for all developers.
> 
> 
> > Further, I notice that, under llvm project repo, there is also a clang-
> tests
> > that using gcc test
> > suits(http://llvm.org/svn/llvm-project/clang-tests/trunk/gcc-4_2-
> testsuite/).
> > Is that test suit used by llvm to guarantee the quality before the
> release ?
> 
> The GCC test-suite, AFAIK, has very poor quality on what's considered
> a pass or a failure, and it's common to release GCC with thousands of
> failures on those tests. Some people may run it, but I honestly don't
> trust it myself, nor have the time to sift through every single test
> to make sure my errors are compiler errors or test errors. You can't
> assume that just because GCC runs *more* tests, that what they're
> testing is more *thorough*. There are also lots of tests that have
> erratic behaviour, which only adds noise to the process.
> 
> The release process also involves passing standard compiler benchmarks
> from the part of the base testers, and higher level applications (like
> Chromium) from the community. Different targets may get different
> community interest, but most targets have an additional phase inside
> companies like, ARM, Mips, Intel, Apple, Google, Sony, Qualcomm, etc.
> They all have internal work-loads that represent a larger piece of
> real world code that the test-suite can offer. Whenever those work
> loads fail, we get bug reports. It's also good practice to add a
> snippet to the test-suite or the regression tests in these cases.

Hear, hear.  The llvm/test/... and clang/test/... suites are no more
than "smoke tests" in my opinion.  We do a LOT more than that internally.
--paulr

> 
> As a separate quality control, there are a few efforts tracking
> trunk/releases to build the Linux kernel, Debian, FreeBSD, Mandriva,
> OpenEmbedded and other large scale projects. Whenever something breaks
> on those projects, bugs are reported and fixed on the next stable
> release possible.
> 
> I think it's a pretty solid validation process for both trunk and
> releases.
> 
> cheers,
> --renato
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev