[llvm-dev] How LLVM guarantee the qualify of the product from the limited test suit?

[Renato Golin via llvm-dev]

On 10 November 2015 at 15:47, Daniel Berlin <dberlin at dberlin.org> wrote:
> GCC has plenty of buildbots, it has no revert-on-breakage policy.

I stand corrected.


> That sounds like a failure on the part of the ARM developers.

Or in my knowledge. :)


> I expect this would change if someone pushed.

Probably.


> Here is, for example, the failure list for i686-pc-linux-gnu for each 4.9
> release:
> https://gcc.gnu.org/gcc-4.9/buildstat.html

These sound a lot more thorough that we're doing at the moment. I'm
not surprised with that number of failures, but I (personally)
wouldn't care much if a sequence of arbitrary passes (ot in -ON)
failed. I'd easily mark it as XFAIL, but as I said, that's personal.


> You say all this as if it is a GCC testsuite issue.

Didn't mean to.


> It sounds completely like a process issue that hasn't been raised and dealt
> with. IE something that could easily happen to LLVM.

Absolutely!

My point is that we already have our complexity, just like GCC.
Merging the two complex validation systems *may* bring more harm than
good, that's all.

It would be a worthy project if someone was willing to take on the
hard work, but I don't know many people / companies where this would
justify, especially because I don't know how much of their suite is
redundant with ours, or within itself, to warrant an extra run.

Hope that's more clear... :)

cheers,
-renato