[llvm-dev] How LLVM guarantee the qualify of the product from the limited test suit?

[Daniel Berlin via llvm-dev]

>
>
>
> 1. GCC trunk is less stable than LLVM because the lack of general
> buildbots.
>

GCC has plenty of buildbots, it has no revert-on-breakage policy.


>  * Testing a new patch means comparing the test results (including
> breakages) against the previous commit, and check the differences.
> This is a poor definition of "pass", especially when the number of
> failures is large


This is an artifact of the lack of a revert-on-breakage policy.


> .
>  * On ARM and AArch64, the number of failures is around a couple of
> thousand (don't know the exact figure). AFAIK, these are not marked
> XFAIL in any way, but are known to be broken for one reason or
> another.
>

That sounds like a failure on the part of the ARM developers.

>
>  * Linaro monthly releases go out with those failures, and the fact
> that they keep on going means the FSF releases do, too.


I expect this would change if someone pushed.

Here is, for example, the failure list for i686-pc-linux-gnu for each 4.9
release:
https://gcc.gnu.org/gcc-4.9/buildstat.html


This is a huge
> cost on the release process, since it needs complicated diff programs
> and often incur in manual analysis.
>


You say all this as if it is a GCC testsuite issue.

It sounds completely like a process issue that hasn't been raised and dealt
with.

IE something that could easily happen to LLVM.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20151110/62c544fd/attachment.html>