[LLVMdev] [LNT] Question about results reliability in LNT infrustructure

[Renato Golin]

On 28 June 2013 19:45, Chris Matthews <chris.matthews at apple.com> wrote:

> Given this tradeoff I think we want to tend towards false positives (over
> false negatives) strictly as a matter of compiler quality.
>

False hits are not binary, but (at least) two-dimensional. You can't say
it's better to have any amount of false positives than any amount of false
negatives (pretty much like the NSA spying on *everybody* to avoid *any*
false negative). You can't also say that N false-positives is the same as N
false-negatives, because a false-hit can be huge in itself, or not.

What we have today is a huge amount of false positives and very few (or
none) false negatives. But even the real positives that we could spot even
with this amount of noise, we don't, because people don't normally look at
regressions. If I had to skim through the regressions on every build, I'd
do nothing else.

Given the proportion, I'd rather have a few small false positives and
reduce considerably the number of false positives with a hammer approach,
and only later try to nail down the options and do some fine tuning, than
doing the fine tuning now while still nobody cares about any result because
they're not trust-worthy.


That said, I’d never object to a professional’s opinion on this problem!
>

Absolutely! And David can help you a lot, there. But I wouldn't try to get
it perfect before we get it acceptable.

cheers,
--renato
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130628/3b936baa/attachment.html>