[LLVMdev] [LNT] Question about results reliability in LNT infrustructure

[Chris Matthews]

I should describe the cost of false negatives and false positives, since I think it matters for how this problem is approached.  False negatives mean we miss a real regression --- we don’t want that.  False positives mean somebody has to spend some time looking at and reproducing the regression when there is not one --- bad too.  Given this tradeoff I think we want to tend towards false positives (over false negatives) strictly as a matter of compiler quality, but if we can throw more data to reduce false positives that is good.  

I have discussed the classification problem before with people off list.  The problem that we face is that the space is pretty big for manual classification, at worse: number of benchmarks * number of architectures * sets of flags * metrics collected.  Perhaps some sensible defaults could overcome that, also to classify well, you probably need a lot of samples as a baseline.  

There certainly are lots of tests for small data. As far as I know though they rely more heavily on assumptions that in our case would have to be proven.  That said, I’d never object to a professional’s opinion on this problem!

Chris Matthews
chris.matthews@.com
(408) 783-6335

On Jun 28, 2013, at 6:28 AM, Renato Golin <renato.golin at linaro.org> wrote:

> On 28 June 2013 14:06, David Tweed <david.tweed at arm.com> wrote:
> That's a viewpoint; another one is that statisticians might well have very good reasons why they spend so long coming up with statistical tests in order to create the most powerful tests so they can deal with marginal quantities of data.
> 
> 
> 87.35% of all statistics are made up, 55.12% of them could have been done a lot simpler, a lot quicker and only 1.99% (AER) actually make your life better.
> 
> I'm glad that Chris already has working solutions, and I'b be happy to see them go live before any professional statistician had a look at it. ;)
> 
> cheers,
> --renato
> 
> 
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130628/62b600f8/attachment.html>