[LNT] r239138 - Fix report submission when there are failed tests.

[Chris Matthews]

Yeah, we would have to get them installed there.  I’m not a big fan of these external pip repos. Nothing but trouble, and I think they add very little value.

I was timing mean and median, on a list of 10 numbers.  Our implementation was ~30us (45 when I added the cast), numpy was ~230us.  I assume the overhead is when converting the list into a numpy array before use. 

Rule #1 of data analysis: always use stats routines made by someone who will lose a lot of money if they are wrong. Last thing you want is to be making conclusions based on output from an algorithm written by a long gone grad student who didn’t care about that corner case that you fell into.  Proof: we didn’t even implement mean correctly for all cases.

This alone make me think even if it is slower we should consider using them.


> On Jun 6, 2015, at 1:37 AM, Kristof Beyls <kristof.beyls at gmail.com> wrote:
> 
> I guess that the biggest problem in switching to use numpy and scipy is making sure all required packages are available at http://lab.llvm.org/packages/ <http://lab.llvm.org/packages/> so the buildbots won't break?
> Is it for specific functions that numpy was much slower than the current implementation?
> 
> 2015-06-05 17:54 GMT+01:00 Chris Matthews <chris.matthews at apple.com <mailto:chris.matthews at apple.com>>:
> Thanks for fixing!  With that assertion I was hoping to flush out all the places where we pass ints and None to the stats routines.  Given the way the routines are written, they produce subtly wrong results in those cases.   I sort of want to switch them to numpy and spicy, but in some informal benchmarking we were much faster (10x) than numpy.  Simplicity seems to win out there.
> 
> > On Jun 5, 2015, at 1:43 AM, James Molloy <james.molloy at arm.com <mailto:james.molloy at arm.com>> wrote:
> >
> > Author: jamesm
> > Date: Fri Jun  5 03:43:27 2015
> > New Revision: 239138
> >
> > URL: http://llvm.org/viewvc/llvm-project?rev=239138&view=rev <http://llvm.org/viewvc/llvm-project?rev=239138&view=rev>
> > Log:
> > Fix report submission when there are failed tests.
> >
> > When tests fail, we get a .exec.status field type. This has integer
> > type, so calling the stats.* functions will now assert and the
> > import will silently fail (because the assert is caught).
> >
> > Ensure we bypass the stats.* functions when we have integer data.
> >
> > Modified:
> >    lnt/trunk/lnt/server/reporting/analysis.py
> >
> > Modified: lnt/trunk/lnt/server/reporting/analysis.py
> > URL: http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/server/reporting/analysis.py?rev=239138&r1=239137&r2=239138&view=diff <http://llvm.org/viewvc/llvm-project/lnt/trunk/lnt/server/reporting/analysis.py?rev=239138&r1=239137&r2=239138&view=diff>
> > ==============================================================================
> > --- lnt/trunk/lnt/server/reporting/analysis.py (original)
> > +++ lnt/trunk/lnt/server/reporting/analysis.py Fri Jun  5 03:43:27 2015
> > @@ -77,7 +77,11 @@ class ComparisonResult:
> >
> >         # If we have multiple values for this run, use that to estimate the
> >         # distribution.
> > -        if samples and len(samples) > 1:
> > +        #
> > +        # We can get integer sample types here - for example if the field is
> > +        # .exec.status. Make sure we don't assert by avoiding the stats
> > +        # functions in this case.
> > +        if samples and len(samples) > 1 and isinstance(samples[0], float):
> >             self.stddev = stats.standard_deviation(samples)
> >             self.MAD = stats.median_absolute_deviation(samples)
> >         else:
> >
> >
> > _______________________________________________
> > llvm-commits mailing list
> > llvm-commits at cs.uiuc.edu <mailto:llvm-commits at cs.uiuc.edu>
> > http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits <http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits>
> 
> 
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu <mailto:llvm-commits at cs.uiuc.edu>
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits <http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits>
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20150609/e144622a/attachment.html>