[PATCH] [LNT] Add geometric mean to run report

[Hal Finkel]

----- Original Message -----
> From: "Chris Matthews" <chris.matthews at apple.com>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "Yi Kong" <kongy.dev at gmail.com>, llvm-commits at cs.uiuc.edu, "Anton Korobeynikov" <anton at korobeynikov.info>
> Sent: Friday, May 16, 2014 12:42:47 PM
> Subject: Re: [PATCH] [LNT] Add geometric mean to run report
> 
> I wonder if we filter, if benchmarks that move around the filter
> point, so that sometimes they are included, and other times not,
> would cause problems in the geomean?

I'm not sure that it would cause too much of a problem; on the other hand, Yi's proposal to add 1 to everything (and subtract it at the end) might be the most numerically-stable adjustment.

 -Hal

> 
> 
> On May 16, 2014, at 10:39 AM, Chris Matthews
> <chris.matthews at apple.com> wrote:
> 
> > I agree that it would be better to filter them in some way.
> > 
> > I really like the idea of trying this though.
> > 
> > On May 16, 2014, at 10:37 AM, Hal Finkel <hfinkel at anl.gov> wrote:
> > 
> >> ----- Original Message -----
> >>> From: "Hal Finkel" <hfinkel at anl.gov>
> >>> To: "Yi Kong" <kongy.dev at gmail.com>
> >>> Cc: llvm-commits at cs.uiuc.edu, "Chris Matthews"
> >>> <chris.matthews at apple.com>, "Anton Korobeynikov"
> >>> <anton at korobeynikov.info>
> >>> Sent: Friday, May 16, 2014 12:29:21 PM
> >>> Subject: Re: [PATCH] [LNT] Add geometric mean to run report
> >>> 
> >>> ----- Original Message -----
> >>>> From: "Yi Kong" <kongy.dev at gmail.com>
> >>>> To: "Chris Matthews" <chris.matthews at apple.com>, "Anton
> >>>> Korobeynikov" <anton at korobeynikov.info>
> >>>> Cc: llvm-commits at cs.uiuc.edu
> >>>> Sent: Friday, May 16, 2014 12:08:54 PM
> >>>> Subject: [PATCH] [LNT] Add geometric mean to run report
> >>>> 
> >>>> Hi all,
> >>>> 
> >>>> This patch implements geometric mean in run report.
> >>> 
> >>> You mean that this is for all tests in the entire test suite
> >>> aggregated into a single number?
> >>> 
> >>>> 
> >>>> The idea of geomean is to represent the entire data set and not
> >>>> to
> >>>> get
> >>>> skewed by long running tests. However the major problem with it
> >>>> is,
> >>>> zero time is not allowed. I used a workaround by adding 1 to
> >>>> each
> >>>> value and subtract 1 from the final geomean result.  The
> >>>> workaround
> >>>> certainly break the geomean number, but we don't really care the
> >>>> number itself, rather the change of it. It's certainly not the
> >>>> ideal.
> >>>> Is there any better solution or statistics we can use?
> >>> 
> >>> I don't understand this at all. Is the geometric mean not
> >>> (mul_i^n
> >>> a_i)^(1/n)?
> >> 
> >> On the other hand, I suppose if you include any zeros, then the
> >> answer is always zero. Why don't you just exclude values less
> >> than 0.1s? Maybe this is no better than what you've proposed.
> >> 
> >> -Hal
> >> 
> >>> 
> >>> Thanks again,
> >>> Hal
> >>> 
> >>>> 
> >>>> Thanks,
> >>>> Yi Kong
> >>>> 
> >>>> _______________________________________________
> >>>> llvm-commits mailing list
> >>>> llvm-commits at cs.uiuc.edu
> >>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
> >>>> 
> >>> 
> >>> --
> >>> Hal Finkel
> >>> Assistant Computational Scientist
> >>> Leadership Computing Facility
> >>> Argonne National Laboratory
> >>> 
> >> 
> >> --
> >> Hal Finkel
> >> Assistant Computational Scientist
> >> Leadership Computing Facility
> >> Argonne National Laboratory
> > 
> 
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory