[PATCH] [LNT] Add geometric mean to run report

[Chris Matthews]

I wonder if we filter, if benchmarks that move around the filter point, so that sometimes they are included, and other times not, would cause problems in the geomean? 


On May 16, 2014, at 10:39 AM, Chris Matthews <chris.matthews at apple.com> wrote:

> I agree that it would be better to filter them in some way.
> 
> I really like the idea of trying this though.
> 
> On May 16, 2014, at 10:37 AM, Hal Finkel <hfinkel at anl.gov> wrote:
> 
>> ----- Original Message -----
>>> From: "Hal Finkel" <hfinkel at anl.gov>
>>> To: "Yi Kong" <kongy.dev at gmail.com>
>>> Cc: llvm-commits at cs.uiuc.edu, "Chris Matthews" <chris.matthews at apple.com>, "Anton Korobeynikov"
>>> <anton at korobeynikov.info>
>>> Sent: Friday, May 16, 2014 12:29:21 PM
>>> Subject: Re: [PATCH] [LNT] Add geometric mean to run report
>>> 
>>> ----- Original Message -----
>>>> From: "Yi Kong" <kongy.dev at gmail.com>
>>>> To: "Chris Matthews" <chris.matthews at apple.com>, "Anton
>>>> Korobeynikov" <anton at korobeynikov.info>
>>>> Cc: llvm-commits at cs.uiuc.edu
>>>> Sent: Friday, May 16, 2014 12:08:54 PM
>>>> Subject: [PATCH] [LNT] Add geometric mean to run report
>>>> 
>>>> Hi all,
>>>> 
>>>> This patch implements geometric mean in run report.
>>> 
>>> You mean that this is for all tests in the entire test suite
>>> aggregated into a single number?
>>> 
>>>> 
>>>> The idea of geomean is to represent the entire data set and not to
>>>> get
>>>> skewed by long running tests. However the major problem with it is,
>>>> zero time is not allowed. I used a workaround by adding 1 to each
>>>> value and subtract 1 from the final geomean result.  The workaround
>>>> certainly break the geomean number, but we don't really care the
>>>> number itself, rather the change of it. It's certainly not the
>>>> ideal.
>>>> Is there any better solution or statistics we can use?
>>> 
>>> I don't understand this at all. Is the geometric mean not (mul_i^n
>>> a_i)^(1/n)?
>> 
>> On the other hand, I suppose if you include any zeros, then the answer is always zero. Why don't you just exclude values less than 0.1s? Maybe this is no better than what you've proposed.
>> 
>> -Hal
>> 
>>> 
>>> Thanks again,
>>> Hal
>>> 
>>>> 
>>>> Thanks,
>>>> Yi Kong
>>>> 
>>>> _______________________________________________
>>>> llvm-commits mailing list
>>>> llvm-commits at cs.uiuc.edu
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>>>> 
>>> 
>>> --
>>> Hal Finkel
>>> Assistant Computational Scientist
>>> Leadership Computing Facility
>>> Argonne National Laboratory
>>> 
>> 
>> -- 
>> Hal Finkel
>> Assistant Computational Scientist
>> Leadership Computing Facility
>> Argonne National Laboratory
>