[PATCH] [LNT] Add geometric mean to run report

[Chris Matthews]

Duncan and I took a look at this, and have made some changes.

A few things we changed:

1. Current was being compared with current, not prev, so it was not going to find much :) 
2. We changed the offset from adding 1 then subtracting 1 to 0.00001, this should skew the new geomean less.
3. We introduced a safe min which maps min([]) to None.
4. Changed the use of filter(None, because that can remove 0’s from the list
5. Cleanups so we could reason a little better about what is going on.

Yi, what do you think of these changes?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: patch_commit_63a31cafebaa.patch
Type: application/octet-stream
Size: 9098 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20140529/d46003d6/attachment.obj>
-------------- next part --------------


On May 16, 2014, at 4:31 PM, Yi Kong <kongy.dev at gmail.com> wrote:

> I'm seeing a small lag(~1 second) on my local testing machine. However
> I expect much larger resource usage with the number of runs on the
> public server.
> 
> I'm holding the patch for now, as I'm not sure whether the workaround
> is good enough. It will not get swinged like arithmetic mean, but
> still biased towards longer tests.
> 
> Cheers,
> Yi Kong
> 
> On 16 May 2014 22:23, Chris Matthews <chris.matthews at apple.com> wrote:
>> Sometimes.  What time impact are you seeing?
>> 
>> 
>> On May 16, 2014, at 12:06 PM, Yi Kong <kongy.dev at gmail.com> wrote:
>> 
>>> Chris, does the LNT server have high work load? I'm testing graphing
>>> support for geomean, but it seems to be quite expensive for both
>>> processor and IO.
>>> 
>>> On 16 May 2014 18:50, Chris Matthews <chris.matthews at apple.com> wrote:
>>>> Patch LGTM.
>>>> 
>>>> On May 16, 2014, at 10:46 AM, Hal Finkel <hfinkel at anl.gov> wrote:
>>>> 
>>>> ----- Original Message -----
>>>> 
>>>> From: "Chris Matthews" <chris.matthews at apple.com>
>>>> To: "Hal Finkel" <hfinkel at anl.gov>
>>>> Cc: "Yi Kong" <kongy.dev at gmail.com>, llvm-commits at cs.uiuc.edu, "Anton
>>>> Korobeynikov" <anton at korobeynikov.info>
>>>> Sent: Friday, May 16, 2014 12:42:47 PM
>>>> Subject: Re: [PATCH] [LNT] Add geometric mean to run report
>>>> 
>>>> I wonder if we filter, if benchmarks that move around the filter
>>>> point, so that sometimes they are included, and other times not,
>>>> would cause problems in the geomean?
>>>> 
>>>> 
>>>> I'm not sure that it would cause too much of a problem; on the other hand,
>>>> Yi's proposal to add 1 to everything (and subtract it at the end) might be
>>>> the most numerically-stable adjustment.
>>>> 
>>>> -Hal
>>>> 
>>>> 
>>>> 
>>>> On May 16, 2014, at 10:39 AM, Chris Matthews
>>>> <chris.matthews at apple.com> wrote:
>>>> 
>>>> I agree that it would be better to filter them in some way.
>>>> 
>>>> I really like the idea of trying this though.
>>>> 
>>>> On May 16, 2014, at 10:37 AM, Hal Finkel <hfinkel at anl.gov> wrote:
>>>> 
>>>> ----- Original Message -----
>>>> 
>>>> From: "Hal Finkel" <hfinkel at anl.gov>
>>>> To: "Yi Kong" <kongy.dev at gmail.com>
>>>> Cc: llvm-commits at cs.uiuc.edu, "Chris Matthews"
>>>> <chris.matthews at apple.com>, "Anton Korobeynikov"
>>>> <anton at korobeynikov.info>
>>>> Sent: Friday, May 16, 2014 12:29:21 PM
>>>> Subject: Re: [PATCH] [LNT] Add geometric mean to run report
>>>> 
>>>> ----- Original Message -----
>>>> 
>>>> From: "Yi Kong" <kongy.dev at gmail.com>
>>>> To: "Chris Matthews" <chris.matthews at apple.com>, "Anton
>>>> Korobeynikov" <anton at korobeynikov.info>
>>>> Cc: llvm-commits at cs.uiuc.edu
>>>> Sent: Friday, May 16, 2014 12:08:54 PM
>>>> Subject: [PATCH] [LNT] Add geometric mean to run report
>>>> 
>>>> Hi all,
>>>> 
>>>> This patch implements geometric mean in run report.
>>>> 
>>>> 
>>>> You mean that this is for all tests in the entire test suite
>>>> aggregated into a single number?
>>>> 
>>>> 
>>>> The idea of geomean is to represent the entire data set and not
>>>> to
>>>> get
>>>> skewed by long running tests. However the major problem with it
>>>> is,
>>>> zero time is not allowed. I used a workaround by adding 1 to
>>>> each
>>>> value and subtract 1 from the final geomean result.  The
>>>> workaround
>>>> certainly break the geomean number, but we don't really care the
>>>> number itself, rather the change of it. It's certainly not the
>>>> ideal.
>>>> Is there any better solution or statistics we can use?
>>>> 
>>>> 
>>>> I don't understand this at all. Is the geometric mean not
>>>> (mul_i^n
>>>> a_i)^(1/n)?
>>>> 
>>>> 
>>>> On the other hand, I suppose if you include any zeros, then the
>>>> answer is always zero. Why don't you just exclude values less
>>>> than 0.1s? Maybe this is no better than what you've proposed.
>>>> 
>>>> -Hal
>>>> 
>>>> 
>>>> Thanks again,
>>>> Hal
>>>> 
>>>> 
>>>> Thanks,
>>>> Yi Kong
>>>> 
>>>> _______________________________________________
>>>> llvm-commits mailing list
>>>> llvm-commits at cs.uiuc.edu
>>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>>>> 
>>>> 
>>>> --
>>>> Hal Finkel
>>>> Assistant Computational Scientist
>>>> Leadership Computing Facility
>>>> Argonne National Laboratory
>>>> 
>>>> 
>>>> --
>>>> Hal Finkel
>>>> Assistant Computational Scientist
>>>> Leadership Computing Facility
>>>> Argonne National Laboratory
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Hal Finkel
>>>> Assistant Computational Scientist
>>>> Leadership Computing Facility
>>>> Argonne National Laboratory
>>>> 
>>>> 
>> 
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits