[PATCH] [LNT] Add geometric mean to run report

[Yi Kong]

I'm seeing a small lag(~1 second) on my local testing machine. However
I expect much larger resource usage with the number of runs on the
public server.

I'm holding the patch for now, as I'm not sure whether the workaround
is good enough. It will not get swinged like arithmetic mean, but
still biased towards longer tests.

Cheers,
Yi Kong

On 16 May 2014 22:23, Chris Matthews <chris.matthews at apple.com> wrote:
> Sometimes.  What time impact are you seeing?
>
>
> On May 16, 2014, at 12:06 PM, Yi Kong <kongy.dev at gmail.com> wrote:
>
>> Chris, does the LNT server have high work load? I'm testing graphing
>> support for geomean, but it seems to be quite expensive for both
>> processor and IO.
>>
>> On 16 May 2014 18:50, Chris Matthews <chris.matthews at apple.com> wrote:
>>> Patch LGTM.
>>>
>>> On May 16, 2014, at 10:46 AM, Hal Finkel <hfinkel at anl.gov> wrote:
>>>
>>> ----- Original Message -----
>>>
>>> From: "Chris Matthews" <chris.matthews at apple.com>
>>> To: "Hal Finkel" <hfinkel at anl.gov>
>>> Cc: "Yi Kong" <kongy.dev at gmail.com>, llvm-commits at cs.uiuc.edu, "Anton
>>> Korobeynikov" <anton at korobeynikov.info>
>>> Sent: Friday, May 16, 2014 12:42:47 PM
>>> Subject: Re: [PATCH] [LNT] Add geometric mean to run report
>>>
>>> I wonder if we filter, if benchmarks that move around the filter
>>> point, so that sometimes they are included, and other times not,
>>> would cause problems in the geomean?
>>>
>>>
>>> I'm not sure that it would cause too much of a problem; on the other hand,
>>> Yi's proposal to add 1 to everything (and subtract it at the end) might be
>>> the most numerically-stable adjustment.
>>>
>>> -Hal
>>>
>>>
>>>
>>> On May 16, 2014, at 10:39 AM, Chris Matthews
>>> <chris.matthews at apple.com> wrote:
>>>
>>> I agree that it would be better to filter them in some way.
>>>
>>> I really like the idea of trying this though.
>>>
>>> On May 16, 2014, at 10:37 AM, Hal Finkel <hfinkel at anl.gov> wrote:
>>>
>>> ----- Original Message -----
>>>
>>> From: "Hal Finkel" <hfinkel at anl.gov>
>>> To: "Yi Kong" <kongy.dev at gmail.com>
>>> Cc: llvm-commits at cs.uiuc.edu, "Chris Matthews"
>>> <chris.matthews at apple.com>, "Anton Korobeynikov"
>>> <anton at korobeynikov.info>
>>> Sent: Friday, May 16, 2014 12:29:21 PM
>>> Subject: Re: [PATCH] [LNT] Add geometric mean to run report
>>>
>>> ----- Original Message -----
>>>
>>> From: "Yi Kong" <kongy.dev at gmail.com>
>>> To: "Chris Matthews" <chris.matthews at apple.com>, "Anton
>>> Korobeynikov" <anton at korobeynikov.info>
>>> Cc: llvm-commits at cs.uiuc.edu
>>> Sent: Friday, May 16, 2014 12:08:54 PM
>>> Subject: [PATCH] [LNT] Add geometric mean to run report
>>>
>>> Hi all,
>>>
>>> This patch implements geometric mean in run report.
>>>
>>>
>>> You mean that this is for all tests in the entire test suite
>>> aggregated into a single number?
>>>
>>>
>>> The idea of geomean is to represent the entire data set and not
>>> to
>>> get
>>> skewed by long running tests. However the major problem with it
>>> is,
>>> zero time is not allowed. I used a workaround by adding 1 to
>>> each
>>> value and subtract 1 from the final geomean result.  The
>>> workaround
>>> certainly break the geomean number, but we don't really care the
>>> number itself, rather the change of it. It's certainly not the
>>> ideal.
>>> Is there any better solution or statistics we can use?
>>>
>>>
>>> I don't understand this at all. Is the geometric mean not
>>> (mul_i^n
>>> a_i)^(1/n)?
>>>
>>>
>>> On the other hand, I suppose if you include any zeros, then the
>>> answer is always zero. Why don't you just exclude values less
>>> than 0.1s? Maybe this is no better than what you've proposed.
>>>
>>> -Hal
>>>
>>>
>>> Thanks again,
>>> Hal
>>>
>>>
>>> Thanks,
>>> Yi Kong
>>>
>>> _______________________________________________
>>> llvm-commits mailing list
>>> llvm-commits at cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>>>
>>>
>>> --
>>> Hal Finkel
>>> Assistant Computational Scientist
>>> Leadership Computing Facility
>>> Argonne National Laboratory
>>>
>>>
>>> --
>>> Hal Finkel
>>> Assistant Computational Scientist
>>> Leadership Computing Facility
>>> Argonne National Laboratory
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Hal Finkel
>>> Assistant Computational Scientist
>>> Leadership Computing Facility
>>> Argonne National Laboratory
>>>
>>>
>