[LLVMdev] RFC:LNT Improvements

[Tobias Grosser]

On 30/04/2014 16:47, Yi Kong wrote:
> On 30/04/14 15:34, Tobias Grosser wrote:
>> On 30/04/2014 16:20, Yi Kong wrote:
>>> On 30 April 2014 07:50, Tobias Grosser <tobias at grosser.es> wrote:
>>>   >> - Show and graph total compile time
>>>   >>    There is no obvious way to scale up the compile time of
>>>   >> individual benchmarks, so total time is the best thing we can do to
>>>   >> minimize error.
>>>   >>    LNT: [PATCH 1/3] Add Total to run view and graph plot
>>>   >
>>>   > I did not see the effect of these changes in your images and also
>>>   > honestly do not fully understand what you are doing. What is the
>>>   > total compile time? Don't we already show the compile time in run
>>>   > view? How is the total time different to this compile time?
>>>
>>> It is hard to spot minor improvements or regressions over a large number
>>> of tests from independent machine noise. So I added a "total time"
>>> analysis to the run report and able to graph its trend, hoping that
>>> noise will cancel out and will help us to easily spot. (Screenshot
>>> attached)
>>
>> I understand the picture, but I still don't get how to compute "total
>> time". Is this a well known term?
>>
>> When looking at the plots of our existing -O3 testers, I also look for
>> some kind of less noisy line. The first thing coming to my mind would
>> just be the median of the set of run samples. Are you doing something
>> similar? Or are you computing a value across different runs?
>
> That's the total time taken to compile/execute. Put it in another way,
> sum of compile/execution time of all tests.

OK. I understand your intentions now.

I currently have little intuition if this works or not. It seems you 
also don't know if this works or not, do you?

My personal hope is that the reliability allows us to get rid of almost 
all noise such that most runs would just report no performance changes 
at all. If this is the case, the actual performance changes would stand 
out nicely and we could highlight them better in LNT.

If this does not work, some aggregated performance numbers as the ones 
you propose may be helpful. The total time is a reasonable first metric 
I suppose, but we may want to verify if statistics don't give us a 
better tool (Anton may be able to help).

Thanks again for your explanation,
Tobias