[cfe-dev] [LLVMdev] [3.6 Release] RC3 has been tagged

[Mehmet Erol Sanliturk]

On Tue, Feb 17, 2015 at 4:23 PM, Hans Wennborg <hans at chromium.org> wrote:

> On Tue, Feb 17, 2015 at 11:47 AM, Jack Howarth
> <howarth.mailing.lists at gmail.com> wrote:
> >     We probably ought to open a meta-bug for all of these new
> > performance regressions introduced in 3.6.
>
> Feel free to open one. The important thing is whether anyone is
> investigating the bugs, though.
>
> > The 20% performance
> > regression observed in the "8 queens puzzle" solver example
> > (http://llvm.org/bugs/show_bug.cgi?id=22058) doesn't seem to be
> > introduced by either the r217953 or r222451 commits. I'll try to do a
> > regression hunt later this week to pin down the offending commit for
> > that regression. Likewise, the reported regression in the -m32
> > performance of the sudoku solver example (reported in
> > http://llvm.org/bugs/show_bug.cgi?id=22589) doesn't match up to those
> > two commits either (but I suspect a register pressure issue since it
> > only shows up at 32-bit).
> >      Jack
> > ps Do we have a complete set of benchmark results for current 3.6svn
> > vs 3.5.1 posted anywhere?
>
> Not that I know. There are test results going into the LNT tool at
> http://llvm.org/perf, but I'm not really familiar with that and also
> don't think it makes it easy to do comparisons between releases.
>
>  - Hans
>



Comparison of Clang Test Results :



Assume Clang compiled some programs , the following data is obtained :


Assume  compilation_time for the Program i is obtained on the SAME computer
with similar execution environments .

Computers for Program i and Program j may be different .




                 Release 3.5.0         Release 3.6.0
                 ----------------      ---------------
Program 1       compilation_time      compilation_time

 .
 .
 .

Program n       compilation_time      compilation_time


Means            Mean_of_Release_3.5.0    Mean_of_Release_3.6.0


You may apply the following tests :


http://en.wikipedia.org/wiki/Levene%27s_test
( If W is significant : Variability of compilation speeds are different
between releases )

http://en.wikipedia.org/wiki/Paired_difference_test
( If D is significant : Average compilation time increased ( or decreased )


A similar analysis may be performed on "execution times" of programs :


                 Release 3.5.0         Release 3.6.0
                 ----------------      ---------------
Program 1       execution_time      execution_time

 .
 .
 .

Program n       execution_time      execution_time


Means            Mean_of_Release_3.5.0    Mean_of_Release_3.6.0


If you use more than 15 programs , test results will be "good" .
If you use more than 60 programs , test results will be "better" .


The tests will give at least a rough idea about performance changes .


Thank you very much .

Mehmet Erol Sanliturk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20150217/065ccd29/attachment.html>