[llvm-dev] Statistics for Effectiveness of Passes on Reference Workloads

[Stefanos Baziotis via llvm-dev]

Hi Johannes,

Yes, I CC'd Hideto on the opening message, I should've CC'd all the
contributors. If the full data is available, that would be a great start!
And thanks for the paper! I didn't know about it.

Not to say that we should not build some functionality upstream
> to do this regularly :)


I have a (bad) feeling that I'll need to do a lot of drudgework to get this
data, so, I might as well help to set something up if other people are
interested.
The closest thing I know is a compile-time-measurement website (?) Nikita
Popov (CC'd) had set up.

Best,
Stefanos

Στις Τρί, 25 Μαΐ 2021 στις 12:01 π.μ., ο/η Johannes Doerfert <
johannesdoerfert at gmail.com> έγραψε:

> Hideto (cc'ed) did look into this last year and he has some data.
> You can find a quick summary in his lighting talk last year [1] and
> also read a little bit about it in our paper draft! (attached),
> especially Section 2.2 and Figure 3 are interesting here.
>
> Not to say that we should not build some functionality upstream
> to do this regularly :)
>
> ~ Johannes
>
> [1] https://www.youtube.com/watch?v=nxfew3hsMFM&t=1435s
>
>
> On 5/24/21 3:51 PM, Stefanos Baziotis via llvm-dev wrote:
> > Hi Min,
> >
> > Honestly, I don't think optimization remarks are good to check _whether_
> a
> > pass did any work. A far better option, to the best of my knowledge, is,
> as
> > I mentioned, the --print-after-all family [1]
> > Optimization remarks may help on the last question I posed and that is to
> > check _why_ a transformation was not applied. Although, if one reaches
> this
> > point, they will need to use way more
> > tools than remarks.
> >
> >   And even for the Passes that use it, my impression was that not every
> >> predicates are annotated with optimization remarks.
> >
> > Definitely not and a lot of remarks are not that descriptive either.
> >
> > Best,
> > Stefanos
> >
> > [1] https://godbolt.org/z/58Ms7qW4s
> >
> > Στις Δευ, 24 Μαΐ 2021 στις 11:07 μ.μ., ο/η Min-Yih Hsu <minyihh at uci.edu>
> > έγραψε:
> >
> >> I think optimization remarks is a good framework for measuring whether a
> >> Pass does any work.
> >> But unfortunately it is not widely adopted by Passes outside loop
> >> transformations. And even for the Passes that use it, my impression was
> >> that not every predicates are annotated with optimization remarks.
> >>
> >> -Min
> >>
> >> On May 24, 2021, at 12:55 PM, Stefanos Baziotis via llvm-dev <
> >> llvm-dev at lists.llvm.org> wrote:
> >>
> >> Hi,
> >>
> >> Has anyone gathered statistics on reference workloads (*) for
> >> (transformation) passes
> >> that are enabled / run by default (e.g., -O3) but most of the
> >> time _don't_ do any effective transformation?
> >> Even better if we also have such statistics for passes that are _not_
> >> enabled by default
> >> (e.g., loop fusion, distribution, interchange, NewGVN).
> >>
> >> And yet even better if people have some idea / data for the reason for
> >> ineffectiveness.
> >> Bad heuristics / decision-making? Are some of these transformations
> >> useless most of the time?
> >> Or maybe they are useful but their implementation in LLVM is not
> powerful
> >> enough. Or maybe
> >> they incur a significant compile-time overhead.
> >>
> >> If not, it would also be helpful if anyone who has tried gathering
> similar
> >> statistics has any
> >> advice on how to approach it (my rough idea is initially
> >> use --print-changed / --print-after-all on these workloads for -O3 and
> >> then try to
> >> slide in passes that are not enabled by default; although that's harder
> to
> >> do it right).
> >>
> >> @Hideto: In the last LLVM meeting you gave a related talk [1]. Do you
> >> maybe have
> >> the full statistics and / or ways to reproduce them?
> >>
> >> Best,
> >> Stefanos
> >>
> >> (*)  SPEC, Polybench, Cryptographic libraries, Genome alignment, Image
> >> Processing, Graph Processing, Web browsers, databases like sqlite etc.
> >>
> >> [1] https://www.youtube.com/watch?v=QvF68tOt_w8
> >> _______________________________________________
> >> LLVM Developers mailing list
> >> llvm-dev at lists.llvm.org
> >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >>
> >>
> >>
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210525/17547abf/attachment.html>