[llvm-dev] Statistics for Effectiveness of Passes on Reference Workloads

[Hideto Ueno via llvm-dev]

Hi all,

> Yes, I CC'd Hideto on the opening message, I should've CC'd all the
contributors. If the full data is available, that would be a great start!
And thanks for the paper! I didn't know about it.

What I did in the last year is to dump results to temporary files at
PasManager.h[0], and then look over all results.
I don't think it is a good way to accumulate the results so it is necessary
to consider another way if we want the functionality in the trunk.

[0]
https://github.com/llvm/llvm-project/blob/d881319cc5606baa7668405a296d0960a83a1e4c/llvm/include/llvm/IR/PassManager.h#L509

On Tue, May 25, 2021 at 6:12 AM Stefanos Baziotis <
stefanos.baziotis at gmail.com> wrote:

> Hi Johannes,
>
> Yes, I CC'd Hideto on the opening message, I should've CC'd all the
> contributors. If the full data is available, that would be a great start!
> And thanks for the paper! I didn't know about it.
>
> Not to say that we should not build some functionality upstream
>> to do this regularly :)
>
>
> I have a (bad) feeling that I'll need to do a lot of drudgework to get
> this data, so, I might as well help to set something up if other people are
> interested.
> The closest thing I know is a compile-time-measurement website (?) Nikita
> Popov (CC'd) had set up.
>
> Best,
> Stefanos
>
> Στις Τρί, 25 Μαΐ 2021 στις 12:01 π.μ., ο/η Johannes Doerfert <
> johannesdoerfert at gmail.com> έγραψε:
>
>> Hideto (cc'ed) did look into this last year and he has some data.
>> You can find a quick summary in his lighting talk last year [1] and
>> also read a little bit about it in our paper draft! (attached),
>> especially Section 2.2 and Figure 3 are interesting here.
>>
>> Not to say that we should not build some functionality upstream
>> to do this regularly :)
>>
>> ~ Johannes
>>
>> [1] https://www.youtube.com/watch?v=nxfew3hsMFM&t=1435s
>>
>>
>> On 5/24/21 3:51 PM, Stefanos Baziotis via llvm-dev wrote:
>> > Hi Min,
>> >
>> > Honestly, I don't think optimization remarks are good to check
>> _whether_ a
>> > pass did any work. A far better option, to the best of my knowledge,
>> is, as
>> > I mentioned, the --print-after-all family [1]
>> > Optimization remarks may help on the last question I posed and that is
>> to
>> > check _why_ a transformation was not applied. Although, if one reaches
>> this
>> > point, they will need to use way more
>> > tools than remarks.
>> >
>> >   And even for the Passes that use it, my impression was that not every
>> >> predicates are annotated with optimization remarks.
>> >
>> > Definitely not and a lot of remarks are not that descriptive either.
>> >
>> > Best,
>> > Stefanos
>> >
>> > [1] https://godbolt.org/z/58Ms7qW4s
>> >
>> > Στις Δευ, 24 Μαΐ 2021 στις 11:07 μ.μ., ο/η Min-Yih Hsu <minyihh at uci.edu
>> >
>> > έγραψε:
>> >
>> >> I think optimization remarks is a good framework for measuring whether
>> a
>> >> Pass does any work.
>> >> But unfortunately it is not widely adopted by Passes outside loop
>> >> transformations. And even for the Passes that use it, my impression was
>> >> that not every predicates are annotated with optimization remarks.
>> >>
>> >> -Min
>> >>
>> >> On May 24, 2021, at 12:55 PM, Stefanos Baziotis via llvm-dev <
>> >> llvm-dev at lists.llvm.org> wrote:
>> >>
>> >> Hi,
>> >>
>> >> Has anyone gathered statistics on reference workloads (*) for
>> >> (transformation) passes
>> >> that are enabled / run by default (e.g., -O3) but most of the
>> >> time _don't_ do any effective transformation?
>> >> Even better if we also have such statistics for passes that are _not_
>> >> enabled by default
>> >> (e.g., loop fusion, distribution, interchange, NewGVN).
>> >>
>> >> And yet even better if people have some idea / data for the reason for
>> >> ineffectiveness.
>> >> Bad heuristics / decision-making? Are some of these transformations
>> >> useless most of the time?
>> >> Or maybe they are useful but their implementation in LLVM is not
>> powerful
>> >> enough. Or maybe
>> >> they incur a significant compile-time overhead.
>> >>
>> >> If not, it would also be helpful if anyone who has tried gathering
>> similar
>> >> statistics has any
>> >> advice on how to approach it (my rough idea is initially
>> >> use --print-changed / --print-after-all on these workloads for -O3 and
>> >> then try to
>> >> slide in passes that are not enabled by default; although that's
>> harder to
>> >> do it right).
>> >>
>> >> @Hideto: In the last LLVM meeting you gave a related talk [1]. Do you
>> >> maybe have
>> >> the full statistics and / or ways to reproduce them?
>> >>
>> >> Best,
>> >> Stefanos
>> >>
>> >> (*)  SPEC, Polybench, Cryptographic libraries, Genome alignment, Image
>> >> Processing, Graph Processing, Web browsers, databases like sqlite etc.
>> >>
>> >> [1] https://www.youtube.com/watch?v=QvF68tOt_w8
>> >> _______________________________________________
>> >> LLVM Developers mailing list
>> >> llvm-dev at lists.llvm.org
>> >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> >>
>> >>
>> >>
>> >
>> > _______________________________________________
>> > LLVM Developers mailing list
>> > llvm-dev at lists.llvm.org
>> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210525/6b584770/attachment.html>