[llvm-dev] Statistics for Effectiveness of Passes on Reference Workloads

Tue May 25 05:42:53 PDT 2021

Hi Hideto,

Thanks, it makes sense. It's similar to --print-after-all / --print-changed.

Best,
Stefanos

Στις Τρί, 25 Μαΐ 2021 στις 4:45 π.μ., ο/η Hideto Ueno <
uenoku.tokotoko at gmail.com> έγραψε:

> Hi all,
>
> > Yes, I CC'd Hideto on the opening message, I should've CC'd all the
> contributors. If the full data is available, that would be a great start!
> And thanks for the paper! I didn't know about it.
>
> What I did in the last year is to dump results to temporary files at
> PasManager.h[0], and then look over all results.
> I don't think it is a good way to accumulate the results so it is
> necessary to consider another way if we want the functionality in the trunk.
>
> [0]
> https://github.com/llvm/llvm-project/blob/d881319cc5606baa7668405a296d0960a83a1e4c/llvm/include/llvm/IR/PassManager.h#L509
>
> On Tue, May 25, 2021 at 6:12 AM Stefanos Baziotis <
> stefanos.baziotis at gmail.com> wrote:
>
>> Hi Johannes,
>>
>> Yes, I CC'd Hideto on the opening message, I should've CC'd all the
>> contributors. If the full data is available, that would be a great start!
>> And thanks for the paper! I didn't know about it.
>>
>> Not to say that we should not build some functionality upstream
>>> to do this regularly :)
>>
>>
>> I have a (bad) feeling that I'll need to do a lot of drudgework to get
>> this data, so, I might as well help to set something up if other people are
>> interested.
>> The closest thing I know is a compile-time-measurement website (?) Nikita
>> Popov (CC'd) had set up.
>>
>> Best,
>> Stefanos
>>
>> Στις Τρί, 25 Μαΐ 2021 στις 12:01 π.μ., ο/η Johannes Doerfert <
>> johannesdoerfert at gmail.com> έγραψε:
>>
>>> Hideto (cc'ed) did look into this last year and he has some data.
>>> You can find a quick summary in his lighting talk last year [1] and
>>> also read a little bit about it in our paper draft! (attached),
>>> especially Section 2.2 and Figure 3 are interesting here.
>>>
>>> Not to say that we should not build some functionality upstream
>>> to do this regularly :)
>>>
>>> ~ Johannes
>>>
>>> [1] https://www.youtube.com/watch?v=nxfew3hsMFM&t=1435s
>>>
>>>
>>> On 5/24/21 3:51 PM, Stefanos Baziotis via llvm-dev wrote:
>>> > Hi Min,
>>> >
>>> > Honestly, I don't think optimization remarks are good to check
>>> _whether_ a
>>> > pass did any work. A far better option, to the best of my knowledge,
>>> is, as
>>> > I mentioned, the --print-after-all family [1]
>>> > Optimization remarks may help on the last question I posed and that is
>>> to
>>> > check _why_ a transformation was not applied. Although, if one reaches
>>> this
>>> > point, they will need to use way more
>>> > tools than remarks.
>>> >
>>> >   And even for the Passes that use it, my impression was that not every
>>> >> predicates are annotated with optimization remarks.
>>> >
>>> > Definitely not and a lot of remarks are not that descriptive either.
>>> >
>>> > Best,
>>> > Stefanos
>>> >
>>> > [1] https://godbolt.org/z/58Ms7qW4s
>>> >
>>> > Στις Δευ, 24 Μαΐ 2021 στις 11:07 μ.μ., ο/η Min-Yih Hsu <
>>> minyihh at uci.edu>
>>> > έγραψε:
>>> >
>>> >> I think optimization remarks is a good framework for measuring
>>> whether a
>>> >> Pass does any work.
>>> >> But unfortunately it is not widely adopted by Passes outside loop
>>> >> transformations. And even for the Passes that use it, my impression
>>> was
>>> >> that not every predicates are annotated with optimization remarks.
>>> >>
>>> >> -Min
>>> >>
>>> >> On May 24, 2021, at 12:55 PM, Stefanos Baziotis via llvm-dev <
>>> >> llvm-dev at lists.llvm.org> wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> Has anyone gathered statistics on reference workloads (*) for
>>> >> (transformation) passes
>>> >> that are enabled / run by default (e.g., -O3) but most of the
>>> >> time _don't_ do any effective transformation?
>>> >> Even better if we also have such statistics for passes that are _not_
>>> >> enabled by default
>>> >> (e.g., loop fusion, distribution, interchange, NewGVN).
>>> >>
>>> >> And yet even better if people have some idea / data for the reason for
>>> >> ineffectiveness.
>>> >> Bad heuristics / decision-making? Are some of these transformations
>>> >> useless most of the time?
>>> >> Or maybe they are useful but their implementation in LLVM is not
>>> powerful
>>> >> enough. Or maybe
>>> >> they incur a significant compile-time overhead.
>>> >>
>>> >> If not, it would also be helpful if anyone who has tried gathering
>>> similar
>>> >> statistics has any
>>> >> advice on how to approach it (my rough idea is initially
>>> >> use --print-changed / --print-after-all on these workloads for -O3 and
>>> >> then try to
>>> >> slide in passes that are not enabled by default; although that's
>>> harder to
>>> >> do it right).
>>> >>
>>> >> @Hideto: In the last LLVM meeting you gave a related talk [1]. Do you
>>> >> maybe have
>>> >> the full statistics and / or ways to reproduce them?
>>> >>
>>> >> Best,
>>> >> Stefanos
>>> >>
>>> >> (*)  SPEC, Polybench, Cryptographic libraries, Genome alignment, Image
>>> >> Processing, Graph Processing, Web browsers, databases like sqlite etc.
>>> >>
>>> >> [1] https://www.youtube.com/watch?v=QvF68tOt_w8
>>> >> _______________________________________________
>>> >> LLVM Developers mailing list
>>> >> llvm-dev at lists.llvm.org
>>> >> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>> >>
>>> >>
>>> >>
>>> >
>>> > _______________________________________________
>>> > LLVM Developers mailing list
>>> > llvm-dev at lists.llvm.org
>>> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210525/57be0151/attachment.html>