[llvm-dev] Statistics for Effectiveness of Passes on Reference Workloads

[Johannes Doerfert via llvm-dev]

Hideto (cc'ed) did look into this last year and he has some data.
You can find a quick summary in his lighting talk last year [1] and
also read a little bit about it in our paper draft! (attached),
especially Section 2.2 and Figure 3 are interesting here.

Not to say that we should not build some functionality upstream
to do this regularly :)

~ Johannes

[1] https://www.youtube.com/watch?v=nxfew3hsMFM&t=1435s


On 5/24/21 3:51 PM, Stefanos Baziotis via llvm-dev wrote:
> Hi Min,
>
> Honestly, I don't think optimization remarks are good to check _whether_ a
> pass did any work. A far better option, to the best of my knowledge, is, as
> I mentioned, the --print-after-all family [1]
> Optimization remarks may help on the last question I posed and that is to
> check _why_ a transformation was not applied. Although, if one reaches this
> point, they will need to use way more
> tools than remarks.
>
>   And even for the Passes that use it, my impression was that not every
>> predicates are annotated with optimization remarks.
>
> Definitely not and a lot of remarks are not that descriptive either.
>
> Best,
> Stefanos
>
> [1] https://godbolt.org/z/58Ms7qW4s
>
> Στις Δευ, 24 Μαΐ 2021 στις 11:07 μ.μ., ο/η Min-Yih Hsu <minyihh at uci.edu>
> έγραψε:
>
>> I think optimization remarks is a good framework for measuring whether a
>> Pass does any work.
>> But unfortunately it is not widely adopted by Passes outside loop
>> transformations. And even for the Passes that use it, my impression was
>> that not every predicates are annotated with optimization remarks.
>>
>> -Min
>>
>> On May 24, 2021, at 12:55 PM, Stefanos Baziotis via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>> Hi,
>>
>> Has anyone gathered statistics on reference workloads (*) for
>> (transformation) passes
>> that are enabled / run by default (e.g., -O3) but most of the
>> time _don't_ do any effective transformation?
>> Even better if we also have such statistics for passes that are _not_
>> enabled by default
>> (e.g., loop fusion, distribution, interchange, NewGVN).
>>
>> And yet even better if people have some idea / data for the reason for
>> ineffectiveness.
>> Bad heuristics / decision-making? Are some of these transformations
>> useless most of the time?
>> Or maybe they are useful but their implementation in LLVM is not powerful
>> enough. Or maybe
>> they incur a significant compile-time overhead.
>>
>> If not, it would also be helpful if anyone who has tried gathering similar
>> statistics has any
>> advice on how to approach it (my rough idea is initially
>> use --print-changed / --print-after-all on these workloads for -O3 and
>> then try to
>> slide in passes that are not enabled by default; although that's harder to
>> do it right).
>>
>> @Hideto: In the last LLVM meeting you gave a related talk [1]. Do you
>> maybe have
>> the full statistics and / or ways to reproduce them?
>>
>> Best,
>> Stefanos
>>
>> (*)  SPEC, Polybench, Cryptographic libraries, Genome alignment, Image
>> Processing, Graph Processing, Web browsers, databases like sqlite etc.
>>
>> [1] https://www.youtube.com/watch?v=QvF68tOt_w8
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Towards_Compile_Time_Reducing_Compiler_Optimization_Selection_via_Machine_Learning.pdf
Type: application/pdf
Size: 1947927 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210524/c9a4b7ec/attachment-0001.pdf>