[llvm-dev] OptBisect implementation for new pass manager

[Greg Bedwell via llvm-dev]

Just to add one more simple use-case to the mix, although it sounds like
the main requirements are already captured.  We're using opt-bisect-limit
from clang and lld in https://github.com/SNSystems/dexter to build up a
picture of how different optimization passes affect the user debugging
experience (see slide 19 onwards in
http://llvm.org/devmtg/2018-04/slides/Bedwell-Measuring_the_User_Debugging_Experience.pdf
) which provides us useful data for writing useful small targeted tests
(resulting in PR38768) as well as hopefully allowing us to tune a useful
-Og mode once we've dealt with the obvious cases of fixable DI loss.

For en example of inspecting a small two-file example at -O2 -flto the tool
would first do the following in order to work out how many bisectable
passes are run for each stage by inspecting the "BISECT: running pass"
lines in the stderr output:

clang++ a.cpp -c -O2 -flto -o a.o -mllvm -opt-bisect-limit=999999
clang++ b.cpp -c -O2 -flto -o b.o -mllvm -opt-bisect-limit=999999
clang++ a.o b.o -fuse-ld=lld -Wl,-mllvm -Wl,-opt-bisect-limit=999999

Then we'll just keep rebuilding the code with increasing opt-bisect-limit
values:

clang++ a.cpp -c -O2 -flto -o a.o -mllvm -opt-bisect-limit=1
clang++ b.cpp -c -O2 -flto -o b.o -mllvm -opt-bisect-limit=0
clang++ a.o b.o -fuse-ld=lld -Wl,-mllvm -Wl,-opt-bisect-limit=0

clang++ a.cpp -c -O2 -flto -o a.o -mllvm -opt-bisect-limit=2
clang++ b.cpp -c -O2 -flto -o b.o -mllvm -opt-bisect-limit=0
clang++ a.o b.o -fuse-ld=lld -Wl,-mllvm -Wl,-opt-bisect-limit=0

and so on until each opt-bisect-limit value is at the maximum, measuring
the 'debugging experience score' for every build along the way.  We look
for the final instance of "BISECT: running pass" on the combined stderr
output to figure out the name of the pass that we're attaching that score
to so that we can build up a picture of which passes actually have an
effect on the score.  Obviously we'll modify the tool to cope with whatever
changes to interface/debug output occur, but I'd be very keen that any new
solutions didn't totally break the use-case!





On Fri, 5 Oct 2018 at 09:31, Chandler Carruth via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> On Thu, Oct 4, 2018 at 4:11 PM Kaylor, Andrew <andrew.kaylor at intel.com>
> wrote:
>
>> We (Intel) use opt-bisect most often in conjunction with clang and
>> frequently with applications that have non-trivial compilation mechanisms.
>> If there isn’t a way to get something hooked up through code generation, it
>> doesn’t really meet the needs of the way we are using it.  As an interim
>> solution it would be tolerable to have two separate bisect-like options,
>> one that works its way through the pre-codegen passes and a different one
>> that bisects codegen.
>>
>
> To an extent, I think that will be a very likely necessary step due to the
> fact that all of codegen is a separate pass manager at the moment.
>
> However, given that the goal is bisection, I would imagine a somewhat
> simpler approach of the first step of bisection be to determine whether the
> regression comes before codegen or in codegen.
>
> That said, I think it is interesting that you are using this directly with
> Clang. I'm honestly somewhat surprised that this works, but hey, nice!
>
>
>>
>>
>> Skipping “necessary” passes is a somewhat different matter, because you
>> can’t skip something like register allocation on your way to executable
>> code.  Most of the passes that are really significant in this way are
>> codegen passes, so we might still be able to work something out. I believe
>> there are a few pre-codegen passes that are required but they do trivial
>> things like removing intrinsics that don’t get lowered.
>>
>
> FWIW, even the "IR" passes like this are actually run as part of codegen
> using the legacy PM in all cases right now. So to an extent, I think this
> will "just work" out of the box once you can select between the bisection
> occuring over the main optimization pipeline (w/ the new PM) or the code
> generation pipeline (still w/ legacy PM).
>
> Perhaps we could have some mechanism that builds a set of clean up passes
>> that get run under some very limited circumstances. Is that what you were
>> suggesting in your comment about running “some minimal second set of
>> passes” after the bisection?
>>
>
> This is one approach that seems like it might be fruitful. It does make it
> hard to use bisection techniques to isolate issues caused by passes
> specifically when intermingled with those "necessary" passes like register
> allocation. I'm not sure how important bisection is for those passes
> currently though. To be honest, while I've used pass bisection a *lot* for
> a lot of different things, I've never really found it useful once I'm in
> MI-land because so few of the passes can realistically be disabled. So I
> was somewhat focused on solving the problem in a more simple way focused on
> the more flexible passes.
>
> There is another approach that I think might be more interesting
> long-term: rather than merely bisection to isolate a single point which
> introduces a bug, I'd like us to generalize the debug counter pattern.
> Right now, you can skip N times and then run M times. What I would like is
> to support an arbitrary number of skip and run counts. If you combine this
> with a test harness that marks compilations that fail, or runs that fail
> differently from the bug in question, you can use this to *detect* and
> preserve the necessary passes because they will fail the "sanity" or
> "baseline" test prior to being "interesting".
>
> This is a pretty common pattern in delta reduction (verifying that the
> reduction remains "interesting" includes verifying that it doesn't fail for
> unrelated reasons), and I think could be used to avoid building in complex
> logic to handle "required" passes.
>
>
> If none of this works, and we *need* to share the logic between `optnone`
> and bisection because we have no other way of expressing "required" passes,
> my inclination would be to actually try to use "optnone" directly rather
> than trying to build a unified "skip" routine that somehow coordinates with
> the pass manager. But I'm not sure how to do that really effectively
> outside of function passes. Or maybe there is some other way of easily
> communicating "required"-ness here, but currently I struggle to see it. So
> I'm somewhat hoping that in practice, one of the above will be enough.
>
>
>>
>> The current opt-bisect mechanism really doesn’t do much more than
>> increment a counter, but it does produce output that indicates what was run
>> and what wasn’t.  This is very useful for finding the proper owner for a
>> bug once the failure has been diagnosed with opt-bisect.  I don’t really
>> care how the mechanism is implemented, but I’d really like to know what the
>> last optimization was that ran in a failing case and which IR unit it was
>> run on.  Knowing the highest pass count that was used is also helpful to
>> establish a starting point.
>>
>
> Having clear output would definitely be useful. Fedor was already thinking
> of replacing the pass manager debug output with the instrumentation, and it
> would seem very easy to similarly wire together helpful output from the
> bisection instrumentation.
>
>
> Thanks for all the thoughts on different ways to use bisection!
> -Chandler
>
>> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181005/2902283a/attachment.html>