[llvm-dev] Bugpoint Redesign
Diego Treviño via llvm-dev
llvm-dev at lists.llvm.org
Wed Jun 12 13:18:27 PDT 2019
Hi everyone,
Thanks so much for all the feedback, I'll keep all your comments and
suggestions in mind. So, for the moment I will focus on building the
IR-Reduction tool, and once that's done, I will work on integrating it to
the existing bugpoint, either as a sub-tool (as Philip suggested) or as
another debug strategy.
And once that's finished I'll be working on implementing all your
thoughtful suggestions to improve the tool's functionality.
Cheers,
Diego
On Tue, Jun 11, 2019 at 11:30 AM Finkel, Hal J. <hfinkel at anl.gov> wrote:
>
> On 6/11/19 12:25 PM, Philip Reames via llvm-dev wrote:
>
> At the moment, bugpoint has three major use cases: crash reduction,
> miscompile reduction, and mutation fuzzing. Out of these, a huge
> proportion of the interface complexity comes from the miscompile handling.
>
> I generally agree with removing the auto-detection logic. I've found it
> to be extraordinarily error prone and confusing.
>
>
> I'm not sure if I said this previously, but +1 to this. I don't recall a
> situation where I wasn't sure whether the problem was the compiler crashing
> or whether the problem was that the code was miscompiled. Even in a CI-type
> setup, a failure in the compiler step and in the run step can be
> distinguished by the relevant scripts.
>
> -Hal
>
>
> Interface wise, I might suggest something in the spirit of sub-tools (i.e.
> git or svn). As possible example:
> bugpoint crash-reduce
> bugpoint miscompile-reduce
> bugpoint mutate
>
> In addition to these high-level commands, it may also be useful to expose
> individual reduction steps. I find myself frequently wanting to run only
> individual reduction steps (and have hacked up my local bugpoint to allow
> this) via a wrapper script. Having first class support for "bugpoint
> reduce-step functions <input.ll>" would be awesome.
>
> Another idea would be to move all of the complexity of test formation into
> a separate command. Rather than having the tool detect which opt to use as
> part of reduction, instead have a generate command which generates a script
> which is then used for reduction. (i.e. make everything use the custom
> mode, while still proving helpers to generate). This is probably more
> natural for crash reduction instead of miscompile reduction, but maybe we
> could make it work for both? Or maybe if we split the two commands (and
> thus their interface) it doesn't really matter.
>
> Philip
>
> p.s. Bugpoint is a fairly critical tool. If we start rewriting it, making
> sure it continues to work through the process will be critical. We don't
> have much in the way of testing for it today, and that would need to
> change.
> On 6/7/19 2:19 PM, Diego Treviño via llvm-dev wrote:
>
> Hey all,
>
> I wanted to share a proposal
> <https://docs.google.com/document/d/171ecPTeXw68fbCghdGw_NPBouWvmvUX8vePlbhhHEdA/edit?usp=sharing>
> to revamp the current go-to IR debugging tool: Bugpoint. i'd love to hear
> any feedback or general thoughts.
>
> Here's the markdown version of the doc:
> ---
> # Bugpoint Redesign
> Author: Diego Treviño (diegotf at google.com)
>
> Date: 2016-06-05
>
> Status: Draft
>
>
> ## Introduction
> As use of bugpoint has grown several areas of improvement have been
> identified through years of use: confusing to use, slow, it doesn’t always
> produce high quality test cases, etc. This document proposes a new approach
> with a narrower focus: minimization of IR test cases.
>
>
> ## Proposed New Design
>
>
> ### Narrow focus: test-case reduction
> The main focus will be a code reduction strategy to obtain much smaller
> test cases that still have the same property as the original one. This will
> be done via classic delta debugging and by adding some IR-specific
> reductions (e.g. replacing globals, removing unused instructions, etc),
> similar to what already exists, but with more in-depth minimization.
>
>
> Granted, if the community differs on this proposal, the legacy code could
> still be present in the tool, but with the caveat of still being documented
> and designed towards delta reduction.
>
>
> ### Command-Line Options
> We are proposing to reduce the plethora of bugpoint’s options to just two:
> an interesting-ness test and the arguments for said test, similar to other
> delta reduction tools such as CReduce, Delta, and Lithium; the tool should
> feel less cluttered, and there should also be no uncertainty about how to
> operate it.
>
>
> The interesting-ness test that’s going to be run to reduce the code is
> given by name:
> `--test=<test_name>`
> If a `--test` option is not given, the program exits; this option is
> similar to bugpoint’s current `-compile-custom` option, which lets the user
> run a custom script.
>
>
> The interesting-ness test would be defined as a script that returns 0 when
> the IR achieves a user-defined behaviour (e.g. failure to compile on clang)
> and a nonzero value when otherwise. Leaving the user the freedom to
> determine what is and isn’t interesting to the tool, and thus, streamlining
> the process of reducing a test-case.
>
>
> If the test accepts any arguments (excluding the input ll/bc file), they
> are given via the following flag:
> `--test_args=<test_arguments>`
> If unspecified, the test is run as given. It’s worth noting that the input
> file would be passed as a parameter to the test, similar how
> `-compile-custom` currently operates.
>
>
> ### Implementation
> The tool would behave similar to CReduce’s functionality in that it would
> have a list of passes that try to minimize the given test-case. We should
> be able to modularize the tool’s behavior, as well as making it easier to
> maintain and expand.
>
>
> The first version of this redesign would try to:
>
>
> * Split the code into chunks and discard those that fail the given test
> * Discard functions, instructions and metadata that don’t influence the
> interesting-ness test
> * Remove unused parameters from functions
> * Eliminate unvisited conditional paths
> * Rename variables to more regular ones (such as “a”, “b”, “c”, etc.)
>
>
> Once these passes are implemented, more meaningful reductions (such as
> type reduction) would be added to the tool, to even further reduce IR.
>
>
> ## Background on historical bugpoint issues
>
>
> ### Root Cause Analysis
> Presently, bugpoint takes a long time to find the source problem in a
> given IR file, mainly due to the fact that it tries to debug the input by
> running various strategies to classify the bug, which in turn run multiple
> optimizer and compilation passes over the input, taking up a lot of time.
> Furthermore, when the IR crashes, it tries to reduce it by performing some
> sub-optimal passes (e.g. a lot of unreachable blocks), and sometimes even
> fails to minimize at all.
>
>
> ### "Quirky" Interface
> Bugpoint’s current interface overwhelms and confuses the user, the help
> screen alone ends up confusing rather providing guidance, as seen below:
>
> 
>
> And, not only are there numerous features and options, but some of them
> also work in unexpected ways and most of the time the user ends up using a
> custom script. Pruning and simplifying the interface will be worth
> considering in order to make the tool more useful in the general case and
> easier to maintain.
>
>
> --
> Cheers,
> Diego
>
> _______________________________________________
> LLVM Developers mailing listllvm-dev at lists.llvm.orghttps://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> _______________________________________________
> LLVM Developers mailing listllvm-dev at lists.llvm.orghttps://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> --
> Hal Finkel
> Lead, Compiler Technology and Programming Languages
> Leadership Computing Facility
> Argonne National Laboratory
>
>
--
Cheers,
Diego
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190612/4eb905f4/attachment.html>
More information about the llvm-dev
mailing list