<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>At the moment, bugpoint has three major use cases: crash
reduction, miscompile reduction, and mutation fuzzing. Out of
these, a huge proportion of the interface complexity comes from
the miscompile handling. <br>
</p>
<p>I generally agree with removing the auto-detection logic. I've
found it to be extraordinarily error prone and confusing. <br>
</p>
<p>Interface wise, I might suggest something in the spirit of
sub-tools (i.e. git or svn). As possible example:<br>
bugpoint crash-reduce<br>
bugpoint miscompile-reduce<br>
bugpoint mutate<br>
</p>
<p>In addition to these high-level commands, it may also be useful
to expose individual reduction steps. I find myself frequently
wanting to run only individual reduction steps (and have hacked up
my local bugpoint to allow this) via a wrapper script. Having
first class support for "bugpoint reduce-step functions
<input.ll>" would be awesome.<br>
</p>
<p>Another idea would be to move all of the complexity of test
formation into a separate command. Rather than having the tool
detect which opt to use as part of reduction, instead have a
generate command which generates a script which is then used for
reduction. (i.e. make everything use the custom mode, while still
proving helpers to generate). This is probably more natural for
crash reduction instead of miscompile reduction, but maybe we
could make it work for both? Or maybe if we split the two
commands (and thus their interface) it doesn't really matter. <br>
</p>
<p>Philip</p>
<p>p.s. Bugpoint is a fairly critical tool. If we start rewriting
it, making sure it continues to work through the process will be
critical. We don't have much in the way of testing for it today,
and that would need to change. <br>
</p>
<div class="moz-cite-prefix">On 6/7/19 2:19 PM, Diego Treviño via
llvm-dev wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAFRRjJSz3y-Z_NzXkq7P1R1m4KMaPovqtcM3sL30St8BJR9Wng@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">Hey all,
<div><br>
</div>
<div>I wanted to share a <a
href="https://docs.google.com/document/d/171ecPTeXw68fbCghdGw_NPBouWvmvUX8vePlbhhHEdA/edit?usp=sharing"
moz-do-not-send="true">proposal</a> to revamp the current
go-to IR debugging tool: Bugpoint. i'd love to hear any
feedback or general thoughts. </div>
<div><br>
</div>
<div>Here's the markdown version of the doc:</div>
<div>---<br clear="all">
<div># Bugpoint Redesign<br>
Author: Diego Treviño (<a href="mailto:diegotf@google.com"
moz-do-not-send="true">diegotf@google.com</a>)<br>
<br>
Date: 2016-06-05<br>
<br>
Status: Draft<br>
<br>
<br>
## Introduction<br>
As use of bugpoint has grown several areas of improvement
have been identified through years of use: confusing to use,
slow, it doesn’t always produce high quality test cases,
etc. This document proposes a new approach with a narrower
focus: minimization of IR test cases.<br>
<br>
<br>
## Proposed New Design<br>
<br>
<br>
### Narrow focus: test-case reduction<br>
The main focus will be a code reduction strategy to obtain
much smaller test cases that still have the same property as
the original one. This will be done via classic delta
debugging and by adding some IR-specific reductions (e.g.
replacing globals, removing unused instructions, etc),
similar to what already exists, but with more in-depth
minimization.<br>
<br>
<br>
Granted, if the community differs on this proposal, the
legacy code could still be present in the tool, but with the
caveat of still being documented and designed towards delta
reduction.<br>
<br>
<br>
### Command-Line Options<br>
We are proposing to reduce the plethora of bugpoint’s
options to just two: an interesting-ness test and the
arguments for said test, similar to other delta reduction
tools such as CReduce, Delta, and Lithium; the tool should
feel less cluttered, and there should also be no uncertainty
about how to operate it.<br>
<br>
<br>
The interesting-ness test that’s going to be run to reduce
the code is given by name:<br>
`--test=<test_name>`<br>
If a `--test` option is not given, the program exits; this
option is similar to bugpoint’s current `-compile-custom`
option, which lets the user run a custom script.<br>
<br>
<br>
The interesting-ness test would be defined as a script that
returns 0 when the IR achieves a user-defined behaviour
(e.g. failure to compile on clang) and a nonzero value when
otherwise. Leaving the user the freedom to determine what is
and isn’t interesting to the tool, and thus, streamlining
the process of reducing a test-case.<br>
<br>
<br>
If the test accepts any arguments (excluding the input ll/bc
file), they are given via the following flag:<br>
`--test_args=<test_arguments>`<br>
If unspecified, the test is run as given. It’s worth noting
that the input file would be passed as a parameter to the
test, similar how `-compile-custom` currently operates.<br>
<br>
<br>
### Implementation<br>
The tool would behave similar to CReduce’s functionality in
that it would have a list of passes that try to minimize the
given test-case. We should be able to modularize the tool’s
behavior, as well as making it easier to maintain and
expand. <br>
<br>
<br>
The first version of this redesign would try to:<br>
<br>
<br>
* Split the code into chunks and discard those that fail the
given test<br>
* Discard functions, instructions and metadata that don’t
influence the interesting-ness test<br>
* Remove unused parameters from functions<br>
* Eliminate unvisited conditional paths<br>
* Rename variables to more regular ones (such as “a”, “b”,
“c”, etc.)<br>
<br>
<br>
Once these passes are implemented, more meaningful
reductions (such as type reduction) would be added to the
tool, to even further reduce IR.<br>
<br>
<br>
## Background on historical bugpoint issues<br>
<br>
<br>
### Root Cause Analysis<br>
Presently, bugpoint takes a long time to find the source
problem in a given IR file, mainly due to the fact that it
tries to debug the input by running various strategies to
classify the bug, which in turn run multiple optimizer and
compilation passes over the input, taking up a lot of time.
Furthermore, when the IR crashes, it tries to reduce it by
performing some sub-optimal passes (e.g. a lot of
unreachable blocks), and sometimes even fails to minimize at
all.<br>
<br>
<br>
### "Quirky" Interface<br>
Bugpoint’s current interface overwhelms and confuses the
user, the help screen alone ends up confusing rather
providing guidance, as seen below:<br>
<br>
![Bugpoint's help option showcase](<a
href="https://lh6.googleusercontent.com/sbpaSVHzpVVZKKAgHL9gvfzTWdgh3ju0KiDYql6WmWZfDYrdauOJMcuo9PP_V1dq8JQfMHOSKTv3lJcSpVytUyU8r5tJ2KTlGB0b2ve7jsZ3nVX8K8ItAbsA0JWkFKw67VJnq99m"
moz-do-not-send="true">https://lh6.googleusercontent.com/sbpaSVHzpVVZKKAgHL9gvfzTWdgh3ju0KiDYql6WmWZfDYrdauOJMcuo9PP_V1dq8JQfMHOSKTv3lJcSpVytUyU8r5tJ2KTlGB0b2ve7jsZ3nVX8K8ItAbsA0JWkFKw67VJnq99m</a>)<br>
<br>
And, not only are there numerous features and options, but
some of them also work in unexpected ways and most of the
time the user ends up using a custom script. Pruning and
simplifying the interface will be worth considering in order
to make the tool more useful in the general case and easier
to maintain.<br>
</div>
<div><br>
</div>
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">Cheers,
<div>Diego</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
LLVM Developers mailing list
<a class="moz-txt-link-abbreviated" href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>
<a class="moz-txt-link-freetext" href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a>
</pre>
</blockquote>
</body>
</html>