<div dir="ltr">+some of the usual debug info folks<br><br>I haven't looked at the code/complexity of dexter to get a sense of what the cost of maintenance would be - but the general idea seems reasonable to me.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Oct 9, 2019 at 8:33 AM Jeremy Morse via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Hi llvm-dev@,<br>
<br>
This is a proposal for LLVM to adopt Sony's Dexter tool [0], import it into the<br>
debuginfo-tests repo, and use it to run integration tests between debuggers<br>
and clang/llvms debuginfo. (Sony has approved donating Dexter to LLVM).<br>
<br>
Background<br>
----------<br>
<br>
The debuginfo-tests repo contains an integration test suite for debug data,<br>
which builds each test case from its source code using clang and runs the<br>
output in gdb/lldb, and for some the Windows-based cdb. Each test case<br>
contains a list of debugger commands which are executed in order; the output of<br>
the commands is used to determine whether the test passes. Directly using<br>
debugger commands prevents the tests from being easily portable however, as the<br>
driver commands are debugger-specific and so each test case will only run with<br>
a specific debugger. gdb and lldb are sufficiently similar that their commands<br>
can be translated to the other, but it's much more difficult to support<br>
Visual Studios debugger or cdb.<br>
<br>
Dexter (Debug Experience Tester) [0] was developed for the purpose of testing<br>
the quality of debug information for combinations of compilers and debuggers.<br>
It was first introduced to the community last year in a lightning talk<br>
about measuring the user debug experience<br>
[<a href="http://llvm.org/devmtg/2018-04/talks.html#Lightning_11" rel="noreferrer" target="_blank">http://llvm.org/devmtg/2018-04/talks.html#Lightning_11</a>]. Tests are written as<br>
comments in the source code, with each comment containing a single test<br>
command. Each test command specifies an expectation about the debug experience,<br>
such as the value of a variable on a given line, which Dexter compares against<br>
what the debugger actually observes when the test is run. Dexter gives a score<br>
based on how similar the expected and actual experience were; for the<br>
purposes of<br>
regression testing anything less than a perfect score is considered a failure.<br>
The test commands are debugger-agnostic, allowing the same test to be run with<br>
any supported debugger. The compiler and debugger to use in the test are<br>
specified on the command line.<br>
<br>
In a previous discussion [<a href="https://reviews.llvm.org/D54187" rel="noreferrer" target="_blank">https://reviews.llvm.org/D54187</a>], the idea of using<br>
Dexter to drive the debuginfo-tests was raised as a solution to adding<br>
cdb support<br>
without needing to maintain an entirely separate suite of tests (as is<br>
currently the case). It can also replace the existing llgdb layer, which<br>
provides a frontend for driving tests with GDB and LLDB; the upshot being that<br>
each test can be written once and then run with any desired debugger. The<br>
Dexter test commands are also more concise than the equivalent set of debugger<br>
commands: where the existing tests require 3 separate statements to break on a<br>
line, print the value of an expression, and then check the output, a single<br>
Dexter command encapsulates all these items.<br>
<br>
Existing Test: ```<br>
struct S { int a[8]; };<br>
<br>
int f(struct S s, unsigned i) {<br>
// DEBUGGER: break 14<br>
return s.a[i];<br>
}<br>
<br>
int main(int argc, const char **argv) {<br>
struct S s = {{0, 1, 2, 3, 4, 5, 6, 7}};<br>
if (f(s, 4) == 4)<br>
return f(s, 0);<br>
return 0;<br>
}<br>
<br>
// DEBUGGER: r<br>
// DEBUGGER: p s<br>
// CHECK: a =<br>
// DEBUGGER: p s.a[0]<br>
// CHECK: = 0<br>
// DEBUGGER: p s.a[1]<br>
// CHECK: = 1<br>
// DEBUGGER: p s.a[7]<br>
```<br>
<br>
Dexter: ```<br>
struct S { int a[8]; };<br>
<br>
int f(struct S s, unsigned i) {<br>
return s.a[i]; // DexLabel('f_ret')<br>
}<br>
<br>
int main(int argc, const char **argv) {<br>
struct S s = {{0, 1, 2, 3, 4, 5, 6, 7}};<br>
if (f(s, 4) == 4)<br>
return f(s, 0);<br>
return 0;<br>
}<br>
<br>
// DexExpectWatchValue('s.a[0]', '0', on_line='f_ret')<br>
// DexExpectWatchValue('s.a[1]', '1', on_line='f_ret')<br>
// DexExpectWatchValue('s.a[7]', '7', on_line='f_ret')<br>
```<br>
<br>
Currently Dexter has commands which can be used to cover most of the existing<br>
debuginfo-test cases (see "rough edges" below). Documentation on what<br>
expectations can be described are here [1]. We're also working on<br>
some sequencing commands ("this state is seen before that state") to better<br>
test optimized debuginfo.<br>
<br>
Proposal<br>
------------<br>
<br>
Our patch is up here: <a href="https://reviews.llvm.org/D68708" rel="noreferrer" target="_blank">https://reviews.llvm.org/D68708</a><br>
You can browse the tree on github here:<br>
<a href="https://github.com/jmorse/llvm-project/tree/dexter-rfc" rel="noreferrer" target="_blank">https://github.com/jmorse/llvm-project/tree/dexter-rfc</a><br>
<br>
It imports Dexter into the debuginfo-tests repository, and converts many<br>
existing debuginfo-tests to run with Dexter. We'd suggest that future<br>
debuginfo integration tests use Dexter to run by default, so that we can<br>
build a testsuite ensuring clang/llvms debuginfo is correct and interpreted<br>
correctly everywhere.<br>
<br>
Rough edges<br>
-----------<br>
<br>
Unifying debuginfo tests is a good goal, but there are limits -- for example,<br>
there's always going to be a place for testing clang-cl specific flags. It<br>
doesn't help that the cdb / windows-debug-engine expression parser is very<br>
different from other debuggers. We've stopped short of making all debuginfo<br>
tests completely generic, as there'll be different opinions on where to draw<br>
the line. Other scope for differences includes different platforms having<br>
different types for STL constructs, and different debuggers pretty printing<br>
differently.<br>
<br>
There's also a moderate sized question mark over GDB -- their python API is<br>
great, but has to be accessed from within a gdb process. This example<br>
implementation [2] uses a python RPC library to do that, which is an annoying<br>
extra dependency. Plus, GDB is GPL3, touching GDBs API from Dexter might raise<br>
licensing questions if LLVM adopted this.<br>
<br>
A small number of existing debuginfo tests (ex: nested-struct.cpp) don't<br>
actually start the built program, and instead just inspect type information.<br>
We haven't tried to support this, partially because that usecase doesn't seem<br>
suitable for Dexter, but mostly because debugger APIs don't seem to support<br>
looking up an arbitrary type on an arbitrary line number.<br>
<br>
We've also focused entirely on Python 3 given that Python 2 expires at the end<br>
of this year. The Dexter tests mark themselves as unsupported if the python<br>
process running lit (usually PYTHON_EXECUTABLE) isn't >= 3.0. We've been<br>
targeting Python 3.6; I'm not aware of any LLVM minimum python version<br>
requirements.<br>
<br>
Roadmap<br>
--------<br>
<br>
Non-goals: As stated, Dexter is an integration test framework, which means that<br>
it is not intended to be used to test the output of specific optimization<br>
passes, or any other transformations other than a complete build and link.<br>
<br>
More can be done to test the debugging experience, and we would like Dexter<br>
to be able to express more behaviours in the future, for example the order in<br>
which information appears or the absence of certain information.<br>
<br>
Beyond straightforward regression tests, Dexter's original function was to<br>
quantify the quality of the debug information in a program. While we're only<br>
proposing using Dexter as a replacement debuginfo-tests driver, we think<br>
there's a future in being able to measure ``how bad'' both debuginfo and<br>
codegen changes affect the debugging experience. More about that in a future<br>
RFC though!<br>
<br>
[0] <a href="https://github.com/snsystems/dexter" rel="noreferrer" target="_blank">https://github.com/snsystems/dexter</a><br>
[1] <a href="https://github.com/SNSystems/dexter/blob/master/Commands.md" rel="noreferrer" target="_blank">https://github.com/SNSystems/dexter/blob/master/Commands.md</a><br>
[2] <a href="https://github.com/SNSystems/dexter/pull/70" rel="noreferrer" target="_blank">https://github.com/SNSystems/dexter/pull/70</a><br>
<br>
--<br>
Thanks,<br>
Jeremy<br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div>