[llvm-dev] [RFC] Adopt Dexter and use it to run debuginfo-tests

[Jeremy Morse via llvm-dev]

 Hi llvm-dev@,

This is a proposal for LLVM to adopt Sony's Dexter tool [0], import it into the
debuginfo-tests repo, and use it to run integration tests between debuggers
and clang/llvms debuginfo. (Sony has approved donating Dexter to LLVM).

Background
----------

The debuginfo-tests repo contains an integration test suite for debug data,
which builds each test case from its source code using clang and runs the
output in gdb/lldb, and for some the Windows-based cdb.  Each test case
contains a list of debugger commands which are executed in order; the output of
the commands is used to determine whether the test passes.  Directly using
debugger commands prevents the tests from being easily portable however, as the
driver commands are debugger-specific and so each test case will only run with
a specific debugger. gdb and lldb are sufficiently similar that their commands
can be translated to the other, but it's much more difficult to support
Visual Studios debugger or cdb.

Dexter (Debug Experience Tester) [0] was developed for the purpose of testing
the quality of debug information for combinations of compilers and debuggers.
It was first introduced to the community last year in a lightning talk
about measuring the user debug experience
[http://llvm.org/devmtg/2018-04/talks.html#Lightning_11]. Tests are written as
comments in the source code, with each comment containing a single test
command. Each test command specifies an expectation about the debug experience,
such as the value of a variable on a given line, which Dexter compares against
what the debugger actually observes when the test is run. Dexter gives a score
based on how similar the expected and actual experience were; for the
purposes of
regression testing anything less than a perfect score is considered a failure.
The test commands are debugger-agnostic, allowing the same test to be run with
any supported debugger. The compiler and debugger to use in the test are
specified on the command line.

In a previous discussion [https://reviews.llvm.org/D54187], the idea of using
Dexter to drive the debuginfo-tests was raised as a solution to adding
cdb support
without needing to maintain an entirely separate suite of tests (as is
currently the case). It can also replace the existing llgdb layer, which
provides a frontend for driving tests with GDB and LLDB; the upshot being that
each test can be written once and then run with any desired debugger. The
Dexter test commands are also more concise than the equivalent set of debugger
commands: where the existing tests require 3 separate statements to break on a
line, print the value of an expression, and then check the output, a single
Dexter command encapsulates all these items.

Existing Test: ```
  struct S { int a[8]; };

  int f(struct S s, unsigned i) {
    // DEBUGGER: break 14
    return s.a[i];
  }

  int main(int argc, const char **argv) {
    struct S s = {{0, 1, 2, 3, 4, 5, 6, 7}};
    if (f(s, 4) == 4)
      return f(s, 0);
    return 0;
  }

  // DEBUGGER: r
  // DEBUGGER: p s
  // CHECK: a =
  // DEBUGGER: p s.a[0]
  // CHECK: = 0
  // DEBUGGER: p s.a[1]
  // CHECK: = 1
  // DEBUGGER: p s.a[7]
```

Dexter: ```
  struct S { int a[8]; };

  int f(struct S s, unsigned i) {
    return s.a[i]; // DexLabel('f_ret')
  }

  int main(int argc, const char **argv) {
    struct S s = {{0, 1, 2, 3, 4, 5, 6, 7}};
    if (f(s, 4) == 4)
      return f(s, 0);
    return 0;
  }

  // DexExpectWatchValue('s.a[0]', '0', on_line='f_ret')
  // DexExpectWatchValue('s.a[1]', '1', on_line='f_ret')
  // DexExpectWatchValue('s.a[7]', '7', on_line='f_ret')
```

Currently Dexter has commands which can be used to cover most of the existing
debuginfo-test cases (see "rough edges" below). Documentation on what
expectations can be described are here [1]. We're also working on
some sequencing commands ("this state is seen before that state") to better
test optimized debuginfo.

Proposal
------------

Our patch is up here: https://reviews.llvm.org/D68708
You can browse the tree on github here:
https://github.com/jmorse/llvm-project/tree/dexter-rfc

It imports Dexter into the debuginfo-tests repository, and converts many
existing debuginfo-tests to run with Dexter. We'd suggest that future
debuginfo integration tests use Dexter to run by default, so that we can
build a testsuite ensuring clang/llvms debuginfo is correct and interpreted
correctly everywhere.

Rough edges
-----------

Unifying debuginfo tests is a good goal, but there are limits -- for example,
there's always going to be a place for testing clang-cl specific flags. It
doesn't help that the cdb / windows-debug-engine expression parser is very
different from other debuggers. We've stopped short of making all debuginfo
tests completely generic, as there'll be different opinions on where to draw
the line. Other scope for differences includes different platforms having
different types for STL constructs, and different debuggers pretty printing
differently.

There's also a moderate sized question mark over GDB -- their python API is
great, but has to be accessed from within a gdb process. This example
implementation [2] uses a python RPC library to do that, which is an annoying
extra dependency. Plus, GDB is GPL3, touching GDBs API from Dexter might raise
licensing questions if LLVM adopted this.

A small number of existing debuginfo tests (ex: nested-struct.cpp) don't
actually start the built program, and instead just inspect type information.
We haven't tried to support this, partially because that usecase doesn't seem
suitable for Dexter, but mostly because debugger APIs don't seem to support
looking up an arbitrary type on an arbitrary line number.

We've also focused entirely on Python 3 given that Python 2 expires at the end
of this year. The Dexter tests mark themselves as unsupported if the python
process running lit (usually PYTHON_EXECUTABLE) isn't >= 3.0. We've been
targeting Python 3.6; I'm not aware of any LLVM minimum python version
requirements.

Roadmap
--------

Non-goals: As stated, Dexter is an integration test framework, which means that
it is not intended to be used to test the output of specific optimization
passes, or any other transformations other than a complete build and link.

More can be done to test the debugging experience, and we would like Dexter
to be able to express more behaviours in the future, for example the order in
which information appears or the absence of certain information.

Beyond straightforward regression tests, Dexter's original function was to
quantify the quality of the debug information in a program. While we're only
proposing using Dexter as a replacement debuginfo-tests driver, we think
there's a future in being able to measure ``how bad'' both debuginfo and
codegen changes affect the debugging experience. More about that in a future
RFC though!

[0] https://github.com/snsystems/dexter
[1] https://github.com/SNSystems/dexter/blob/master/Commands.md
[2] https://github.com/SNSystems/dexter/pull/70

--
Thanks,
Jeremy