[llvm-dev] [RFC] Adopt Dexter and use it to run debuginfo-tests

[Jeremy Morse via llvm-dev]

Hi Adrian,

On Wed, Oct 9, 2019 at 4:59 PM Adrian Prantl <aprantl at apple.com> wrote:
> this is great news! I'm excited about having a single debugger driver in debuginfo-tests.
> I'm very much in favor of this proposal.

Much appreciated!,

> Can you share experiences as to how hard it is to get tests to work under multiple platforms and toolchains in practice?

Sometimes it's easy, othertimes not. Almost all the large blob of
Dexter tests I wrote about a year ago [0] to stimulate different
passes worked under Windows / cdb immediately. The difficulties I've
encountered have been:
 * Tests that rely on (for example) -fno-inline / -fno-unroll-loops
not playing well with clang-cl,
 * Different STL types on Windows -- gdb and lldb quite naturally
pretty-print things like vectors, it's much more difficult to do that
in a implementation-independent way,
 * I don't think cdb's expression parser is as expressive as gdb/lldb,
or at least I couldn't convince it to apply a subscript operator to a
std::deque object easily.

Difficulties examining state are probably the largest sticking point:
one can easily write a test that makes use of language and debugger
idioms, but they're all implemented differently on different platforms
of course.Finding the sweet spot between forcing tests to be more
explicit, and baking too much complexity into the test too, is an open
question.

> I see that many tests in the patch hardcode things like specific debuggers/linux/cflags (which presumably look different on windows). Does dexter have an #ifdef feature to allow for slightly different results or for disable one sub-test on a specific platform?

This is mostly because we wanted to freeze things in a condition where
they worked for the RFC. We're hoping that later we can have llvm-lit
autoselect a working builder/debugger and put that in the %dexter
expansion, then filter out any known bad combinations with XFAIL /
UNSUPPORTED in the tests. llvm-lit can almost certainly handle
per-test configuration -- we haven't approached the idea of making
debug-experience expectations conditional yet.

[0] https://github.com/SNSystems/dexter/tree/master/tests/nostdlib/llvm_passes

--
Thanks,
Jeremy