[llvm-dev] Status of debuginfo-tests

[Robinson, Paul via llvm-dev]

Let me say up front that I sympathize deeply with the problem; debug
info is an interface, and it is frequently unclear whether the goal of
some bit of work is to test the producer or test the consumer of the
interface.  In fact we end up using the producer to test the consumer,
and (in the case at hand) using the consumer to test the producer.
There are distinct analogies to testing compilers by seeing what the
linker thinks, and testing linkers by seeing whether they can handle
what the compiler produces.

> I understand the desire to keep them as similar as possible, but I'm
> still not really sold that massaging fickle text output into a different
> text format is going to make things more scalable.  I'd like there to be
> as few layers of text processing as possible.

If the text output is fickle, then I'd think hiding the fickleness behind
a wrapper we control would be preferable to updating dozens of tests when 
something changes.  Or if a debugger changes its presentation in version 
N+1, but people are still running tests with version N, persuading the 
wrapper to handle both would be less overall work than making every test 
accommodate both formats.

>  If someone files a bug
> report that includes a WinDbg command log, I'd like to be able to paste
> those statements into a test.

That sounds like your goal is to turn the bug report into a Windows-only 
test, and not a common test.  Is that actually what you want?
If you still want it to be a common test, then you still need to do the
work to write the non-Windows side, and make sure it's actually still
exercising the same thing; not clear you are saving anything by being
able to copy-paste a WinDbg report.

> I also expect that on Windows we will end up having far more debug info
> tests than other platforms, specifically because we don't have the
> ability to write tests against the debugger (as it's proprietary /
> closed source).

I don't see how that follows. Sony runs the GDB test suite using clang as
the compiler, and while that is certainly perverting a debugger test suite
into being a compiler test suite, it has value in being a body of tests
that exercise a variety of debug-info features.  GDB being open source is
completely irrelevant to this use-case.  We treat it as closed.  We have
local changes to the test suite, but not to GDB.  The expected results
from the suite are based on GDB+GCC, which we treat as an oracle; then we
don't bother with tests of debugger features that clearly don't depend on
debug info, such as thread handling.

Whatever CodeView/PDB tests you want to write, you can use MS tools as
your oracle.  Maybe you can't leverage an existing test suite, but it
doesn't mean you can't write tests.

> So the language used in the current set of debug info tests is very
> simple, because GDB and LLDB already have test suites that test more
> complicated things.  But the problem is, we don't have those other test
> suites to fall back on, so we will need much more.

This is an argument in favor of a completely separate WinDbg-based
executable test suite, rather than pumping up debuginfo-tests.

> There's also the issue that we may want to test entirely different
> things.  For example, in the hypothetical example I posted earlier, we
> compile once and link twice with 2 different linkers.  But we might even
> want to compile twice and link twice (compile same program with cl and
> clang-cl, then link both with lld).  The fundamental difference here is
> that we have two different things that can emit debug info - the compiler
> and linker - and we need the flexibility to test both independently of
> each other.

Iterating over many combinations is a distinct testing problem.  It helps
to have the test suite designed to handle this up front.  My experience
is that different combinations will have slightly different pass/fail
results and you need to be ready for that as well.

> I don't see a useful abstraction that glosses over these differences
> that isn't a ton of work for minimal gain, given the frequency with
> which we'd need to fall back to a custom test anyway.

As I mentioned above, you seem to be heading in the direction of a 
completely separate project, rather than being able to usefully
leverage anything from debuginfo-tests other than the basic idea.
--paulr