[llvm-dev] Status of debuginfo-tests

[Zachary Turner via llvm-dev]

On Fri, Sep 8, 2017 at 9:00 AM Adrian Prantl <aprantl at apple.com> wrote:

>
> > Eventually, some tests will inevitably need to Windows or Posix
> specific, so you're going to have to have all this extra stuff (the new
> substitutions, the different command lines, the custom output formats,
> etc.  So I think something like this provides maximal encouragement of
> sharing whenever possible (since you can almost always share source code),
> while still allowing each format to test real input and real output.
>
> I understand the desire to allow for Windows-specific tests and I think it
> would be good to add them to the repository in a windows subdirectory.
>
> Looking at the example you posted, the two variants are so structurally
> similar that I believe it would be a better to come up with a common
> abstraction from a readability / maintenance effort perspective. Basically,
> the only thing that the RUN lines do is compile and link executables from
> source code using the default target and run the test_debuginfo command. I
> think it would be better to define a new command substitution
> %clang-compile-link(?) in LIT that has different implementations on windows
> and posix. The set of debugger commands used by the tests is so tiny that
> it should not be a lot of work to implement a wrapper for the windows
> debugger (it took me about a day to write the python wrapper for LLDB
> including learning how to use the Python API) and it should also be
> possible to either do a sed-style massaging of the output or relax the
> CHECKs to work with both formats.
>
> I really want to avoid duplicating the debugger commands and checks, and I
> also want to maintain the ability to put the commands and CHECKs into the
> source code, since this makes the tests much easier to understand. Using a
> common abstraction will save us a lot of time in the long run, make
> maintenance and adding new tests cheaper, and won't prevent you from also
> having windows-specific tests that may use an expanded vocabulary.
>
> What do you think?
> -- adrian
>

I understand the desire to keep them as similar as possible, but I'm still
not really sold that massaging fickle text output into a different text
format is going to make things more scalable.  I'd like there to be as few
layers of text processing as possible.  If someone files a bug report that
includes a WinDbg command log, I'd like to be able to paste those
statements into a test.

I also expect that on Windows we will end up having far more debug info
tests than other platforms, specifically because we don't have the ability
to write tests against the debugger (as it's proprietary / closed source).
So the language used in the current set of debug info tests is very simple,
because GDB and LLDB already have test suites that test more complicated
things.  But the problem is, we don't have those other test suites to fall
back on, so we will need much more.  For example, we may end up wanting a
test that exercises custom debug visualizers, or a test that a certain
proprietary debugger feature works, or a test that builtin debug
visualizers of STL types work.  To write a check for the latter, you need
to know the layout of the type, which depends on the standard library
implementation, so it's already going to be different.

If all we're doing is printing an integer, then I agree we can write a
common test.  But I don't think this is going to be the case outside of 1
or 2 trivial tests.

There's also the issue that we may want to test entirely different things.
For example, in the hypothetical example I posted earlier, we compile once
and link twice with 2 different linkers.  But we might even want to compile
twice and link twice (compile same program with cl and clang-cl, then link
both with lld).  The fundamental difference here is that we have two
different things that can emit debug info - the compiler and linker - and
we need the flexibility to test both independently of each other.

On a posixy platform, you only care the compiler and don't care what the
linker is.  On the other hand, it has its own set of unique aspects.  You
might decide to compile and link many times, so that you can test
-gsplit-dwarf, -gdebug-info-kind=limited, -gdebug-info-kind=full,
-gdb-index, etc. against a single program.

I don't see a useful abstraction that glosses over these differences that
isn't a ton of work for minimal gain, given the frequency with which we'd
need to fall back to a custom test anyway.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170908/2a9c5966/attachment.html>