[LLVMdev] RFC: Improving our DWARF (and ELF) emission testing capabilities

[David Blaikie]

On Tue, Jan 22, 2013 at 3:23 PM, Robinson, Paul
<Paul.Robinson at am.sony.com> wrote:
>>>> 2. Relying of assembly directive emissions (i.e. .cfi_*), which is
>>>> cumbersome and misses a lot of things like actual DWARF encoding.
>>>
>>> I'm not sure what you mean by "actual DWARF encoding" here.
>>> (disclaimer: I've only recently started dabbling with debug info, so I
>>> may be missing obvious things)
>>
>> I mean that it doesn't test the whole way, and there's quite a bit of
>> DWARF-related functionality in MC. So when a test relies on matching
>> directives in ASM output, there's quite a bit of code in MC it doesn't
>> exercise.
>
> Hmmm. "Proper" testing would exercise each component involved, as well
> as possibly longer paths that maybe are not exactly the sum of the parts.
> Debug info changes are quite likely to involve most or all of:
> - Clang's C/C++ to IR (metadata)

Simple enough: changes to Clang require tests in Clang (OK, so there's
no mechanism in place to avoid tests in Clang testing optimization,
for example, but we try to be good about it)

> - LLVM's IR to assembler source
> - assembler source to object file
> - LLVM's IR to object file (which partly bypasses or can be different
>   from the previous two paths)

& these generally go in LLVM - yeah, they could be separate, but I'd
expect the assembler and object file emission to be tested separately
already - the only benefit to testing particular IR->object &
separately testing particular IR->assembly is probably not worthwhile.
If we could test against the precursor to those outputs then we might
get the advantage of only having the right tests fail for the right
reasons (debug info tests wouldn't fail when we broken the
assembler/object emission).

> Properly speaking they should each get their own tests.
> Not to mention a unit-test (or debuginfo-test) to exercise the complete
> Clang -> object (-> debugger) sequence.

Just because it makes me twitch (though I admit debating test taxonomy
terminology verges on a religious topic): these tests are the
antithesis of unit tests. Taking source code, compiling it with
clang/LLVM, loading it in a debugger and interacting with the debugger
is a scenario test.

A unit test would be API level, say building IR by calling Clang APIs
& then passing it into DebugInfo generation & watching the MI calls
that resulted (preferably stubbing them out in some way).

> I try to be good about this, but as a developer I find that sort of
> thing tedious. Which mostly proves that I suck at QA, and have to depend
> on reviewers to keep me on the straight and narrow.  This works to the
> extent that those reviewers are willing to be critical of my efforts,
> and insist on adequate (instead of minimal) testing.  But testing is
> an art unto itself, and most developers aren't good at it.

We do what we can (because we must).

- David