[LLVMdev] RFC: Improving our DWARF (and ELF) emission testing capabilities

[Robinson, Paul]

David Blaikie [dblaikie at gmail.com] wrote:
> On Tue, Jan 22, 2013 at 3:23 PM, Robinson, Paul
> <Paul.Robinson at am.sony.com> wrote:
>>>>> 2. Relying of assembly directive emissions (i.e. .cfi_*), which is
>>>>> cumbersome and misses a lot of things like actual DWARF encoding.
>>>>
>>>> I'm not sure what you mean by "actual DWARF encoding" here.
>>>> (disclaimer: I've only recently started dabbling with debug info, so I
>>>> may be missing obvious things)
>>>
>>> I mean that it doesn't test the whole way, and there's quite a bit of
>>> DWARF-related functionality in MC. So when a test relies on matching
>>> directives in ASM output, there's quite a bit of code in MC it doesn't
>>> exercise.
>>
>> Hmmm. "Proper" testing would exercise each component involved, as well
>> as possibly longer paths that maybe are not exactly the sum of the parts.
>> Debug info changes are quite likely to involve most or all of:
>> - Clang's C/C++ to IR (metadata)
>
> Simple enough: changes to Clang require tests in Clang (OK, so there's
> no mechanism in place to avoid tests in Clang testing optimization,
> for example, but we try to be good about it)
>
>> - LLVM's IR to assembler source
>> - assembler source to object file
>> - LLVM's IR to object file (which partly bypasses or can be different
>>   from the previous two paths)
>
> & these generally go in LLVM - yeah, they could be separate, but I'd
> expect the assembler and object file emission to be tested separately
> already - the only benefit to testing particular IR->object &
> separately testing particular IR->assembly is probably not worthwhile.

I cite PR13303/PR14524, where asm and direct-object output differ.
This came up early in my LLVM career and has doubtless poisoned my
outlook for life....

In many cases I think the same test _source_ can be used to check both
asm and object, with appropriate RUN lines, and whether you want to
count that as the same or separate depends on how you like to game the
counts.  What matters to me is both paths get tested.

> If we could test against the precursor to those outputs then we might
> get the advantage of only having the right tests fail for the right
> reasons (debug info tests wouldn't fail when we broken the
> assembler/object emission).
>
>> Properly speaking they should each get their own tests.
>> Not to mention a unit-test (or debuginfo-test) to exercise the complete
>> Clang -> object (-> debugger) sequence.
>
> Just because it makes me twitch (though I admit debating test taxonomy
> terminology verges on a religious topic): these tests are the
> antithesis of unit tests. Taking source code, compiling it with
> clang/LLVM, loading it in a debugger and interacting with the debugger
> is a scenario test.
>
> A unit test would be API level, say building IR by calling Clang APIs
> & then passing it into DebugInfo generation & watching the MI calls
> that resulted (preferably stubbing them out in some way).

Not sure why I said unit test there...
I'd think of compiling .cpp->.o as a Clang/LLVM integration test, while
I'd think of running the debugger on the object as a system test
(because gdb is not part of what this community delivers).
I also think I'd spectacularly fail the CSSP test.  :-)

>> I try to be good about this, but as a developer I find that sort of
>> thing tedious. Which mostly proves that I suck at QA, and have to depend
>> on reviewers to keep me on the straight and narrow.  This works to the
>> extent that those reviewers are willing to be critical of my efforts,
>> and insist on adequate (instead of minimal) testing.  But testing is
>> an art unto itself, and most developers aren't good at it.
>
> We do what we can (because we must).

Amen, brother.
--paulr