[LLVMdev] RFC: Improving our DWARF (and ELF) emission testing capabilities

[Eli Bendersky]

>> 1. llvm-dwarfdump: the best approach when it works. But unfortunately
>> lib/DebugInfo supports only a (small) subset of DWARF. Tricky sections
>> like debug_frame aren't supported.
>
> Ideally I'd like to see support added whenever a code change is made
> to a feature - so long as we hold ourselves to a "test new changes"
> that can gate/encourage the necessary feature support in
> llvm-dwarfdump.
>
> Since no one's likely to go back & write a bunch of regression tests
> for all the existing code it seems premature to add new features to
> llvm-dwarfdump before there's a use-case. It does sometimes mean bug
> fixes appear to be costly because they include adding the missing test
> infrastructure support, but that's essentially where the cost is
> anyway.

See test/MC/ELF/cfi-register.s for a test I consider unmaintainable
since it just matches an elf-dump and requires manual decoding of the
data for every change and addition. When tests are too hard to write,
fewer tests get written.

>> 2. Relying of assembly directive emissions (i.e. .cfi_*), which is
>> cumbersome and misses a lot of things like actual DWARF encoding.
>
> I'm not sure what you mean by "actual DWARF encoding" here.
> (disclaimer: I've only recently started dabbling with debug info, so I
> may be missing obvious things)

I mean that it doesn't test the whole way, and there's quite a bit of
DWARF-related functionality in MC. So when a test relies on matching
directives in ASM output, there's quite a bit of code in MC it doesn't
exercise.

>
>> 3. Using elf-dump and examining the raw binary dumps. This makes tests
>> nearly unmaintainable.
>>
>> The latter is also why IMHO our ELF emission in general isn't well
>> tested. elf-dump is just too rudimentary and relies on simple (=dumb)
>> binary contents dumps.
>>
>> The long-term solution for DWARF would be to enhance lib/DebugInfo to
>> the point where it can handle all interesting DWARF sections. But this
>> is a lofty goal, since DWARF parsing is notoriously hard and this
>> would require a large investment of time and effort. And in the
>> meantime, we just don't write good enough tests (and enough of them)
>> for this very important feature.
>
> Are there particular recent commits you've been concerned about the
> test quality of? I've been trying to keep an eye on this but, again,
> don't necessarily fully understand the ramifications of some changes.

See basically every test employing elf-dump for non-trivial things.

>
>> Therefore, as an interim stage, I propose to adopt some external tool
>> that parses DWARF and emits decoded textual dumps which makes tests
>> easy to write.
>>
>> Concretely, I have a pure Python library named pyelftools
>> (https://bitbucket.org/eliben/pyelftools) which provides comprehensive
>> ELF and DWARF parsing capabilities and has a dumper that's fully
>> compatible with the readelf command. Using pyelftools would allow us
>> to immediately improve the quality of our tests, and as lib/DebugInfo
>> matures llvm-dwarfdump can gradually replace the dumper without
>> changing the actual tests.
>
> I would be a little hesitant about test execution performance if
> involved invoking new python processes for each debug info test. But
> numbers could convince me. Beyond that I can't rationally claim any
> particular need to support llvm-dwarfdump as the tool of choice over
> any 3rd party tool.

This is already done with elf-dump (a Python script) which is used for
a lot of tests for lack better options.


Eli