[cfe-dev] Proposal for an ABI testsuite for clang

[Renato Golin]

On 3 July 2014 06:02, Sean Silva <chisophugis at gmail.com> wrote:
>> > a) it tests what we actually care about, which is that the code is
>> > compatible with a reference toolchain. At the end of the day,
>> > compatibility with existing compiled code is more important than
>> > actually sticking to the ABI specification.
>>
>> I understand why you say this, and comparing to a reference toolchain is
>> also valuable, but having checks against the ABI specification itself is
>> independently valuable. This is especially true when the Clang/LLVM
>> toolchain *is* the vendor-provided reference toolchain. I support testing
>> both modes, but I really like the testing-against-the-ABI-spec aspect of
>> what was proposed. This would be a unique contribution to open-source
>> compiler development in general, let alone LLVM.
>
>
> The two things aren't mutually exclusive. If you want to check against the
> spec with the approach I suggested, then just generate an output file which
> is what you think is correct according to the spec and use that as a
> baseline (instead of generating the baseline output file by compiling the
> code in the test suite with a reference compiler).

Sean,

I have to strongly disagree with you up there. Testing the ABI is more
important than testing compatibility with other compilers. IF the
other compiler doesn't follow the ABI, it should be fixed, that's a no
brainer, and any decent compiler community will not argue against ABI
breakages. Testing against other compilers is *also* important, but
not to the detriment of ABI checking.

Also, checking the ABI against expected results make it explicit, and
helps a lot on debugging problems when they happen. An expected output
can have comments pointing out the ABI paragraph and document, as well
as having some regular expression / pattern matching structure that
"explains" the document in a programming-friendly way. This is
invaluable, and we don't normally have that because people don't
normally have time to write it. If someone did this, we should not
discard and request a poorer check.

But checking against another compiler using patterns and documentation
won't work, because there isn't any, just the good old diff. In that
case, I agree, diffing against a golden standard is the way to go, and
I also agree we should have that *in addition* to ABI tests.

My tuppence.

--renato