[cfe-dev] Proposal for an ABI testsuite for clang

[Sean Silva]

On Thu, Jul 3, 2014 at 2:41 AM, Renato Golin <renato.golin at linaro.org>
wrote:

> On 3 July 2014 06:02, Sean Silva <chisophugis at gmail.com> wrote:
> >> > a) it tests what we actually care about, which is that the code is
> >> > compatible with a reference toolchain. At the end of the day,
> >> > compatibility with existing compiled code is more important than
> >> > actually sticking to the ABI specification.
> >>
> >> I understand why you say this, and comparing to a reference toolchain is
> >> also valuable, but having checks against the ABI specification itself is
> >> independently valuable. This is especially true when the Clang/LLVM
> >> toolchain *is* the vendor-provided reference toolchain. I support
> testing
> >> both modes, but I really like the testing-against-the-ABI-spec aspect of
> >> what was proposed. This would be a unique contribution to open-source
> >> compiler development in general, let alone LLVM.
> >
> >
> > The two things aren't mutually exclusive. If you want to check against
> the
> > spec with the approach I suggested, then just generate an output file
> which
> > is what you think is correct according to the spec and use that as a
> > baseline (instead of generating the baseline output file by compiling the
> > code in the test suite with a reference compiler).
>
> Sean,
>
> I have to strongly disagree with you up there. Testing the ABI is more
> important than testing compatibility with other compilers.


They are both important. My suggestion addresses both of them (see below).
Think of my suggestion as a primitive that can be used for both purposes
(see at the end of this post also).

However, in the OP's PDF, it says that what actually motivated this work is
consistency with previous compilers: "In some industries, *consistency is
more important than correctness*. Hence the need for a test-suite to *guard
against changes*." (emphasis mine)
I am not involved with this project internally at Sony, but I know that an
extremely important use case for our clients is binary-only libraries.
Hence ensuring ABI compatibility with previous platform compilers is
important.

The ability to use a "spec" baseline or a "golden master" baseline by just
swapping the baseline file is key to reconciling the two conflicting
desires:
1. the open source project wants to be as "by-the-spec" ABI-compliant as
possible
2. maintainers of platform compilers want to ensure that their platform
compilers are compatible with previous code compiled for the platform.

Since it seems that #2 is the thing that actually motivated this work, I
think it is important to take that into account.

Also keep in mind that the test-suite generator program that generates the
"right answers by the spec" is also subject to human fallibility in the
same way as the the ABI code and tests inside clang itself (although the
narrower focus of the generator might reduce the chance of error). The
extent to which this test suite actually provides an independent check on
Clang's ABI correctness depends on how it was developed; I would be
interested to know:

- how many times during the development of this test suite clang's behavior
diverged from the "expected" behavior as computed by the generator program,
and the generator program was later found to be in error.

- (less likely) how many times the generator program produced a test that
happened to align with a "by-the-spec incorrect" bug in clang, and the
agreement was interpreted as both being correct. (we can't really know this
number, of course)

- what would happen if this test suite does indeed find a bug with a
previous reference toolchain? Would that necessitate an internal fork of
the test suite that has a deliberately "incorrect" (by the spec) test?

I think that checking against another independently developed compiler
targeting the same ABI will provide at least as much of an independent
check on Clang's correctness as a written-from-scratch generator program,
if not moreso. Of course, checking two compilers against a generator
transitively gives you that benefit, but is less accommodating of the
requirement of checking actual code compatibility against a given baseline
(that may have a bug; and would you bet that it doesn't?).


> IF the
> other compiler doesn't follow the ABI, it should be fixed, that's a no
> brainer, and any decent compiler community will not argue against ABI
> breakages. Testing against other compilers is *also* important, but
> not to the detriment of ABI checking.
>
> Also, checking the ABI against expected results make it explicit, and
> helps a lot on debugging problems when they happen. An expected output
> can have comments pointing out the ABI paragraph and document, as well
> as having some regular expression / pattern matching structure that
> "explains" the document in a programming-friendly way.


I don't think that what you are asking for is mutually exclusive with what
I'm suggesting. I think you're imagining the baseline output as just a flat
file containing a bunch of meaningless numbers printed in it. Realistically
each item printed in the output file would likely be keyed on a unique
identifier of some sort, and it would simply be a matter of going to the
place in the test file that contains that unique identifier. The citation
comments that you are suggesting could reside on the source code in exactly
the way you suggest. I'm not sure what you mean by '"explains" the document
in a programming-friendly way' so I can't comment on that.

-- Sean Silva


> This is
> invaluable, and we don't normally have that because people don't
> normally have time to write it. If someone did this, we should not
> discard and request a poorer check.
>
> But checking against another compiler using patterns and documentation
> won't work, because there isn't any, just the good old diff. In that
> case, I agree, diffing against a golden standard is the way to go, and
> I also agree we should have that *in addition* to ABI tests.
>
> My tuppence.
>
> --renato
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20140703/6ea194a3/attachment.html>