[cfe-dev] Proposal for an ABI testsuite for clang

[Sean Silva]

On Wed, Jul 2, 2014 at 10:11 PM, Hal Finkel <hfinkel at anl.gov> wrote:

> ----- Original Message -----
> > From: "Sean Silva" <chisophugis at gmail.com>
> > To: "Sunil Srivastava" <sunil_srivastava at playstation.sony.com>
> > Cc: cfe-dev at cs.uiuc.edu
> > Sent: Wednesday, July 2, 2014 10:58:18 PM
> > Subject: Re: [cfe-dev] Proposal for an ABI testsuite for clang
> >
> >
> >
> >
> > Since the goal is compatibility with a previous version, why does the
> > test suite itself contain the "right answers" that it checks
> > against? Shouldn't the "right answers" be determined by a "golden
> > master" version of the toolchain?
> >
> >
> >
> > Roughly what I'm saying is instead of "check(..., expected)" instead
> > do "print(...)". Then compare what is printed with the output
> > produced by the program when compiled with the reference toolchain.
> > This approach seems more useful, since:
> >
> >
> > a) it tests what we actually care about, which is that the code is
> > compatible with a reference toolchain. At the end of the day,
> > compatibility with existing compiled code is more important than
> > actually sticking to the ABI specification.
>
> I understand why you say this, and comparing to a reference toolchain is
> also valuable, but having checks against the ABI specification itself is
> independently valuable. This is especially true when the Clang/LLVM
> toolchain *is* the vendor-provided reference toolchain. I support testing
> both modes, but I really like the testing-against-the-ABI-spec aspect of
> what was proposed. This would be a unique contribution to open-source
> compiler development in general, let alone LLVM.
>

The two things aren't mutually exclusive. If you want to check against the
spec with the approach I suggested, then just generate an output file which
is what you think is correct according to the spec and use that as a
baseline (instead of generating the baseline output file by compiling the
code in the test suite with a reference compiler).

-- Sean Silva


>
>  -Hal
>
> >
> >
> >
> > b) it lets you use whatever baseline you want. E.g. possibly some
> > internal toolchain that unfortunately shipped with a bug; or the
> > MSVC toolchain, which would be immediately useful for the folks
> > working on MS ABI compatibility. In fact, this test suite could
> > easily be used to create an instant checklist of incompatibilities
> > between *any* two compilers that claim to be compatible (not even
> > from the same vendor).
> >
> >
> > c) Allows expansion to any other target or ABI by simply expanding
> > the set of "every possible ..." to cover any extra details relevant
> > to the new target or ABI. Hence the test suite is naturally target
> > and ABI agnostic. Adding new checks for a different ABI
> > automatically strengthens the checks for all ABI's.
> >
> >
> > As an example of c), consider the following: Suppose that to extend
> > the test suite to MS ABI, you find that you need to generate a new
> > code construct that exercises a particular aspect of the MS mangling
> > which the existing Itanium cases didn't cover. If you compile this
> > construct with two different Itanium ABI compilers and the mangled
> > names differ, then that is an incompatibility.
> >
> >
> > In this sense, I think the goal of the ABI compatibility test suite
> > should be to collect/generate source code constructs that exercise a
> > particular dimension along which two C++ compilers could have an ABI
> > incompatibility. This is mostly agnostic to the target or the
> > compiler being used (a small amount of stuff would need some target
> > hooks, like getting a mangled symbol name, but the bulk should be
> > portable and platform agnostic).
> >
> >
> > I think that what you guys have already done (the steps you list
> > under "A set of ‘interesting’ classes for the test-suite is
> > generated by:") is excellent progress in this direction of
> > collecting examples that exercise dimensions of potential
> > incompatibility since by construction the Itanium ABI (or any ABI
> > spec) tries to cover all the relevant dimensions which could cause
> > an ABI incompatibility.
> >
> >
> > -- Sean Silva
> >
> >
> >
> >
> >
> >
> >
> > On Wed, Jul 2, 2014 at 4:51 PM, Srivastava, Sunil <
> > sunil_srivastava at playstation.sony.com > wrote:
> >
> >
> >
> >
> >
> >
> > Hi,
> >
> > At the SN Systems division of Sony, we have developed an IA64 ABI
> > test-suite for clang/llvm based on the lit framework. We would like
> > to submit this to the LLVM community. The test-suite currently
> > supports clang in both LP64/x86-64 and ILP32/x86 targets, with the
> > ability of adding others. The tests perform target-side execution
> > and work with both cross and native targets.
> >
> > Please find attached a pdf document that describes the design of this
> > test-suite. In summary, the test-suite covers:
> > · struct layout rules, including bit-fields,
> > · Object layout, base classes, vtables, VTTs, construction vtables,
> > · Name mangling,
> > · Contents of typeinfo variables,
> > · Array cookies,
> > · and a few other items.
> >
> > Please consider accepting this test-suite as an llvm project. We look
> > forward to addressing any comments and questions you have on this
> > topic.
> >
> > Best regards,
> >
> >
> >
> > Sunil Srivastava
> >
> > SN Systems / Sony Computer Entertainment
> >
> >
> > _______________________________________________
> > cfe-dev mailing list
> > cfe-dev at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >
> >
> >
> > _______________________________________________
> > cfe-dev mailing list
> > cfe-dev at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >
>
> --
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20140702/afed6800/attachment.html>