[cfe-dev] Proposal for an ABI testsuite for clang

[Hal Finkel]

----- Original Message -----
> From: "Sean Silva" <chisophugis at gmail.com>
> To: "Sunil Srivastava" <sunil_srivastava at playstation.sony.com>
> Cc: cfe-dev at cs.uiuc.edu
> Sent: Wednesday, July 2, 2014 10:58:18 PM
> Subject: Re: [cfe-dev] Proposal for an ABI testsuite for clang
> 
> 
> 
> 
> Since the goal is compatibility with a previous version, why does the
> test suite itself contain the "right answers" that it checks
> against? Shouldn't the "right answers" be determined by a "golden
> master" version of the toolchain?
> 
> 
> 
> Roughly what I'm saying is instead of "check(..., expected)" instead
> do "print(...)". Then compare what is printed with the output
> produced by the program when compiled with the reference toolchain.
> This approach seems more useful, since:
> 
> 
> a) it tests what we actually care about, which is that the code is
> compatible with a reference toolchain. At the end of the day,
> compatibility with existing compiled code is more important than
> actually sticking to the ABI specification.

I understand why you say this, and comparing to a reference toolchain is also valuable, but having checks against the ABI specification itself is independently valuable. This is especially true when the Clang/LLVM toolchain *is* the vendor-provided reference toolchain. I support testing both modes, but I really like the testing-against-the-ABI-spec aspect of what was proposed. This would be a unique contribution to open-source compiler development in general, let alone LLVM.

 -Hal

> 
> 
> 
> b) it lets you use whatever baseline you want. E.g. possibly some
> internal toolchain that unfortunately shipped with a bug; or the
> MSVC toolchain, which would be immediately useful for the folks
> working on MS ABI compatibility. In fact, this test suite could
> easily be used to create an instant checklist of incompatibilities
> between *any* two compilers that claim to be compatible (not even
> from the same vendor).
> 
> 
> c) Allows expansion to any other target or ABI by simply expanding
> the set of "every possible ..." to cover any extra details relevant
> to the new target or ABI. Hence the test suite is naturally target
> and ABI agnostic. Adding new checks for a different ABI
> automatically strengthens the checks for all ABI's.
> 
> 
> As an example of c), consider the following: Suppose that to extend
> the test suite to MS ABI, you find that you need to generate a new
> code construct that exercises a particular aspect of the MS mangling
> which the existing Itanium cases didn't cover. If you compile this
> construct with two different Itanium ABI compilers and the mangled
> names differ, then that is an incompatibility.
> 
> 
> In this sense, I think the goal of the ABI compatibility test suite
> should be to collect/generate source code constructs that exercise a
> particular dimension along which two C++ compilers could have an ABI
> incompatibility. This is mostly agnostic to the target or the
> compiler being used (a small amount of stuff would need some target
> hooks, like getting a mangled symbol name, but the bulk should be
> portable and platform agnostic).
> 
> 
> I think that what you guys have already done (the steps you list
> under "A set of ‘interesting’ classes for the test-suite is
> generated by:") is excellent progress in this direction of
> collecting examples that exercise dimensions of potential
> incompatibility since by construction the Itanium ABI (or any ABI
> spec) tries to cover all the relevant dimensions which could cause
> an ABI incompatibility.
> 
> 
> -- Sean Silva
> 
> 
> 
> 
> 
> 
> 
> On Wed, Jul 2, 2014 at 4:51 PM, Srivastava, Sunil <
> sunil_srivastava at playstation.sony.com > wrote:
> 
> 
> 
> 
> 
> 
> Hi,
> 
> At the SN Systems division of Sony, we have developed an IA64 ABI
> test-suite for clang/llvm based on the lit framework. We would like
> to submit this to the LLVM community. The test-suite currently
> supports clang in both LP64/x86-64 and ILP32/x86 targets, with the
> ability of adding others. The tests perform target-side execution
> and work with both cross and native targets.
> 
> Please find attached a pdf document that describes the design of this
> test-suite. In summary, the test-suite covers:
> · struct layout rules, including bit-fields,
> · Object layout, base classes, vtables, VTTs, construction vtables,
> · Name mangling,
> · Contents of typeinfo variables,
> · Array cookies,
> · and a few other items.
> 
> Please consider accepting this test-suite as an llvm project. We look
> forward to addressing any comments and questions you have on this
> topic.
> 
> Best regards,
> 
> 
> 
> Sunil Srivastava
> 
> SN Systems / Sony Computer Entertainment
> 
> 
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> 
> 
> 
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory