[cfe-dev] Proposal for an ABI testsuite for clang

[Srivastava, Sunil]

Sean Silva wrote:

?  Roughly what I'm saying is instead of "check(..., expected)" instead do "print(...)". Then compare what is printed with the output produced by the program when compiled with the reference toolchain. This approach seems more useful...

Good point. In fact the tests do have a verbose mode where they print one-line-per-check, even when they pass, giving details including the line number in the test. That can be saved as the 'golden master' to compare against future compilers, or possibly even to modify the test to pass on your compiler. This does not work in every case, but we plan to improve that.


?  Suppose that to extend the test suite to MS ABI...
I wouldn't be that ambitious. The checking code relies on the basic concepts like IA64 vtbls and VTTs. As they exist now, most tests will probably fail to link with MSVC, or crash badly. It will be significant amount of work to extend this to MS ABI.

David Majnemer wrote:

?  I do hope that the intent is not to ensure that clang has bug-for-bug compatibility with earlier versions.  We've fixed many bugs over time which would necessitate ABI breaks from earlier versions of clang.
We do not presume to impose any such restriction on future changes, even though many compiler vendors maintain such bug-to-bug compatibility. If clang undergoes an ABI breaking change, the tests will have to change. We have tackled similar changes already internally.

Nico Weber wrote:

?  1.) How much of your 400 test files is from the manual abi method, how much from your test case generator?
About 20% are hand written

?  2.) Test runtime probably scales fairly well with the number of cores, right? You say it takes one hour to run on one core, so an 8-core system should take less than 10 minutes?
Yes. It scales very well. Memory or disk contention may make it less than linear, but still very well.

?  3.) How many bugs did you find with this so far?
I believe there were, but I can't give a number since I was not directly involved in reporting them. Perhaps Paul Robinson or Katya can answer that.

Sunil Srivastava
SN Systems / Sony Computer Entertainment


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20140705/ed530602/attachment.html>