[llvm-dev] RFC: Adding GCC C Torture Suite to External Test Suites

[Flash Sheridan via llvm-dev]

My somewhat ancient experience suggests that this is an excellent idea; C-Torture files have an uncanny ability to elicit interesting bugs even in an unrelated compiler.  The eeriest one it elicited during testing of the PalmSource ARM compiler occurred at the same optimization level (O2) as in GCC, even though our optimization levels meant something completely different (they went all the way up to O5, rather than O3), and our back- and middle-ends were licensed from well-respected commercial vendors.  (Details in my Software: Practice and Experience article [0].)
    One more painful piece of experience testing later compilers and languages:  Be cautious, and remain calm, when updating to the latest third-party tests.  Sometimes a clever and tricky new file, which elicited a bug now fixed in the target compiler, will also elicit a bug in yours.  It’s easy to overreact to this, as though it’s a sudden regression, rather than an old problem newly revealed.

___
[0] http://onlinelibrary.wiley.com/doi/10.1002/spe.812/abstract, 
preprint at http://pobox.com/~flash/Practical_Testing_of_C99.pdf, 
compiler testing bibliography: http://pobox.com/~flash/compiler_testing_bibliography.html.

- - -
Flash Sheridan
Compiler Quality Assurance Lead at Axoni for AxLang (http://axoni.com/axlang)
http://pobox.com/~flash
http://www.linkedin.com/in/flashsheridan

We’re hiring, e.g. compiler engineer: https://jobs.lever.co/axoni/1356bb24-c82f-4a0f-a6f5-783d6a265d25?lever-via=9vamXsNUHv
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190906/b8231555/attachment.html>