[cfe-dev] Proposal: Integrate static analysis test suites

[<Alexander G. Riccio> via cfe-dev]

First time Clang contributor here,

I'd like to add the "C Test Suite for Source Code Analyzer v2", a
relatively small test suite (102 cases/flaws), some of which Clang
doesn't yet detect*. See link at bottom.

Immediate questions:
0. Does the Clang community/project like the idea?
1. What's the procedure for including new tests? (not the technical,
but the community/project).
2. How do I include failing tests without breaking things? Some of
these tests will fail - that's why I'm proposing their inclusion - but
they shouldn't yet cause the regression testing system to complain.
3. How does Clang handle licensing of third party code? Some of these
tests are clearly in the public domain (developed at NIST, says "in
the public domain"), but others are less clearly licensed.


Should the community accept that testsuite, and I successfully add
that test suite, then I'd like to step it up a bit, and include the
"Juliet Test Suite for C/C++". "Juliet" is a huge test suite by the
NSA Center for Assured Software & NIST's Software Assurance Metrics
And Tool Evaluation project, which has 25,477 test cases (!!) for 118
CWEs. I don't think any other open source compiler could compete with
Clang after this. There's a ton of literature on the "Juliet" suite,
and listing it here is not necessary.

This project would be my first Clang contribution :)

Personally, I'm interested in static analysis, and this is the first
step in understanding & improving Clang's static analysis
capabilities.

I have some ideas on how to detect the currently undetected bugs, and
I'm curious to see where things lead.


Secondary questions:
1. How should I break the new tests up into patches? Should I just
whack the whole 102 case suite into a single patch, or a bunch of
smaller ones?
2. How does the Clang/LLVM static analysis testing infrastructure
work? I'm going to have to figure this out myself anyways, but where
should I start? Any tips on adding new tests?

*If I remember correctly,
https://samate.nist.gov/SRD/view_testcase.php?tID=149055 passes
analysis without complaint. I manually spot checked a very small
number of tests.

"C Test Suite for Source Code Analyzer v2" (valid code):
https://samate.nist.gov/SRD/view.php?tsID=101
"C Test Suite for Source Code Analyzer v2" (invalid code):
https://samate.nist.gov/SRD/view.php?tsID=100

"Juliet Test Suite for C/C++" (files):
https://samate.nist.gov/SRD/testsuites/juliet/Juliet_Test_Suite_v1.2_for_C_Cpp.zip
"Juliet Test Suite for C/C++" (docs):
https://samate.nist.gov/SRD/resources/Juliet_Test_Suite_v1.2_for_C_Cpp_-_User_Guide.pdf


Sincerely,
Alexander Riccio