[cfe-dev] Proposal: Integrate static analysis test suites

[Anna Zaks via cfe-dev]

> On Dec 17, 2015, at 11:01 AM, <Alexander G. Riccio> <alexander at riccio.com> <Alexander G. Riccio> wrote:
> 
> However, if the goal is to have the tests
> because you would like to make efforts to have the compiler diagnose
> their cases properly, that's far more interesting and a good reason to
> bring in the tests.
> 
> That's exactly my intention. Improving the static analyzer to detect these cases, that will be interesting.
> placeholder text
> 
> If the other tests are not clearly licensed, we
> should try to get NIST to clarify the license of them before
> inclusion.
> 
> That sounds like the best idea, as a government agency, they almost certainly have lawyers.
> 
> I think the next step is to integrate the working (error correctly diagnosed) tests, only those that are obviously in the public domain, and propose them as a big batched patch. This shouldn't itself be controversial. 
> 
> How exactly do I submit a patch? I see that the LLVM developer policy says to send it to the mailing list (cfe-commits), but I also see that Phabricator comes into this somewhere <http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20151214/145026.html>?
> 

Devin has started writing scripts for running additional analyzer tests as described in this thread:
http://clang-developers.42468.n3.nabble.com/analyzer-Adding-build-bot-for-static-analyzer-reference-results-td4047770.html

The idea was to check out the tests/projects from the existing repos instead of copying them. Would it be possible to do the same with these tests?

Sorry for not replying sooner!
Anna.
> Sincerely,
> Alexander Riccio
> --
> "Change the world or go home."
> about.me/ariccio <http://about.me/ariccio>
> 
>  <http://about.me/ariccio>
> If left to my own devices, I will build more.
> ⁂
> 
> On Thu, Dec 10, 2015 at 4:04 PM, Aaron Ballman <aaron at aaronballman.com <mailto:aaron at aaronballman.com>> wrote:
> On Mon, Dec 7, 2015 at 9:50 PM, <Alexander G. Riccio> via cfe-dev
> <cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>> wrote:
> > First time Clang contributor here,
> >
> > I'd like to add the "C Test Suite for Source Code Analyzer v2", a
> > relatively small test suite (102 cases/flaws), some of which Clang
> > doesn't yet detect*. See link at bottom.
> >
> > Immediate questions:
> > 0. Does the Clang community/project like the idea?
> 
> I've included a few other devs (CCed) to get further opinions.
> 
> I like the idea of being able to diagnose the issues covered by the
> test suite, but I don't think including the test suite by itself is
> particularly useful without that goal in mind. Also, one question I
> would have has to do with the licensing of the tests themselves and
> whether we would need to do anything special there.
> 
> > 1. What's the procedure for including new tests? (not the technical,
> > but the community/project).
> 
> Getting the discussion going about the desired goal (as you are doing)
> is the right first step.
> 
> > 2. How do I include failing tests without breaking things? Some of
> > these tests will fail - that's why I'm proposing their inclusion - but
> > they shouldn't yet cause the regression testing system to complain.
> 
> Agreed, any test cases that are failing would have to fail gracefully.
> I assume that by failure, you mean "should diagnose in some way, but
> currently does not". I would probably split the tests into two types:
> one set of tests that properly diagnose the issue (can be checked with
> FileCheck or -verify, depending on the kind of tests we're talking
> about), and one set of tests where we do not diagnose, but want to see
> them someday (which can be tested with expect-no-diagnostics, for
> example). This way, we can ensure test cases continue to diagnose when
> we want them to, and we can be alerted when new diagnostics start to
> catch previously uncaught tests. This is assuming that it makes sense
> to include all of the tests at once, which may not make sense in
> practice.
> 
> > 3. How does Clang handle licensing of third party code? Some of these
> > tests are clearly in the public domain (developed at NIST, says "in
> > the public domain"), but others are less clearly licensed.
> 
> Oh look, you asked the same question I asked. ;-) If the tests are in
> the public domain and clearly state as such, I think we can go ahead
> and include them. If the other tests are not clearly licensed, we
> should try to get NIST to clarify the license of them before
> inclusion. Depending on the license, we may be able to include them
> under their original license. If we cannot clarify the license, I
> would guess that we simply should not include those tests as part of
> our test suite. Note: I could be totally wrong, IANAL. :-)
> 
> > Should the community accept that testsuite, and I successfully add
> > that test suite, then I'd like to step it up a bit, and include the
> > "Juliet Test Suite for C/C++". "Juliet" is a huge test suite by the
> > NSA Center for Assured Software & NIST's Software Assurance Metrics
> > And Tool Evaluation project, which has 25,477 test cases (!!) for 118
> > CWEs. I don't think any other open source compiler could compete with
> > Clang after this. There's a ton of literature on the "Juliet" suite,
> > and listing it here is not necessary.
> >
> > This project would be my first Clang contribution :)
> >
> > Personally, I'm interested in static analysis, and this is the first
> > step in understanding & improving Clang's static analysis
> > capabilities.
> >
> > I have some ideas on how to detect the currently undetected bugs, and
> > I'm curious to see where things lead.
> 
> Adding the tests by themselves is not necessarily interesting to the
> project unless they exercise the compiler in ways it's not currently
> being exercised. So just having tests for the sake of having the tests
> is not too useful (IMO). However, if the goal is to have the tests
> because you would like to make efforts to have the compiler diagnose
> their cases properly, that's far more interesting and a good reason to
> bring in the tests.
> 
> One possible approach if you are interested in having the compiler
> diagnose the cases is to bring the tests in one at a time. Start with
> the initial batch of "these are diagnosed properly", then move on to
> "this test is diagnosed properly because of this patch." Eventually
> we'll get to the stage where all of the tests are diagnosed properly.
> 
> > Secondary questions:
> > 1. How should I break the new tests up into patches? Should I just
> > whack the whole 102 case suite into a single patch, or a bunch of
> > smaller ones?
> 
> See comments above.
> 
> > 2. How does the Clang/LLVM static analysis testing infrastructure
> > work? I'm going to have to figure this out myself anyways, but where
> > should I start? Any tips on adding new tests?
> 
> http://clang-analyzer.llvm.org/checker_dev_manual.html <http://clang-analyzer.llvm.org/checker_dev_manual.html>
> 
> Another good place for some of these checkers may be clang-tidy, or
> the compiler frontend itself. It's likely to depend on case-by-case
> code patterns.
> 
> http://clang.llvm.org/extra/clang-tidy/ <http://clang.llvm.org/extra/clang-tidy/>
> 
> Thank you for looking into this!
> 
> ~Aaron
> 
> >
> > *If I remember correctly,
> > https://samate.nist.gov/SRD/view_testcase.php?tID=149055 <https://samate.nist.gov/SRD/view_testcase.php?tID=149055> passes
> > analysis without complaint. I manually spot checked a very small
> > number of tests.
> >
> > "C Test Suite for Source Code Analyzer v2" (valid code):
> > https://samate.nist.gov/SRD/view.php?tsID=101 <https://samate.nist.gov/SRD/view.php?tsID=101>
> > "C Test Suite for Source Code Analyzer v2" (invalid code):
> > https://samate.nist.gov/SRD/view.php?tsID=100 <https://samate.nist.gov/SRD/view.php?tsID=100>
> >
> > "Juliet Test Suite for C/C++" (files):
> > https://samate.nist.gov/SRD/testsuites/juliet/Juliet_Test_Suite_v1.2_for_C_Cpp.zip <https://samate.nist.gov/SRD/testsuites/juliet/Juliet_Test_Suite_v1.2_for_C_Cpp.zip>
> > "Juliet Test Suite for C/C++" (docs):
> > https://samate.nist.gov/SRD/resources/Juliet_Test_Suite_v1.2_for_C_Cpp_-_User_Guide.pdf <https://samate.nist.gov/SRD/resources/Juliet_Test_Suite_v1.2_for_C_Cpp_-_User_Guide.pdf>
> >
> >
> > Sincerely,
> > Alexander Riccio
> > _______________________________________________
> > cfe-dev mailing list
> > cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev>
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20151227/88bcf7e8/attachment.html>