<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Dec 17, 2015, at 11:01 AM, <Alexander G. Riccio> <<a href="mailto:alexander@riccio.com" class="">alexander@riccio.com</a>> <Alexander G. Riccio> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">However, if the goal is to have the tests<br class="">because you would like to make efforts to have the compiler diagnose<br class="">their cases properly, that's far more interesting and a good reason to<br class="">bring in the tests.</blockquote><div class=""><br class=""></div><div class="">That's exactly my intention. Improving the static analyzer to detect these cases, that will be interesting.</div><div class="">placeholder text</div><div class=""><br class=""></div><div class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">If the other tests are not clearly licensed, we<br class="">should try to get NIST to clarify the license of them before<br class="">inclusion.</blockquote></div><div class=""><br class=""></div><div class="">That sounds like the best idea, as a government agency, they almost certainly have lawyers.</div><div class=""><br class=""></div><div class="">I think the next step is to integrate the working (error correctly diagnosed) tests, only those that are obviously in the public domain, and propose them as a big batched patch. This shouldn't itself be controversial. </div><div class=""><br class=""></div><div class="">How exactly do I submit a patch? I see that the LLVM developer policy says to send it to the mailing list (cfe-commits), but I also see that <a href="http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20151214/145026.html" class="">Phabricator comes into this somewhere</a>?</div></div><div class="gmail_extra"><br clear="all" class=""></div></div></blockquote><br class=""><div>Devin has started writing scripts for running additional analyzer tests as described in this thread:</div><div><a href="http://clang-developers.42468.n3.nabble.com/analyzer-Adding-build-bot-for-static-analyzer-reference-results-td4047770.html" class="">http://clang-developers.42468.n3.nabble.com/analyzer-Adding-build-bot-for-static-analyzer-reference-results-td4047770.html</a></div></div><div><div><br class=""></div><div>The idea was to check out the tests/projects from the existing repos instead of copying them. Would it be possible to do the same with these tests?</div><div><br class=""></div><div>Sorry for not replying sooner!</div><div>Anna.</div><blockquote type="cite" class=""><div class="gmail_extra"><div class=""><div class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div dir="ltr" class="">Sincerely,<br class="">Alexander Riccio<br class="">--<br class="">"Change the world or go home."<div class=""><a href="http://about.me/ariccio" target="_blank" class="">about.me/ariccio</a></div><div class=""><a href="http://about.me/ariccio" target="_blank" class=""><br class=""></a></div><div class="">If left to my own devices, I will build more.</div><div class="">⁂<br class=""></div></div></div></div></div></div></div>
<br class=""><div class="gmail_quote">On Thu, Dec 10, 2015 at 4:04 PM, Aaron Ballman <span dir="ltr" class=""><<a href="mailto:aaron@aaronballman.com" target="_blank" class="">aaron@aaronballman.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Mon, Dec 7, 2015 at 9:50 PM, <Alexander G. Riccio> via cfe-dev<br class="">
<<a href="mailto:cfe-dev@lists.llvm.org" class="">cfe-dev@lists.llvm.org</a>> wrote:<br class="">
> First time Clang contributor here,<br class="">
><br class="">
> I'd like to add the "C Test Suite for Source Code Analyzer v2", a<br class="">
> relatively small test suite (102 cases/flaws), some of which Clang<br class="">
> doesn't yet detect*. See link at bottom.<br class="">
><br class="">
> Immediate questions:<br class="">
> 0. Does the Clang community/project like the idea?<br class="">
<br class="">
</span>I've included a few other devs (CCed) to get further opinions.<br class="">
<br class="">
I like the idea of being able to diagnose the issues covered by the<br class="">
test suite, but I don't think including the test suite by itself is<br class="">
particularly useful without that goal in mind. Also, one question I<br class="">
would have has to do with the licensing of the tests themselves and<br class="">
whether we would need to do anything special there.<br class="">
<span class=""><br class="">
> 1. What's the procedure for including new tests? (not the technical,<br class="">
> but the community/project).<br class="">
<br class="">
</span>Getting the discussion going about the desired goal (as you are doing)<br class="">
is the right first step.<br class="">
<span class=""><br class="">
> 2. How do I include failing tests without breaking things? Some of<br class="">
> these tests will fail - that's why I'm proposing their inclusion - but<br class="">
> they shouldn't yet cause the regression testing system to complain.<br class="">
<br class="">
</span>Agreed, any test cases that are failing would have to fail gracefully.<br class="">
I assume that by failure, you mean "should diagnose in some way, but<br class="">
currently does not". I would probably split the tests into two types:<br class="">
one set of tests that properly diagnose the issue (can be checked with<br class="">
FileCheck or -verify, depending on the kind of tests we're talking<br class="">
about), and one set of tests where we do not diagnose, but want to see<br class="">
them someday (which can be tested with expect-no-diagnostics, for<br class="">
example). This way, we can ensure test cases continue to diagnose when<br class="">
we want them to, and we can be alerted when new diagnostics start to<br class="">
catch previously uncaught tests. This is assuming that it makes sense<br class="">
to include all of the tests at once, which may not make sense in<br class="">
practice.<br class="">
<span class=""><br class="">
> 3. How does Clang handle licensing of third party code? Some of these<br class="">
> tests are clearly in the public domain (developed at NIST, says "in<br class="">
> the public domain"), but others are less clearly licensed.<br class="">
<br class="">
</span>Oh look, you asked the same question I asked. ;-) If the tests are in<br class="">
the public domain and clearly state as such, I think we can go ahead<br class="">
and include them. If the other tests are not clearly licensed, we<br class="">
should try to get NIST to clarify the license of them before<br class="">
inclusion. Depending on the license, we may be able to include them<br class="">
under their original license. If we cannot clarify the license, I<br class="">
would guess that we simply should not include those tests as part of<br class="">
our test suite. Note: I could be totally wrong, IANAL. :-)<br class="">
<span class=""><br class="">
> Should the community accept that testsuite, and I successfully add<br class="">
> that test suite, then I'd like to step it up a bit, and include the<br class="">
> "Juliet Test Suite for C/C++". "Juliet" is a huge test suite by the<br class="">
> NSA Center for Assured Software & NIST's Software Assurance Metrics<br class="">
> And Tool Evaluation project, which has 25,477 test cases (!!) for 118<br class="">
> CWEs. I don't think any other open source compiler could compete with<br class="">
> Clang after this. There's a ton of literature on the "Juliet" suite,<br class="">
> and listing it here is not necessary.<br class="">
><br class="">
> This project would be my first Clang contribution :)<br class="">
><br class="">
> Personally, I'm interested in static analysis, and this is the first<br class="">
> step in understanding & improving Clang's static analysis<br class="">
> capabilities.<br class="">
><br class="">
> I have some ideas on how to detect the currently undetected bugs, and<br class="">
> I'm curious to see where things lead.<br class="">
<br class="">
</span>Adding the tests by themselves is not necessarily interesting to the<br class="">
project unless they exercise the compiler in ways it's not currently<br class="">
being exercised. So just having tests for the sake of having the tests<br class="">
is not too useful (IMO). However, if the goal is to have the tests<br class="">
because you would like to make efforts to have the compiler diagnose<br class="">
their cases properly, that's far more interesting and a good reason to<br class="">
bring in the tests.<br class="">
<br class="">
One possible approach if you are interested in having the compiler<br class="">
diagnose the cases is to bring the tests in one at a time. Start with<br class="">
the initial batch of "these are diagnosed properly", then move on to<br class="">
"this test is diagnosed properly because of this patch." Eventually<br class="">
we'll get to the stage where all of the tests are diagnosed properly.<br class="">
<span class=""><br class="">
> Secondary questions:<br class="">
> 1. How should I break the new tests up into patches? Should I just<br class="">
> whack the whole 102 case suite into a single patch, or a bunch of<br class="">
> smaller ones?<br class="">
<br class="">
</span>See comments above.<br class="">
<span class=""><br class="">
> 2. How does the Clang/LLVM static analysis testing infrastructure<br class="">
> work? I'm going to have to figure this out myself anyways, but where<br class="">
> should I start? Any tips on adding new tests?<br class="">
<br class="">
</span><a href="http://clang-analyzer.llvm.org/checker_dev_manual.html" rel="noreferrer" target="_blank" class="">http://clang-analyzer.llvm.org/checker_dev_manual.html</a><br class="">
<br class="">
Another good place for some of these checkers may be clang-tidy, or<br class="">
the compiler frontend itself. It's likely to depend on case-by-case<br class="">
code patterns.<br class="">
<br class="">
<a href="http://clang.llvm.org/extra/clang-tidy/" rel="noreferrer" target="_blank" class="">http://clang.llvm.org/extra/clang-tidy/</a><br class="">
<br class="">
Thank you for looking into this!<br class="">
<br class="">
~Aaron<br class="">
<span class=""><br class="">
><br class="">
> *If I remember correctly,<br class="">
> <a href="https://samate.nist.gov/SRD/view_testcase.php?tID=149055" rel="noreferrer" target="_blank" class="">https://samate.nist.gov/SRD/view_testcase.php?tID=149055</a> passes<br class="">
> analysis without complaint. I manually spot checked a very small<br class="">
> number of tests.<br class="">
><br class="">
> "C Test Suite for Source Code Analyzer v2" (valid code):<br class="">
> <a href="https://samate.nist.gov/SRD/view.php?tsID=101" rel="noreferrer" target="_blank" class="">https://samate.nist.gov/SRD/view.php?tsID=101</a><br class="">
> "C Test Suite for Source Code Analyzer v2" (invalid code):<br class="">
> <a href="https://samate.nist.gov/SRD/view.php?tsID=100" rel="noreferrer" target="_blank" class="">https://samate.nist.gov/SRD/view.php?tsID=100</a><br class="">
><br class="">
> "Juliet Test Suite for C/C++" (files):<br class="">
> <a href="https://samate.nist.gov/SRD/testsuites/juliet/Juliet_Test_Suite_v1.2_for_C_Cpp.zip" rel="noreferrer" target="_blank" class="">https://samate.nist.gov/SRD/testsuites/juliet/Juliet_Test_Suite_v1.2_for_C_Cpp.zip</a><br class="">
> "Juliet Test Suite for C/C++" (docs):<br class="">
> <a href="https://samate.nist.gov/SRD/resources/Juliet_Test_Suite_v1.2_for_C_Cpp_-_User_Guide.pdf" rel="noreferrer" target="_blank" class="">https://samate.nist.gov/SRD/resources/Juliet_Test_Suite_v1.2_for_C_Cpp_-_User_Guide.pdf</a><br class="">
><br class="">
><br class="">
> Sincerely,<br class="">
> Alexander Riccio<br class="">
</span>> _______________________________________________<br class="">
> cfe-dev mailing list<br class="">
> <a href="mailto:cfe-dev@lists.llvm.org" class="">cfe-dev@lists.llvm.org</a><br class="">
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br class="">
</blockquote></div><br class=""></div>
</blockquote></div><br class=""></body></html>