[PATCH] D16115: [test-suite] Add ClangAnalyzerBenchmarks directory to test-suite repository

[Devin Coughlin via llvm-commits]

dcoughlin created this revision.
dcoughlin added reviewers: MatzeB, cmatthews, ddunbar.
dcoughlin added subscribers: cfe-commits, llvm-commits.

Hi All,

This patch proposes adding a new directory in the llvm test-suite repository to hold projects and build commands to detect regressions in the clang static analyzer. This directory will be used by a Green Dragon build bot (discussed in <http://lists.llvm.org/pipermail/cfe-dev/2015-September/045199.html> that will periodically re-analyze the projects specified in this directory to detect when analysis results change unexpectedly.

Rather than check in the source code for these projects, we propose to instead check in (1) scripts to download, cache, and build the project and (2) expected reference results in plist form from running the analyzer over the project. There are pre-existing scripts in utils/analyzer in the clang project that understand the layout of this directory (see <http://reviews.llvm.org/D14345>). Internally we have found this approach to be effective at discovering unintended regressions on real-world projects.

http://reviews.llvm.org/D16115

Files:
  ClangAnalyzerBenchmarks/README.txt
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-0jQzwH.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-0leXPD.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-1SCq_n.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-1q8xEn.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-1rHTRW.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-25RkrY.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-3LOwRg.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-3eHerS.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-3wD6sI.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-4Itn6Q.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-5eYzQk.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-5xKe72.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-70n0rJ.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-AJaHrD.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-A_g1PK.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-BSiP9f.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-BSjjch.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-D86sqa.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-DNg7FR.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-ETVLuZ.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-FSqthW.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-FfhYET.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-Fk0wPl.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-GGuLER.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-Gd9Qk8.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-GzYIE8.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-Hxm3jG.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-IEZlZo.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-Jaelfi.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-Jao0Kv.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-KSlvnA.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-KZzQTG.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-KlLu18.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-LCQ9Nf.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-LFTkeC.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-LnaNYn.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-LtYUpC.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-NTClRG.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-O1rOZT.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-OQlhF8.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-Ok4dLX.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-PQUmRQ.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-QDn98D.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-ReQ_GL.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-RjfTIP.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-SnWTTb.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-StC4tP.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-SvYOOI.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-TVmrcU.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-TZ1H1p.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-TfLDsd.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-TigCu1.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-USeWg5.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-UTVx9V.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-VDaNX6.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-WJPjNs.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-WtKK5Z.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-XebXE5.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-YL5xpz.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-Ylm0Go.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-Zn3w9M.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-ZrBRO3.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-_6fciA.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-aJqoCc.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-ak55z2.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-b7Xvct.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-cI5v8I.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-cir2X2.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-d0igml.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-eqoyVD.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-fKa4Rk.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-gkuOso.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-hY88qG.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-i7I6_j.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-iBFRMw.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-j8J8u3.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-jP0jEK.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-jwkvXk.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-kX__b0.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-mSwyc0.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-mfm0do.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-oYIpvl.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-o_yGd7.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-ofrSfi.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-oijYTP.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-oxvtbW.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-ryurLe.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-sbBgmn.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-tCJJyr.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-u9OAPv.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-ufHmc_.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-ustv4K.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-v3HsbB.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-voOhUi.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-wa7yxJ.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-wjqQWg.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-xDsepu.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-z5FdWe.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/RefScanBuildResults/2016-01-11-134007-41609-1/report-zwVix9.plist
  ClangAnalyzerBenchmarks/openssl-1.0.0t/cleanup_run_static_analyzer.sh
  ClangAnalyzerBenchmarks/openssl-1.0.0t/download_project.sh
  ClangAnalyzerBenchmarks/openssl-1.0.0t/run_static_analyzer.cmd
  ClangAnalyzerBenchmarks/projectMap.csv

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D16115.44594.patch
Type: text/x-patch
Size: 2106986 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20160112/8042c4dc/attachment-0001.bin>