[zorg] r281761 - [zorg] Change static analyzer benchmark jenkins bot to be non-strict.

[Devin Coughlin via llvm-commits]

Author: dcoughlin
Date: Fri Sep 16 13:28:49 2016
New Revision: 281761

URL: http://llvm.org/viewvc/llvm-project?rev=281761&view=rev
Log:
[zorg] Change static analyzer benchmark jenkins bot to be non-strict.

For now, we'll just check to make sure there are no failures/crashes rather
than that the analysis results exactly match the expected results.

The analysis results are heavily dependent on headers, so this will make it
easier to reproduce and troubleshoot bot failures without needing to
exactly recreate SDK/headers on the bots.

Modified:
    zorg/trunk/test/jenkins/test_build.py
    zorg/trunk/zorg/jenkins/build.py

Modified: zorg/trunk/test/jenkins/test_build.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/test/jenkins/test_build.py?rev=281761&r1=281760&r2=281761&view=diff
==============================================================================
--- zorg/trunk/test/jenkins/test_build.py (original)
+++ zorg/trunk/test/jenkins/test_build.py Fri Sep 16 13:28:49 2016
@@ -150,7 +150,7 @@
 # RUN: python %{src_root}/zorg/jenkins/build.py static-analyzer-benchmarks | FileCheck --check-prefix CHECK-STATIC-ANALYZER-BENCHMARKS %s
 # CHECK-STATIC-ANALYZER-BENCHMARKS: @@@ Static Analyzer Benchmarks @@@
 # CHECK-STATIC-ANALYZER-BENCHMARKS: cd [[WORKSPACE:.*]]/test-suite-ClangAnalyzer/
-# CHECK-STATIC-ANALYZER-BENCHMARKS: '[[WORKSPACE]]/utils-analyzer/SATestBuild.py' '--strictness' '2'
+# CHECK-STATIC-ANALYZER-BENCHMARKS: '[[WORKSPACE]]/utils-analyzer/SATestBuild.py' '--strictness' '0'
 # CHECK-STATIC-ANALYZER-BENCHMARKS: @@@@@@
 
 # RUN: python %{src_root}/zorg/jenkins/build.py cmake all --globalisel | FileCheck --check-prefix CHECK-GISEL %s

Modified: zorg/trunk/zorg/jenkins/build.py
URL: http://llvm.org/viewvc/llvm-project/zorg/trunk/zorg/jenkins/build.py?rev=281761&r1=281760&r2=281761&view=diff
==============================================================================
--- zorg/trunk/zorg/jenkins/build.py (original)
+++ zorg/trunk/zorg/jenkins/build.py Fri Sep 16 13:28:49 2016
@@ -476,7 +476,7 @@ def static_analyzer_benchmarks_builder()
                               old_path)
 
   benchmark_cmd = [benchmark_script,
-                   "--strictness", "2"
+                   "--strictness", "0"
                   ]
   run_cmd(benchmarks_dir, benchmark_cmd, env=env)