[PATCH] D144101: [test-suite] Increase the --filter-short threshold

[Sjoerd Meijer via Phabricator via llvm-commits]

SjoerdMeijer created this revision.
SjoerdMeijer added reviewers: fhahn, arichardson, MatzeB, aemerson.
Herald added a subscriber: StephenFan.
Herald added a project: All.
SjoerdMeijer requested review of this revision.

This proposes to increase the --filter-short threshold from 0.6 to 1.0 seconds.  The reason is that the LLVM test-suite is quite noisy and this is one mitigation for that. I.e., if we filter out all apps with a runtime of less than a second, then the variation between runs is a lot less and we get less noisy results. I appreciate this replaces one arbitrary number with another arbitray number, but a runtime of at least 1 second for a benchmark seemed reasonable to me.

As a result, less results/benchmarks will be reported. On the system that I tested it, this means that for the MultiSource benchmarks the following apps will be excluded in addition to the ones that already got excluded with --filter-short:

  Program                                         exec_time
  Benchmarks/DOE-ProxyApps-C/CoMD/CoMD            0.92
  Benchmarks...VC/Expansion-flt/Expansion-flt     0.84
  Applications/viterbi/viterbi                    0.78
  Benchmarks...ataFlow-flt/GlobalDataFlow-flt     0.76
  Benchmarks/Trimaran/enc-rc4/enc-rc4             0.74
  Benchmarks/BitBench/five11/five11               0.97
  Benchmarks/Prolangs-C++/life/life               0.65
  Benchmarks...VC/Symbolics-flt/Symbolics-flt     0.73
  Benchmarks...Fhourstones-3.1/fhourstones3.1     0.69
  Benchmarks/VersaBench/dbms/dbms                 0.85
  Benchmarks.../DOE-ProxyApps-C++/CLAMR/CLAMR     0.81
  Benchmarks...roxyApps-C/SimpleMOC/SimpleMOC     0.88
  Benchmarks...alencing-dbl/Equivalencing-dbl     0.63
  Benchmarks/nbench/nbench                        0.89
  Benchmarks/ASCI_Purple/SMG2000/smg2000          0.94
  Benchmarks...aran/netbench-crc/netbench-crc     0.66


https://reviews.llvm.org/D144101

Files:
  utils/compare.py


Index: utils/compare.py
===================================================================
--- utils/compare.py
+++ utils/compare.py
@@ -154,7 +154,7 @@
 def filter_failed(data, key='Exec'):
     return data.loc[data[key] == "pass"]
 
-def filter_short(data, key='Exec_Time', threshold=0.6):
+def filter_short(data, key='Exec_Time', threshold=1.0):
     return data.loc[data[key] >= threshold]
 
 def filter_same_hash(data, key='hash'):


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D144101.497657.patch
Type: text/x-patch
Size: 438 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20230215/c086753f/attachment.bin>