[PATCH] D75235: [debuginfo-tests][dexter] Add a test tool --calculate-average option

[Pierre van Houtryve via Phabricator via llvm-commits]

Pierre-vh created this revision.
Pierre-vh added a reviewer: jmorse.
Herald added a project: LLVM.
Herald added a subscriber: llvm-commits.

During our experiments with DExTer, we found that it lacked a way to calculate the average score of every test ran inside a directory. For instance, if I have a directory full of DExTer tests and I run them all using something like
`python3 dexter.py /foo/mytests --builder clang --debugger lldb --cflags="-g -O2"`, I have no way to know the average score of all the tests that I just ran. So, we implemented this feature  under`--calculate-average`, and we now propose to upstream it:

When using that option, DExTer will calculate the average score of every test run and print it after all the tests have run,.

Example output:

  $ python3 dexter/dexter.py test ~/dexter/tests/nostdlib/llvm_passes/GVN --builder clang --debugger lldb --calculate-average
  
  NumGVNEqProp: (0.1429)
  NumGVNLoad: (0.1429)
  NumGVNPRE: (0.1613)
  NumPRELoad: (0.1549)
  
  @avg: (0.1505)


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D75235

Files:
  debuginfo-tests/dexter/dex/tools/test/Tool.py


Index: debuginfo-tests/dexter/dex/tools/test/Tool.py
===================================================================
--- debuginfo-tests/dexter/dex/tools/test/Tool.py
+++ debuginfo-tests/dexter/dex/tools/test/Tool.py
@@ -6,6 +6,7 @@
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 """Test tool."""
 
+import math
 import os
 import csv
 import pickle
@@ -94,6 +95,9 @@
                             help='exit with status FAIL(2) if the test result'
                                 ' is less than this value.',
                             metavar='<float>')
+        parser.add_argument('--calculate-average',
+                            action="store_true",
+                            help='calculate the average score of every test run')
         super(Tool, self).add_tool_arguments(parser, defaults)
 
     def _build_test_case(self):
@@ -226,6 +230,19 @@
         if not options.verbose:
             self.context.o.auto('\n')
 
+        if options.calculate_average:
+            # Calculate and print the average score
+            score_sum = 0.0
+            num_tests = 0
+            for test_case in self._test_cases:
+                score = test_case.score
+                if not test_case.error and not math.isnan(score):
+                    score_sum += test_case.score
+                    num_tests += 1
+
+            if num_tests != 0:
+                print("@avg: ({:.4f})".format(score_sum/num_tests))
+
         summary_path = os.path.join(options.results_directory, 'summary.csv')
         with open(summary_path, mode='w', newline='') as fp:
             writer = csv.writer(fp, delimiter=',')


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D75235.246899.patch
Type: text/x-patch
Size: 1642 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20200227/4360b3e9/attachment.bin>