[PATCH] D17717: [LNT][test-suite] Implement a new option: --single-result

[James Molloy via llvm-commits]

jmolloy created this revision.
jmolloy added reviewers: cmatthews, MatzeB, kristof.beyls.
jmolloy added a subscriber: llvm-commits.
jmolloy set the repository for this revision to rL LLVM.
Herald added a subscriber: aemerson.

The idea behind this option is to ease bisecting using tools like 'llvmlab bisect'. With --only-test we can already narrow down the tests we want to run to one test, however we still produce a report.json and exit success no matter if the test passed or failed.

With --single-result, we only perform one test (it sets --only-test) but the exit status of LNT is determined by a predicate which allows it to be used directly with llvmlab exec / llvmlab bisect.

The predicate is set with --single-result-predicate, and is a python expression. It is evaluated in a context that contains 'status', a boolean representing the pass or fail status of the test, and all of the metrics exposed by LIT. Where metrics have different names in LIT and LNT parlance (for example 'exec' in LNT and 'exec_time' in LIT) both names are exposed. This has a sideeffect of working around a feature of Python - 'exec' is a keyword so cannot be used as a variable!

The default predicate is simply "status". This causes the exit status of LNT to correspond to the pass/fail status of the test - useful for conformance testing. Predicates such as "status and exec_time > 6.0" allow for simple performance bisections.

We've been using this feature internally for a week or so now and have found our bisections have become a lot easier. An example bisect command would be:


```
$ llvmlab bisect -b ARM-aarch32 --min-rev=261265 --max-rev=261369 \
    lnt runtest test-suite \
    --cc '%(path)s/bin/clang' \
    --sandbox SANDBOX \
    --test-suite /work/llvm-test-suite \
    --use-lit lit \
    --run-under 'taskset -c 5' \
    --cflags '-O3 -mthumb -mcpu=cortex-a57' \
    --single-result MultiSource/Benchmarks/TSVC/Expansion-flt/Expansion-flt.test \
    --single-result-predicate 'exec_time > 8.0'
```

Repository:
  rL LLVM

http://reviews.llvm.org/D17717

Files:
  lnt/tests/test_suite.py

Index: lnt/tests/test_suite.py
===================================================================
--- lnt/tests/test_suite.py
+++ lnt/tests/test_suite.py
@@ -1,4 +1,4 @@
-import subprocess, tempfile, json, os, shlex, platform, pipes
+import subprocess, tempfile, json, os, shlex, platform, pipes, sys
 
 from optparse import OptionParser, OptionGroup
 
@@ -151,6 +151,20 @@
                          help="Do not submit the stat of this type [%default]",
                          action='append', choices=KNOWN_SAMPLE_KEYS,
                          type='choice', default=[])
+        group.add_option("", "--single-result", dest="single_result",
+                         help=("only execute one test and apply "
+                               "--single-result-predicate to calculate the "
+                               "exit status. Do not submit to a LNT instance "
+                               "or create a JSON report (useful for bisection)"))
+        group.add_option("", "--single-result-predicate",
+                         dest="single_result_predicate",
+                         help=("the predicate to apply to calculate the exit "
+                               "status (with --single-result). This is a Python "
+                               "expression that will be eval()ed with all metrics "
+                               "available as variables and a boolean 'status'. "
+                               "Example: \n"
+                               "  status and exec_time < 3.0"),
+                         default="status")
         parser.add_option_group(group)
 
         group = OptionGroup(parser, "Test tools")
@@ -213,6 +227,9 @@
                     "invalid --test-externals argument, does not exist: %r" % (
                         opts.test_suite_externals,))
 
+        if self.opts.single_result:
+            self.opts.only_test = os.path.dirname(self.opts.single_result)
+                
         opts.cmake = resolve_command_path(opts.cmake)
         if not isexecfile(opts.cmake):
             parser.error("CMake tool not found (looked for %s)" % opts.cmake)
@@ -465,6 +482,19 @@
             raw_name = test_data['name'].split(' :: ', 1)[1]
             name = 'nts.' + raw_name.rsplit('.test', 1)[0]
             is_pass = self._is_pass_code(test_data['code'])
+
+            # If --single-result is given, exit based on --single-result-predicate
+            if self.opts.single_result and \
+               raw_name == self.opts.single_result+'.test':
+                env = {'status': is_pass}
+                if 'metrics' in test_data:
+                    for k,v in test_data['metrics'].items():
+                        env[k] = v
+                        if k in LIT_METRIC_TO_LNT:
+                            env[LIT_METRIC_TO_LNT[k]] = v
+                status = eval(self.opts.single_result_predicate, {}, env)
+                sys.exit(0 if status else 1)
+
             if 'metrics' in test_data:
                 for k,v in test_data['metrics'].items():
                     if k not in LIT_METRIC_TO_LNT or LIT_METRIC_TO_LNT[k] in ignore:
@@ -487,6 +517,10 @@
                                             [self._get_lnt_code(test_data['code'])],
                                             test_info))
 
+        if self.opts.single_result:
+            # If we got this far, the result we were looking for didn't exist.
+            raise RuntimeError("Result %s did not exist!" %
+                               self.opts.single_result)
 
         # FIXME: Add more machine info!
         run_info = {


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D17717.49366.patch
Type: text/x-patch
Size: 3581 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20160229/d3463f91/attachment.bin>