<div dir="ltr">Greg,<div><br></div><div>I had to revert this commit as it broke the XML output for test results for Apple's internal buildbots.</div><div><br></div><div>Alex</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 24 January 2017 at 09:58, Greg Parker via llvm-commits <span dir="ltr"><<a href="mailto:llvm-commits@lists.llvm.org" target="_blank">llvm-commits@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Author: gparker<br>
Date: Tue Jan 24 03:58:02 2017<br>
New Revision: 292904<br>
<br>
URL: <a href="http://llvm.org/viewvc/llvm-project?rev=292904&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project?rev=292904&view=rev</a><br>
Log:<br>
[lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED<br>
<br>
A `lit` condition line is now a comma-separated list of boolean expressions.<br>
Comma-separated expressions act as if each expression were on its own<br>
condition line:<br>
For REQUIRES, if every expression is true then the test will run.<br>
For UNSUPPORTED, if every expression is false then the test will run.<br>
For XFAIL, if every expression is false then the test is expected to succeed.<br>
As a special case "XFAIL: *" expects the test to fail.<br>
<br>
Examples:<br>
# Test is expected fail on 64-bit Apple simulators and pass everywhere else<br>
XFAIL: x86_64 && apple && !macosx<br>
# Test is unsupported on Windows and on non-Ubuntu Linux<br>
# and supported everywhere else<br>
UNSUPPORTED: linux && !ubuntu, system-windows<br>
<br>
Syntax:<br>
* '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false.<br>
* Each test feature is a true identifier.<br>
* Substrings of the target triple are true identifiers for UNSUPPORTED<br>
and XFAIL, but not for REQUIRES. (This matches the current behavior.)<br>
* All other identifiers are false.<br>
* Identifiers are [-+=._a-zA-Z0-9]+<br>
<br>
Differential Revision: <a href="https://reviews.llvm.org/D18185" rel="noreferrer" target="_blank">https://reviews.llvm.org/<wbr>D18185</a><br>
<br>
Added:<br>
llvm/trunk/utils/lit/lit/<wbr>BooleanExpression.py<br>
llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/requires-<wbr>star.txt<br>
llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/requires-<wbr>triple.txt<br>
llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/<wbr>unsupported-expr-false.txt<br>
llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/<wbr>unsupported-expr-true.txt<br>
llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/<wbr>unsupported-star.txt<br>
llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/xfail-<wbr>expr-false.txt<br>
llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/xfail-<wbr>expr-true.txt<br>
llvm/trunk/utils/lit/tests/<wbr>boolean-parsing.py<br>
Modified:<br>
llvm/trunk/docs/TestingGuide.<wbr>rst<br>
llvm/trunk/utils/lit/lit/Test.<wbr>py<br>
llvm/trunk/utils/lit/lit/<wbr>TestRunner.py<br>
llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/requires-<wbr>missing.txt<br>
llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/requires-<wbr>present.txt<br>
llvm/trunk/utils/lit/tests/<wbr>shtest-format.py<br>
llvm/trunk/utils/lit/tests/<wbr>unit/TestRunner.py<br>
<br>
Modified: llvm/trunk/docs/TestingGuide.<wbr>rst<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/TestingGuide.rst?rev=292904&r1=292903&r2=292904&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/docs/<wbr>TestingGuide.rst?rev=292904&<wbr>r1=292903&r2=292904&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/docs/TestingGuide.<wbr>rst (original)<br>
+++ llvm/trunk/docs/TestingGuide.<wbr>rst Tue Jan 24 03:58:02 2017<br>
@@ -387,23 +387,49 @@ depends on special features of sub-archi<br>
triple, test with the specific FileCheck and put it into the specific<br>
directory that will filter out all other architectures.<br>
<br>
-REQUIRES and REQUIRES-ANY directive<br>
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<wbr>~~~~~~<br>
<br>
-Some tests can be enabled only in specific situation - like having<br>
-debug build. Use ``REQUIRES`` directive to specify those requirements.<br>
+Constraining test execution<br>
+---------------------------<br>
+<br>
+Some tests can be run only in specific configurations, such as<br>
+with debug builds or on particular platforms. Use ``REQUIRES``<br>
+and ``UNSUPPORTED`` to control when the test is enabled.<br>
+<br>
+Some tests are expected to fail. For example, there may be a known bug<br>
+that the test detect. Use ``XFAIL`` to mark a test as an expected failure.<br>
+An ``XFAIL`` test will be successful if its execution fails, and<br>
+will be a failure if its execution succeeds.<br>
<br>
.. code-block:: llvm<br>
<br>
- ; This test will be only enabled in the build with asserts<br>
+ ; This test will be only enabled in the build with asserts.<br>
; REQUIRES: asserts<br>
+ ; This test is disabled on Linux.<br>
+ ; UNSUPPORTED: -linux-<br>
+ ; This test is expected to fail on PowerPC.<br>
+ ; XFAIL: powerpc<br>
+<br>
+``REQUIRES`` and ``UNSUPPORTED`` and ``XFAIL`` all accept a comma-separated<br>
+list of boolean expressions. The values in each expression may be:<br>
+<br>
+- Features added to ``config.available_features`` by<br>
+ configuration files such as ``lit.cfg``.<br>
+- Substrings of the target triple (``UNSUPPORTED`` and ``XFAIL`` only).<br>
+<br>
+| ``REQUIRES`` enables the test if all expressions are true.<br>
+| ``UNSUPPORTED`` disables the test if any expression is true.<br>
+| ``XFAIL`` expects the test to fail if any expression is true.<br>
+<br>
+As a special case, ``XFAIL: *`` is expected to fail everywhere.<br>
+<br>
+.. code-block:: llvm<br>
<br>
-You can separate requirements by a comma.<br>
-``REQUIRES`` means all listed requirements must be satisfied.<br>
-``REQUIRES-ANY`` means at least one must be satisfied.<br>
+ ; This test is disabled on Windows,<br>
+ ; and is disabled on Linux, except for Android Linux.<br>
+ ; UNSUPPORTED: windows, linux && !android<br>
+ ; This test is expected to fail on both PowerPC and ARM.<br>
+ ; XFAIL: powerpc || arm<br>
<br>
-List of features that can be used in ``REQUIRES`` and ``REQUIRES-ANY`` can be<br>
-found in lit.cfg files.<br>
<br>
Substitutions<br>
-------------<br>
@@ -520,24 +546,6 @@ their name. For example:<br>
This program runs its arguments and then inverts the result code from it.<br>
Zero result codes become 1. Non-zero result codes become 0.<br>
<br>
-Sometimes it is necessary to mark a test case as "expected fail" or<br>
-XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:``<br>
-on a line near the top of the file. This signals that the test case<br>
-should succeed if the test fails. Such test cases are counted separately<br>
-by the testing tool. To specify an expected fail, use the XFAIL keyword<br>
-in the comments of the test program followed by a colon and one or more<br>
-failure patterns. Each failure pattern can be either ``*`` (to specify<br>
-fail everywhere), or a part of a target triple (indicating the test<br>
-should fail on that platform), or the name of a configurable feature<br>
-(for example, ``loadable_module``). If there is a match, the test is<br>
-expected to fail. If not, the test is expected to succeed. To XFAIL<br>
-everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL``<br>
-line:<br>
-<br>
-.. code-block:: llvm<br>
-<br>
- ; XFAIL: darwin,sun<br>
-<br>
To make the output more useful, :program:`lit` will scan<br>
the lines of the test case for ones that contain a pattern that matches<br>
``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number<br>
<br>
Added: llvm/trunk/utils/lit/lit/<wbr>BooleanExpression.py<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/lit/BooleanExpression.py?rev=292904&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/utils/lit/<wbr>lit/BooleanExpression.py?rev=<wbr>292904&view=auto</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/utils/lit/lit/<wbr>BooleanExpression.py (added)<br>
+++ llvm/trunk/utils/lit/lit/<wbr>BooleanExpression.py Tue Jan 24 03:58:02 2017<br>
@@ -0,0 +1,251 @@<br>
+import re<br>
+<br>
+class BooleanExpression:<br>
+ # A simple evaluator of boolean expressions.<br>
+ #<br>
+ # Grammar:<br>
+ # expr :: or_expr<br>
+ # or_expr :: and_expr ('||' and_expr)*<br>
+ # and_expr :: not_expr ('&&' not_expr)*<br>
+ # not_expr :: '!' not_expr<br>
+ # '(' or_expr ')'<br>
+ # identifier<br>
+ # identifier :: [-+=._a-zA-Z0-9]+<br>
+<br>
+ # Evaluates `string` as a boolean expression.<br>
+ # Returns True or False. Throws a ValueError on syntax error.<br>
+ #<br>
+ # Variables in `variables` are true.<br>
+ # Substrings of `triple` are true.<br>
+ # 'true' is true.<br>
+ # All other identifiers are false.<br>
+ @staticmethod<br>
+ def evaluate(string, variables, triple=""):<br>
+ try:<br>
+ parser = BooleanExpression(string, set(variables), triple)<br>
+ return parser.parseAll()<br>
+ except ValueError as e:<br>
+ raise ValueError(str(e) + ('\nin expression: %r' % string))<br>
+<br>
+ #####<br>
+<br>
+ def __init__(self, string, variables, triple=""):<br>
+ self.tokens = BooleanExpression.tokenize(<wbr>string)<br>
+ self.variables = variables<br>
+ self.variables.add('true')<br>
+ self.triple = triple<br>
+ self.value = None<br>
+ self.token = None<br>
+<br>
+ # Singleton end-of-expression marker.<br>
+ END = object()<br>
+<br>
+ # Tokenization pattern.<br>
+ Pattern = re.compile(r'\A\s*([()]|[-+=._<wbr>a-zA-Z0-9]+|&&|\|\||!)\s*(.*)\<wbr>Z')<br>
+<br>
+ @staticmethod<br>
+ def tokenize(string):<br>
+ while True:<br>
+ m = re.match(BooleanExpression.<wbr>Pattern, string)<br>
+ if m is None:<br>
+ if string == "":<br>
+ yield BooleanExpression.END;<br>
+ return<br>
+ else:<br>
+ raise ValueError("couldn't parse text: %r" % string)<br>
+<br>
+ token = m.group(1)<br>
+ string = m.group(2)<br>
+ yield token<br>
+<br>
+ def quote(self, token):<br>
+ if token is BooleanExpression.END:<br>
+ return '<end of expression>'<br>
+ else:<br>
+ return repr(token)<br>
+<br>
+ def accept(self, t):<br>
+ if self.token == t:<br>
+ self.token = next(self.tokens)<br>
+ return True<br>
+ else:<br>
+ return False<br>
+<br>
+ def expect(self, t):<br>
+ if self.token == t:<br>
+ if self.token != BooleanExpression.END:<br>
+ self.token = next(self.tokens)<br>
+ else:<br>
+ raise ValueError("expected: %s\nhave: %s" %<br>
+ (self.quote(t), self.quote(self.token)))<br>
+<br>
+ def isIdentifier(self, t):<br>
+ if (t is BooleanExpression.END or t == '&&' or t == '||' or<br>
+ t == '!' or t == '(' or t == ')'):<br>
+ return False<br>
+ return True<br>
+<br>
+ def parseNOT(self):<br>
+ if self.accept('!'):<br>
+ self.parseNOT()<br>
+ self.value = not self.value<br>
+ elif self.accept('('):<br>
+ self.parseOR()<br>
+ self.expect(')')<br>
+ elif not self.isIdentifier(self.token):<br>
+ raise ValueError("expected: '!' or '(' or identifier\nhave: %s" %<br>
+ self.quote(self.token))<br>
+ else:<br>
+ self.value = (self.token in self.variables or<br>
+ self.token in self.triple)<br>
+ self.token = next(self.tokens)<br>
+<br>
+ def parseAND(self):<br>
+ self.parseNOT()<br>
+ while self.accept('&&'):<br>
+ left = self.value<br>
+ self.parseNOT()<br>
+ right = self.value<br>
+ # this is technically the wrong associativity, but it<br>
+ # doesn't matter for this limited expression grammar<br>
+ self.value = left and right<br>
+<br>
+ def parseOR(self):<br>
+ self.parseAND()<br>
+ while self.accept('||'):<br>
+ left = self.value<br>
+ self.parseAND()<br>
+ right = self.value<br>
+ # this is technically the wrong associativity, but it<br>
+ # doesn't matter for this limited expression grammar<br>
+ self.value = left or right<br>
+<br>
+ def parseAll(self):<br>
+ self.token = next(self.tokens)<br>
+ self.parseOR()<br>
+ self.expect(BooleanExpression.<wbr>END)<br>
+ return self.value<br>
+<br>
+<br>
+#######<br>
+# Tests<br>
+<br>
+import unittest<br>
+<br>
+class TestBooleanExpression(<wbr>unittest.TestCase):<br>
+ def test_variables(self):<br>
+ variables = {'its-true', 'false-lol-true', 'under_score',<br>
+ 'e=quals', 'd1g1ts'}<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('<wbr>true', variables))<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('<wbr>its-true', variables))<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('<wbr>false-lol-true', variables))<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('<wbr>under_score', variables))<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('e=<wbr>quals', variables))<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('<wbr>d1g1ts', variables))<br>
+<br>
+ self.assertFalse(<wbr>BooleanExpression.evaluate('<wbr>false', variables))<br>
+ self.assertFalse(<wbr>BooleanExpression.evaluate('<wbr>True', variables))<br>
+ self.assertFalse(<wbr>BooleanExpression.evaluate('<wbr>true-ish', variables))<br>
+ self.assertFalse(<wbr>BooleanExpression.evaluate('<wbr>not_true', variables))<br>
+ self.assertFalse(<wbr>BooleanExpression.evaluate('<wbr>tru', variables))<br>
+<br>
+ def test_triple(self):<br>
+ triple = 'arch-vendor-os'<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('<wbr>arch-', {}, triple))<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('<wbr>ar', {}, triple))<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('<wbr>ch-vend', {}, triple))<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('-<wbr>vendor-', {}, triple))<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('-<wbr>os', {}, triple))<br>
+ self.assertFalse(<wbr>BooleanExpression.evaluate('<wbr>arch-os', {}, triple))<br>
+<br>
+ def test_operators(self):<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('<wbr>true || true', {}))<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('<wbr>true || false', {}))<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('<wbr>false || true', {}))<br>
+ self.assertFalse(<wbr>BooleanExpression.evaluate('<wbr>false || false', {}))<br>
+<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('<wbr>true && true', {}))<br>
+ self.assertFalse(<wbr>BooleanExpression.evaluate('<wbr>true && false', {}))<br>
+ self.assertFalse(<wbr>BooleanExpression.evaluate('<wbr>false && true', {}))<br>
+ self.assertFalse(<wbr>BooleanExpression.evaluate('<wbr>false && false', {}))<br>
+<br>
+ self.assertFalse(<wbr>BooleanExpression.evaluate('!<wbr>true', {}))<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('!<wbr>false', {}))<br>
+<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate(' ((!((false) )) ) ', {}))<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('<wbr>true && (true && (true))', {}))<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('!<wbr>false && !false && !! !false', {}))<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('<wbr>false && false || true', {}))<br>
+ self.assertTrue(<wbr>BooleanExpression.evaluate('(<wbr>false && false) || true', {}))<br>
+ self.assertFalse(<wbr>BooleanExpression.evaluate('<wbr>false && (false || true)', {}))<br>
+<br>
+ # Evaluate boolean expression `expr`.<br>
+ # Fail if it does not throw a ValueError containing the text `error`.<br>
+ def checkException(self, expr, error):<br>
+ try:<br>
+ BooleanExpression.evaluate(<wbr>expr, {})<br>
+ self.fail("expression %r didn't cause an exception" % expr)<br>
+ except ValueError as e:<br>
+ if -1 == str(e).find(error):<br>
+ self.fail(("expression %r caused the wrong ValueError\n" +<br>
+ "actual error was:\n%s\n" +<br>
+ "expected error was:\n%s\n") % (expr, e, error))<br>
+ except BaseException as e:<br>
+ self.fail(("expression %r caused the wrong exception; actual " +<br>
+ "exception was: \n%r") % (expr, e))<br>
+<br>
+ def test_errors(self):<br>
+ self.checkException("ba#d",<br>
+ "couldn't parse text: '#d'\n" +<br>
+ "in expression: 'ba#d'")<br>
+<br>
+ self.checkException("true and true",<br>
+ "expected: <end of expression>\n" +<br>
+ "have: 'and'\n" +<br>
+ "in expression: 'true and true'")<br>
+<br>
+ self.checkException("|| true",<br>
+ "expected: '!' or '(' or identifier\n" +<br>
+ "have: '||'\n" +<br>
+ "in expression: '|| true'")<br>
+<br>
+ self.checkException("true &&",<br>
+ "expected: '!' or '(' or identifier\n" +<br>
+ "have: <end of expression>\n" +<br>
+ "in expression: 'true &&'")<br>
+<br>
+ self.checkException("",<br>
+ "expected: '!' or '(' or identifier\n" +<br>
+ "have: <end of expression>\n" +<br>
+ "in expression: ''")<br>
+<br>
+ self.checkException("*",<br>
+ "couldn't parse text: '*'\n" +<br>
+ "in expression: '*'")<br>
+<br>
+ self.checkException("no wait stop",<br>
+ "expected: <end of expression>\n" +<br>
+ "have: 'wait'\n" +<br>
+ "in expression: 'no wait stop'")<br>
+<br>
+ self.checkException("no-$-<wbr>please",<br>
+ "couldn't parse text: '$-please'\n" +<br>
+ "in expression: 'no-$-please'")<br>
+<br>
+ self.checkException("(((true && true) || true)",<br>
+ "expected: ')'\n" +<br>
+ "have: <end of expression>\n" +<br>
+ "in expression: '(((true && true) || true)'")<br>
+<br>
+ self.checkException("true (true)",<br>
+ "expected: <end of expression>\n" +<br>
+ "have: '('\n" +<br>
+ "in expression: 'true (true)'")<br>
+<br>
+ self.checkException("( )",<br>
+ "expected: '!' or '(' or identifier\n" +<br>
+ "have: ')'\n" +<br>
+ "in expression: '( )'")<br>
+<br>
+if __name__ == '__main__':<br>
+ unittest.main()<br>
<br>
Modified: llvm/trunk/utils/lit/lit/Test.<wbr>py<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/lit/Test.py?rev=292904&r1=292903&r2=292904&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/utils/lit/<wbr>lit/Test.py?rev=292904&r1=<wbr>292903&r2=292904&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/utils/lit/lit/Test.<wbr>py (original)<br>
+++ llvm/trunk/utils/lit/lit/Test.<wbr>py Tue Jan 24 03:58:02 2017<br>
@@ -2,6 +2,8 @@ import os<br>
from xml.sax.saxutils import escape<br>
from json import JSONEncoder<br>
<br>
+from lit.BooleanExpression import BooleanExpression<br>
+<br>
# Test result codes.<br>
<br>
class ResultCode(object):<br>
@@ -180,10 +182,24 @@ class Test:<br>
self.path_in_suite = path_in_suite<br>
self.config = config<br>
self.file_path = file_path<br>
- # A list of conditions under which this test is expected to fail. These<br>
- # can optionally be provided by test format handlers, and will be<br>
- # honored when the test result is supplied.<br>
+<br>
+ # A list of conditions under which this test is expected to fail.<br>
+ # Each condition is a boolean expression of features and target<br>
+ # triple parts. These can optionally be provided by test format<br>
+ # handlers, and will be honored when the test result is supplied.<br>
self.xfails = []<br>
+<br>
+ # A list of conditions that must be satisfied before running the test.<br>
+ # Each condition is a boolean expression of features. All of them<br>
+ # must be True for the test to run.<br>
+ # FIXME should target triple parts count here too?<br>
+ self.requires = []<br>
+<br>
+ # A list of conditions that prevent execution of the test.<br>
+ # Each condition is a boolean expression of features and target<br>
+ # triple parts. All of them must be False for the test to run.<br>
+ self.unsupported = []<br>
+<br>
# The test result, once complete.<br>
self.result = None<br>
<br>
@@ -196,11 +212,16 @@ class Test:<br>
self.result = result<br>
<br>
# Apply the XFAIL handling to resolve the result exit code.<br>
- if self.isExpectedToFail():<br>
- if self.result.code == PASS:<br>
- self.result.code = XPASS<br>
- elif self.result.code == FAIL:<br>
- self.result.code = XFAIL<br>
+ try:<br>
+ if self.isExpectedToFail():<br>
+ if self.result.code == PASS:<br>
+ self.result.code = XPASS<br>
+ elif self.result.code == FAIL:<br>
+ self.result.code = XFAIL<br>
+ except ValueError as e:<br>
+ # Syntax error in an XFAIL line.<br>
+ self.result.code = UNRESOLVED<br>
+ self.result.output = str(e)<br>
<br>
def getFullName(self):<br>
return <a href="http://self.suite.config.name" rel="noreferrer" target="_blank">self.suite.config.name</a> + ' :: ' + '/'.join(self.path_in_suite)<br>
@@ -224,24 +245,91 @@ class Test:<br>
configuration. This check relies on the test xfails property which by<br>
some test formats may not be computed until the test has first been<br>
executed.<br>
+ Throws ValueError if an XFAIL line has a syntax error.<br>
"""<br>
<br>
+ features = self.config.available_features<br>
+ triple = getattr(self.suite.config, 'target_triple', "")<br>
+<br>
# Check if any of the xfails match an available feature or the target.<br>
for item in self.xfails:<br>
# If this is the wildcard, it always fails.<br>
if item == '*':<br>
return True<br>
<br>
- # If this is an exact match for one of the features, it fails.<br>
- if item in self.config.available_<wbr>features:<br>
- return True<br>
-<br>
- # If this is a part of the target triple, it fails.<br>
- if item and item in self.suite.config.target_<wbr>triple:<br>
- return True<br>
+ # If this is a True expression of features and target triple parts,<br>
+ # it fails.<br>
+ try:<br>
+ if BooleanExpression.evaluate(<wbr>item, features, triple):<br>
+ return True<br>
+ except ValueError as e:<br>
+ raise ValueError('Error in XFAIL list:\n%s' % str(e))<br>
<br>
return False<br>
<br>
+ def isWithinFeatureLimits(self):<br>
+ """<br>
+ isWithinFeatureLimits() -> bool<br>
+<br>
+ A test is within the feature limits set by run_only_tests if<br>
+ 1. the test's requirements ARE satisfied by the available features<br>
+ 2. the test's requirements ARE NOT satisfied after the limiting<br>
+ features are removed from the available features<br>
+<br>
+ Throws ValueError if a REQUIRES line has a syntax error.<br>
+ """<br>
+<br>
+ if not self.config.limit_to_features:<br>
+ return True # No limits. Run it.<br>
+<br>
+ # Check the requirements as-is (#1)<br>
+ if self.<wbr>getMissingRequiredFeatures():<br>
+ return False<br>
+<br>
+ # Check the requirements after removing the limiting features (#2)<br>
+ featuresMinusLimits = [f for f in self.config.available_features<br>
+ if not f in self.config.limit_to_features]<br>
+ if not self.<wbr>getMissingRequiredFeaturesFrom<wbr>List(featuresMinusLimits):<br>
+ return False<br>
+<br>
+ return True<br>
+<br>
+ def getMissingRequiredFeaturesFrom<wbr>List(self, features):<br>
+ try:<br>
+ return [item for item in self.requires<br>
+ if not BooleanExpression.evaluate(<wbr>item, features)]<br>
+ except ValueError as e:<br>
+ raise ValueError('Error in REQUIRES list:\n%s' % str(e))<br>
+<br>
+ def getMissingRequiredFeatures(<wbr>self):<br>
+ """<br>
+ getMissingRequiredFeatures() -> list of strings<br>
+<br>
+ Returns a list of features from REQUIRES that are not satisfied."<br>
+ Throws ValueError if a REQUIRES line has a syntax error.<br>
+ """<br>
+<br>
+ features = self.config.available_features<br>
+ return self.<wbr>getMissingRequiredFeaturesFrom<wbr>List(features)<br>
+<br>
+ def getUnsupportedFeatures(self):<br>
+ """<br>
+ getUnsupportedFeatures() -> list of strings<br>
+<br>
+ Returns a list of features from UNSUPPORTED that are present<br>
+ in the test configuration's features or target triple.<br>
+ Throws ValueError if an UNSUPPORTED line has a syntax error.<br>
+ """<br>
+<br>
+ features = self.config.available_features<br>
+ triple = getattr(self.suite.config, 'target_triple', "")<br>
+<br>
+ try:<br>
+ return [item for item in self.unsupported<br>
+ if BooleanExpression.evaluate(<wbr>item, features, triple)]<br>
+ except ValueError as e:<br>
+ raise ValueError('Error in UNSUPPORTED list:\n%s' % str(e))<br>
+<br>
def isEarlyTest(self):<br>
"""<br>
isEarlyTest() -> bool<br>
<br>
Modified: llvm/trunk/utils/lit/lit/<wbr>TestRunner.py<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/lit/TestRunner.py?rev=292904&r1=292903&r2=292904&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/utils/lit/<wbr>lit/TestRunner.py?rev=292904&<wbr>r1=292903&r2=292904&view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/utils/lit/lit/<wbr>TestRunner.py (original)<br>
+++ llvm/trunk/utils/lit/lit/<wbr>TestRunner.py Tue Jan 24 03:58:02 2017<br>
@@ -9,6 +9,7 @@ import lit.ShUtil as ShUtil<br>
import lit.Test as Test<br>
import lit.util<br>
from lit.util import to_bytes, to_string<br>
+from lit.BooleanExpression import BooleanExpression<br>
<br>
class InternalShellError(Exception):<br>
def __init__(self, command, message):<br>
@@ -746,14 +747,35 @@ class ParserKind(object):<br>
command.<br>
<br>
TAG: A keyword taking no value. Ex 'END.'<br>
- COMMAND: A Keyword taking a list of shell commands. Ex 'RUN:'<br>
- LIST: A keyword taking a comma separated list of value. Ex 'XFAIL:'<br>
+ COMMAND: A keyword taking a list of shell commands. Ex 'RUN:'<br>
+ LIST: A keyword taking a comma-separated list of values.<br>
+ BOOLEAN_EXPR: A keyword taking a comma-separated list of<br>
+ boolean expressions. Ex 'XFAIL:'<br>
CUSTOM: A keyword with custom parsing semantics.<br>
"""<br>
TAG = 0<br>
COMMAND = 1<br>
LIST = 2<br>
- CUSTOM = 3<br>
+ BOOLEAN_EXPR = 3<br>
+ CUSTOM = 4<br>
+<br>
+ @staticmethod<br>
+ def allowedKeywordSuffixes(value):<br>
+ return { ParserKind.TAG: ['.'],<br>
+ ParserKind.COMMAND: [':'],<br>
+ ParserKind.LIST: [':'],<br>
+ ParserKind.BOOLEAN_EXPR: [':'],<br>
+ ParserKind.CUSTOM: [':', '.']<br>
+ } [value]<br>
+<br>
+ @staticmethod<br>
+ def str(value):<br>
+ return { ParserKind.TAG: 'TAG',<br>
+ ParserKind.COMMAND: 'COMMAND',<br>
+ ParserKind.LIST: 'LIST',<br>
+ ParserKind.BOOLEAN_EXPR: 'BOOLEAN_EXPR',<br>
+ ParserKind.CUSTOM: 'CUSTOM'<br>
+ } [value]<br>
<br>
<br>
class IntegratedTestKeywordParser(<wbr>object):<br>
@@ -765,15 +787,18 @@ class IntegratedTestKeywordParser(<wbr>object<br>
ParserKind.CUSTOM.<br>
"""<br>
def __init__(self, keyword, kind, parser=None, initial_value=None):<br>
- if not keyword.endswith('.') and not keyword.endswith(':'):<br>
- raise ValueError("keyword '%s' must end with either '.' or ':' "<br>
- % keyword)<br>
- if keyword.endswith('.') and kind in \<br>
- [ParserKind.LIST, ParserKind.COMMAND]:<br>
- raise ValueError("Keyword '%s' should end in ':'" % keyword)<br>
+ allowedSuffixes = ParserKind.<wbr>allowedKeywordSuffixes(kind)<br>
+ if len(keyword) == 0 or keyword[-1] not in allowedSuffixes:<br>
+ if len(allowedSuffixes) == 1:<br>
+ raise ValueError("Keyword '%s' of kind '%s' must end in '%s'"<br>
+ % (keyword, ParserKind.str(kind),<br>
+ allowedSuffixes[0]))<br>
+ else:<br>
+ raise ValueError("Keyword '%s' of kind '%s' must end in "<br>
+ " one of '%s'"<br>
+ % (keyword, ParserKind.str(kind),<br>
+ ' '.join(allowedSuffixes)))<br>
<br>
- elif keyword.endswith(':') and kind in [ParserKind.TAG]:<br>
- raise ValueError("Keyword '%s' should end in '.'" % keyword)<br>
if parser is not None and kind != ParserKind.CUSTOM:<br>
raise ValueError("custom parsers can only be specified with "<br>
"ParserKind.CUSTOM")<br>
@@ -787,9 +812,9 @@ class IntegratedTestKeywordParser(<wbr>object<br>
self.parser = self._handleCommand<br>
elif kind == ParserKind.LIST:<br>
self.parser = self._handleList<br>
+ elif kind == ParserKind.BOOLEAN_EXPR:<br>
+ self.parser = self._handleBooleanExpr<br>
elif kind == ParserKind.TAG:<br>
- if not keyword.endswith('.'):<br>
- raise ValueError("keyword '%s' should end with '.'" % keyword)<br>
self.parser = self._handleTag<br>
elif kind == ParserKind.CUSTOM:<br>
if parser is None:<br>
@@ -799,8 +824,12 @@ class IntegratedTestKeywordParser(<wbr>object<br>
raise ValueError("Unknown kind '%s'" % kind)<br>
<br>
def parseLine(self, line_number, line):<br>
- self.parsed_lines += [(line_number, line)]<br>
- self.value = self.parser(line_number, line, self.value)<br>
+ try:<br>
+ self.parsed_lines += [(line_number, line)]<br>
+ self.value = self.parser(line_number, line, self.value)<br>
+ except ValueError as e:<br>
+ raise ValueError(str(e) + ("\nin %s directive on test line %d" %<br>
+ (self.keyword, line_number)))<br>
<br>
def getValue(self):<br>
return self.value<br>
@@ -841,12 +870,38 @@ class IntegratedTestKeywordParser(<wbr>object<br>
output.extend([s.strip() for s in line.split(',')])<br>
return output<br>
<br>
+ @staticmethod<br>
+ def _handleBooleanExpr(line_<wbr>number, line, output):<br>
+ """A parser for BOOLEAN_EXPR type keywords"""<br>
+ if output is None:<br>
+ output = []<br>
+ output.extend([s.strip() for s in line.split(',')])<br>
+ # Evaluate each expression to verify syntax.<br>
+ # We don't want any results, just the raised ValueError.<br>
+ for s in output:<br>
+ if s != '*':<br>
+ BooleanExpression.evaluate(s, [])<br>
+ return output<br>
+<br>
+ @staticmethod<br>
+ def _handleRequiresAny(line_<wbr>number, line, output):<br>
+ """A custom parser to transform REQUIRES-ANY: into REQUIRES:"""<br>
+<br>
+ # Extract the conditions specified in REQUIRES-ANY: as written.<br>
+ conditions = []<br>
+ IntegratedTestKeywordParser._<wbr>handleList(line_number, line, conditions)<br>
+<br>
+ # Output a `REQUIRES: a || b || c` expression in its place.<br>
+ expression = ' || '.join(conditions)<br>
+ IntegratedTestKeywordParser._<wbr>handleBooleanExpr(line_number,<br>
+ expression, output)<br>
+ return output<br>
<br>
def parseIntegratedTestScript(<wbr>test, additional_parsers=[],<br>
require_script=True):<br>
"""parseIntegratedTestScript - Scan an LLVM/Clang style integrated test<br>
script and extract the lines to 'RUN' as well as 'XFAIL' and 'REQUIRES'<br>
- 'REQUIRES-ANY' and 'UNSUPPORTED' information.<br>
+ and 'UNSUPPORTED' information.<br>
<br>
If additional parsers are specified then the test is also scanned for the<br>
keywords they specify and all matches are passed to the custom parser.<br>
@@ -855,26 +910,26 @@ def parseIntegratedTestScript(<wbr>test, addi<br>
may be returned. This can be used for test formats where the actual script<br>
is optional or ignored.<br>
"""<br>
- # Collect the test lines from the script.<br>
- sourcepath = test.getSourcePath()<br>
+<br>
+ # Install the built-in keyword parsers.<br>
script = []<br>
- requires = []<br>
- requires_any = []<br>
- unsupported = []<br>
builtin_parsers = [<br>
IntegratedTestKeywordParser('<wbr>RUN:', ParserKind.COMMAND,<br>
initial_value=script),<br>
- IntegratedTestKeywordParser('<wbr>XFAIL:', ParserKind.LIST,<br>
+ IntegratedTestKeywordParser('<wbr>XFAIL:', ParserKind.BOOLEAN_EXPR,<br>
initial_value=test.xfails),<br>
- IntegratedTestKeywordParser('<wbr>REQUIRES:', ParserKind.LIST,<br>
- initial_value=requires),<br>
- IntegratedTestKeywordParser('<wbr>REQUIRES-ANY:', ParserKind.LIST,<br>
- initial_value=requires_any),<br>
- IntegratedTestKeywordParser('<wbr>UNSUPPORTED:', ParserKind.LIST,<br>
- initial_value=unsupported),<br>
+ IntegratedTestKeywordParser('<wbr>REQUIRES:', ParserKind.BOOLEAN_EXPR,<br>
+ initial_value=test.requires),<br>
+ IntegratedTestKeywordParser('<wbr>REQUIRES-ANY:', ParserKind.CUSTOM,<br>
+ IntegratedTestKeywordParser._<wbr>handleRequiresAny,<br>
+ initial_value=test.requires),<br>
+ IntegratedTestKeywordParser('<wbr>UNSUPPORTED:', ParserKind.BOOLEAN_EXPR,<br>
+ initial_value=test.<wbr>unsupported),<br>
IntegratedTestKeywordParser('<wbr>END.', ParserKind.TAG)<br>
]<br>
keyword_parsers = {p.keyword: p for p in builtin_parsers}<br>
+<br>
+ # Install user-defined additional parsers.<br>
for parser in additional_parsers:<br>
if not isinstance(parser, IntegratedTestKeywordParser):<br>
raise ValueError('additional parser must be an instance of '<br>
@@ -883,7 +938,9 @@ def parseIntegratedTestScript(<wbr>test, addi<br>
raise ValueError("Parser for keyword '%s' already exists"<br>
% parser.keyword)<br>
keyword_parsers[parser.<wbr>keyword] = parser<br>
-<br>
+<br>
+ # Collect the test lines from the script.<br>
+ sourcepath = test.getSourcePath()<br>
for line_number, command_type, ln in \<br>
parseIntegratedTestScriptComma<wbr>nds(sourcepath,<br>
keyword_parsers.keys()):<br>
@@ -901,46 +958,30 @@ def parseIntegratedTestScript(<wbr>test, addi<br>
return lit.Test.Result(Test.<wbr>UNRESOLVED,<br>
"Test has unterminated run lines (with '\\')")<br>
<br>
- # Check that we have the required features:<br>
- missing_required_features = [f for f in requires<br>
- if f not in test.config.available_<wbr>features]<br>
+ # Enforce REQUIRES:<br>
+ missing_required_features = test.<wbr>getMissingRequiredFeatures()<br>
if missing_required_features:<br>
msg = ', '.join(missing_required_<wbr>features)<br>
return lit.Test.Result(Test.<wbr>UNSUPPORTED,<br>
- "Test requires the following features: %s"<br>
- % msg)<br>
- requires_any_features = [f for f in requires_any<br>
- if f in test.config.available_<wbr>features]<br>
- if requires_any and not requires_any_features:<br>
- msg = ' ,'.join(requires_any)<br>
- return lit.Test.Result(Test.<wbr>UNSUPPORTED,<br>
- "Test requires any of the following features: "<br>
- "%s" % msg)<br>
- unsupported_features = [f for f in unsupported<br>
- if f in test.config.available_<wbr>features]<br>
+ "Test requires the following unavailable "<br>
+ "features: %s" % msg)<br>
+<br>
+ # Enforce UNSUPPORTED:<br>
+ unsupported_features = test.getUnsupportedFeatures()<br>
if unsupported_features:<br>
msg = ', '.join(unsupported_features)<br>
return lit.Test.Result(<br>
Test.UNSUPPORTED,<br>
- "Test is unsupported with the following features: %s" % msg)<br>
+ "Test does not support the following features "<br>
+ "and/or targets: %s" % msg)<br>
<br>
- unsupported_targets = [f for f in unsupported<br>
- if f in test.suite.config.target_<wbr>triple]<br>
- if unsupported_targets:<br>
- return lit.Test.Result(<br>
- Test.UNSUPPORTED,<br>
- "Test is unsupported with the following triple: %s" % (<br>
- test.suite.config.target_<wbr>triple,))<br>
+ # Enforce limit_to_features.<br>
+ if not test.isWithinFeatureLimits():<br>
+ msg = ', '.join(test.config.limit_to_<wbr>features)<br>
+ return lit.Test.Result(Test.<wbr>UNSUPPORTED,<br>
+ "Test does not require any of the features "<br>
+ "specified in limit_to_features: %s" % msg)<br>
<br>
- if test.config.limit_to_features:<br>
- # Check that we have one of the limit_to_features features in requires.<br>
- limit_to_features_tests = [f for f in test.config.limit_to_features<br>
- if f in requires]<br>
- if not limit_to_features_tests:<br>
- msg = ', '.join(test.config.limit_to_<wbr>features)<br>
- return lit.Test.Result(<br>
- Test.UNSUPPORTED,<br>
- "Test requires one of the limit_to_features features %s" % msg)<br>
return script<br>
<br>
<br>
<br>
Modified: llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/requires-<wbr>missing.txt<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-missing.txt?rev=292904&r1=292903&r2=292904&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/utils/lit/<wbr>tests/Inputs/shtest-format/<wbr>requires-missing.txt?rev=<wbr>292904&r1=292903&r2=292904&<wbr>view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/requires-<wbr>missing.txt (original)<br>
+++ llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/requires-<wbr>missing.txt Tue Jan 24 03:58:02 2017<br>
@@ -1,2 +1,5 @@<br>
-RUN: true<br>
-REQUIRES: a-missing-feature<br>
+# REQUIRES with a false clause. Test should not run.<br>
+REQUIRES: true<br>
+REQUIRES: a-missing-feature, true<br>
+REQUIRES: true<br>
+RUN: false<br>
<br>
Modified: llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/requires-<wbr>present.txt<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-present.txt?rev=292904&r1=292903&r2=292904&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/utils/lit/<wbr>tests/Inputs/shtest-format/<wbr>requires-present.txt?rev=<wbr>292904&r1=292903&r2=292904&<wbr>view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/requires-<wbr>present.txt (original)<br>
+++ llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/requires-<wbr>present.txt Tue Jan 24 03:58:02 2017<br>
@@ -1,2 +1,4 @@<br>
+# REQUIRES with only true clauses. Test should run.<br>
+REQUIRES: a-present-feature, true, !not-true<br>
+REQUIRES: true<br>
RUN: true<br>
-REQUIRES: a-present-feature<br>
<br>
Added: llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/requires-<wbr>star.txt<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-star.txt?rev=292904&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/utils/lit/<wbr>tests/Inputs/shtest-format/<wbr>requires-star.txt?rev=292904&<wbr>view=auto</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/requires-<wbr>star.txt (added)<br>
+++ llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/requires-<wbr>star.txt Tue Jan 24 03:58:02 2017<br>
@@ -0,0 +1,3 @@<br>
+# '*' only works in XFAIL<br>
+REQUIRES: *<br>
+RUN: false<br>
<br>
Added: llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/requires-<wbr>triple.txt<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-triple.txt?rev=292904&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/utils/lit/<wbr>tests/Inputs/shtest-format/<wbr>requires-triple.txt?rev=<wbr>292904&view=auto</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/requires-<wbr>triple.txt (added)<br>
+++ llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/requires-<wbr>triple.txt Tue Jan 24 03:58:02 2017<br>
@@ -0,0 +1,3 @@<br>
+# REQUIRES line that uses target triple, which doesn't work. Test should not run<br>
+REQUIRES: x86_64<br>
+RUN: false<br>
<br>
Added: llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/<wbr>unsupported-expr-false.txt<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-false.txt?rev=292904&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/utils/lit/<wbr>tests/Inputs/shtest-format/<wbr>unsupported-expr-false.txt?<wbr>rev=292904&view=auto</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/<wbr>unsupported-expr-false.txt (added)<br>
+++ llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/<wbr>unsupported-expr-false.txt Tue Jan 24 03:58:02 2017<br>
@@ -0,0 +1,9 @@<br>
+# UNSUPPORTED with only false clauses. Test should run.<br>
+UNSUPPORTED: false<br>
+UNSUPPORTED: false, not-true<br>
+UNSUPPORTED: false<br>
+UNSUPPORTED: still-not-true<br>
+UNSUPPORTED: false<br>
+UNSUPPORTED: false<br>
+UNSUPPORTED: false<br>
+RUN: true<br>
<br>
Added: llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/<wbr>unsupported-expr-true.txt<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-true.txt?rev=292904&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/utils/lit/<wbr>tests/Inputs/shtest-format/<wbr>unsupported-expr-true.txt?rev=<wbr>292904&view=auto</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/<wbr>unsupported-expr-true.txt (added)<br>
+++ llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/<wbr>unsupported-expr-true.txt Tue Jan 24 03:58:02 2017<br>
@@ -0,0 +1,4 @@<br>
+# UNSUPPORTED with a true clause. Test should not run.<br>
+UNSUPPORTED: false<br>
+UNSUPPORTED: false, false, false, _64-unk && a-present-feature, false<br>
+RUN: false<br>
<br>
Added: llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/<wbr>unsupported-star.txt<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-star.txt?rev=292904&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/utils/lit/<wbr>tests/Inputs/shtest-format/<wbr>unsupported-star.txt?rev=<wbr>292904&view=auto</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/<wbr>unsupported-star.txt (added)<br>
+++ llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/<wbr>unsupported-star.txt Tue Jan 24 03:58:02 2017<br>
@@ -0,0 +1,3 @@<br>
+# '*' only works in XFAIL<br>
+UNSUPPORTED: *<br>
+RUN: false<br>
<br>
Added: llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/xfail-<wbr>expr-false.txt<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-false.txt?rev=292904&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/utils/lit/<wbr>tests/Inputs/shtest-format/<wbr>xfail-expr-false.txt?rev=<wbr>292904&view=auto</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/xfail-<wbr>expr-false.txt (added)<br>
+++ llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/xfail-<wbr>expr-false.txt Tue Jan 24 03:58:02 2017<br>
@@ -0,0 +1,3 @@<br>
+# XFAIL with only false clauses. Test should run.<br>
+XFAIL: false, a-missing-feature || ! a-present-feature || ! x86_64, false<br>
+RUN: true<br>
<br>
Added: llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/xfail-<wbr>expr-true.txt<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-true.txt?rev=292904&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/utils/lit/<wbr>tests/Inputs/shtest-format/<wbr>xfail-expr-true.txt?rev=<wbr>292904&view=auto</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/xfail-<wbr>expr-true.txt (added)<br>
+++ llvm/trunk/utils/lit/tests/<wbr>Inputs/shtest-format/xfail-<wbr>expr-true.txt Tue Jan 24 03:58:02 2017<br>
@@ -0,0 +1,4 @@<br>
+# XFAIL with a true clause. Test should not run.<br>
+XFAIL: false<br>
+XFAIL: false, a-present-feature && ! a-missing-feature && x86_64<br>
+RUN: false<br>
<br>
Added: llvm/trunk/utils/lit/tests/<wbr>boolean-parsing.py<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/boolean-parsing.py?rev=292904&view=auto" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/utils/lit/<wbr>tests/boolean-parsing.py?rev=<wbr>292904&view=auto</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/utils/lit/tests/<wbr>boolean-parsing.py (added)<br>
+++ llvm/trunk/utils/lit/tests/<wbr>boolean-parsing.py Tue Jan 24 03:58:02 2017<br>
@@ -0,0 +1,4 @@<br>
+# Test the boolean expression parser<br>
+# used for REQUIRES and UNSUPPORTED and XFAIL<br>
+<br>
+# RUN: %{python} -m lit.BooleanExpression<br>
<br>
Modified: llvm/trunk/utils/lit/tests/<wbr>shtest-format.py<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/shtest-format.py?rev=292904&r1=292903&r2=292904&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/utils/lit/<wbr>tests/shtest-format.py?rev=<wbr>292904&r1=292903&r2=292904&<wbr>view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/utils/lit/tests/<wbr>shtest-format.py (original)<br>
+++ llvm/trunk/utils/lit/tests/<wbr>shtest-format.py Tue Jan 24 03:58:02 2017<br>
@@ -50,7 +50,14 @@<br>
# CHECK: PASS: shtest-format :: requires-any-present.txt<br>
# CHECK: UNSUPPORTED: shtest-format :: requires-missing.txt<br>
# CHECK: PASS: shtest-format :: requires-present.txt<br>
+# CHECK: UNRESOLVED: shtest-format :: requires-star.txt<br>
+# CHECK: UNSUPPORTED: shtest-format :: requires-triple.txt<br>
+# CHECK: PASS: shtest-format :: unsupported-expr-false.txt<br>
+# CHECK: UNSUPPORTED: shtest-format :: unsupported-expr-true.txt<br>
+# CHECK: UNRESOLVED: shtest-format :: unsupported-star.txt<br>
# CHECK: UNSUPPORTED: shtest-format :: unsupported_dir/some-test.txt<br>
+# CHECK: PASS: shtest-format :: xfail-expr-false.txt<br>
+# CHECK: XFAIL: shtest-format :: xfail-expr-true.txt<br>
# CHECK: XFAIL: shtest-format :: xfail-feature.txt<br>
# CHECK: XFAIL: shtest-format :: xfail-target.txt<br>
# CHECK: XFAIL: shtest-format :: xfail.txt<br>
@@ -70,9 +77,9 @@<br>
# CHECK: shtest-format :: external_shell/fail_with_bad_<wbr>encoding.txt<br>
# CHECK: shtest-format :: fail.txt<br>
<br>
-# CHECK: Expected Passes : 5<br>
-# CHECK: Expected Failures : 3<br>
-# CHECK: Unsupported Tests : 3<br>
-# CHECK: Unresolved Tests : 1<br>
+# CHECK: Expected Passes : 7<br>
+# CHECK: Expected Failures : 4<br>
+# CHECK: Unsupported Tests : 5<br>
+# CHECK: Unresolved Tests : 3<br>
# CHECK: Unexpected Passes : 1<br>
# CHECK: Unexpected Failures: 3<br>
<br>
Modified: llvm/trunk/utils/lit/tests/<wbr>unit/TestRunner.py<br>
URL: <a href="http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/unit/TestRunner.py?rev=292904&r1=292903&r2=292904&view=diff" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/llvm/trunk/utils/lit/<wbr>tests/unit/TestRunner.py?rev=<wbr>292904&r1=292903&r2=292904&<wbr>view=diff</a><br>
==============================<wbr>==============================<wbr>==================<br>
--- llvm/trunk/utils/lit/tests/<wbr>unit/TestRunner.py (original)<br>
+++ llvm/trunk/utils/lit/tests/<wbr>unit/TestRunner.py Tue Jan 24 03:58:02 2017<br>
@@ -108,6 +108,63 @@ class TestIntegratedTestKeywordParse<wbr>r(un<br>
value = custom_parser.getValue()<br>
self.assertItemsEqual(value, ['a', 'b', 'c'])<br>
<br>
+ def test_bad_keywords(self):<br>
+ def custom_parse(line_number, line, output):<br>
+ return output<br>
+<br>
+ try:<br>
+ IntegratedTestKeywordParser("<wbr>TAG_NO_SUFFIX", ParserKind.TAG),<br>
+ self.fail("TAG_NO_SUFFIX failed to raise an exception")<br>
+ except ValueError as e:<br>
+ pass<br>
+ except BaseException as e:<br>
+ self.fail("TAG_NO_SUFFIX raised the wrong exception: %r" % e)<br>
+<br>
+ try:<br>
+ IntegratedTestKeywordParser("<wbr>TAG_WITH_COLON:", ParserKind.TAG),<br>
+ self.fail("TAG_WITH_COLON: failed to raise an exception")<br>
+ except ValueError as e:<br>
+ pass<br>
+ except BaseException as e:<br>
+ self.fail("TAG_WITH_COLON: raised the wrong exception: %r" % e)<br>
+<br>
+ try:<br>
+ IntegratedTestKeywordParser("<wbr>LIST_WITH_DOT.", ParserKind.LIST),<br>
+ self.fail("LIST_WITH_DOT. failed to raise an exception")<br>
+ except ValueError as e:<br>
+ pass<br>
+ except BaseException as e:<br>
+ self.fail("LIST_WITH_DOT. raised the wrong exception: %r" % e)<br>
+<br>
+ try:<br>
+ IntegratedTestKeywordParser("<wbr>CUSTOM_NO_SUFFIX",<br>
+ ParserKind.CUSTOM, custom_parse),<br>
+ self.fail("CUSTOM_NO_SUFFIX failed to raise an exception")<br>
+ except ValueError as e:<br>
+ pass<br>
+ except BaseException as e:<br>
+ self.fail("CUSTOM_NO_SUFFIX raised the wrong exception: %r" % e)<br>
+<br>
+ # Both '.' and ':' are allowed for CUSTOM keywords.<br>
+ try:<br>
+ IntegratedTestKeywordParser("<wbr>CUSTOM_WITH_DOT.",<br>
+ ParserKind.CUSTOM, custom_parse),<br>
+ except BaseException as e:<br>
+ self.fail("CUSTOM_WITH_DOT. raised an exception: %r" % e)<br>
+ try:<br>
+ IntegratedTestKeywordParser("<wbr>CUSTOM_WITH_COLON:",<br>
+ ParserKind.CUSTOM, custom_parse),<br>
+ except BaseException as e:<br>
+ self.fail("CUSTOM_WITH_COLON: raised an exception: %r" % e)<br>
+<br>
+ try:<br>
+ IntegratedTestKeywordParser("<wbr>CUSTOM_NO_PARSER:",<br>
+ ParserKind.CUSTOM),<br>
+ self.fail("CUSTOM_NO_PARSER: failed to raise an exception")<br>
+ except ValueError as e:<br>
+ pass<br>
+ except BaseException as e:<br>
+ self.fail("CUSTOM_NO_PARSER: raised the wrong exception: %r" % e)<br>
<br>
if __name__ == '__main__':<br>
TestIntegratedTestKeywordParse<wbr>r.load_keyword_parser_lit_<wbr>tests()<br>
<br>
<br>
______________________________<wbr>_________________<br>
llvm-commits mailing list<br>
<a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/llvm-commits</a><br>
</blockquote></div><br></div>