[llvm] r292896 - [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED
Greg Parker via llvm-commits
llvm-commits at lists.llvm.org
Tue Jan 24 00:45:52 PST 2017
Author: gparker
Date: Tue Jan 24 02:45:50 2017
New Revision: 292896
URL: http://llvm.org/viewvc/llvm-project?rev=292896&view=rev
Log:
[lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED
A `lit` condition line is now a comma-separated list of boolean expressions.
Comma-separated expressions act as if each expression were on its own
condition line:
For REQUIRES, if every expression is true then the test will run.
For UNSUPPORTED, if every expression is false then the test will run.
For XFAIL, if every expression is false then the test is expected to succeed.
As a special case "XFAIL: *" expects the test to fail.
Examples:
# Test is expected fail on 64-bit Apple simulators and pass everywhere else
XFAIL: x86_64 && apple && !macosx
# Test is unsupported on Windows and on non-Ubuntu Linux
# and supported everywhere else
UNSUPPORTED: linux && !ubuntu, system-windows
Syntax:
* '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false.
* Each test feature is a true identifier.
* Substrings of the target triple are true identifiers for UNSUPPORTED
and XFAIL, but not for REQUIRES. (This matches the current behavior.)
* All other identifiers are false.
* Identifiers are [-+=._a-zA-Z0-9]+
Differential Revision: https://reviews.llvm.org/D18185
Added:
llvm/trunk/utils/lit/lit/BooleanExpression.py
llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-star.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-triple.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-false.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-true.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-star.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-false.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-true.txt
llvm/trunk/utils/lit/tests/boolean-parsing.py
Removed:
llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any-missing.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any-present.txt
Modified:
llvm/trunk/docs/TestingGuide.rst
llvm/trunk/utils/lit/lit/Test.py
llvm/trunk/utils/lit/lit/TestRunner.py
llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-missing.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-present.txt
llvm/trunk/utils/lit/tests/shtest-format.py
llvm/trunk/utils/lit/tests/unit/TestRunner.py
Modified: llvm/trunk/docs/TestingGuide.rst
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/TestingGuide.rst?rev=292896&r1=292895&r2=292896&view=diff
==============================================================================
--- llvm/trunk/docs/TestingGuide.rst (original)
+++ llvm/trunk/docs/TestingGuide.rst Tue Jan 24 02:45:50 2017
@@ -387,23 +387,49 @@ depends on special features of sub-archi
triple, test with the specific FileCheck and put it into the specific
directory that will filter out all other architectures.
-REQUIRES and REQUIRES-ANY directive
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Some tests can be enabled only in specific situation - like having
-debug build. Use ``REQUIRES`` directive to specify those requirements.
+Constraining test execution
+---------------------------
+
+Some tests can be run only in specific configurations, such as
+with debug builds or on particular platforms. Use ``REQUIRES``
+and ``UNSUPPORTED`` to control when the test is enabled.
+
+Some tests are expected to fail. For example, there may be a known bug
+that the test detect. Use ``XFAIL`` to mark a test as an expected failure.
+An ``XFAIL`` test will be successful if its execution fails, and
+will be a failure if its execution succeeds.
.. code-block:: llvm
- ; This test will be only enabled in the build with asserts
+ ; This test will be only enabled in the build with asserts.
; REQUIRES: asserts
+ ; This test is disabled on Linux.
+ ; UNSUPPORTED: -linux-
+ ; This test is expected to fail on PowerPC.
+ ; XFAIL: powerpc
+
+``REQUIRES`` and ``UNSUPPORTED`` and ``XFAIL`` all accept a comma-separated
+list of boolean expressions. The values in each expression may be:
+
+- Features added to ``config.available_features`` by
+ configuration files such as ``lit.cfg``.
+- Substrings of the target triple (``UNSUPPORTED`` and ``XFAIL`` only).
+
+| ``REQUIRES`` enables the test if all expressions are true.
+| ``UNSUPPORTED`` disables the test if any expression is true.
+| ``XFAIL`` expects the test to fail if any expression is true.
+
+As a special case, ``XFAIL: *`` is expected to fail everywhere.
+
+.. code-block:: llvm
-You can separate requirements by a comma.
-``REQUIRES`` means all listed requirements must be satisfied.
-``REQUIRES-ANY`` means at least one must be satisfied.
+ ; This test is disabled on Windows,
+ ; and is disabled on Linux, except for Android Linux.
+ ; UNSUPPORTED: windows, linux && !android
+ ; This test is expected to fail on both PowerPC and ARM.
+ ; XFAIL: powerpc || arm
-List of features that can be used in ``REQUIRES`` and ``REQUIRES-ANY`` can be
-found in lit.cfg files.
Substitutions
-------------
@@ -520,24 +546,6 @@ their name. For example:
This program runs its arguments and then inverts the result code from it.
Zero result codes become 1. Non-zero result codes become 0.
-Sometimes it is necessary to mark a test case as "expected fail" or
-XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:``
-on a line near the top of the file. This signals that the test case
-should succeed if the test fails. Such test cases are counted separately
-by the testing tool. To specify an expected fail, use the XFAIL keyword
-in the comments of the test program followed by a colon and one or more
-failure patterns. Each failure pattern can be either ``*`` (to specify
-fail everywhere), or a part of a target triple (indicating the test
-should fail on that platform), or the name of a configurable feature
-(for example, ``loadable_module``). If there is a match, the test is
-expected to fail. If not, the test is expected to succeed. To XFAIL
-everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL``
-line:
-
-.. code-block:: llvm
-
- ; XFAIL: darwin,sun
-
To make the output more useful, :program:`lit` will scan
the lines of the test case for ones that contain a pattern that matches
``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number
Added: llvm/trunk/utils/lit/lit/BooleanExpression.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/lit/BooleanExpression.py?rev=292896&view=auto
==============================================================================
--- llvm/trunk/utils/lit/lit/BooleanExpression.py (added)
+++ llvm/trunk/utils/lit/lit/BooleanExpression.py Tue Jan 24 02:45:50 2017
@@ -0,0 +1,251 @@
+import re
+
+class BooleanExpression:
+ # A simple evaluator of boolean expressions.
+ #
+ # Grammar:
+ # expr :: or_expr
+ # or_expr :: and_expr ('||' and_expr)*
+ # and_expr :: not_expr ('&&' not_expr)*
+ # not_expr :: '!' not_expr
+ # '(' or_expr ')'
+ # identifier
+ # identifier :: [-+=._a-zA-Z0-9]+
+
+ # Evaluates `string` as a boolean expression.
+ # Returns True or False. Throws a ValueError on syntax error.
+ #
+ # Variables in `variables` are true.
+ # Substrings of `triple` are true.
+ # 'true' is true.
+ # All other identifiers are false.
+ @staticmethod
+ def evaluate(string, variables, triple=""):
+ try:
+ parser = BooleanExpression(string, set(variables), triple)
+ return parser.parseAll()
+ except ValueError as e:
+ raise ValueError(str(e) + ('\nin expression: %r' % string))
+
+ #####
+
+ def __init__(self, string, variables, triple=""):
+ self.tokens = BooleanExpression.tokenize(string)
+ self.variables = variables
+ self.variables.add('true')
+ self.triple = triple
+ self.value = None
+ self.token = None
+
+ # Singleton end-of-expression marker.
+ END = object()
+
+ # Tokenization pattern.
+ Pattern = re.compile(r'\A\s*([()]|[-+=._a-zA-Z0-9]+|&&|\|\||!)\s*(.*)\Z')
+
+ @staticmethod
+ def tokenize(string):
+ while True:
+ m = re.match(BooleanExpression.Pattern, string)
+ if m is None:
+ if string == "":
+ yield BooleanExpression.END;
+ return
+ else:
+ raise ValueError("couldn't parse text: %r" % string)
+
+ token = m.group(1)
+ string = m.group(2)
+ yield token
+
+ def quote(self, token):
+ if token is BooleanExpression.END:
+ return '<end of expression>'
+ else:
+ return repr(token)
+
+ def accept(self, t):
+ if self.token == t:
+ self.token = next(self.tokens)
+ return True
+ else:
+ return False
+
+ def expect(self, t):
+ if self.token == t:
+ if self.token != BooleanExpression.END:
+ self.token = next(self.tokens)
+ else:
+ raise ValueError("expected: %s\nhave: %s" %
+ (self.quote(t), self.quote(self.token)))
+
+ def isIdentifier(self, t):
+ if (t is BooleanExpression.END or t == '&&' or t == '||' or
+ t == '!' or t == '(' or t == ')'):
+ return False
+ return True
+
+ def parseNOT(self):
+ if self.accept('!'):
+ self.parseNOT()
+ self.value = not self.value
+ elif self.accept('('):
+ self.parseOR()
+ self.expect(')')
+ elif not self.isIdentifier(self.token):
+ raise ValueError("expected: '!' or '(' or identifier\nhave: %s" %
+ self.quote(self.token))
+ else:
+ self.value = (self.token in self.variables or
+ self.token in self.triple)
+ self.token = next(self.tokens)
+
+ def parseAND(self):
+ self.parseNOT()
+ while self.accept('&&'):
+ left = self.value
+ self.parseNOT()
+ right = self.value
+ # this is technically the wrong associativity, but it
+ # doesn't matter for this limited expression grammar
+ self.value = left and right
+
+ def parseOR(self):
+ self.parseAND()
+ while self.accept('||'):
+ left = self.value
+ self.parseAND()
+ right = self.value
+ # this is technically the wrong associativity, but it
+ # doesn't matter for this limited expression grammar
+ self.value = left or right
+
+ def parseAll(self):
+ self.token = next(self.tokens)
+ self.parseOR()
+ self.expect(BooleanExpression.END)
+ return self.value
+
+
+#######
+# Tests
+
+import unittest
+
+class TestBooleanExpression(unittest.TestCase):
+ def test_variables(self):
+ variables = {'its-true', 'false-lol-true', 'under_score',
+ 'e=quals', 'd1g1ts'}
+ self.assertTrue(BooleanExpression.evaluate('true', variables))
+ self.assertTrue(BooleanExpression.evaluate('its-true', variables))
+ self.assertTrue(BooleanExpression.evaluate('false-lol-true', variables))
+ self.assertTrue(BooleanExpression.evaluate('under_score', variables))
+ self.assertTrue(BooleanExpression.evaluate('e=quals', variables))
+ self.assertTrue(BooleanExpression.evaluate('d1g1ts', variables))
+
+ self.assertFalse(BooleanExpression.evaluate('false', variables))
+ self.assertFalse(BooleanExpression.evaluate('True', variables))
+ self.assertFalse(BooleanExpression.evaluate('true-ish', variables))
+ self.assertFalse(BooleanExpression.evaluate('not_true', variables))
+ self.assertFalse(BooleanExpression.evaluate('tru', variables))
+
+ def test_triple(self):
+ triple = 'arch-vendor-os'
+ self.assertTrue(BooleanExpression.evaluate('arch-', {}, triple))
+ self.assertTrue(BooleanExpression.evaluate('ar', {}, triple))
+ self.assertTrue(BooleanExpression.evaluate('ch-vend', {}, triple))
+ self.assertTrue(BooleanExpression.evaluate('-vendor-', {}, triple))
+ self.assertTrue(BooleanExpression.evaluate('-os', {}, triple))
+ self.assertFalse(BooleanExpression.evaluate('arch-os', {}, triple))
+
+ def test_operators(self):
+ self.assertTrue(BooleanExpression.evaluate('true || true', {}))
+ self.assertTrue(BooleanExpression.evaluate('true || false', {}))
+ self.assertTrue(BooleanExpression.evaluate('false || true', {}))
+ self.assertFalse(BooleanExpression.evaluate('false || false', {}))
+
+ self.assertTrue(BooleanExpression.evaluate('true && true', {}))
+ self.assertFalse(BooleanExpression.evaluate('true && false', {}))
+ self.assertFalse(BooleanExpression.evaluate('false && true', {}))
+ self.assertFalse(BooleanExpression.evaluate('false && false', {}))
+
+ self.assertFalse(BooleanExpression.evaluate('!true', {}))
+ self.assertTrue(BooleanExpression.evaluate('!false', {}))
+
+ self.assertTrue(BooleanExpression.evaluate(' ((!((false) )) ) ', {}))
+ self.assertTrue(BooleanExpression.evaluate('true && (true && (true))', {}))
+ self.assertTrue(BooleanExpression.evaluate('!false && !false && !! !false', {}))
+ self.assertTrue(BooleanExpression.evaluate('false && false || true', {}))
+ self.assertTrue(BooleanExpression.evaluate('(false && false) || true', {}))
+ self.assertFalse(BooleanExpression.evaluate('false && (false || true)', {}))
+
+ # Evaluate boolean expression `expr`.
+ # Fail if it does not throw a ValueError containing the text `error`.
+ def checkException(self, expr, error):
+ try:
+ BooleanExpression.evaluate(expr, {})
+ self.fail("expression %r didn't cause an exception" % expr)
+ except ValueError as e:
+ if -1 == str(e).find(error):
+ self.fail(("expression %r caused the wrong ValueError\n" +
+ "actual error was:\n%s\n" +
+ "expected error was:\n%s\n") % (expr, e, error))
+ except BaseException as e:
+ self.fail(("expression %r caused the wrong exception; actual " +
+ "exception was: \n%r") % (expr, e))
+
+ def test_errors(self):
+ self.checkException("ba#d",
+ "couldn't parse text: '#d'\n" +
+ "in expression: 'ba#d'")
+
+ self.checkException("true and true",
+ "expected: <end of expression>\n" +
+ "have: 'and'\n" +
+ "in expression: 'true and true'")
+
+ self.checkException("|| true",
+ "expected: '!' or '(' or identifier\n" +
+ "have: '||'\n" +
+ "in expression: '|| true'")
+
+ self.checkException("true &&",
+ "expected: '!' or '(' or identifier\n" +
+ "have: <end of expression>\n" +
+ "in expression: 'true &&'")
+
+ self.checkException("",
+ "expected: '!' or '(' or identifier\n" +
+ "have: <end of expression>\n" +
+ "in expression: ''")
+
+ self.checkException("*",
+ "couldn't parse text: '*'\n" +
+ "in expression: '*'")
+
+ self.checkException("no wait stop",
+ "expected: <end of expression>\n" +
+ "have: 'wait'\n" +
+ "in expression: 'no wait stop'")
+
+ self.checkException("no-$-please",
+ "couldn't parse text: '$-please'\n" +
+ "in expression: 'no-$-please'")
+
+ self.checkException("(((true && true) || true)",
+ "expected: ')'\n" +
+ "have: <end of expression>\n" +
+ "in expression: '(((true && true) || true)'")
+
+ self.checkException("true (true)",
+ "expected: <end of expression>\n" +
+ "have: '('\n" +
+ "in expression: 'true (true)'")
+
+ self.checkException("( )",
+ "expected: '!' or '(' or identifier\n" +
+ "have: ')'\n" +
+ "in expression: '( )'")
+
+if __name__ == '__main__':
+ unittest.main()
Modified: llvm/trunk/utils/lit/lit/Test.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/lit/Test.py?rev=292896&r1=292895&r2=292896&view=diff
==============================================================================
--- llvm/trunk/utils/lit/lit/Test.py (original)
+++ llvm/trunk/utils/lit/lit/Test.py Tue Jan 24 02:45:50 2017
@@ -2,6 +2,8 @@ import os
from xml.sax.saxutils import escape
from json import JSONEncoder
+from lit.BooleanExpression import BooleanExpression
+
# Test result codes.
class ResultCode(object):
@@ -180,10 +182,24 @@ class Test:
self.path_in_suite = path_in_suite
self.config = config
self.file_path = file_path
- # A list of conditions under which this test is expected to fail. These
- # can optionally be provided by test format handlers, and will be
- # honored when the test result is supplied.
+
+ # A list of conditions under which this test is expected to fail.
+ # Each condition is a boolean expression of features and target
+ # triple parts. These can optionally be provided by test format
+ # handlers, and will be honored when the test result is supplied.
self.xfails = []
+
+ # A list of conditions that must be satisfied before running the test.
+ # Each condition is a boolean expression of features. All of them
+ # must be True for the test to run.
+ # FIXME should target triple parts count here too?
+ self.requires = []
+
+ # A list of conditions that prevent execution of the test.
+ # Each condition is a boolean expression of features and target
+ # triple parts. All of them must be False for the test to run.
+ self.unsupported = []
+
# The test result, once complete.
self.result = None
@@ -196,11 +212,16 @@ class Test:
self.result = result
# Apply the XFAIL handling to resolve the result exit code.
- if self.isExpectedToFail():
- if self.result.code == PASS:
- self.result.code = XPASS
- elif self.result.code == FAIL:
- self.result.code = XFAIL
+ try:
+ if self.isExpectedToFail():
+ if self.result.code == PASS:
+ self.result.code = XPASS
+ elif self.result.code == FAIL:
+ self.result.code = XFAIL
+ except ValueError as e:
+ # Syntax error in an XFAIL line.
+ self.result.code = UNRESOLVED
+ self.result.output = str(e)
def getFullName(self):
return self.suite.config.name + ' :: ' + '/'.join(self.path_in_suite)
@@ -224,24 +245,91 @@ class Test:
configuration. This check relies on the test xfails property which by
some test formats may not be computed until the test has first been
executed.
+ Throws ValueError if an XFAIL line has a syntax error.
"""
+ features = self.config.available_features
+ triple = getattr(self.suite.config, 'target_triple', "")
+
# Check if any of the xfails match an available feature or the target.
for item in self.xfails:
# If this is the wildcard, it always fails.
if item == '*':
return True
- # If this is an exact match for one of the features, it fails.
- if item in self.config.available_features:
- return True
-
- # If this is a part of the target triple, it fails.
- if item and item in self.suite.config.target_triple:
- return True
+ # If this is a True expression of features and target triple parts,
+ # it fails.
+ try:
+ if BooleanExpression.evaluate(item, features, triple):
+ return True
+ except ValueError as e:
+ raise ValueError('Error in XFAIL list:\n%s' % str(e))
return False
+ def isWithinFeatureLimits(self):
+ """
+ isWithinFeatureLimits() -> bool
+
+ A test is within the feature limits set by run_only_tests if
+ 1. the test's requirements ARE satisfied by the available features
+ 2. the test's requirements ARE NOT satisfied after the limiting
+ features are removed from the available features
+
+ Throws ValueError if a REQUIRES line has a syntax error.
+ """
+
+ if not self.config.limit_to_features:
+ return True # No limits. Run it.
+
+ # Check the requirements as-is (#1)
+ if self.getMissingRequiredFeatures():
+ return False
+
+ # Check the requirements after removing the limiting features (#2)
+ featuresMinusLimits = [f for f in self.config.available_features
+ if not f in self.config.limit_to_features]
+ if not self.getMissingRequiredFeaturesFromList(featuresMinusLimits):
+ return False
+
+ return True
+
+ def getMissingRequiredFeaturesFromList(self, features):
+ try:
+ return [item for item in self.requires
+ if not BooleanExpression.evaluate(item, features)]
+ except ValueError as e:
+ raise ValueError('Error in REQUIRES list:\n%s' % str(e))
+
+ def getMissingRequiredFeatures(self):
+ """
+ getMissingRequiredFeatures() -> list of strings
+
+ Returns a list of features from REQUIRES that are not satisfied."
+ Throws ValueError if a REQUIRES line has a syntax error.
+ """
+
+ features = self.config.available_features
+ return self.getMissingRequiredFeaturesFromList(features)
+
+ def getUnsupportedFeatures(self):
+ """
+ getUnsupportedFeatures() -> list of strings
+
+ Returns a list of features from UNSUPPORTED that are present
+ in the test configuration's features or target triple.
+ Throws ValueError if an UNSUPPORTED line has a syntax error.
+ """
+
+ features = self.config.available_features
+ triple = getattr(self.suite.config, 'target_triple', "")
+
+ try:
+ return [item for item in self.unsupported
+ if BooleanExpression.evaluate(item, features, triple)]
+ except ValueError as e:
+ raise ValueError('Error in UNSUPPORTED list:\n%s' % str(e))
+
def isEarlyTest(self):
"""
isEarlyTest() -> bool
Modified: llvm/trunk/utils/lit/lit/TestRunner.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/lit/TestRunner.py?rev=292896&r1=292895&r2=292896&view=diff
==============================================================================
--- llvm/trunk/utils/lit/lit/TestRunner.py (original)
+++ llvm/trunk/utils/lit/lit/TestRunner.py Tue Jan 24 02:45:50 2017
@@ -9,6 +9,7 @@ import lit.ShUtil as ShUtil
import lit.Test as Test
import lit.util
from lit.util import to_bytes, to_string
+from lit.BooleanExpression import BooleanExpression
class InternalShellError(Exception):
def __init__(self, command, message):
@@ -746,14 +747,35 @@ class ParserKind(object):
command.
TAG: A keyword taking no value. Ex 'END.'
- COMMAND: A Keyword taking a list of shell commands. Ex 'RUN:'
- LIST: A keyword taking a comma separated list of value. Ex 'XFAIL:'
+ COMMAND: A keyword taking a list of shell commands. Ex 'RUN:'
+ LIST: A keyword taking a comma-separated list of values.
+ BOOLEAN_EXPR: A keyword taking a comma-separated list of
+ boolean expressions. Ex 'XFAIL:'
CUSTOM: A keyword with custom parsing semantics.
"""
TAG = 0
COMMAND = 1
LIST = 2
- CUSTOM = 3
+ BOOLEAN_EXPR = 3
+ CUSTOM = 4
+
+ @staticmethod
+ def allowedKeywordSuffixes(value):
+ return { ParserKind.TAG: ['.'],
+ ParserKind.COMMAND: [':'],
+ ParserKind.LIST: [':'],
+ ParserKind.BOOLEAN_EXPR: [':'],
+ ParserKind.CUSTOM: [':', '.']
+ } [value]
+
+ @staticmethod
+ def str(value):
+ return { ParserKind.TAG: 'TAG',
+ ParserKind.COMMAND: 'COMMAND',
+ ParserKind.LIST: 'LIST',
+ ParserKind.BOOLEAN_EXPR: 'BOOLEAN_EXPR',
+ ParserKind.CUSTOM: 'CUSTOM'
+ } [value]
class IntegratedTestKeywordParser(object):
@@ -765,15 +787,18 @@ class IntegratedTestKeywordParser(object
ParserKind.CUSTOM.
"""
def __init__(self, keyword, kind, parser=None, initial_value=None):
- if not keyword.endswith('.') and not keyword.endswith(':'):
- raise ValueError("keyword '%s' must end with either '.' or ':' "
- % keyword)
- if keyword.endswith('.') and kind in \
- [ParserKind.LIST, ParserKind.COMMAND]:
- raise ValueError("Keyword '%s' should end in ':'" % keyword)
+ allowedSuffixes = ParserKind.allowedKeywordSuffixes(kind)
+ if len(keyword) == 0 or keyword[-1] not in allowedSuffixes:
+ if len(allowedSuffixes) == 1:
+ raise ValueError("Keyword '%s' of kind '%s' must end in '%s'"
+ % (keyword, ParserKind.str(kind),
+ allowedSuffixes[0]))
+ else:
+ raise ValueError("Keyword '%s' of kind '%s' must end in "
+ " one of '%s'"
+ % (keyword, ParserKind.str(kind),
+ ' '.join(allowedSuffixes)))
- elif keyword.endswith(':') and kind in [ParserKind.TAG]:
- raise ValueError("Keyword '%s' should end in '.'" % keyword)
if parser is not None and kind != ParserKind.CUSTOM:
raise ValueError("custom parsers can only be specified with "
"ParserKind.CUSTOM")
@@ -787,9 +812,9 @@ class IntegratedTestKeywordParser(object
self.parser = self._handleCommand
elif kind == ParserKind.LIST:
self.parser = self._handleList
+ elif kind == ParserKind.BOOLEAN_EXPR:
+ self.parser = self._handleBooleanExpr
elif kind == ParserKind.TAG:
- if not keyword.endswith('.'):
- raise ValueError("keyword '%s' should end with '.'" % keyword)
self.parser = self._handleTag
elif kind == ParserKind.CUSTOM:
if parser is None:
@@ -799,8 +824,12 @@ class IntegratedTestKeywordParser(object
raise ValueError("Unknown kind '%s'" % kind)
def parseLine(self, line_number, line):
- self.parsed_lines += [(line_number, line)]
- self.value = self.parser(line_number, line, self.value)
+ try:
+ self.parsed_lines += [(line_number, line)]
+ self.value = self.parser(line_number, line, self.value)
+ except ValueError as e:
+ raise ValueError(str(e) + ("\nin %s directive on test line %d" %
+ (self.keyword, line_number)))
def getValue(self):
return self.value
@@ -841,12 +870,24 @@ class IntegratedTestKeywordParser(object
output.extend([s.strip() for s in line.split(',')])
return output
+ @staticmethod
+ def _handleBooleanExpr(line_number, line, output):
+ """A parser for BOOLEAN_EXPR type keywords"""
+ if output is None:
+ output = []
+ output.extend([s.strip() for s in line.split(',')])
+ # Evaluate each expression to verify syntax.
+ # We don't want any results, just the raised ValueError.
+ for s in output:
+ if s != '*':
+ BooleanExpression.evaluate(s, [])
+ return output
def parseIntegratedTestScript(test, additional_parsers=[],
require_script=True):
"""parseIntegratedTestScript - Scan an LLVM/Clang style integrated test
script and extract the lines to 'RUN' as well as 'XFAIL' and 'REQUIRES'
- 'REQUIRES-ANY' and 'UNSUPPORTED' information.
+ and 'UNSUPPORTED' information.
If additional parsers are specified then the test is also scanned for the
keywords they specify and all matches are passed to the custom parser.
@@ -855,26 +896,23 @@ def parseIntegratedTestScript(test, addi
may be returned. This can be used for test formats where the actual script
is optional or ignored.
"""
- # Collect the test lines from the script.
- sourcepath = test.getSourcePath()
+
+ # Install the built-in keyword parsers.
script = []
- requires = []
- requires_any = []
- unsupported = []
builtin_parsers = [
IntegratedTestKeywordParser('RUN:', ParserKind.COMMAND,
initial_value=script),
- IntegratedTestKeywordParser('XFAIL:', ParserKind.LIST,
+ IntegratedTestKeywordParser('XFAIL:', ParserKind.BOOLEAN_EXPR,
initial_value=test.xfails),
- IntegratedTestKeywordParser('REQUIRES:', ParserKind.LIST,
- initial_value=requires),
- IntegratedTestKeywordParser('REQUIRES-ANY:', ParserKind.LIST,
- initial_value=requires_any),
- IntegratedTestKeywordParser('UNSUPPORTED:', ParserKind.LIST,
- initial_value=unsupported),
+ IntegratedTestKeywordParser('REQUIRES:', ParserKind.BOOLEAN_EXPR,
+ initial_value=test.requires),
+ IntegratedTestKeywordParser('UNSUPPORTED:', ParserKind.BOOLEAN_EXPR,
+ initial_value=test.unsupported),
IntegratedTestKeywordParser('END.', ParserKind.TAG)
]
keyword_parsers = {p.keyword: p for p in builtin_parsers}
+
+ # Install user-defined additional parsers.
for parser in additional_parsers:
if not isinstance(parser, IntegratedTestKeywordParser):
raise ValueError('additional parser must be an instance of '
@@ -884,6 +922,18 @@ def parseIntegratedTestScript(test, addi
% parser.keyword)
keyword_parsers[parser.keyword] = parser
+ # Install a helpful error-generating parser for the no-longer-supported
+ # REQUIRES-ANY: keyword, if no other parser for it exists.
+ if 'REQUIRES-ANY:' not in keyword_parsers:
+ def requires_any_error_parser(line_number, line, output):
+ raise ValueError('`REQUIRES-ANY: a, b, c` not supported. Use '
+ '`REQUIRES: a || b || c` instead.')
+ parser = IntegratedTestKeywordParser('REQUIRES-ANY:', ParserKind.CUSTOM,
+ requires_any_error_parser)
+ keyword_parsers[parser.keyword] = parser
+
+ # Collect the test lines from the script.
+ sourcepath = test.getSourcePath()
for line_number, command_type, ln in \
parseIntegratedTestScriptCommands(sourcepath,
keyword_parsers.keys()):
@@ -901,46 +951,30 @@ def parseIntegratedTestScript(test, addi
return lit.Test.Result(Test.UNRESOLVED,
"Test has unterminated run lines (with '\\')")
- # Check that we have the required features:
- missing_required_features = [f for f in requires
- if f not in test.config.available_features]
+ # Enforce REQUIRES:
+ missing_required_features = test.getMissingRequiredFeatures()
if missing_required_features:
msg = ', '.join(missing_required_features)
return lit.Test.Result(Test.UNSUPPORTED,
- "Test requires the following features: %s"
- % msg)
- requires_any_features = [f for f in requires_any
- if f in test.config.available_features]
- if requires_any and not requires_any_features:
- msg = ' ,'.join(requires_any)
- return lit.Test.Result(Test.UNSUPPORTED,
- "Test requires any of the following features: "
- "%s" % msg)
- unsupported_features = [f for f in unsupported
- if f in test.config.available_features]
+ "Test requires the following unavailable "
+ "features: %s" % msg)
+
+ # Enforce UNSUPPORTED:
+ unsupported_features = test.getUnsupportedFeatures()
if unsupported_features:
msg = ', '.join(unsupported_features)
return lit.Test.Result(
Test.UNSUPPORTED,
- "Test is unsupported with the following features: %s" % msg)
+ "Test does not support the following features "
+ "and/or targets: %s" % msg)
- unsupported_targets = [f for f in unsupported
- if f in test.suite.config.target_triple]
- if unsupported_targets:
- return lit.Test.Result(
- Test.UNSUPPORTED,
- "Test is unsupported with the following triple: %s" % (
- test.suite.config.target_triple,))
+ # Enforce limit_to_features.
+ if not test.isWithinFeatureLimits():
+ msg = ', '.join(test.config.limit_to_features)
+ return lit.Test.Result(Test.UNSUPPORTED,
+ "Test does not require any of the features "
+ "specified in limit_to_features: %s" % msg)
- if test.config.limit_to_features:
- # Check that we have one of the limit_to_features features in requires.
- limit_to_features_tests = [f for f in test.config.limit_to_features
- if f in requires]
- if not limit_to_features_tests:
- msg = ', '.join(test.config.limit_to_features)
- return lit.Test.Result(
- Test.UNSUPPORTED,
- "Test requires one of the limit_to_features features %s" % msg)
return script
Removed: llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any-missing.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any-missing.txt?rev=292895&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any-missing.txt (original)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any-missing.txt (removed)
@@ -1,2 +0,0 @@
-RUN: true
-REQUIRES-ANY: a-missing-feature, a-missing-feature-2
Removed: llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any-present.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any-present.txt?rev=292895&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any-present.txt (original)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any-present.txt (removed)
@@ -1,2 +0,0 @@
-RUN: true
-REQUIRES-ANY: a-missing-feature, a-present-feature
Added: llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any.txt?rev=292896&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any.txt (added)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any.txt Tue Jan 24 02:45:50 2017
@@ -0,0 +1,3 @@
+# REQUIRES-ANY is no longer supported. Test should not run.
+REQUIRES-ANY: true, a-present-feature
+RUN: false
Modified: llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-missing.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-missing.txt?rev=292896&r1=292895&r2=292896&view=diff
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-missing.txt (original)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-missing.txt Tue Jan 24 02:45:50 2017
@@ -1,2 +1,5 @@
-RUN: true
-REQUIRES: a-missing-feature
+# REQUIRES with a false clause. Test should not run.
+REQUIRES: true
+REQUIRES: a-missing-feature, true
+REQUIRES: true
+RUN: false
Modified: llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-present.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-present.txt?rev=292896&r1=292895&r2=292896&view=diff
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-present.txt (original)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-present.txt Tue Jan 24 02:45:50 2017
@@ -1,2 +1,4 @@
+# REQUIRES with only true clauses. Test should run.
+REQUIRES: a-present-feature, true, !not-true
+REQUIRES: true
RUN: true
-REQUIRES: a-present-feature
Added: llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-star.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-star.txt?rev=292896&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-star.txt (added)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-star.txt Tue Jan 24 02:45:50 2017
@@ -0,0 +1,3 @@
+# '*' only works in XFAIL
+REQUIRES: *
+RUN: false
Added: llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-triple.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-triple.txt?rev=292896&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-triple.txt (added)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-triple.txt Tue Jan 24 02:45:50 2017
@@ -0,0 +1,3 @@
+# REQUIRES line that uses target triple, which doesn't work. Test should not run
+REQUIRES: x86_64
+RUN: false
Added: llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-false.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-false.txt?rev=292896&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-false.txt (added)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-false.txt Tue Jan 24 02:45:50 2017
@@ -0,0 +1,9 @@
+# UNSUPPORTED with only false clauses. Test should run.
+UNSUPPORTED: false
+UNSUPPORTED: false, not-true
+UNSUPPORTED: false
+UNSUPPORTED: still-not-true
+UNSUPPORTED: false
+UNSUPPORTED: false
+UNSUPPORTED: false
+RUN: true
Added: llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-true.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-true.txt?rev=292896&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-true.txt (added)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-true.txt Tue Jan 24 02:45:50 2017
@@ -0,0 +1,4 @@
+# UNSUPPORTED with a true clause. Test should not run.
+UNSUPPORTED: false
+UNSUPPORTED: false, false, false, _64-unk && a-present-feature, false
+RUN: false
Added: llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-star.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-star.txt?rev=292896&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-star.txt (added)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-star.txt Tue Jan 24 02:45:50 2017
@@ -0,0 +1,3 @@
+# '*' only works in XFAIL
+UNSUPPORTED: *
+RUN: false
Added: llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-false.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-false.txt?rev=292896&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-false.txt (added)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-false.txt Tue Jan 24 02:45:50 2017
@@ -0,0 +1,3 @@
+# XFAIL with only false clauses. Test should run.
+XFAIL: false, a-missing-feature || ! a-present-feature || ! x86_64, false
+RUN: true
Added: llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-true.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-true.txt?rev=292896&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-true.txt (added)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-true.txt Tue Jan 24 02:45:50 2017
@@ -0,0 +1,4 @@
+# XFAIL with a true clause. Test should not run.
+XFAIL: false
+XFAIL: false, a-present-feature && ! a-missing-feature && x86_64
+RUN: false
Added: llvm/trunk/utils/lit/tests/boolean-parsing.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/boolean-parsing.py?rev=292896&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/boolean-parsing.py (added)
+++ llvm/trunk/utils/lit/tests/boolean-parsing.py Tue Jan 24 02:45:50 2017
@@ -0,0 +1,4 @@
+# Test the boolean expression parser
+# used for REQUIRES and UNSUPPORTED and XFAIL
+
+# RUN: %{python} -m lit.BooleanExpression
Modified: llvm/trunk/utils/lit/tests/shtest-format.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/shtest-format.py?rev=292896&r1=292895&r2=292896&view=diff
==============================================================================
--- llvm/trunk/utils/lit/tests/shtest-format.py (original)
+++ llvm/trunk/utils/lit/tests/shtest-format.py Tue Jan 24 02:45:50 2017
@@ -46,11 +46,18 @@
# CHECK: UNRESOLVED: shtest-format :: no-test-line.txt
# CHECK: PASS: shtest-format :: pass.txt
-# CHECK: UNSUPPORTED: shtest-format :: requires-any-missing.txt
-# CHECK: PASS: shtest-format :: requires-any-present.txt
+# CHECK: UNRESOLVED: shtest-format :: requires-any.txt
+# CHECK: ValueError: `REQUIRES-ANY: a, b, c` not supported
# CHECK: UNSUPPORTED: shtest-format :: requires-missing.txt
# CHECK: PASS: shtest-format :: requires-present.txt
+# CHECK: UNRESOLVED: shtest-format :: requires-star.txt
+# CHECK: UNSUPPORTED: shtest-format :: requires-triple.txt
+# CHECK: PASS: shtest-format :: unsupported-expr-false.txt
+# CHECK: UNSUPPORTED: shtest-format :: unsupported-expr-true.txt
+# CHECK: UNRESOLVED: shtest-format :: unsupported-star.txt
# CHECK: UNSUPPORTED: shtest-format :: unsupported_dir/some-test.txt
+# CHECK: PASS: shtest-format :: xfail-expr-false.txt
+# CHECK: XFAIL: shtest-format :: xfail-expr-true.txt
# CHECK: XFAIL: shtest-format :: xfail-feature.txt
# CHECK: XFAIL: shtest-format :: xfail-target.txt
# CHECK: XFAIL: shtest-format :: xfail.txt
@@ -70,9 +77,9 @@
# CHECK: shtest-format :: external_shell/fail_with_bad_encoding.txt
# CHECK: shtest-format :: fail.txt
-# CHECK: Expected Passes : 5
-# CHECK: Expected Failures : 3
-# CHECK: Unsupported Tests : 3
-# CHECK: Unresolved Tests : 1
+# CHECK: Expected Passes : 6
+# CHECK: Expected Failures : 4
+# CHECK: Unsupported Tests : 4
+# CHECK: Unresolved Tests : 4
# CHECK: Unexpected Passes : 1
# CHECK: Unexpected Failures: 3
Modified: llvm/trunk/utils/lit/tests/unit/TestRunner.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/unit/TestRunner.py?rev=292896&r1=292895&r2=292896&view=diff
==============================================================================
--- llvm/trunk/utils/lit/tests/unit/TestRunner.py (original)
+++ llvm/trunk/utils/lit/tests/unit/TestRunner.py Tue Jan 24 02:45:50 2017
@@ -108,6 +108,71 @@ class TestIntegratedTestKeywordParser(un
value = custom_parser.getValue()
self.assertItemsEqual(value, ['a', 'b', 'c'])
+ def test_bad_keywords(self):
+ def custom_parse(line_number, line, output):
+ return output
+
+ try:
+ IntegratedTestKeywordParser("TAG_NO_SUFFIX", ParserKind.TAG),
+ self.fail("TAG_NO_SUFFIX failed to raise an exception")
+ except ValueError as e:
+ pass
+ except BaseException as e:
+ self.fail("TAG_NO_SUFFIX raised the wrong exception: %r" % e)
+
+ try:
+ IntegratedTestKeywordParser("TAG_WITH_COLON:", ParserKind.TAG),
+ self.fail("TAG_WITH_COLON: failed to raise an exception")
+ except ValueError as e:
+ pass
+ except BaseException as e:
+ self.fail("TAG_WITH_COLON: raised the wrong exception: %r" % e)
+
+ try:
+ IntegratedTestKeywordParser("LIST_WITH_DOT.", ParserKind.LIST),
+ self.fail("LIST_WITH_DOT. failed to raise an exception")
+ except ValueError as e:
+ pass
+ except BaseException as e:
+ self.fail("LIST_WITH_DOT. raised the wrong exception: %r" % e)
+
+ try:
+ IntegratedTestKeywordParser("CUSTOM_NO_SUFFIX",
+ ParserKind.CUSTOM, custom_parse),
+ self.fail("CUSTOM_NO_SUFFIX failed to raise an exception")
+ except ValueError as e:
+ pass
+ except BaseException as e:
+ self.fail("CUSTOM_NO_SUFFIX raised the wrong exception: %r" % e)
+
+ # Both '.' and ':' are allowed for CUSTOM keywords.
+ try:
+ IntegratedTestKeywordParser("CUSTOM_WITH_DOT.",
+ ParserKind.CUSTOM, custom_parse),
+ except BaseException as e:
+ self.fail("CUSTOM_WITH_DOT. raised an exception: %r" % e)
+ try:
+ IntegratedTestKeywordParser("CUSTOM_WITH_COLON:",
+ ParserKind.CUSTOM, custom_parse),
+ except BaseException as e:
+ self.fail("CUSTOM_WITH_COLON: raised an exception: %r" % e)
+
+ try:
+ IntegratedTestKeywordParser("CUSTOM_NO_PARSER:",
+ ParserKind.CUSTOM),
+ self.fail("CUSTOM_NO_PARSER: failed to raise an exception")
+ except ValueError as e:
+ pass
+ except BaseException as e:
+ self.fail("CUSTOM_NO_PARSER: raised the wrong exception: %r" % e)
+
+ # REQUIRES-ANY: has a built-in parser that generates an error,
+ # but it may be overridden by a custom parser.
+ try:
+ IntegratedTestKeywordParser("REQUIRES-ANY:",
+ ParserKind.CUSTOM, custom_parse),
+ except BaseException as e:
+ self.fail("REQUIRES-ANY: raised an exception: %r" % e)
if __name__ == '__main__':
TestIntegratedTestKeywordParser.load_keyword_parser_lit_tests()
More information about the llvm-commits
mailing list