[llvm] r292900 - Revert "[lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED"
Greg Parker via llvm-commits
llvm-commits at lists.llvm.org
Tue Jan 24 00:58:20 PST 2017
Author: gparker
Date: Tue Jan 24 02:58:20 2017
New Revision: 292900
URL: http://llvm.org/viewvc/llvm-project?rev=292900&view=rev
Log:
Revert "[lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED"
This change needs to be better-coordinated with libc++.
Added:
llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any-missing.txt
- copied unchanged from r292895, llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any-missing.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any-present.txt
- copied unchanged from r292895, llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any-present.txt
Removed:
llvm/trunk/utils/lit/lit/BooleanExpression.py
llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-star.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-triple.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-false.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-true.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-star.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-false.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-true.txt
llvm/trunk/utils/lit/tests/boolean-parsing.py
Modified:
llvm/trunk/docs/TestingGuide.rst
llvm/trunk/utils/lit/lit/Test.py
llvm/trunk/utils/lit/lit/TestRunner.py
llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-missing.txt
llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-present.txt
llvm/trunk/utils/lit/tests/shtest-format.py
llvm/trunk/utils/lit/tests/unit/TestRunner.py
Modified: llvm/trunk/docs/TestingGuide.rst
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/TestingGuide.rst?rev=292900&r1=292899&r2=292900&view=diff
==============================================================================
--- llvm/trunk/docs/TestingGuide.rst (original)
+++ llvm/trunk/docs/TestingGuide.rst Tue Jan 24 02:58:20 2017
@@ -387,49 +387,23 @@ depends on special features of sub-archi
triple, test with the specific FileCheck and put it into the specific
directory that will filter out all other architectures.
+REQUIRES and REQUIRES-ANY directive
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Constraining test execution
----------------------------
-
-Some tests can be run only in specific configurations, such as
-with debug builds or on particular platforms. Use ``REQUIRES``
-and ``UNSUPPORTED`` to control when the test is enabled.
-
-Some tests are expected to fail. For example, there may be a known bug
-that the test detect. Use ``XFAIL`` to mark a test as an expected failure.
-An ``XFAIL`` test will be successful if its execution fails, and
-will be a failure if its execution succeeds.
+Some tests can be enabled only in specific situation - like having
+debug build. Use ``REQUIRES`` directive to specify those requirements.
.. code-block:: llvm
- ; This test will be only enabled in the build with asserts.
+ ; This test will be only enabled in the build with asserts
; REQUIRES: asserts
- ; This test is disabled on Linux.
- ; UNSUPPORTED: -linux-
- ; This test is expected to fail on PowerPC.
- ; XFAIL: powerpc
-
-``REQUIRES`` and ``UNSUPPORTED`` and ``XFAIL`` all accept a comma-separated
-list of boolean expressions. The values in each expression may be:
-
-- Features added to ``config.available_features`` by
- configuration files such as ``lit.cfg``.
-- Substrings of the target triple (``UNSUPPORTED`` and ``XFAIL`` only).
-
-| ``REQUIRES`` enables the test if all expressions are true.
-| ``UNSUPPORTED`` disables the test if any expression is true.
-| ``XFAIL`` expects the test to fail if any expression is true.
-
-As a special case, ``XFAIL: *`` is expected to fail everywhere.
-
-.. code-block:: llvm
- ; This test is disabled on Windows,
- ; and is disabled on Linux, except for Android Linux.
- ; UNSUPPORTED: windows, linux && !android
- ; This test is expected to fail on both PowerPC and ARM.
- ; XFAIL: powerpc || arm
+You can separate requirements by a comma.
+``REQUIRES`` means all listed requirements must be satisfied.
+``REQUIRES-ANY`` means at least one must be satisfied.
+List of features that can be used in ``REQUIRES`` and ``REQUIRES-ANY`` can be
+found in lit.cfg files.
Substitutions
-------------
@@ -546,6 +520,24 @@ their name. For example:
This program runs its arguments and then inverts the result code from it.
Zero result codes become 1. Non-zero result codes become 0.
+Sometimes it is necessary to mark a test case as "expected fail" or
+XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:``
+on a line near the top of the file. This signals that the test case
+should succeed if the test fails. Such test cases are counted separately
+by the testing tool. To specify an expected fail, use the XFAIL keyword
+in the comments of the test program followed by a colon and one or more
+failure patterns. Each failure pattern can be either ``*`` (to specify
+fail everywhere), or a part of a target triple (indicating the test
+should fail on that platform), or the name of a configurable feature
+(for example, ``loadable_module``). If there is a match, the test is
+expected to fail. If not, the test is expected to succeed. To XFAIL
+everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL``
+line:
+
+.. code-block:: llvm
+
+ ; XFAIL: darwin,sun
+
To make the output more useful, :program:`lit` will scan
the lines of the test case for ones that contain a pattern that matches
``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number
Removed: llvm/trunk/utils/lit/lit/BooleanExpression.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/lit/BooleanExpression.py?rev=292899&view=auto
==============================================================================
--- llvm/trunk/utils/lit/lit/BooleanExpression.py (original)
+++ llvm/trunk/utils/lit/lit/BooleanExpression.py (removed)
@@ -1,251 +0,0 @@
-import re
-
-class BooleanExpression:
- # A simple evaluator of boolean expressions.
- #
- # Grammar:
- # expr :: or_expr
- # or_expr :: and_expr ('||' and_expr)*
- # and_expr :: not_expr ('&&' not_expr)*
- # not_expr :: '!' not_expr
- # '(' or_expr ')'
- # identifier
- # identifier :: [-+=._a-zA-Z0-9]+
-
- # Evaluates `string` as a boolean expression.
- # Returns True or False. Throws a ValueError on syntax error.
- #
- # Variables in `variables` are true.
- # Substrings of `triple` are true.
- # 'true' is true.
- # All other identifiers are false.
- @staticmethod
- def evaluate(string, variables, triple=""):
- try:
- parser = BooleanExpression(string, set(variables), triple)
- return parser.parseAll()
- except ValueError as e:
- raise ValueError(str(e) + ('\nin expression: %r' % string))
-
- #####
-
- def __init__(self, string, variables, triple=""):
- self.tokens = BooleanExpression.tokenize(string)
- self.variables = variables
- self.variables.add('true')
- self.triple = triple
- self.value = None
- self.token = None
-
- # Singleton end-of-expression marker.
- END = object()
-
- # Tokenization pattern.
- Pattern = re.compile(r'\A\s*([()]|[-+=._a-zA-Z0-9]+|&&|\|\||!)\s*(.*)\Z')
-
- @staticmethod
- def tokenize(string):
- while True:
- m = re.match(BooleanExpression.Pattern, string)
- if m is None:
- if string == "":
- yield BooleanExpression.END;
- return
- else:
- raise ValueError("couldn't parse text: %r" % string)
-
- token = m.group(1)
- string = m.group(2)
- yield token
-
- def quote(self, token):
- if token is BooleanExpression.END:
- return '<end of expression>'
- else:
- return repr(token)
-
- def accept(self, t):
- if self.token == t:
- self.token = next(self.tokens)
- return True
- else:
- return False
-
- def expect(self, t):
- if self.token == t:
- if self.token != BooleanExpression.END:
- self.token = next(self.tokens)
- else:
- raise ValueError("expected: %s\nhave: %s" %
- (self.quote(t), self.quote(self.token)))
-
- def isIdentifier(self, t):
- if (t is BooleanExpression.END or t == '&&' or t == '||' or
- t == '!' or t == '(' or t == ')'):
- return False
- return True
-
- def parseNOT(self):
- if self.accept('!'):
- self.parseNOT()
- self.value = not self.value
- elif self.accept('('):
- self.parseOR()
- self.expect(')')
- elif not self.isIdentifier(self.token):
- raise ValueError("expected: '!' or '(' or identifier\nhave: %s" %
- self.quote(self.token))
- else:
- self.value = (self.token in self.variables or
- self.token in self.triple)
- self.token = next(self.tokens)
-
- def parseAND(self):
- self.parseNOT()
- while self.accept('&&'):
- left = self.value
- self.parseNOT()
- right = self.value
- # this is technically the wrong associativity, but it
- # doesn't matter for this limited expression grammar
- self.value = left and right
-
- def parseOR(self):
- self.parseAND()
- while self.accept('||'):
- left = self.value
- self.parseAND()
- right = self.value
- # this is technically the wrong associativity, but it
- # doesn't matter for this limited expression grammar
- self.value = left or right
-
- def parseAll(self):
- self.token = next(self.tokens)
- self.parseOR()
- self.expect(BooleanExpression.END)
- return self.value
-
-
-#######
-# Tests
-
-import unittest
-
-class TestBooleanExpression(unittest.TestCase):
- def test_variables(self):
- variables = {'its-true', 'false-lol-true', 'under_score',
- 'e=quals', 'd1g1ts'}
- self.assertTrue(BooleanExpression.evaluate('true', variables))
- self.assertTrue(BooleanExpression.evaluate('its-true', variables))
- self.assertTrue(BooleanExpression.evaluate('false-lol-true', variables))
- self.assertTrue(BooleanExpression.evaluate('under_score', variables))
- self.assertTrue(BooleanExpression.evaluate('e=quals', variables))
- self.assertTrue(BooleanExpression.evaluate('d1g1ts', variables))
-
- self.assertFalse(BooleanExpression.evaluate('false', variables))
- self.assertFalse(BooleanExpression.evaluate('True', variables))
- self.assertFalse(BooleanExpression.evaluate('true-ish', variables))
- self.assertFalse(BooleanExpression.evaluate('not_true', variables))
- self.assertFalse(BooleanExpression.evaluate('tru', variables))
-
- def test_triple(self):
- triple = 'arch-vendor-os'
- self.assertTrue(BooleanExpression.evaluate('arch-', {}, triple))
- self.assertTrue(BooleanExpression.evaluate('ar', {}, triple))
- self.assertTrue(BooleanExpression.evaluate('ch-vend', {}, triple))
- self.assertTrue(BooleanExpression.evaluate('-vendor-', {}, triple))
- self.assertTrue(BooleanExpression.evaluate('-os', {}, triple))
- self.assertFalse(BooleanExpression.evaluate('arch-os', {}, triple))
-
- def test_operators(self):
- self.assertTrue(BooleanExpression.evaluate('true || true', {}))
- self.assertTrue(BooleanExpression.evaluate('true || false', {}))
- self.assertTrue(BooleanExpression.evaluate('false || true', {}))
- self.assertFalse(BooleanExpression.evaluate('false || false', {}))
-
- self.assertTrue(BooleanExpression.evaluate('true && true', {}))
- self.assertFalse(BooleanExpression.evaluate('true && false', {}))
- self.assertFalse(BooleanExpression.evaluate('false && true', {}))
- self.assertFalse(BooleanExpression.evaluate('false && false', {}))
-
- self.assertFalse(BooleanExpression.evaluate('!true', {}))
- self.assertTrue(BooleanExpression.evaluate('!false', {}))
-
- self.assertTrue(BooleanExpression.evaluate(' ((!((false) )) ) ', {}))
- self.assertTrue(BooleanExpression.evaluate('true && (true && (true))', {}))
- self.assertTrue(BooleanExpression.evaluate('!false && !false && !! !false', {}))
- self.assertTrue(BooleanExpression.evaluate('false && false || true', {}))
- self.assertTrue(BooleanExpression.evaluate('(false && false) || true', {}))
- self.assertFalse(BooleanExpression.evaluate('false && (false || true)', {}))
-
- # Evaluate boolean expression `expr`.
- # Fail if it does not throw a ValueError containing the text `error`.
- def checkException(self, expr, error):
- try:
- BooleanExpression.evaluate(expr, {})
- self.fail("expression %r didn't cause an exception" % expr)
- except ValueError as e:
- if -1 == str(e).find(error):
- self.fail(("expression %r caused the wrong ValueError\n" +
- "actual error was:\n%s\n" +
- "expected error was:\n%s\n") % (expr, e, error))
- except BaseException as e:
- self.fail(("expression %r caused the wrong exception; actual " +
- "exception was: \n%r") % (expr, e))
-
- def test_errors(self):
- self.checkException("ba#d",
- "couldn't parse text: '#d'\n" +
- "in expression: 'ba#d'")
-
- self.checkException("true and true",
- "expected: <end of expression>\n" +
- "have: 'and'\n" +
- "in expression: 'true and true'")
-
- self.checkException("|| true",
- "expected: '!' or '(' or identifier\n" +
- "have: '||'\n" +
- "in expression: '|| true'")
-
- self.checkException("true &&",
- "expected: '!' or '(' or identifier\n" +
- "have: <end of expression>\n" +
- "in expression: 'true &&'")
-
- self.checkException("",
- "expected: '!' or '(' or identifier\n" +
- "have: <end of expression>\n" +
- "in expression: ''")
-
- self.checkException("*",
- "couldn't parse text: '*'\n" +
- "in expression: '*'")
-
- self.checkException("no wait stop",
- "expected: <end of expression>\n" +
- "have: 'wait'\n" +
- "in expression: 'no wait stop'")
-
- self.checkException("no-$-please",
- "couldn't parse text: '$-please'\n" +
- "in expression: 'no-$-please'")
-
- self.checkException("(((true && true) || true)",
- "expected: ')'\n" +
- "have: <end of expression>\n" +
- "in expression: '(((true && true) || true)'")
-
- self.checkException("true (true)",
- "expected: <end of expression>\n" +
- "have: '('\n" +
- "in expression: 'true (true)'")
-
- self.checkException("( )",
- "expected: '!' or '(' or identifier\n" +
- "have: ')'\n" +
- "in expression: '( )'")
-
-if __name__ == '__main__':
- unittest.main()
Modified: llvm/trunk/utils/lit/lit/Test.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/lit/Test.py?rev=292900&r1=292899&r2=292900&view=diff
==============================================================================
--- llvm/trunk/utils/lit/lit/Test.py (original)
+++ llvm/trunk/utils/lit/lit/Test.py Tue Jan 24 02:58:20 2017
@@ -2,8 +2,6 @@ import os
from xml.sax.saxutils import escape
from json import JSONEncoder
-from lit.BooleanExpression import BooleanExpression
-
# Test result codes.
class ResultCode(object):
@@ -182,24 +180,10 @@ class Test:
self.path_in_suite = path_in_suite
self.config = config
self.file_path = file_path
-
- # A list of conditions under which this test is expected to fail.
- # Each condition is a boolean expression of features and target
- # triple parts. These can optionally be provided by test format
- # handlers, and will be honored when the test result is supplied.
+ # A list of conditions under which this test is expected to fail. These
+ # can optionally be provided by test format handlers, and will be
+ # honored when the test result is supplied.
self.xfails = []
-
- # A list of conditions that must be satisfied before running the test.
- # Each condition is a boolean expression of features. All of them
- # must be True for the test to run.
- # FIXME should target triple parts count here too?
- self.requires = []
-
- # A list of conditions that prevent execution of the test.
- # Each condition is a boolean expression of features and target
- # triple parts. All of them must be False for the test to run.
- self.unsupported = []
-
# The test result, once complete.
self.result = None
@@ -212,16 +196,11 @@ class Test:
self.result = result
# Apply the XFAIL handling to resolve the result exit code.
- try:
- if self.isExpectedToFail():
- if self.result.code == PASS:
- self.result.code = XPASS
- elif self.result.code == FAIL:
- self.result.code = XFAIL
- except ValueError as e:
- # Syntax error in an XFAIL line.
- self.result.code = UNRESOLVED
- self.result.output = str(e)
+ if self.isExpectedToFail():
+ if self.result.code == PASS:
+ self.result.code = XPASS
+ elif self.result.code == FAIL:
+ self.result.code = XFAIL
def getFullName(self):
return self.suite.config.name + ' :: ' + '/'.join(self.path_in_suite)
@@ -245,90 +224,23 @@ class Test:
configuration. This check relies on the test xfails property which by
some test formats may not be computed until the test has first been
executed.
- Throws ValueError if an XFAIL line has a syntax error.
"""
- features = self.config.available_features
- triple = getattr(self.suite.config, 'target_triple', "")
-
# Check if any of the xfails match an available feature or the target.
for item in self.xfails:
# If this is the wildcard, it always fails.
if item == '*':
return True
- # If this is a True expression of features and target triple parts,
- # it fails.
- try:
- if BooleanExpression.evaluate(item, features, triple):
- return True
- except ValueError as e:
- raise ValueError('Error in XFAIL list:\n%s' % str(e))
-
- return False
-
- def isWithinFeatureLimits(self):
- """
- isWithinFeatureLimits() -> bool
-
- A test is within the feature limits set by run_only_tests if
- 1. the test's requirements ARE satisfied by the available features
- 2. the test's requirements ARE NOT satisfied after the limiting
- features are removed from the available features
-
- Throws ValueError if a REQUIRES line has a syntax error.
- """
-
- if not self.config.limit_to_features:
- return True # No limits. Run it.
-
- # Check the requirements as-is (#1)
- if self.getMissingRequiredFeatures():
- return False
-
- # Check the requirements after removing the limiting features (#2)
- featuresMinusLimits = [f for f in self.config.available_features
- if not f in self.config.limit_to_features]
- if not self.getMissingRequiredFeaturesFromList(featuresMinusLimits):
- return False
-
- return True
-
- def getMissingRequiredFeaturesFromList(self, features):
- try:
- return [item for item in self.requires
- if not BooleanExpression.evaluate(item, features)]
- except ValueError as e:
- raise ValueError('Error in REQUIRES list:\n%s' % str(e))
-
- def getMissingRequiredFeatures(self):
- """
- getMissingRequiredFeatures() -> list of strings
-
- Returns a list of features from REQUIRES that are not satisfied."
- Throws ValueError if a REQUIRES line has a syntax error.
- """
-
- features = self.config.available_features
- return self.getMissingRequiredFeaturesFromList(features)
-
- def getUnsupportedFeatures(self):
- """
- getUnsupportedFeatures() -> list of strings
-
- Returns a list of features from UNSUPPORTED that are present
- in the test configuration's features or target triple.
- Throws ValueError if an UNSUPPORTED line has a syntax error.
- """
+ # If this is an exact match for one of the features, it fails.
+ if item in self.config.available_features:
+ return True
- features = self.config.available_features
- triple = getattr(self.suite.config, 'target_triple', "")
+ # If this is a part of the target triple, it fails.
+ if item and item in self.suite.config.target_triple:
+ return True
- try:
- return [item for item in self.unsupported
- if BooleanExpression.evaluate(item, features, triple)]
- except ValueError as e:
- raise ValueError('Error in UNSUPPORTED list:\n%s' % str(e))
+ return False
def isEarlyTest(self):
"""
Modified: llvm/trunk/utils/lit/lit/TestRunner.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/lit/TestRunner.py?rev=292900&r1=292899&r2=292900&view=diff
==============================================================================
--- llvm/trunk/utils/lit/lit/TestRunner.py (original)
+++ llvm/trunk/utils/lit/lit/TestRunner.py Tue Jan 24 02:58:20 2017
@@ -9,7 +9,6 @@ import lit.ShUtil as ShUtil
import lit.Test as Test
import lit.util
from lit.util import to_bytes, to_string
-from lit.BooleanExpression import BooleanExpression
class InternalShellError(Exception):
def __init__(self, command, message):
@@ -747,35 +746,14 @@ class ParserKind(object):
command.
TAG: A keyword taking no value. Ex 'END.'
- COMMAND: A keyword taking a list of shell commands. Ex 'RUN:'
- LIST: A keyword taking a comma-separated list of values.
- BOOLEAN_EXPR: A keyword taking a comma-separated list of
- boolean expressions. Ex 'XFAIL:'
+ COMMAND: A Keyword taking a list of shell commands. Ex 'RUN:'
+ LIST: A keyword taking a comma separated list of value. Ex 'XFAIL:'
CUSTOM: A keyword with custom parsing semantics.
"""
TAG = 0
COMMAND = 1
LIST = 2
- BOOLEAN_EXPR = 3
- CUSTOM = 4
-
- @staticmethod
- def allowedKeywordSuffixes(value):
- return { ParserKind.TAG: ['.'],
- ParserKind.COMMAND: [':'],
- ParserKind.LIST: [':'],
- ParserKind.BOOLEAN_EXPR: [':'],
- ParserKind.CUSTOM: [':', '.']
- } [value]
-
- @staticmethod
- def str(value):
- return { ParserKind.TAG: 'TAG',
- ParserKind.COMMAND: 'COMMAND',
- ParserKind.LIST: 'LIST',
- ParserKind.BOOLEAN_EXPR: 'BOOLEAN_EXPR',
- ParserKind.CUSTOM: 'CUSTOM'
- } [value]
+ CUSTOM = 3
class IntegratedTestKeywordParser(object):
@@ -787,18 +765,15 @@ class IntegratedTestKeywordParser(object
ParserKind.CUSTOM.
"""
def __init__(self, keyword, kind, parser=None, initial_value=None):
- allowedSuffixes = ParserKind.allowedKeywordSuffixes(kind)
- if len(keyword) == 0 or keyword[-1] not in allowedSuffixes:
- if len(allowedSuffixes) == 1:
- raise ValueError("Keyword '%s' of kind '%s' must end in '%s'"
- % (keyword, ParserKind.str(kind),
- allowedSuffixes[0]))
- else:
- raise ValueError("Keyword '%s' of kind '%s' must end in "
- " one of '%s'"
- % (keyword, ParserKind.str(kind),
- ' '.join(allowedSuffixes)))
+ if not keyword.endswith('.') and not keyword.endswith(':'):
+ raise ValueError("keyword '%s' must end with either '.' or ':' "
+ % keyword)
+ if keyword.endswith('.') and kind in \
+ [ParserKind.LIST, ParserKind.COMMAND]:
+ raise ValueError("Keyword '%s' should end in ':'" % keyword)
+ elif keyword.endswith(':') and kind in [ParserKind.TAG]:
+ raise ValueError("Keyword '%s' should end in '.'" % keyword)
if parser is not None and kind != ParserKind.CUSTOM:
raise ValueError("custom parsers can only be specified with "
"ParserKind.CUSTOM")
@@ -812,9 +787,9 @@ class IntegratedTestKeywordParser(object
self.parser = self._handleCommand
elif kind == ParserKind.LIST:
self.parser = self._handleList
- elif kind == ParserKind.BOOLEAN_EXPR:
- self.parser = self._handleBooleanExpr
elif kind == ParserKind.TAG:
+ if not keyword.endswith('.'):
+ raise ValueError("keyword '%s' should end with '.'" % keyword)
self.parser = self._handleTag
elif kind == ParserKind.CUSTOM:
if parser is None:
@@ -824,12 +799,8 @@ class IntegratedTestKeywordParser(object
raise ValueError("Unknown kind '%s'" % kind)
def parseLine(self, line_number, line):
- try:
- self.parsed_lines += [(line_number, line)]
- self.value = self.parser(line_number, line, self.value)
- except ValueError as e:
- raise ValueError(str(e) + ("\nin %s directive on test line %d" %
- (self.keyword, line_number)))
+ self.parsed_lines += [(line_number, line)]
+ self.value = self.parser(line_number, line, self.value)
def getValue(self):
return self.value
@@ -870,24 +841,12 @@ class IntegratedTestKeywordParser(object
output.extend([s.strip() for s in line.split(',')])
return output
- @staticmethod
- def _handleBooleanExpr(line_number, line, output):
- """A parser for BOOLEAN_EXPR type keywords"""
- if output is None:
- output = []
- output.extend([s.strip() for s in line.split(',')])
- # Evaluate each expression to verify syntax.
- # We don't want any results, just the raised ValueError.
- for s in output:
- if s != '*':
- BooleanExpression.evaluate(s, [])
- return output
def parseIntegratedTestScript(test, additional_parsers=[],
require_script=True):
"""parseIntegratedTestScript - Scan an LLVM/Clang style integrated test
script and extract the lines to 'RUN' as well as 'XFAIL' and 'REQUIRES'
- and 'UNSUPPORTED' information.
+ 'REQUIRES-ANY' and 'UNSUPPORTED' information.
If additional parsers are specified then the test is also scanned for the
keywords they specify and all matches are passed to the custom parser.
@@ -896,23 +855,26 @@ def parseIntegratedTestScript(test, addi
may be returned. This can be used for test formats where the actual script
is optional or ignored.
"""
-
- # Install the built-in keyword parsers.
+ # Collect the test lines from the script.
+ sourcepath = test.getSourcePath()
script = []
+ requires = []
+ requires_any = []
+ unsupported = []
builtin_parsers = [
IntegratedTestKeywordParser('RUN:', ParserKind.COMMAND,
initial_value=script),
- IntegratedTestKeywordParser('XFAIL:', ParserKind.BOOLEAN_EXPR,
+ IntegratedTestKeywordParser('XFAIL:', ParserKind.LIST,
initial_value=test.xfails),
- IntegratedTestKeywordParser('REQUIRES:', ParserKind.BOOLEAN_EXPR,
- initial_value=test.requires),
- IntegratedTestKeywordParser('UNSUPPORTED:', ParserKind.BOOLEAN_EXPR,
- initial_value=test.unsupported),
+ IntegratedTestKeywordParser('REQUIRES:', ParserKind.LIST,
+ initial_value=requires),
+ IntegratedTestKeywordParser('REQUIRES-ANY:', ParserKind.LIST,
+ initial_value=requires_any),
+ IntegratedTestKeywordParser('UNSUPPORTED:', ParserKind.LIST,
+ initial_value=unsupported),
IntegratedTestKeywordParser('END.', ParserKind.TAG)
]
keyword_parsers = {p.keyword: p for p in builtin_parsers}
-
- # Install user-defined additional parsers.
for parser in additional_parsers:
if not isinstance(parser, IntegratedTestKeywordParser):
raise ValueError('additional parser must be an instance of '
@@ -922,18 +884,6 @@ def parseIntegratedTestScript(test, addi
% parser.keyword)
keyword_parsers[parser.keyword] = parser
- # Install a helpful error-generating parser for the no-longer-supported
- # REQUIRES-ANY: keyword, if no other parser for it exists.
- if 'REQUIRES-ANY:' not in keyword_parsers:
- def requires_any_error_parser(line_number, line, output):
- raise ValueError('`REQUIRES-ANY: a, b, c` not supported. Use '
- '`REQUIRES: a || b || c` instead.')
- parser = IntegratedTestKeywordParser('REQUIRES-ANY:', ParserKind.CUSTOM,
- requires_any_error_parser)
- keyword_parsers[parser.keyword] = parser
-
- # Collect the test lines from the script.
- sourcepath = test.getSourcePath()
for line_number, command_type, ln in \
parseIntegratedTestScriptCommands(sourcepath,
keyword_parsers.keys()):
@@ -951,30 +901,46 @@ def parseIntegratedTestScript(test, addi
return lit.Test.Result(Test.UNRESOLVED,
"Test has unterminated run lines (with '\\')")
- # Enforce REQUIRES:
- missing_required_features = test.getMissingRequiredFeatures()
+ # Check that we have the required features:
+ missing_required_features = [f for f in requires
+ if f not in test.config.available_features]
if missing_required_features:
msg = ', '.join(missing_required_features)
return lit.Test.Result(Test.UNSUPPORTED,
- "Test requires the following unavailable "
- "features: %s" % msg)
-
- # Enforce UNSUPPORTED:
- unsupported_features = test.getUnsupportedFeatures()
+ "Test requires the following features: %s"
+ % msg)
+ requires_any_features = [f for f in requires_any
+ if f in test.config.available_features]
+ if requires_any and not requires_any_features:
+ msg = ' ,'.join(requires_any)
+ return lit.Test.Result(Test.UNSUPPORTED,
+ "Test requires any of the following features: "
+ "%s" % msg)
+ unsupported_features = [f for f in unsupported
+ if f in test.config.available_features]
if unsupported_features:
msg = ', '.join(unsupported_features)
return lit.Test.Result(
Test.UNSUPPORTED,
- "Test does not support the following features "
- "and/or targets: %s" % msg)
+ "Test is unsupported with the following features: %s" % msg)
- # Enforce limit_to_features.
- if not test.isWithinFeatureLimits():
- msg = ', '.join(test.config.limit_to_features)
- return lit.Test.Result(Test.UNSUPPORTED,
- "Test does not require any of the features "
- "specified in limit_to_features: %s" % msg)
+ unsupported_targets = [f for f in unsupported
+ if f in test.suite.config.target_triple]
+ if unsupported_targets:
+ return lit.Test.Result(
+ Test.UNSUPPORTED,
+ "Test is unsupported with the following triple: %s" % (
+ test.suite.config.target_triple,))
+ if test.config.limit_to_features:
+ # Check that we have one of the limit_to_features features in requires.
+ limit_to_features_tests = [f for f in test.config.limit_to_features
+ if f in requires]
+ if not limit_to_features_tests:
+ msg = ', '.join(test.config.limit_to_features)
+ return lit.Test.Result(
+ Test.UNSUPPORTED,
+ "Test requires one of the limit_to_features features %s" % msg)
return script
Removed: llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any.txt?rev=292899&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any.txt (original)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-any.txt (removed)
@@ -1,3 +0,0 @@
-# REQUIRES-ANY is no longer supported. Test should not run.
-REQUIRES-ANY: true, a-present-feature
-RUN: false
Modified: llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-missing.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-missing.txt?rev=292900&r1=292899&r2=292900&view=diff
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-missing.txt (original)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-missing.txt Tue Jan 24 02:58:20 2017
@@ -1,5 +1,2 @@
-# REQUIRES with a false clause. Test should not run.
-REQUIRES: true
-REQUIRES: a-missing-feature, true
-REQUIRES: true
-RUN: false
+RUN: true
+REQUIRES: a-missing-feature
Modified: llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-present.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-present.txt?rev=292900&r1=292899&r2=292900&view=diff
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-present.txt (original)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-present.txt Tue Jan 24 02:58:20 2017
@@ -1,4 +1,2 @@
-# REQUIRES with only true clauses. Test should run.
-REQUIRES: a-present-feature, true, !not-true
-REQUIRES: true
RUN: true
+REQUIRES: a-present-feature
Removed: llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-star.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-star.txt?rev=292899&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-star.txt (original)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-star.txt (removed)
@@ -1,3 +0,0 @@
-# '*' only works in XFAIL
-REQUIRES: *
-RUN: false
Removed: llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-triple.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-triple.txt?rev=292899&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-triple.txt (original)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-triple.txt (removed)
@@ -1,3 +0,0 @@
-# REQUIRES line that uses target triple, which doesn't work. Test should not run
-REQUIRES: x86_64
-RUN: false
Removed: llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-false.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-false.txt?rev=292899&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-false.txt (original)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-false.txt (removed)
@@ -1,9 +0,0 @@
-# UNSUPPORTED with only false clauses. Test should run.
-UNSUPPORTED: false
-UNSUPPORTED: false, not-true
-UNSUPPORTED: false
-UNSUPPORTED: still-not-true
-UNSUPPORTED: false
-UNSUPPORTED: false
-UNSUPPORTED: false
-RUN: true
Removed: llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-true.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-true.txt?rev=292899&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-true.txt (original)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-true.txt (removed)
@@ -1,4 +0,0 @@
-# UNSUPPORTED with a true clause. Test should not run.
-UNSUPPORTED: false
-UNSUPPORTED: false, false, false, _64-unk && a-present-feature, false
-RUN: false
Removed: llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-star.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-star.txt?rev=292899&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-star.txt (original)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-star.txt (removed)
@@ -1,3 +0,0 @@
-# '*' only works in XFAIL
-UNSUPPORTED: *
-RUN: false
Removed: llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-false.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-false.txt?rev=292899&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-false.txt (original)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-false.txt (removed)
@@ -1,3 +0,0 @@
-# XFAIL with only false clauses. Test should run.
-XFAIL: false, a-missing-feature || ! a-present-feature || ! x86_64, false
-RUN: true
Removed: llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-true.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-true.txt?rev=292899&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-true.txt (original)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-true.txt (removed)
@@ -1,4 +0,0 @@
-# XFAIL with a true clause. Test should not run.
-XFAIL: false
-XFAIL: false, a-present-feature && ! a-missing-feature && x86_64
-RUN: false
Removed: llvm/trunk/utils/lit/tests/boolean-parsing.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/boolean-parsing.py?rev=292899&view=auto
==============================================================================
--- llvm/trunk/utils/lit/tests/boolean-parsing.py (original)
+++ llvm/trunk/utils/lit/tests/boolean-parsing.py (removed)
@@ -1,4 +0,0 @@
-# Test the boolean expression parser
-# used for REQUIRES and UNSUPPORTED and XFAIL
-
-# RUN: %{python} -m lit.BooleanExpression
Modified: llvm/trunk/utils/lit/tests/shtest-format.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/shtest-format.py?rev=292900&r1=292899&r2=292900&view=diff
==============================================================================
--- llvm/trunk/utils/lit/tests/shtest-format.py (original)
+++ llvm/trunk/utils/lit/tests/shtest-format.py Tue Jan 24 02:58:20 2017
@@ -46,18 +46,11 @@
# CHECK: UNRESOLVED: shtest-format :: no-test-line.txt
# CHECK: PASS: shtest-format :: pass.txt
-# CHECK: UNRESOLVED: shtest-format :: requires-any.txt
-# CHECK: ValueError: `REQUIRES-ANY: a, b, c` not supported
+# CHECK: UNSUPPORTED: shtest-format :: requires-any-missing.txt
+# CHECK: PASS: shtest-format :: requires-any-present.txt
# CHECK: UNSUPPORTED: shtest-format :: requires-missing.txt
# CHECK: PASS: shtest-format :: requires-present.txt
-# CHECK: UNRESOLVED: shtest-format :: requires-star.txt
-# CHECK: UNSUPPORTED: shtest-format :: requires-triple.txt
-# CHECK: PASS: shtest-format :: unsupported-expr-false.txt
-# CHECK: UNSUPPORTED: shtest-format :: unsupported-expr-true.txt
-# CHECK: UNRESOLVED: shtest-format :: unsupported-star.txt
# CHECK: UNSUPPORTED: shtest-format :: unsupported_dir/some-test.txt
-# CHECK: PASS: shtest-format :: xfail-expr-false.txt
-# CHECK: XFAIL: shtest-format :: xfail-expr-true.txt
# CHECK: XFAIL: shtest-format :: xfail-feature.txt
# CHECK: XFAIL: shtest-format :: xfail-target.txt
# CHECK: XFAIL: shtest-format :: xfail.txt
@@ -77,9 +70,9 @@
# CHECK: shtest-format :: external_shell/fail_with_bad_encoding.txt
# CHECK: shtest-format :: fail.txt
-# CHECK: Expected Passes : 6
-# CHECK: Expected Failures : 4
-# CHECK: Unsupported Tests : 4
-# CHECK: Unresolved Tests : 4
+# CHECK: Expected Passes : 5
+# CHECK: Expected Failures : 3
+# CHECK: Unsupported Tests : 3
+# CHECK: Unresolved Tests : 1
# CHECK: Unexpected Passes : 1
# CHECK: Unexpected Failures: 3
Modified: llvm/trunk/utils/lit/tests/unit/TestRunner.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/unit/TestRunner.py?rev=292900&r1=292899&r2=292900&view=diff
==============================================================================
--- llvm/trunk/utils/lit/tests/unit/TestRunner.py (original)
+++ llvm/trunk/utils/lit/tests/unit/TestRunner.py Tue Jan 24 02:58:20 2017
@@ -108,71 +108,6 @@ class TestIntegratedTestKeywordParser(un
value = custom_parser.getValue()
self.assertItemsEqual(value, ['a', 'b', 'c'])
- def test_bad_keywords(self):
- def custom_parse(line_number, line, output):
- return output
-
- try:
- IntegratedTestKeywordParser("TAG_NO_SUFFIX", ParserKind.TAG),
- self.fail("TAG_NO_SUFFIX failed to raise an exception")
- except ValueError as e:
- pass
- except BaseException as e:
- self.fail("TAG_NO_SUFFIX raised the wrong exception: %r" % e)
-
- try:
- IntegratedTestKeywordParser("TAG_WITH_COLON:", ParserKind.TAG),
- self.fail("TAG_WITH_COLON: failed to raise an exception")
- except ValueError as e:
- pass
- except BaseException as e:
- self.fail("TAG_WITH_COLON: raised the wrong exception: %r" % e)
-
- try:
- IntegratedTestKeywordParser("LIST_WITH_DOT.", ParserKind.LIST),
- self.fail("LIST_WITH_DOT. failed to raise an exception")
- except ValueError as e:
- pass
- except BaseException as e:
- self.fail("LIST_WITH_DOT. raised the wrong exception: %r" % e)
-
- try:
- IntegratedTestKeywordParser("CUSTOM_NO_SUFFIX",
- ParserKind.CUSTOM, custom_parse),
- self.fail("CUSTOM_NO_SUFFIX failed to raise an exception")
- except ValueError as e:
- pass
- except BaseException as e:
- self.fail("CUSTOM_NO_SUFFIX raised the wrong exception: %r" % e)
-
- # Both '.' and ':' are allowed for CUSTOM keywords.
- try:
- IntegratedTestKeywordParser("CUSTOM_WITH_DOT.",
- ParserKind.CUSTOM, custom_parse),
- except BaseException as e:
- self.fail("CUSTOM_WITH_DOT. raised an exception: %r" % e)
- try:
- IntegratedTestKeywordParser("CUSTOM_WITH_COLON:",
- ParserKind.CUSTOM, custom_parse),
- except BaseException as e:
- self.fail("CUSTOM_WITH_COLON: raised an exception: %r" % e)
-
- try:
- IntegratedTestKeywordParser("CUSTOM_NO_PARSER:",
- ParserKind.CUSTOM),
- self.fail("CUSTOM_NO_PARSER: failed to raise an exception")
- except ValueError as e:
- pass
- except BaseException as e:
- self.fail("CUSTOM_NO_PARSER: raised the wrong exception: %r" % e)
-
- # REQUIRES-ANY: has a built-in parser that generates an error,
- # but it may be overridden by a custom parser.
- try:
- IntegratedTestKeywordParser("REQUIRES-ANY:",
- ParserKind.CUSTOM, custom_parse),
- except BaseException as e:
- self.fail("REQUIRES-ANY: raised an exception: %r" % e)
if __name__ == '__main__':
TestIntegratedTestKeywordParser.load_keyword_parser_lit_tests()
More information about the llvm-commits
mailing list