[llvm] r293007 - Reinstate "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAIL

Greg Parker via llvm-commits llvm-commits at lists.llvm.org
Tue Jan 24 18:26:04 PST 2017


Author: gparker
Date: Tue Jan 24 20:26:03 2017
New Revision: 293007

URL: http://llvm.org/viewvc/llvm-project?rev=293007&view=rev
Log:
Reinstate "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAIL
and UNSUPPORTED"

This reverts the revert in r292942.

Added:
    llvm/trunk/utils/lit/lit/BooleanExpression.py
      - copied unchanged from r292941, llvm/trunk/utils/lit/lit/BooleanExpression.py
    llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-star.txt
      - copied unchanged from r292941, llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-star.txt
    llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-triple.txt
      - copied unchanged from r292941, llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-triple.txt
    llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-false.txt
      - copied unchanged from r292941, llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-false.txt
    llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-true.txt
      - copied unchanged from r292941, llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-expr-true.txt
    llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-star.txt
      - copied unchanged from r292941, llvm/trunk/utils/lit/tests/Inputs/shtest-format/unsupported-star.txt
    llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-false.txt
      - copied unchanged from r292941, llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-false.txt
    llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-true.txt
      - copied unchanged from r292941, llvm/trunk/utils/lit/tests/Inputs/shtest-format/xfail-expr-true.txt
    llvm/trunk/utils/lit/tests/boolean-parsing.py
      - copied unchanged from r292941, llvm/trunk/utils/lit/tests/boolean-parsing.py
Modified:
    llvm/trunk/docs/TestingGuide.rst
    llvm/trunk/utils/lit/lit/Test.py
    llvm/trunk/utils/lit/lit/TestRunner.py
    llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-missing.txt
    llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-present.txt
    llvm/trunk/utils/lit/tests/shtest-format.py
    llvm/trunk/utils/lit/tests/unit/TestRunner.py

Modified: llvm/trunk/docs/TestingGuide.rst
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/docs/TestingGuide.rst?rev=293007&r1=293006&r2=293007&view=diff
==============================================================================
--- llvm/trunk/docs/TestingGuide.rst (original)
+++ llvm/trunk/docs/TestingGuide.rst Tue Jan 24 20:26:03 2017
@@ -387,23 +387,49 @@ depends on special features of sub-archi
 triple, test with the specific FileCheck and put it into the specific
 directory that will filter out all other architectures.
 
-REQUIRES and REQUIRES-ANY directive
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Some tests can be enabled only in specific situation - like having
-debug build. Use ``REQUIRES`` directive to specify those requirements.
+Constraining test execution
+---------------------------
+
+Some tests can be run only in specific configurations, such as
+with debug builds or on particular platforms. Use ``REQUIRES``
+and ``UNSUPPORTED`` to control when the test is enabled.
+
+Some tests are expected to fail. For example, there may be a known bug
+that the test detect. Use ``XFAIL`` to mark a test as an expected failure.
+An ``XFAIL`` test will be successful if its execution fails, and
+will be a failure if its execution succeeds.
 
 .. code-block:: llvm
 
-    ; This test will be only enabled in the build with asserts
+    ; This test will be only enabled in the build with asserts.
     ; REQUIRES: asserts
+    ; This test is disabled on Linux.
+    ; UNSUPPORTED: -linux-
+    ; This test is expected to fail on PowerPC.
+    ; XFAIL: powerpc
+
+``REQUIRES`` and ``UNSUPPORTED`` and ``XFAIL`` all accept a comma-separated
+list of boolean expressions. The values in each expression may be:
+
+- Features added to ``config.available_features`` by 
+  configuration files such as ``lit.cfg``.
+- Substrings of the target triple (``UNSUPPORTED`` and ``XFAIL`` only).
+
+| ``REQUIRES`` enables the test if all expressions are true.
+| ``UNSUPPORTED`` disables the test if any expression is true.
+| ``XFAIL`` expects the test to fail if any expression is true.
+
+As a special case, ``XFAIL: *`` is expected to fail everywhere.
+
+.. code-block:: llvm
 
-You can separate requirements by a comma.
-``REQUIRES`` means all listed requirements must be satisfied.
-``REQUIRES-ANY`` means at least one must be satisfied.
+    ; This test is disabled on Windows,
+    ; and is disabled on Linux, except for Android Linux.
+    ; UNSUPPORTED: windows, linux && !android
+    ; This test is expected to fail on both PowerPC and ARM.
+    ; XFAIL: powerpc || arm
 
-List of features that can be used in ``REQUIRES`` and ``REQUIRES-ANY`` can be
-found in lit.cfg files.
 
 Substitutions
 -------------
@@ -520,24 +546,6 @@ their name. For example:
    This program runs its arguments and then inverts the result code from it.
    Zero result codes become 1. Non-zero result codes become 0.
 
-Sometimes it is necessary to mark a test case as "expected fail" or
-XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:``
-on a line near the top of the file. This signals that the test case
-should succeed if the test fails. Such test cases are counted separately
-by the testing tool. To specify an expected fail, use the XFAIL keyword
-in the comments of the test program followed by a colon and one or more
-failure patterns. Each failure pattern can be either ``*`` (to specify
-fail everywhere), or a part of a target triple (indicating the test
-should fail on that platform), or the name of a configurable feature
-(for example, ``loadable_module``). If there is a match, the test is
-expected to fail. If not, the test is expected to succeed. To XFAIL
-everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL``
-line:
-
-.. code-block:: llvm
-
-    ; XFAIL: darwin,sun
-
 To make the output more useful, :program:`lit` will scan
 the lines of the test case for ones that contain a pattern that matches
 ``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number

Modified: llvm/trunk/utils/lit/lit/Test.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/lit/Test.py?rev=293007&r1=293006&r2=293007&view=diff
==============================================================================
--- llvm/trunk/utils/lit/lit/Test.py (original)
+++ llvm/trunk/utils/lit/lit/Test.py Tue Jan 24 20:26:03 2017
@@ -2,6 +2,8 @@ import os
 from xml.sax.saxutils import escape
 from json import JSONEncoder
 
+from lit.BooleanExpression import BooleanExpression
+
 # Test result codes.
 
 class ResultCode(object):
@@ -180,10 +182,24 @@ class Test:
         self.path_in_suite = path_in_suite
         self.config = config
         self.file_path = file_path
-        # A list of conditions under which this test is expected to fail. These
-        # can optionally be provided by test format handlers, and will be
-        # honored when the test result is supplied.
+
+        # A list of conditions under which this test is expected to fail.
+        # Each condition is a boolean expression of features and target
+        # triple parts. These can optionally be provided by test format
+        # handlers, and will be honored when the test result is supplied.
         self.xfails = []
+
+        # A list of conditions that must be satisfied before running the test.
+        # Each condition is a boolean expression of features. All of them
+        # must be True for the test to run.
+        # FIXME should target triple parts count here too?
+        self.requires = []
+
+        # A list of conditions that prevent execution of the test.
+        # Each condition is a boolean expression of features and target
+        # triple parts. All of them must be False for the test to run.
+        self.unsupported = []
+
         # The test result, once complete.
         self.result = None
 
@@ -196,11 +212,16 @@ class Test:
         self.result = result
 
         # Apply the XFAIL handling to resolve the result exit code.
-        if self.isExpectedToFail():
-            if self.result.code == PASS:
-                self.result.code = XPASS
-            elif self.result.code == FAIL:
-                self.result.code = XFAIL
+        try:
+            if self.isExpectedToFail():
+                if self.result.code == PASS:
+                    self.result.code = XPASS
+                elif self.result.code == FAIL:
+                    self.result.code = XFAIL
+        except ValueError as e:
+            # Syntax error in an XFAIL line.
+            self.result.code = UNRESOLVED
+            self.result.output = str(e)
         
     def getFullName(self):
         return self.suite.config.name + ' :: ' + '/'.join(self.path_in_suite)
@@ -224,24 +245,91 @@ class Test:
         configuration. This check relies on the test xfails property which by
         some test formats may not be computed until the test has first been
         executed.
+        Throws ValueError if an XFAIL line has a syntax error.
         """
 
+        features = self.config.available_features
+        triple = getattr(self.suite.config, 'target_triple', "")
+
         # Check if any of the xfails match an available feature or the target.
         for item in self.xfails:
             # If this is the wildcard, it always fails.
             if item == '*':
                 return True
 
-            # If this is an exact match for one of the features, it fails.
-            if item in self.config.available_features:
-                return True
-
-            # If this is a part of the target triple, it fails.
-            if item and item in self.suite.config.target_triple:
-                return True
+            # If this is a True expression of features and target triple parts,
+            # it fails.
+            try:
+                if BooleanExpression.evaluate(item, features, triple):
+                    return True
+            except ValueError as e:
+                raise ValueError('Error in XFAIL list:\n%s' % str(e))
 
         return False
 
+    def isWithinFeatureLimits(self):
+        """
+        isWithinFeatureLimits() -> bool
+
+        A test is within the feature limits set by run_only_tests if
+        1. the test's requirements ARE satisfied by the available features
+        2. the test's requirements ARE NOT satisfied after the limiting
+           features are removed from the available features
+
+        Throws ValueError if a REQUIRES line has a syntax error.
+        """
+
+        if not self.config.limit_to_features:
+            return True  # No limits. Run it.
+
+        # Check the requirements as-is (#1)
+        if self.getMissingRequiredFeatures():
+            return False
+
+        # Check the requirements after removing the limiting features (#2)
+        featuresMinusLimits = [f for f in self.config.available_features
+                               if not f in self.config.limit_to_features]
+        if not self.getMissingRequiredFeaturesFromList(featuresMinusLimits):
+            return False
+
+        return True
+
+    def getMissingRequiredFeaturesFromList(self, features):
+        try:
+            return [item for item in self.requires
+                    if not BooleanExpression.evaluate(item, features)]
+        except ValueError as e:
+            raise ValueError('Error in REQUIRES list:\n%s' % str(e))
+
+    def getMissingRequiredFeatures(self):
+        """
+        getMissingRequiredFeatures() -> list of strings
+
+        Returns a list of features from REQUIRES that are not satisfied."
+        Throws ValueError if a REQUIRES line has a syntax error.
+        """
+
+        features = self.config.available_features
+        return self.getMissingRequiredFeaturesFromList(features)
+
+    def getUnsupportedFeatures(self):
+        """
+        getUnsupportedFeatures() -> list of strings
+
+        Returns a list of features from UNSUPPORTED that are present
+        in the test configuration's features or target triple.
+        Throws ValueError if an UNSUPPORTED line has a syntax error.
+        """
+
+        features = self.config.available_features
+        triple = getattr(self.suite.config, 'target_triple', "")
+
+        try:
+            return [item for item in self.unsupported
+                    if BooleanExpression.evaluate(item, features, triple)]
+        except ValueError as e:
+            raise ValueError('Error in UNSUPPORTED list:\n%s' % str(e))
+
     def isEarlyTest(self):
         """
         isEarlyTest() -> bool

Modified: llvm/trunk/utils/lit/lit/TestRunner.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/lit/TestRunner.py?rev=293007&r1=293006&r2=293007&view=diff
==============================================================================
--- llvm/trunk/utils/lit/lit/TestRunner.py (original)
+++ llvm/trunk/utils/lit/lit/TestRunner.py Tue Jan 24 20:26:03 2017
@@ -9,6 +9,7 @@ import lit.ShUtil as ShUtil
 import lit.Test as Test
 import lit.util
 from lit.util import to_bytes, to_string
+from lit.BooleanExpression import BooleanExpression
 
 class InternalShellError(Exception):
     def __init__(self, command, message):
@@ -746,14 +747,35 @@ class ParserKind(object):
     command.
 
     TAG: A keyword taking no value. Ex 'END.'
-    COMMAND: A Keyword taking a list of shell commands. Ex 'RUN:'
-    LIST: A keyword taking a comma separated list of value. Ex 'XFAIL:'
+    COMMAND: A keyword taking a list of shell commands. Ex 'RUN:'
+    LIST: A keyword taking a comma-separated list of values.
+    BOOLEAN_EXPR: A keyword taking a comma-separated list of 
+        boolean expressions. Ex 'XFAIL:'
     CUSTOM: A keyword with custom parsing semantics.
     """
     TAG = 0
     COMMAND = 1
     LIST = 2
-    CUSTOM = 3
+    BOOLEAN_EXPR = 3
+    CUSTOM = 4
+
+    @staticmethod
+    def allowedKeywordSuffixes(value):
+        return { ParserKind.TAG:          ['.'],
+                 ParserKind.COMMAND:      [':'],
+                 ParserKind.LIST:         [':'],
+                 ParserKind.BOOLEAN_EXPR: [':'],
+                 ParserKind.CUSTOM:       [':', '.']
+               } [value]
+
+    @staticmethod
+    def str(value):
+        return { ParserKind.TAG:          'TAG',
+                 ParserKind.COMMAND:      'COMMAND',
+                 ParserKind.LIST:         'LIST',
+                 ParserKind.BOOLEAN_EXPR: 'BOOLEAN_EXPR',
+                 ParserKind.CUSTOM:       'CUSTOM'
+               } [value]
 
 
 class IntegratedTestKeywordParser(object):
@@ -765,15 +787,18 @@ class IntegratedTestKeywordParser(object
             ParserKind.CUSTOM.
     """
     def __init__(self, keyword, kind, parser=None, initial_value=None):
-        if not keyword.endswith('.') and not keyword.endswith(':'):
-            raise ValueError("keyword '%s' must end with either '.' or ':' "
-                             % keyword)
-        if keyword.endswith('.') and kind in \
-                [ParserKind.LIST, ParserKind.COMMAND]:
-            raise ValueError("Keyword '%s' should end in ':'" % keyword)
+        allowedSuffixes = ParserKind.allowedKeywordSuffixes(kind)
+        if len(keyword) == 0 or keyword[-1] not in allowedSuffixes:
+            if len(allowedSuffixes) == 1:
+                raise ValueError("Keyword '%s' of kind '%s' must end in '%s'"
+                                 % (keyword, ParserKind.str(kind),
+                                    allowedSuffixes[0]))
+            else:
+                raise ValueError("Keyword '%s' of kind '%s' must end in "
+                                 " one of '%s'"
+                                 % (keyword, ParserKind.str(kind),
+                                    ' '.join(allowedSuffixes)))
 
-        elif keyword.endswith(':') and kind in [ParserKind.TAG]:
-            raise ValueError("Keyword '%s' should end in '.'" % keyword)
         if parser is not None and kind != ParserKind.CUSTOM:
             raise ValueError("custom parsers can only be specified with "
                              "ParserKind.CUSTOM")
@@ -787,9 +812,9 @@ class IntegratedTestKeywordParser(object
             self.parser = self._handleCommand
         elif kind == ParserKind.LIST:
             self.parser = self._handleList
+        elif kind == ParserKind.BOOLEAN_EXPR:
+            self.parser = self._handleBooleanExpr
         elif kind == ParserKind.TAG:
-            if not keyword.endswith('.'):
-                raise ValueError("keyword '%s' should end with '.'" % keyword)
             self.parser = self._handleTag
         elif kind == ParserKind.CUSTOM:
             if parser is None:
@@ -799,8 +824,12 @@ class IntegratedTestKeywordParser(object
             raise ValueError("Unknown kind '%s'" % kind)
 
     def parseLine(self, line_number, line):
-        self.parsed_lines += [(line_number, line)]
-        self.value = self.parser(line_number, line, self.value)
+        try:
+            self.parsed_lines += [(line_number, line)]
+            self.value = self.parser(line_number, line, self.value)
+        except ValueError as e:
+            raise ValueError(str(e) + ("\nin %s directive on test line %d" %
+                                       (self.keyword, line_number)))
 
     def getValue(self):
         return self.value
@@ -841,12 +870,38 @@ class IntegratedTestKeywordParser(object
         output.extend([s.strip() for s in line.split(',')])
         return output
 
+    @staticmethod
+    def _handleBooleanExpr(line_number, line, output):
+        """A parser for BOOLEAN_EXPR type keywords"""
+        if output is None:
+            output = []
+        output.extend([s.strip() for s in line.split(',')])
+        # Evaluate each expression to verify syntax.
+        # We don't want any results, just the raised ValueError.
+        for s in output:
+            if s != '*':
+                BooleanExpression.evaluate(s, [])
+        return output
+
+    @staticmethod
+    def _handleRequiresAny(line_number, line, output):
+        """A custom parser to transform REQUIRES-ANY: into REQUIRES:"""
+
+        # Extract the conditions specified in REQUIRES-ANY: as written.
+        conditions = []
+        IntegratedTestKeywordParser._handleList(line_number, line, conditions)
+
+        # Output a `REQUIRES: a || b || c` expression in its place.
+        expression = ' || '.join(conditions)
+        IntegratedTestKeywordParser._handleBooleanExpr(line_number,
+                                                       expression, output)
+        return output
 
 def parseIntegratedTestScript(test, additional_parsers=[],
                               require_script=True):
     """parseIntegratedTestScript - Scan an LLVM/Clang style integrated test
     script and extract the lines to 'RUN' as well as 'XFAIL' and 'REQUIRES'
-    'REQUIRES-ANY' and 'UNSUPPORTED' information.
+    and 'UNSUPPORTED' information.
 
     If additional parsers are specified then the test is also scanned for the
     keywords they specify and all matches are passed to the custom parser.
@@ -855,26 +910,26 @@ def parseIntegratedTestScript(test, addi
     may be returned. This can be used for test formats where the actual script
     is optional or ignored.
     """
-    # Collect the test lines from the script.
-    sourcepath = test.getSourcePath()
+
+    # Install the built-in keyword parsers.
     script = []
-    requires = []
-    requires_any = []
-    unsupported = []
     builtin_parsers = [
         IntegratedTestKeywordParser('RUN:', ParserKind.COMMAND,
                                     initial_value=script),
-        IntegratedTestKeywordParser('XFAIL:', ParserKind.LIST,
+        IntegratedTestKeywordParser('XFAIL:', ParserKind.BOOLEAN_EXPR,
                                     initial_value=test.xfails),
-        IntegratedTestKeywordParser('REQUIRES:', ParserKind.LIST,
-                                    initial_value=requires),
-        IntegratedTestKeywordParser('REQUIRES-ANY:', ParserKind.LIST,
-                                    initial_value=requires_any),
-        IntegratedTestKeywordParser('UNSUPPORTED:', ParserKind.LIST,
-                                    initial_value=unsupported),
+        IntegratedTestKeywordParser('REQUIRES:', ParserKind.BOOLEAN_EXPR,
+                                    initial_value=test.requires),
+        IntegratedTestKeywordParser('REQUIRES-ANY:', ParserKind.CUSTOM,
+                                    IntegratedTestKeywordParser._handleRequiresAny, 
+                                    initial_value=test.requires), 
+        IntegratedTestKeywordParser('UNSUPPORTED:', ParserKind.BOOLEAN_EXPR,
+                                    initial_value=test.unsupported),
         IntegratedTestKeywordParser('END.', ParserKind.TAG)
     ]
     keyword_parsers = {p.keyword: p for p in builtin_parsers}
+    
+    # Install user-defined additional parsers.
     for parser in additional_parsers:
         if not isinstance(parser, IntegratedTestKeywordParser):
             raise ValueError('additional parser must be an instance of '
@@ -883,7 +938,9 @@ def parseIntegratedTestScript(test, addi
             raise ValueError("Parser for keyword '%s' already exists"
                              % parser.keyword)
         keyword_parsers[parser.keyword] = parser
-
+        
+    # Collect the test lines from the script.
+    sourcepath = test.getSourcePath()
     for line_number, command_type, ln in \
             parseIntegratedTestScriptCommands(sourcepath,
                                               keyword_parsers.keys()):
@@ -901,46 +958,30 @@ def parseIntegratedTestScript(test, addi
         return lit.Test.Result(Test.UNRESOLVED,
                                "Test has unterminated run lines (with '\\')")
 
-    # Check that we have the required features:
-    missing_required_features = [f for f in requires
-                                 if f not in test.config.available_features]
+    # Enforce REQUIRES:
+    missing_required_features = test.getMissingRequiredFeatures()
     if missing_required_features:
         msg = ', '.join(missing_required_features)
         return lit.Test.Result(Test.UNSUPPORTED,
-                               "Test requires the following features: %s"
-                               % msg)
-    requires_any_features = [f for f in requires_any
-                             if f in test.config.available_features]
-    if requires_any and not requires_any_features:
-        msg = ' ,'.join(requires_any)
-        return lit.Test.Result(Test.UNSUPPORTED,
-                               "Test requires any of the following features: "
-                               "%s" % msg)
-    unsupported_features = [f for f in unsupported
-                            if f in test.config.available_features]
+                               "Test requires the following unavailable "
+                               "features: %s" % msg)
+
+    # Enforce UNSUPPORTED:
+    unsupported_features = test.getUnsupportedFeatures()
     if unsupported_features:
         msg = ', '.join(unsupported_features)
         return lit.Test.Result(
             Test.UNSUPPORTED,
-            "Test is unsupported with the following features: %s" % msg)
+            "Test does not support the following features "
+            "and/or targets: %s" % msg)
 
-    unsupported_targets = [f for f in unsupported
-                           if f in test.suite.config.target_triple]
-    if unsupported_targets:
-        return lit.Test.Result(
-            Test.UNSUPPORTED,
-            "Test is unsupported with the following triple: %s" % (
-             test.suite.config.target_triple,))
+    # Enforce limit_to_features.
+    if not test.isWithinFeatureLimits():
+        msg = ', '.join(test.config.limit_to_features)
+        return lit.Test.Result(Test.UNSUPPORTED,
+                               "Test does not require any of the features "
+                               "specified in limit_to_features: %s" % msg)
 
-    if test.config.limit_to_features:
-        # Check that we have one of the limit_to_features features in requires.
-        limit_to_features_tests = [f for f in test.config.limit_to_features
-                                   if f in requires]
-        if not limit_to_features_tests:
-            msg = ', '.join(test.config.limit_to_features)
-            return lit.Test.Result(
-                Test.UNSUPPORTED,
-                "Test requires one of the limit_to_features features %s" % msg)
     return script
 
 

Modified: llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-missing.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-missing.txt?rev=293007&r1=293006&r2=293007&view=diff
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-missing.txt (original)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-missing.txt Tue Jan 24 20:26:03 2017
@@ -1,2 +1,5 @@
-RUN: true
-REQUIRES: a-missing-feature
+# REQUIRES with a false clause. Test should not run.
+REQUIRES: true
+REQUIRES: a-missing-feature, true
+REQUIRES: true
+RUN: false

Modified: llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-present.txt
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-present.txt?rev=293007&r1=293006&r2=293007&view=diff
==============================================================================
--- llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-present.txt (original)
+++ llvm/trunk/utils/lit/tests/Inputs/shtest-format/requires-present.txt Tue Jan 24 20:26:03 2017
@@ -1,2 +1,4 @@
+# REQUIRES with only true clauses. Test should run.
+REQUIRES: a-present-feature, true, !not-true
+REQUIRES: true
 RUN: true
-REQUIRES: a-present-feature

Modified: llvm/trunk/utils/lit/tests/shtest-format.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/shtest-format.py?rev=293007&r1=293006&r2=293007&view=diff
==============================================================================
--- llvm/trunk/utils/lit/tests/shtest-format.py (original)
+++ llvm/trunk/utils/lit/tests/shtest-format.py Tue Jan 24 20:26:03 2017
@@ -50,7 +50,14 @@
 # CHECK: PASS: shtest-format :: requires-any-present.txt
 # CHECK: UNSUPPORTED: shtest-format :: requires-missing.txt
 # CHECK: PASS: shtest-format :: requires-present.txt
+# CHECK: UNRESOLVED: shtest-format :: requires-star.txt
+# CHECK: UNSUPPORTED: shtest-format :: requires-triple.txt
+# CHECK: PASS: shtest-format :: unsupported-expr-false.txt
+# CHECK: UNSUPPORTED: shtest-format :: unsupported-expr-true.txt
+# CHECK: UNRESOLVED: shtest-format :: unsupported-star.txt
 # CHECK: UNSUPPORTED: shtest-format :: unsupported_dir/some-test.txt
+# CHECK: PASS: shtest-format :: xfail-expr-false.txt
+# CHECK: XFAIL: shtest-format :: xfail-expr-true.txt
 # CHECK: XFAIL: shtest-format :: xfail-feature.txt
 # CHECK: XFAIL: shtest-format :: xfail-target.txt
 # CHECK: XFAIL: shtest-format :: xfail.txt
@@ -70,9 +77,9 @@
 # CHECK: shtest-format :: external_shell/fail_with_bad_encoding.txt
 # CHECK: shtest-format :: fail.txt
 
-# CHECK: Expected Passes    : 5
-# CHECK: Expected Failures  : 3
-# CHECK: Unsupported Tests  : 3
-# CHECK: Unresolved Tests   : 1
+# CHECK: Expected Passes    : 7
+# CHECK: Expected Failures  : 4
+# CHECK: Unsupported Tests  : 5
+# CHECK: Unresolved Tests   : 3
 # CHECK: Unexpected Passes  : 1
 # CHECK: Unexpected Failures: 3

Modified: llvm/trunk/utils/lit/tests/unit/TestRunner.py
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/utils/lit/tests/unit/TestRunner.py?rev=293007&r1=293006&r2=293007&view=diff
==============================================================================
--- llvm/trunk/utils/lit/tests/unit/TestRunner.py (original)
+++ llvm/trunk/utils/lit/tests/unit/TestRunner.py Tue Jan 24 20:26:03 2017
@@ -108,6 +108,63 @@ class TestIntegratedTestKeywordParser(un
         value = custom_parser.getValue()
         self.assertItemsEqual(value, ['a', 'b', 'c'])
 
+    def test_bad_keywords(self):
+        def custom_parse(line_number, line, output):
+            return output
+        
+        try:
+            IntegratedTestKeywordParser("TAG_NO_SUFFIX", ParserKind.TAG),
+            self.fail("TAG_NO_SUFFIX failed to raise an exception")
+        except ValueError as e:
+            pass
+        except BaseException as e:
+            self.fail("TAG_NO_SUFFIX raised the wrong exception: %r" % e)
+
+        try:
+            IntegratedTestKeywordParser("TAG_WITH_COLON:", ParserKind.TAG),
+            self.fail("TAG_WITH_COLON: failed to raise an exception")
+        except ValueError as e:
+            pass
+        except BaseException as e:
+            self.fail("TAG_WITH_COLON: raised the wrong exception: %r" % e)
+
+        try:
+            IntegratedTestKeywordParser("LIST_WITH_DOT.", ParserKind.LIST),
+            self.fail("LIST_WITH_DOT. failed to raise an exception")
+        except ValueError as e:
+            pass
+        except BaseException as e:
+            self.fail("LIST_WITH_DOT. raised the wrong exception: %r" % e)
+
+        try:
+            IntegratedTestKeywordParser("CUSTOM_NO_SUFFIX",
+                                        ParserKind.CUSTOM, custom_parse),
+            self.fail("CUSTOM_NO_SUFFIX failed to raise an exception")
+        except ValueError as e:
+            pass
+        except BaseException as e:
+            self.fail("CUSTOM_NO_SUFFIX raised the wrong exception: %r" % e)
+
+        # Both '.' and ':' are allowed for CUSTOM keywords.
+        try:
+            IntegratedTestKeywordParser("CUSTOM_WITH_DOT.",
+                                        ParserKind.CUSTOM, custom_parse),
+        except BaseException as e:
+            self.fail("CUSTOM_WITH_DOT. raised an exception: %r" % e)
+        try:
+            IntegratedTestKeywordParser("CUSTOM_WITH_COLON:",
+                                        ParserKind.CUSTOM, custom_parse),
+        except BaseException as e:
+            self.fail("CUSTOM_WITH_COLON: raised an exception: %r" % e)
+
+        try:
+            IntegratedTestKeywordParser("CUSTOM_NO_PARSER:",
+                                        ParserKind.CUSTOM),
+            self.fail("CUSTOM_NO_PARSER: failed to raise an exception")
+        except ValueError as e:
+            pass
+        except BaseException as e:
+            self.fail("CUSTOM_NO_PARSER: raised the wrong exception: %r" % e)
 
 if __name__ == '__main__':
     TestIntegratedTestKeywordParser.load_keyword_parser_lit_tests()




More information about the llvm-commits mailing list