[libcxx] r174440 - [tests] Add support for REQUIRES and XFAIL lines in libc++ tests.

[David Blaikie]

On Tue, Feb 5, 2013 at 2:08 PM, Daniel Dunbar <daniel at zuster.org> wrote:
>
>
>
> On Tue, Feb 5, 2013 at 1:07 PM, David Blaikie <dblaikie at gmail.com> wrote:
>>
>> On Tue, Feb 5, 2013 at 1:03 PM, Daniel Dunbar <daniel at zuster.org> wrote:
>> > Author: ddunbar
>> > Date: Tue Feb  5 15:03:25 2013
>> > New Revision: 174440
>> >
>> > URL: http://llvm.org/viewvc/llvm-project?rev=174440&view=rev
>> > Log:
>> > [tests] Add support for REQUIRES and XFAIL lines in libc++ tests.
>> >
>> >  - We parse up to the first non-empty non-comment (C++ style) line,
>> > otherwise
>> >    the format and semantics match what is used for LLVM/Clang tests.
>>
>> Any particular reason for the difference?
>
>
> Most tests don't have metadata

I'm not sure which metadata you're referring to which would influence
this aspect. (no doubt there is - I'm just not sure what it is you're
referring to)

> and we don't have a legacy requirement, this saves parsing time.

If that's a preference for all tests should we document it somewhere &
intend to cleanup the existing test suites for consistency? (it still
seems a little subtle that those first few comments are 'special' & I
don't know if the saved parsing time would be noticeable (any
numbers?), but people probably do this by default anyway so it doesn't
seem like a big issue - but the inconsistency could be annoyingly
surprising)

>
>  - Daniel
>
>>
>> >  - For now, the only interesting thing to test against is a user
>> > supplied
>> >    target_triple test parameter.
>> >
>> > Modified:
>> >     libcxx/trunk/test/lit.cfg
>> >
>> > Modified: libcxx/trunk/test/lit.cfg
>> > URL:
>> > http://llvm.org/viewvc/llvm-project/libcxx/trunk/test/lit.cfg?rev=174440&r1=174439&r2=174440&view=diff
>> >
>> > ==============================================================================
>> > --- libcxx/trunk/test/lit.cfg (original)
>> > +++ libcxx/trunk/test/lit.cfg Tue Feb  5 15:03:25 2013
>> > @@ -11,6 +11,26 @@ import subprocess
>> >  import errno
>> >  import time
>> >
>> > +# FIXME: For now, this is cribbed from lit.TestRunner, to avoid
>> > introducing a
>> > +# dependency there. What we more ideally would like to do is lift the
>> > "xfail"
>> > +# and "requires" handling to be a core lit framework feature.
>> > +def isExpectedFail(test, xfails):
>> > +    # Check if any of the xfails match an available feature or the
>> > target.
>> > +    for item in xfails:
>> > +        # If this is the wildcard, it always fails.
>> > +        if item == '*':
>> > +            return True
>> > +
>> > +        # If this is an exact match for one of the features, it fails.
>> > +        if item in test.config.available_features:
>> > +            return True
>> > +
>> > +        # If this is a part of the target triple, it fails.
>> > +        if item in test.suite.config.target_triple:
>> > +            return True
>> > +
>> > +    return False
>> > +
>> >  class LibcxxTestFormat(lit.formats.FileBasedTest):
>> >      """
>> >      Custom test format handler for use with the test format use by
>> > libc++.
>> > @@ -55,6 +75,52 @@ class LibcxxTestFormat(lit.formats.FileB
>> >                  time.sleep(0.1)
>> >
>> >      def _execute(self, test, lit_config):
>> > +        # Extract test metadata from the test file.
>> > +        xfails = []
>> > +        requires = []
>> > +        with open(test.getSourcePath()) as f:
>> > +            for ln in f:
>> > +                if 'XFAIL:' in ln:
>> > +                    items = ln[ln.index('XFAIL:') + 6:].split(',')
>> > +                    xfails.extend([s.strip() for s in items])
>> > +                elif 'REQUIRES:' in ln:
>> > +                    items = ln[ln.index('REQUIRES:') + 9:].split(',')
>> > +                    requires.extend([s.strip() for s in items])
>> > +                elif not ln.startswith("//") and ln.strip():
>> > +                    # Stop at the first non-empty line that is not a
>> > C++
>> > +                    # comment.
>> > +                    break
>> > +
>> > +        # Check that we have the required features.
>> > +        #
>> > +        # FIXME: For now, this is cribbed from lit.TestRunner, to avoid
>> > +        # introducing a dependency there. What we more ideally would
>> > like to do
>> > +        # is lift the "xfail" and "requires" handling to be a core lit
>> > framework
>> > +        # feature.
>> > +        missing_required_features = [f for f in requires
>> > +                                     if f not in
>> > test.config.available_features]
>> > +        if missing_required_features:
>> > +            return (lit.Test.UNSUPPORTED,
>> > +                    "Test requires the following features: %s" % (
>> > +                      ', '.join(missing_required_features),))
>> > +
>> > +        # Determine if this test is an expected failure.
>> > +        isXFail = isExpectedFail(test, xfails)
>> > +
>> > +        # Evaluate the test.
>> > +        result, report = self._evaluate_test(test, lit_config)
>> > +
>> > +        # Convert the test result based on whether this is an expected
>> > failure.
>> > +        if isXFail:
>> > +            if result != lit.Test.FAIL:
>> > +                report += "\n\nTest was expected to FAIL, but did
>> > not.\n"
>> > +                result = lit.Test.XPASS
>> > +            else:
>> > +                result = lit.Test.XFAIL
>> > +
>> > +        return result, report
>> > +
>> > +    def _evaluate_test(self, test, lit_config):
>> >          name = test.path_in_suite[-1]
>> >          source_path = test.getSourcePath()
>> >          source_dir = os.path.dirname(source_path)
>> > @@ -210,4 +276,5 @@ config.test_format = LibcxxTestFormat(
>> >      ld_flags = ['-nodefaultlibs'] + library_paths + ['-lc++'] +
>> > libraries,
>> >      exec_env = exec_env)
>> >
>> > -config.target_triple = None
>> > +config.target_triple = lit.params.get(
>> > +    'target_triple', 'unknown-unknown-unknown')
>> >
>> >
>> > _______________________________________________
>> > cfe-commits mailing list
>> > cfe-commits at cs.uiuc.edu
>> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-commits
>
>