[libcxx] r174440 - [tests] Add support for REQUIRES and XFAIL lines in libc++ tests.

[Daniel Dunbar]

On Tue, Feb 5, 2013 at 1:07 PM, David Blaikie <dblaikie at gmail.com> wrote:

> On Tue, Feb 5, 2013 at 1:03 PM, Daniel Dunbar <daniel at zuster.org> wrote:
> > Author: ddunbar
> > Date: Tue Feb  5 15:03:25 2013
> > New Revision: 174440
> >
> > URL: http://llvm.org/viewvc/llvm-project?rev=174440&view=rev
> > Log:
> > [tests] Add support for REQUIRES and XFAIL lines in libc++ tests.
> >
> >  - We parse up to the first non-empty non-comment (C++ style) line,
> otherwise
> >    the format and semantics match what is used for LLVM/Clang tests.
>
> Any particular reason for the difference?
>

Most tests don't have metadata and we don't have a legacy requirement, this
saves parsing time.

 - Daniel


> >  - For now, the only interesting thing to test against is a user supplied
> >    target_triple test parameter.
> >
> > Modified:
> >     libcxx/trunk/test/lit.cfg
> >
> > Modified: libcxx/trunk/test/lit.cfg
> > URL:
> http://llvm.org/viewvc/llvm-project/libcxx/trunk/test/lit.cfg?rev=174440&r1=174439&r2=174440&view=diff
> >
> ==============================================================================
> > --- libcxx/trunk/test/lit.cfg (original)
> > +++ libcxx/trunk/test/lit.cfg Tue Feb  5 15:03:25 2013
> > @@ -11,6 +11,26 @@ import subprocess
> >  import errno
> >  import time
> >
> > +# FIXME: For now, this is cribbed from lit.TestRunner, to avoid
> introducing a
> > +# dependency there. What we more ideally would like to do is lift the
> "xfail"
> > +# and "requires" handling to be a core lit framework feature.
> > +def isExpectedFail(test, xfails):
> > +    # Check if any of the xfails match an available feature or the
> target.
> > +    for item in xfails:
> > +        # If this is the wildcard, it always fails.
> > +        if item == '*':
> > +            return True
> > +
> > +        # If this is an exact match for one of the features, it fails.
> > +        if item in test.config.available_features:
> > +            return True
> > +
> > +        # If this is a part of the target triple, it fails.
> > +        if item in test.suite.config.target_triple:
> > +            return True
> > +
> > +    return False
> > +
> >  class LibcxxTestFormat(lit.formats.FileBasedTest):
> >      """
> >      Custom test format handler for use with the test format use by
> libc++.
> > @@ -55,6 +75,52 @@ class LibcxxTestFormat(lit.formats.FileB
> >                  time.sleep(0.1)
> >
> >      def _execute(self, test, lit_config):
> > +        # Extract test metadata from the test file.
> > +        xfails = []
> > +        requires = []
> > +        with open(test.getSourcePath()) as f:
> > +            for ln in f:
> > +                if 'XFAIL:' in ln:
> > +                    items = ln[ln.index('XFAIL:') + 6:].split(',')
> > +                    xfails.extend([s.strip() for s in items])
> > +                elif 'REQUIRES:' in ln:
> > +                    items = ln[ln.index('REQUIRES:') + 9:].split(',')
> > +                    requires.extend([s.strip() for s in items])
> > +                elif not ln.startswith("//") and ln.strip():
> > +                    # Stop at the first non-empty line that is not a C++
> > +                    # comment.
> > +                    break
> > +
> > +        # Check that we have the required features.
> > +        #
> > +        # FIXME: For now, this is cribbed from lit.TestRunner, to avoid
> > +        # introducing a dependency there. What we more ideally would
> like to do
> > +        # is lift the "xfail" and "requires" handling to be a core lit
> framework
> > +        # feature.
> > +        missing_required_features = [f for f in requires
> > +                                     if f not in
> test.config.available_features]
> > +        if missing_required_features:
> > +            return (lit.Test.UNSUPPORTED,
> > +                    "Test requires the following features: %s" % (
> > +                      ', '.join(missing_required_features),))
> > +
> > +        # Determine if this test is an expected failure.
> > +        isXFail = isExpectedFail(test, xfails)
> > +
> > +        # Evaluate the test.
> > +        result, report = self._evaluate_test(test, lit_config)
> > +
> > +        # Convert the test result based on whether this is an expected
> failure.
> > +        if isXFail:
> > +            if result != lit.Test.FAIL:
> > +                report += "\n\nTest was expected to FAIL, but did
> not.\n"
> > +                result = lit.Test.XPASS
> > +            else:
> > +                result = lit.Test.XFAIL
> > +
> > +        return result, report
> > +
> > +    def _evaluate_test(self, test, lit_config):
> >          name = test.path_in_suite[-1]
> >          source_path = test.getSourcePath()
> >          source_dir = os.path.dirname(source_path)
> > @@ -210,4 +276,5 @@ config.test_format = LibcxxTestFormat(
> >      ld_flags = ['-nodefaultlibs'] + library_paths + ['-lc++'] +
> libraries,
> >      exec_env = exec_env)
> >
> > -config.target_triple = None
> > +config.target_triple = lit.params.get(
> > +    'target_triple', 'unknown-unknown-unknown')
> >
> >
> > _______________________________________________
> > cfe-commits mailing list
> > cfe-commits at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-commits
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20130205/c70843e0/attachment.html>