[Lldb-commits] [lldb] e3a7c77 - [lldb/Lit] Change the lldbtest format to behave more like shell test.

[Jonas Devlieghere via lldb-commits]

Yep, Michał also reported this here: https://reviews.llvm.org/D73384

I hope to take a look at that today.

On Thu, Jan 30, 2020 at 1:43 PM Vedant Kumar <vedant_kumar at apple.com> wrote:
>
>
>
> > On Jan 24, 2020, at 4:18 PM, Jonas Devlieghere via lldb-commits <lldb-commits at lists.llvm.org> wrote:
> >
> >
> > Author: Jonas Devlieghere
> > Date: 2020-01-24T16:17:55-08:00
> > New Revision: e3a7c7713cd87e37a95a544373cd21f6f06ab94e
> >
> > URL: https://github.com/llvm/llvm-project/commit/e3a7c7713cd87e37a95a544373cd21f6f06ab94e
> > DIFF: https://github.com/llvm/llvm-project/commit/e3a7c7713cd87e37a95a544373cd21f6f06ab94e.diff
> >
> > LOG: [lldb/Lit] Change the lldbtest format to behave more like shell test.
> >
> > The current lldbtest format has a number of shortcomings, all related to
> > how we omit information based on why the test fails. For example, a
> > successful test would print nothing, even when `-a` is passed to lit.
> > It's not up to the test format to decide whether to print something or
> > not, that's handled by lit itself. For other test results we would
> > sometimes print stdout & stderr, but not always, such as when a timeout
> > was reached or we couldn't parse the dotest output.
> >
> > This patch changes the lldbtest format and makes it behave more like
> > lit. We now always print the dotest invocation, the exit code, the
> > output to stdout & stderr. If you're used to dealing with ShTests in
> > lit, this will feel all very familiar.
> >
> > Differential revision: https://reviews.llvm.org/D73384
> >
> > Added:
> >
> >
> > Modified:
> >    lldb/test/API/lldbtest.py
> >
> > Removed:
> >
> >
> >
> > ################################################################################
> > diff  --git a/lldb/test/API/lldbtest.py b/lldb/test/API/lldbtest.py
> > index 349a67f22fb3..864d5ea1df03 100644
> > --- a/lldb/test/API/lldbtest.py
> > +++ b/lldb/test/API/lldbtest.py
> > @@ -86,33 +86,46 @@ def execute(self, test, litConfig):
> >                 shutil.copy(python, copied_python)
> >             cmd[0] = copied_python
> >
> > +        timeoutInfo = None
> >         try:
> >             out, err, exitCode = lit.util.executeCommand(
> >                 cmd,
> >                 env=test.config.environment,
> >                 timeout=litConfig.maxIndividualTestTime)
> >         except lit.util.ExecuteCommandTimeoutException:
> > -            return (lit.Test.TIMEOUT, 'Reached timeout of {} seconds'.format(
> > -                litConfig.maxIndividualTestTime))
> > +            timeoutInfo = 'Reached timeout of {} seconds'.format(
> > +                litConfig.maxIndividualTestTime)
> > +
> > +        output = """Script:\n--\n%s\n--\nExit Code: %d\n""" % (
> > +            ' '.join(cmd), exitCode)
>
> Heads up, when a test times out this prints:
>
>   File "/Users/vsk/src/llvm-project-master/lldb/test/API/lldbtest.py", line 100, in execute
>     ' '.join(cmd), exitCode)
> UnboundLocalError: local variable 'exitCode' referenced before assignment
>
>
> > +        if timeoutInfo is not None:
> > +            output += """Timeout: %s\n""" % (timeoutInfo,)
> > +        output += "\n"
> > +
> > +        if out:
> > +            output += """Command Output (stdout):\n--\n%s\n--\n""" % (out,)
> > +        if err:
> > +            output += """Command Output (stderr):\n--\n%s\n--\n""" % (err,)
> > +
> > +        if timeoutInfo:
> > +            return lit.Test.TIMEOUT, output
> >
> >         if exitCode:
> >             # Match FAIL but not XFAIL.
> >             for line in out.splitlines() + err.splitlines():
> >                 if line.startswith('FAIL:'):
> > -                    return lit.Test.FAIL, out + err
> > +                    return lit.Test.FAIL, output
> >
> >             if 'XPASS:' in out or 'XPASS:' in err:
> > -                return lit.Test.XPASS, out + err
> > +                return lit.Test.XPASS, output
> >
> >         has_unsupported_tests = 'UNSUPPORTED:' in out or 'UNSUPPORTED:' in err
> >         has_passing_tests = 'PASS:' in out or 'PASS:' in err
> >         if has_unsupported_tests and not has_passing_tests:
> > -            return lit.Test.UNSUPPORTED, out + err
> > +            return lit.Test.UNSUPPORTED, output
> >
> >         passing_test_line = 'RESULT: PASSED'
> >         if passing_test_line not in out and passing_test_line not in err:
> > -            msg = ('Unable to find %r in dotest output (exit code %d):\n\n%s%s'
> > -                   % (passing_test_line, exitCode, out, err))
> > -            return lit.Test.UNRESOLVED, msg
> > +            return lit.Test.UNRESOLVED, output
> >
> > -        return lit.Test.PASS, ''
> > +        return lit.Test.PASS, output
> >
> >
> >
> > _______________________________________________
> > lldb-commits mailing list
> > lldb-commits at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
>