[lldb-dev] Is anyone using the gtest Xcode project?

[Zachary Turner]

On Fri, Mar 13, 2015 at 4:01 PM Jonathan Roelofs <jonathan at codesourcery.com>
wrote:

> +ddunbar
>
> On 3/13/15 9:53 AM, jingham at apple.com wrote:
> >>> Depending on how different the different things are.  Compiler tests
> >>> tend to have input, output and some machine that converts the input to
> >>> the output.  That is one very particular model of testing.  Debugger
> >>> tests need to do: get to stage 1, if that succeeded, get to stage 2,
> >>> if that succeeded, etc.  Plus there's generally substantial setup code
> >>> to get somewhere interesting, so while you are there you generally try
> >>> to test a bunch of similar things.  Plus, the tests often have points
> >>> where there are several success cases, but each one requires a
> >>> different "next action", stepping being the prime example of this.
> >>> These are very different models and I don't see that trying to smush
> >>> the two together would be a fruitful exercise.
>
> I think LIT does make the assumption that one "test file" has one "test
> result". But this is a place where we could extend LIT a bit. I don't
> think it would be very painful.
>
> For me, this would be very useful for a few of the big libc++abi tests,
> like the demangler one, as currently I have to #ifdef out a couple of
> the cases that can't possibly work on my platform. It would be much
> nicer if that particular test file outputted multiple test results of
> which I could XFAIL the ones I know won't ever work. (For anyone who is
> curious, the one that comes to mind needs the c99 %a printf format,
> which my libc doesn't have. It's a baremetal target, and binary size is
> really important).
>
> How much actual benefit is there in having lots of results per test
> case, rather than having them all &&'d together to one result?
>
> Out of curiosity, does lldb's existing testsuite allow you to run
> individual test results in test cases where there are more than one test
> result?


 I think I'm not following this line of discussion.  So it's possible you
and Jim are talking about different things here.

If I understand correctly (and maybe I don't), what Jim is saying is that a
debugger test might need to do something like:

1) Set 5 breakpoints
2) Continue
3) Depending on which breakpoint gets hit, take one of 5 possible "next"
actions.

But I'm having trouble coming up with an example of why this might be
useful.  Jim, can you make this a little more concrete with a specific
example of a test that does this, how the test works, and what the
different success / failure cases are so we can be sure everyone is on the
same page?

In the case of the libc++ abi tests, I'm not sure what is meant by
"multiple results per test case".  Do you mean (for example) you'd like to
be able to XFAIL individual run lines based on some condition?  If so, LLDB
definitely needs that.  One example which LLDB uses almost everywhere is
that of running the same test with dSYM or DWARF debug info.  On Apple
platforms, tests generally need to run with both dSYM and DWARF debug info
(literally just repeat the same test twice), and on non Apple platforms,
only DWARF tests ever need to be run.  So there would need to be a way to
express this.

There are plenty of other one-off examples.  Debuggers have a lot of
platform specific code, and the different platforms support different
amounts of functionality (especially for things like Android / Windows that
are works in progress).  So we frequently have the need to have a single
test file which has, say 10 tests in it.  And specific tests can be XFAILed
or even disabled individually based on conditions (usually which platform
is running the test suite, but not always).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150314/04957429/attachment.html>