[lldb-dev] Is anyone using the gtest Xcode project?

[Jonathan Roelofs]

On 3/13/15 9:10 PM, Zachary Turner wrote:
>
>
> On Fri, Mar 13, 2015 at 4:01 PM Jonathan Roelofs
> <jonathan at codesourcery.com <mailto:jonathan at codesourcery.com>> wrote:
>
>     +ddunbar
>
>     On 3/13/15 9:53 AM, jingham at apple.com <mailto:jingham at apple.com> wrote:
>      >>> Depending on how different the different things are.  Compiler
>     tests
>      >>> tend to have input, output and some machine that converts the
>     input to
>      >>> the output.  That is one very particular model of testing.
>     Debugger
>      >>> tests need to do: get to stage 1, if that succeeded, get to
>     stage 2,
>      >>> if that succeeded, etc.  Plus there's generally substantial
>     setup code
>      >>> to get somewhere interesting, so while you are there you
>     generally try
>      >>> to test a bunch of similar things.  Plus, the tests often have
>     points
>      >>> where there are several success cases, but each one requires a
>      >>> different "next action", stepping being the prime example of this.
>      >>> These are very different models and I don't see that trying to
>     smush
>      >>> the two together would be a fruitful exercise.
>
>     I think LIT does make the assumption that one "test file" has one "test
>     result". But this is a place where we could extend LIT a bit. I don't
>     think it would be very painful.
>
>     For me, this would be very useful for a few of the big libc++abi tests,
>     like the demangler one, as currently I have to #ifdef out a couple of
>     the cases that can't possibly work on my platform. It would be much
>     nicer if that particular test file outputted multiple test results of
>     which I could XFAIL the ones I know won't ever work. (For anyone who is
>     curious, the one that comes to mind needs the c99 %a printf format,
>     which my libc doesn't have. It's a baremetal target, and binary size is
>     really important).
>
>     How much actual benefit is there in having lots of results per test
>     case, rather than having them all &&'d together to one result?
>
>     Out of curiosity, does lldb's existing testsuite allow you to run
>     individual test results in test cases where there are more than one test
>     result?
>
>
>   I think I'm not following this line of discussion.  So it's possible
> you and Jim are talking about different things here.

I think that's the case... I was imagining the "logic of the test" 
something like this:

   1) Set 5 breakpoints
   2) Continue
   3) Assert that the debugger stopped at the first breakpoint
   4) Continue
   5) Assert that the debugger stopped at the second breakpoint
   6) etc.

Reading Jim's description again, with the help of your speculative 
example, it sounds like the test logic itself isn't straightline 
code.... that's okay too. What I was speaking to is a perceived 
difference in what the "results" of running such a test are.

In llvm, the assertions are CHECK lines. In libc++, the assertions are 
calls to `assert` from assert.h, as well as `static_assert`s. In both 
cases, failing any one of those checks in a test makes the whole test 
fail. For some reason I had the impression that in lldb there wasn't a 
single test result per *.py test. Perhaps that's not the case? Either 
way, what I want to emphasize is that LIT doesn't care about the "logic 
of the test", as long as there is one test result per test (and even 
that condition could be amended, if it would be useful for lldb).

>
> If I understand correctly (and maybe I don't), what Jim is saying is
> that a debugger test might need to do something like:
>
> 1) Set 5 breakpoints
> 2) Continue
> 3) Depending on which breakpoint gets hit, take one of 5 possible "next"
> actions.
>
> But I'm having trouble coming up with an example of why this might be
> useful.  Jim, can you make this a little more concrete with a specific
> example of a test that does this, how the test works, and what the
> different success / failure cases are so we can be sure everyone is on
> the same page?
>
> In the case of the libc++ abi tests, I'm not sure what is meant by
> "multiple results per test case".  Do you mean (for example) you'd like
> to be able to XFAIL individual run lines based on some condition?  If

I think this means I should make the libc++abi example even more 
concrete.... In libc++/libc++abi tests, the "RUN" line is implicit 
(well, aside from the few ShTest tests ericwf has added recently). Every 
*.pass.cpp test is a file that the test harness knows it has to compile, 
run, and check its exit status. That being said, 
libcxxabi/test/test_demangle.pass.cpp has a huge array like this:

       20 const char* cases[][2] =
       21 {
       22     {"_Z1A", "A"},
       23     {"_Z1Av", "A()"},
       24     {"_Z1A1B1C", "A(B, C)"},
       25     {"_Z4testI1A1BE1Cv", "C test<A, B>()"},

    snip

    29594     {"_Zli2_xy", "operator\"\" _x(unsigned long long)"},
    29595     {"_Z1fIiEDcT_", "decltype(auto) f<int>(int)"},
    29596 };

Then there's some logic in `main()` that runs, __cxa_demangle on 
`cases[i][0]`, and asserts that it's the same as `cases[i][1]`. If any 
of those assertions fail, the entire test is marked as failing, and no 
further lines in that array are verified. For the sake of discussion, 
let's call each of entries in `cases` a "subtest", and the entirety of 
test_demangle.pass.cpp a test.

The sticky issue is that there are a few subtests in this test that 
don't make sense on various platforms, so currently, they are #ifdef'd 
out. If the LIT TestFormat and the tests themselves had a way to 
communicate that a subtest failed, but to continue running other 
subtests after that, then we could XFAIL these weird subtests individually.

Keep in mind though that I'm not really advocating we go and change 
test_demangle.pass.cpp to suit that model, because #ifdef's work 
reasonably well there, and there are relatively few subtests that have 
these platform differences... That's just the first example of the 
test/subtest relationship that I could think of.

> so, LLDB definitely needs that.  One example which LLDB uses almost
> everywhere is that of running the same test with dSYM or DWARF debug
> info.  On Apple platforms, tests generally need to run with both dSYM
> and DWARF debug info (literally just repeat the same test twice), and on
> non Apple platforms, only DWARF tests ever need to be run.  So there
> would need to be a way to express this.

Can you point me to an example of this?

>
> There are plenty of other one-off examples.  Debuggers have a lot of
> platform specific code, and the different platforms support different
> amounts of functionality (especially for things like Android / Windows
> that are works in progress).  So we frequently have the need to have a
> single test file which has, say 10 tests in it.  And specific tests can
> be XFAILed or even disabled individually based on conditions (usually
> which platform is running the test suite, but not always).

-- 
Jon Roelofs
jonathan at codesourcery.com
CodeSourcery / Mentor Embedded