[lldb-dev] Testing through api vs. commands

[Zachary Turner via lldb-dev]

We'll probably rewrite tests that we find are failing specifically as a
result of issues like this, but I agree it's not worth re-writing
everything else except on an as-needed basis.

To make the distinction explicit and enforce it kind of at an
organnizational level, would it be worth creating folders under lldb/test
like lldb/test/commands, and recommending that all HandleCommand tests to
go there?

Possibly unrelated question: But in regards to this api test vs.
HandleCommand test situation, is that what the purpose of the
@python_api_test decorator is (and is this decorator even still useful)?

On Fri, Sep 11, 2015 at 11:42 AM Jim Ingham <jingham at apple.com> wrote:

> I have held from the beginning that the only tests that should be written
> using HandleCommand are those that explicitly test command behavior, and if
> it is possible to write a test using the SB API you should always do it
> that way for the very reasons you cite.  Not everybody agreed with me at
> first, so we ended up with a bunch of tests that do complex things using
> HandleCommand where they really ought not to.  I'm not sure it is worth the
> time to go rewrite all those tests, but we shouldn't write any new tests
> that way.
>
> Jim
>
>
> > On Sep 11, 2015, at 11:33 AM, Zachary Turner via lldb-dev <
> lldb-dev at lists.llvm.org> wrote:
> >
> > The past few weeks I've spent a lot of time xfailing the rest of the
> failing tests on windows so we can enable tests to run on the bots.
> >
> > One thing I ran into more frequently than I would have liked is that the
> tests were failing not because the functionality is broken, but because the
> substrings being grepped for in the output had a slightly different format
> on Windows.  The pattern for tests is frequently something like:
> >
> > result = runCommand(<some command>)
> > self.expect(<result matches some regex>)
> >
> > A good example of this is that when you do a backtrace, on windows you
> might see a fully demangled function name such as a.out`void foo(int x),
> whereas on other platforms you might just see a.out`foo.
> >
> > I saw the reverse situation as well, where a test was passing but
> shouldn't have because it was actually broken, but due to the impreciseness
> of grepping output the grep was suceeding.  Specifically, this was
> happening on a test that verified function parameters.  it launched a
> program with 3 arguments, and then looked for "(int)argc=3" in the frame
> info.  It was broken on Windows because argc was pointing to junk memory,
> so it was actually printing "(int)argc=3248902" in the output.  Test was
> passing, even though it was broken.
> >
> > Rather than make the regexes more complicated, I think the right fix
> here is to stop using the command system and grepping to write tests.  Just
> go through the api for everything, including verifying the result.  In the
> second case, for example, you launch the process, set the breakpoint, wait
> for it to stop, find the argument named argc, and verify that it's value is
> 3.
> >
> > I don't want to propose going back and rewriting every single test to do
> this, but I want to see how people feel about moving toward this model
> going forward as the default method of writing tests.
> >
> > I do still think we need some tests that verify commands run, but I
> think those tests should focus not on doing complicated interactions with
> the debugger, and instead just verifying that things parse correctly and
> the command is configured correctly, with the underlying functionality
> being tested by the api tests.
> >
> > Thoughts?
> >
> >
> > _______________________________________________
> > lldb-dev mailing list
> > lldb-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150911/58709840/attachment.html>