[lldb-dev] Testing through api vs. commands

[Zachary Turner via lldb-dev]

The past few weeks I've spent a lot of time xfailing the rest of the
failing tests on windows so we can enable tests to run on the bots.

One thing I ran into more frequently than I would have liked is that the
tests were failing not because the functionality is broken, but because the
substrings being grepped for in the output had a slightly different format
on Windows.  The pattern for tests is frequently something like:

result = runCommand(<some command>)
self.expect(<result matches some regex>)

A good example of this is that when you do a backtrace, on windows you
might see a fully demangled function name such as a.out`void foo(int x),
whereas on other platforms you might just see a.out`foo.

I saw the reverse situation as well, where a test was passing but shouldn't
have because it was actually broken, but due to the impreciseness of
grepping output the grep was suceeding.  Specifically, this was happening
on a test that verified function parameters.  it launched a program with 3
arguments, and then looked for "(int)argc=3" in the frame info.  It was
broken on Windows because argc was pointing to junk memory, so it was
actually printing "(int)argc=3248902" in the output.  Test was passing,
even though it was broken.

Rather than make the regexes more complicated, I think the right fix here
is to stop using the command system and grepping to write tests.  Just go
through the api for everything, including verifying the result.  In the
second case, for example, you launch the process, set the breakpoint, wait
for it to stop, find the argument named argc, and verify that it's value is
3.

I don't want to propose going back and rewriting every single test to do
this, but I want to see how people feel about moving toward this model
going forward as the default method of writing tests.

I do still think we need some tests that verify commands run, but I think
those tests should focus not on doing complicated interactions with the
debugger, and instead just verifying that things parse correctly and the
command is configured correctly, with the underlying functionality being
tested by the api tests.

Thoughts?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150911/5fb3034d/attachment.html>