[lldb-dev] Testing through api vs. commands

[Zachary Turner via lldb-dev]

On Thu, Sep 17, 2015 at 10:08 AM <dawn at burble.org> wrote:

> On Tue, Sep 15, 2015 at 11:36:00PM +0000, Zachary Turner wrote:
> > a) they should be explicitly marked as interface tests.
>
> What's the decorator for this?
>
There's not one currently.


>
> > d) Results of these interface tests should also not be *verified* by the
> > use of self.expect, but itself through the API.  (Relying on the text to
> be
> > formatted a specific way is obviously problematic, as opposed to just
> > verifying the actual state of the debugger)
>
> How do you rely "on the text to be formatted a specific way"?
>

Quick example: One of the most common patterns in the test are to do some
things, print a backtrace, and run a regex against the output of the
backtrace.   I've found at least 3 different examples of how this causes
things to fail on Windows:

1) When we demangle a function, we print the full signature.  like
foo.exe`void func(int)`.  On linux/mac, it would print foo`func.  So when
the regex is built as <module>`<func> this will fail on windows, because
our demangled names look different.  The proper way to do this is to get an
SBFrame, get the instruction pointer for it, get an SBSymbol for the
instruction pointer, then get the short name of the symbol and compare it
against 'func' or whatever your function name is.

2) Another example is that there's some tests that verify argument passing
on the command line to the inferior.  Again, it does this by printing a
backtrace, and grepping for "argc=3" anywhere in the backtrace.  This is an
incorrect check for any number of reasons.  On Windows it's incorrect in
that it *succeeds* when it should fail, because argc is actually pointing
to junk memory, and it's value is 3290823098 or something that happens to
start with a 3.  It could also be incorrect.

3) A third example is that when we encounter an exception on Windows
(access violation, etc) the debugger will break.  A backtrace will show
stop reason = exception, as it should.  But we would like to print the
exception code as well, because this is important on Windows.  This differs
from the situation on other platforms, where you might print the name of a
signal, for example.  So you have to carefully construct a regex to make
sure it doesn't mess up platforms which you don't even work on, which is
problematic.

This problem will probably become worse as time goes on, because people who
are accustomed to debugging on Windows (and Linux, and MacOSX, etc) have
certain expectations of how things should look, and to present the best
user experience we would like to match those expectations wherever it is
reasonable to do so.  But since many tests are basically tests of the
formatted command output, it makes it almost impossible to veer from the
current format without breaking tests.

So ultimately, core functionality testing should be independent of command
testing.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150917/c5f4f71c/attachment-0001.html>