[Lldb-commits] [PATCH] D16334: Fix TestSignedTypes.py by removing a bogus step-over

[Tamas Berghammer via lldb-commits]

tberghammer added a subscriber: tberghammer.
tberghammer added a comment.

In http://reviews.llvm.org/D16334#331318, @zturner wrote:

> FWIW, I think Adrian's original point is that testing the behavior of signed types shouldn't depend on step over functionality.  It's good practice in general to make tests depend on as little debugger functionality as possibly to reliably test the thing you want to test.  Because the more functionality you depend on, the more fickle your test becomes.  Why does a bug in one platform's implementation of step over break a test about whether signed ints work?
>
> So, I'm all for removing this test's dependency on step-over (TestUnsignedTypes doesn't use step over, for example) if there's a way to reliably test the functionality without step over.
>
> But I still think it's important to know what CL broke all these tests.


In general I agree with your concept of trying to make the tests standalone without depending on a lot of other functionality, but I see a major issue. Currently our test coverage is low even for the basic functionality (backtrace, frame variables, stepping, etc.) and a lot of failure in these areas are detected by completely unrelated tests because they are depending on them and as a result doing some sort of stress testing (each test try a slightly different situation). If we make all of our tests standalone without increasing the number of tests by a lot (I guess 2-5x needed) I expect that we will lose a lot of coverage. I am more happy if an unrelated test fails because we introduced a bug then ending up with a lot more undetected bugs.


http://reviews.llvm.org/D16334