[Lldb-commits] [PATCH] D16334: Fix TestSignedTypes.py by removing a bogus step-over

[Zachary Turner via lldb-commits]

So, I don't actually disagree that the idea of doing a "step and see if it
still works" is ok.  Because I think that -- by itself -- is useful as an
additional check.  I'm not against doing extra work in a test when it adds
something concretely useful to the test.  So I still think we should reduce
all test cases as much as possible, but as long as we can justify each
operation as adding additional robustness to that particular test, I'm fine
with it.

That said, I don't think we should discourage finding better ways to test
something either.  If someone comes up with a different approach to writing
a test for a particular feature, and that approach relies on less debugger
functionality, but still tests the feature in question in , I think that
should be encouraged.  More granular tests give more clues as to the nature
of the underlying problem, less false positives on failures, and make the
test suite run faster.

On Thu, Jan 21, 2016 at 12:04 PM Jim Ingham <jingham at apple.com> wrote:

> I'm not sure this is a terribly productive discussion.
>
> Since I know that the debugger is stateful, when I write a test where I
> get to point A and do thing X, I will often add - while I'm there - "step
> again and see if it still works" or something morally equivalent to that.
> I have found that to be a method of test writing for debuggers that very
> often catches bugs.  The fact that this test will then break if "step"
> breaks doesn't bother me because a) this might be the only example where
> step breaks in this particular way, so that was actually a plus, and if
> something basic like step breaks we're going to fix it right away anyway
> 'cause it is step not working...
>
> I am also not against writing more focused tests when that is
> appropriate.  But I am also pretty sure formal considerations are unlikely
> to outweigh this pretty consistent experience of writing tests for
> debuggers.
>
> Jim
>
> > On Jan 21, 2016, at 11:54 AM, Zachary Turner via lldb-commits <
> lldb-commits at lists.llvm.org> wrote:
> >
> > zturner added a comment.
> >
> > Right, but I don't agree in this case that "different" has to mean "we
> > discourage the use of reduced test cases".  I have a hard time imagining
> a
> > scenario where not having reduced test cases is an advantage.  It's also
> > easy to explain to people.  "Write reduced test cases".  It's easy to
> > understand.  Moreso than "Reduced test cases are good, but not always
> > because fuzziness, but we can't really put into words exactly what amount
> > of fuzziness we want, or what it should look like, so I guess anything
> goes
> >
> > - don't ask don't tell".
> >
> > If we want fuzziness, we should do fuzz testing.  Feature tests and
> > regression tests should be reduced.  The whole reason the tests in
> > lldb/tests were written is because people were testing a specific feature
> > or bugfix.  I don't see a reason to add fuzziness to this kind of test.
> > All it does is pollute failure logs.  Fuzziness is better tested by fuzz
> > tests.
> >
> >
> > http://reviews.llvm.org/D16334
> >
> >
> >
> > _______________________________________________
> > lldb-commits mailing list
> > lldb-commits at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-commits/attachments/20160121/19a21f1b/attachment.html>