[Lldb-commits] [PATCH] D16334: Fix TestSignedTypes.py by removing a bogus step-over

[Tamas Berghammer via lldb-commits]

tberghammer added a comment.

In http://reviews.llvm.org/D16334#331428, @zturner wrote:

> In http://reviews.llvm.org/D16334#331420, @tberghammer wrote:
>
> > In http://reviews.llvm.org/D16334#331368, @zturner wrote:
> >
> > > I don't know, I still disagree.  If something in step-over breaks, I dont' want to dig through a list of 30 other tests that have nothing to do with the problem, only to find out 2 days later that the problem is actually in step over.  The only reason this helps is because the test suite is insufficient as it is.  But it doesn't need to be!
> >
> >
> > I agree but first we should fix the test coverage and then fix the individual tests. Doing it in the opposite way will cause a significant drop in quality (we will fix individual tests but not increase the coverage enough).
> >
> > > The real solution is for people to start thinking about tests more.  I've hounded on this time and time again, but it seems most of the time tests only get added when I catch a CL go by with no tests and request them.  Sometimes they don't even get added then.  "Oh yea this is on my radar, I'll loop back around to it."  <Months go by, no tests>.  Hundreds of CLs have gone in over the past few months, and probably 10 tests have gone in.  *That's* the problem.  People should be spending as much time thinking about how to write tests as they are about how to write the thing they're implementing.  Almost every CL can be tested.  Everything, no matter how small, can be tested.  If the SB tests are too heavyweight, that's what the unit tests are for.  IF there's no SB API that does what you need to do to test it, add the SB API.  "But I have to design the API first" is not an excuse.  Design it then.
> >
> >
> > I think we need a different API for tests then the SB API which can be changed more freely without have to worry about backward compatibility. When adding a new feature I try to avoid adding an SB API until I know for sure what data I have to expose because a wrong decision early on will carry forward (how many deprecated SB API calls we have?).
>
>
> Do you have a concrete example of where you don't want to add an SB API, but a unit test isn't ideal?


Recently we added several language specific commands for render script and for go. I don't think we want to add SB API support for these at the moment because nobody uses them so maintaining it could be problematic.

In http://reviews.llvm.org/D16334#331429, @zturner wrote:

> In http://reviews.llvm.org/D16334#331420, @tberghammer wrote:
>
> > It is true that every CL can be tested but a lot of change is going in to address a specific edge case generated by a specific compiler in a strange situation. To create a reliable test from it we have to commit in a compiled binary with the strange/incorrect debug info and then it will be a platform and architecture specific test what is also very hard to debug because you most likely can't recompile it with your own compiler. I am not sure we want to go down in this road.
>
>
> You can test cases like this easily with unit tests and dependency injection.  Just make a mock interface that returns the exact values you need when queried for certain symbols or whatever.


To do this we have to mock a lot of thing including the full Process and Thread classes what I am not too fancy doing because I expect that a completely unrelated change will break the mock.

Personally I am more happy to debug a complicated test then to maintain a huge number of tests but I know that lot of people have different preferences. Also I prefer using stress tests and fuzzing instead of trying to create a specific test for each edge case we can think about because it keeps the number of tests low and relatively easy to maintain with having a high probability of detecting issues.


http://reviews.llvm.org/D16334