[Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support

[Jim Ingham via lldb-commits]

> On Sep 14, 2016, at 5:13 PM, Zachary Turner <zturner at google.com> wrote:
> 
> I'm only saying that we should have an open mind.  Obviously there are (valid!) concerns.  If we can't solve them then we can't solve them.  The goal (my goal anyway) is always to make things better, not to use X because it's X.  There's value in consistency, but that doesn't mean that the value from consistency always outweighs the value you get from doing a custom thing.
> 
> So what I'm saying is: IF we can find a way to have one test suite across all of LLVM and it's subprojects, AND it is sufficiently powerful to test all the things we need to test while remaining maintainable in the long term, we should absolutely jump on the opportunity.
> 
> But this is one of those things that requires a lot of upfront time investment before you can actually know if it can work for 100% of things.  Obviously some people don't want to invest their time that way when they're already satisfied, and I don't blame them.
> 
> But for the people who do, and who think they can solve the problem, what's the harm?  Obviously the burden is on those people to prove that their vision can be realized.
> 
> But if it is successful, then there's no denying the benefits.  1) Tests become easier to write.  2) Tests become easier to debug.  3) Consistency encourages people who have traditionally stayed away from LLDB to contribute.  4) All the people pouring their effort into the custom thing can now pour it into the shared thing, so everybody benefits.

I disagree that one of the benefits will be #2.  I worry that what will really happen is "easy tests will become slightly easier to write, and complex tests will be fragile when they aren't impossible" which will mean we end up only writing easy tests.  Lots of easy tests, but we won't test complex multi-threaded stepping behavior, etc and so we'll end up breaking that sort of thing, which we won't find out about quickly because the failures are intermittent, and hard to write bugs about.

Jim


> 
> I don't blame you for being scared of command tests.  I don't support their use in the current LLDB test suite either, for exactly the same reasons you and Jason have expressed.  But I do think it's possible to come up with something that a) doesn't suffer from the same problems, b) allows testing a ton of extra functionality that is not currently testable through the api, and c) doesn't rely on python at all.  If I'm wrong I'll eat crow :)
> 
> On Wed, Sep 14, 2016 at 5:00 PM Jim Ingham <jingham at apple.com> wrote:
> Also, w.r.t:
> 
> >  Aside from write imperative control flow constructs, which I see as a positive rather than a negative.
> 
> I wrote a bunch of tests to test that stepping behavior for swift and C was reasonable.  When stepping through source code, there is not one correct way to write the line tables, and in fact clang & swiftc change how they describe the source through the line tables all the time.  So you have to do: I stepped, and sometimes I'll get to A, sometimes to B, both are "right" but I have to do different things in either case.  If A, step again before the next test, if B go to the next test.
> 
> You could "fix" that by only doing one step per test, and taking each of these as a success.  But then you wouldn't test that series of steps don't accumulate errors, you'd only test "run to a breakpoint and step once."  That would not be good.  So your positive would be very much a negative for this kind of test.
> 
> Traditionally the answer to this has been: we know we have to keep the current testsuite around but we're just adding other new different ways to write tests.  Now you are saying something very different.  Do you really mean that?
> 
> Jim
>