[Lldb-commits] [PATCH] D42656: [testsuite] Remove flakey test relying on `pexpect`

[Jim Ingham via lldb-commits]

So far, all the "flakey" tests I've analyzed - of which there have been not a few over the years - have either been:

1) Why was pexpect so hard to get right when expect has been rock solid for decades...
2) Tests whose setup conditions are hard to get right, or hard to have happen on whatever machine we have to run them on in a timely fashion
3) Badly written tests - very often these are command-scraping tests that test for incidental accidents of the output
4) Actual flakiness in liblldb.

Of these, #2 is the most common, followed (sadly for our pride but to the credit of the testsuite) were real non-deterministic bugs in lldb.

So I'm not much convinced that the supposed flakiness of dotest provides a substantial reason to go write a whole parallel test harness.

Jim


> On Jan 29, 2018, at 5:51 PM, Zachary Turner <zturner at google.com> wrote:
> 
> We’ve had many instances of flakiness in non pexpect tests (on all platforms). There’s no obvious pattern to when a test will be flaky. Whether those are due to dotest or liblldb is an open question, but one good way of answering those types of questions is to replace one source of unknown-flakiness with a source of known-not-flakiness and seeing if the flakiness goes away.
> 
> The new-and-not-tested code you’re referring to would be about 5 lines of c++ that also directly calls the api, just like your dotest example. So that aspect doesn’t feel like a convincing argument 
> On Mon, Jan 29, 2018 at 5:28 PM Jim Ingham via Phabricator <reviews at reviews.llvm.org> wrote:
> jingham added a comment.
> 
> lldb testcases are know not to be flakey if they don't use pexpect, which these wouldn't.  The setup machinery for running a dotest based test is pretty well tested at this point.
> 
> And the lldb-test test would not just magically come into being by writing the lit-form text you suggested.  You will have to write a lldb-test func that marshals the input (both --complete-string and you'll also need a cursor position to test completion inside a line).  That's new and not tested code.  Whereas the dotest test relies on the API it is directly testing, and trust that the basic machinery of dotest is going to continue to function.
> 
> 
> https://reviews.llvm.org/D42656
> 
> 
>