[Lldb-commits] [PATCH] D42656: [testsuite] Remove flakey test relying on `pexpect`

Mon Jan 29 15:08:16 PST 2018

davide added a comment.

In https://reviews.llvm.org/D42656#991239, @davide wrote:

> In https://reviews.llvm.org/D42656#991065, @jingham wrote:
>
> > There are a whole bunch of other tests that test completion in this file that use the exact same mechanism but don't seem to be flakey.  Why is this one test flakey?
>
>
> So, I take a look at this to reply to your question. They're all flakey.
>  Most of them fails non-deterministically depending on the load of the system. I just wrote a driver that spawns randomly threads that spin loops & checked out multiple copies of lldb and ran test suites in parallel.
>  I see many of the failures testing completion randomly failing.

As an example of a test which just failed on my machine because of a timeout (non-deterministic, happens around ~1/50 times on my machine)

  TIMEOUT: test_step_over_3_times_dsym (python_api/thread/TestThreadAPI.py)

  [TestThreadAPI.py FAILED] (TIMEOUT)
  Command invoked: /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python /Users/dcci/work/llvm/llvm/tools/lldb/test/dotest.py -q --arch=x86_64 --executable /Users/dcci/work/llvm/build/bin/lldb -s /Users/dcci/work/llvm/build/lldb-test-traces -S nm -u CXXFLAGS -u CFLAGS -C /Users/dcci/work/llvm/build/bin/clang --codesign-identity lldb_codesign --server /Users/dcci/work/llvm/build/bin/debugserver --results-port 52049 -S nm --inferior -p TestThreadAPI.py /Users/dcci/work/llvm/llvm/tools/lldb/packages/Python/lldbsuite/test --event-add-entries worker_index=6:int

I'll probably go ahead and either unXFAIL or SKIP the tests that I find being non-reliable cause of `pexpect`. Having tests that pass most of the time under load but not always is quite unfortunate, in particular now that we want to enable continuous integration for downstream projects using lldb.

Best,

-

Davide

https://reviews.llvm.org/D42656