[lldb-dev] Test suite itself creating failures

[Thirumurthi, Ashok]

Note that the buildbots use dosep.py.  Stressing the tests as in dotest.py is useful as it also ensures that lldb is robust.  We have other stress tests (i.e. running builds in parallel with test loops) that we occasionally run in private jobs on the buildbots.  Just trying to strike a balance.

As long as the tests are run in the same order by dotest.py (at least by default), we'll have deterministic failures.  This is a good use-case for pdb since we can break in one of hundreds of tests.  Good tests should clean up after themselves, and that also helps the tests serve as examples of lldb usage.  

Calling these points out on lldb.org could help.  What do you think?

- Ashok

-----Original Message-----
From: lldb-dev-bounces at cs.uiuc.edu [mailto:lldb-dev-bounces at cs.uiuc.edu] On Behalf Of Richard Mitton
Sent: Tuesday, September 24, 2013 10:00 PM
To: lldb-dev at cs.uiuc.edu
Subject: [lldb-dev] Test suite itself creating failures

Hi all,

So I was looking into why TestInferiorAssert was (still) failing on my machine, and it turned out the root cause was in fact that tests are not run in isolation; dotest.py runs multiple tests using the same LLDB context for each one. So if a test doesn't clean up after itself properly, it can cause following tests to incorrectly fail.

Is this really a good idea? Wouldn't it make more sense to make it so tests are always run individually, to guarantee consistent results?

--
Richard Mitton
richard at codersnotes.com

_______________________________________________
lldb-dev mailing list
lldb-dev at cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev