[lldb-dev] MacOSX tests now run multi-threaded

[Todd Fiala]

For the record, these are the set of tests that currently fail for me on
MacOSX 10.9.3 as of svn r212548 on MacOSX:

Ran 297 tests.
Failing Tests (11)
FAIL: LLDB (suite) :: TestDataFormatterObjC.py (Darwin
tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: Thu
Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 i386)
FAIL: LLDB (suite) :: TestDataFormatterStdMap.py (Darwin
tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: Thu
Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 i386)
FAIL: LLDB (suite) :: TestDataFormatterStdVector.py (Darwin
tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: Thu
Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 i386)
FAIL: LLDB (suite) :: TestProcessLaunch.py (Darwin
tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: Thu
Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 i386)
FAIL: LLDB (suite) :: TestRegisterVariables.py (Darwin
tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: Thu
Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 i386)
FAIL: LLDB (suite) :: TestObjCMethods.py (Darwin
tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: Thu
Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 i386)
FAIL: LLDB (suite) :: TestHiddenIvars.py (Darwin
tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: Thu
Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 i386)
FAIL: LLDB (suite) :: TestObjCDynamicSBType.py (Darwin
tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: Thu
Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 i386)
FAIL: LLDB (suite) :: TestObjCDynamicValue.py (Darwin
tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: Thu
Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 i386)
FAIL: LLDB (suite) :: TestRealDefinition.py (Darwin
tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: Thu
Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 i386)
FAIL: LLDB (suite) :: TestTargetAPI.py (Darwin
tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: Thu
Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 i386)


Also, there were no performance regressions detected on Linux multithreaded
test run for the change in implementation of the multi-core testing.  What
I changed was implementing the worker threads via Python's
multiprocessing.Pool class instead of using the threading module's threads.
 All the MacOSX threads were locked on the Python global interpreter lock.
 Not entirely sure what the diff was that prevented that in Linux and
FreeBSD, but no matter.  The Pool implementation is slightly cleaner anyway.

-Todd


On Tue, Jul 8, 2014 at 9:46 AM, Todd Fiala <tfiala at google.com> wrote:

> Hi all,
>
> I've updated the test runner for lldb so it now runs multithreaded on
> MacOSX with similar performance gains to the Linux/FreeBSD test speedups.
>
> On my MacBookPro (mid-2012 Retina), test runs when from ~28 minutes to 7.5
> minutes.
> On my MacPro (late-2013), test runs went from ~25 minutes to 3.5 minutes.
>
> It's totally dependent on the number of cores, so YMMV, but it is faster.
>
> Also, I've been discussing ways of mitigating tests that are
> load-sensitive (which multithreaded test running exposes).  I'm looking at
> adding a new test declaration that lists a test as load sensitive
> (@load_sensitive_test), and somehow allowing that to fail in the
> multithreaded pass without failing the test run.  Then, run all the
> load-sensitive tests in a follow-up single-threaded pass if they failed
> under load.  Then, only mark them as failed if they fail under the
> single-threaded pass.  I'll have more to say on that when I get some time
> behind it.  If we do something like that, it should eliminate the tests
> that are hard to simplify and work under load, without forcing us to run
> all tests in a *much longer* single-threaded manner.
>
> --
> Todd Fiala | Software Engineer |  tfiala at google.com |  650-943-3180
>



-- 
Todd Fiala | Software Engineer | tfiala at google.com | 650-943-3180
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20140708/65edccb7/attachment.html>