[lldb-dev] MacOSX tests now run multi-threaded

Tue Jul 8 13:50:06 PDT 2014

FWIW I got the same set of 11 errors on MacOSX 10.10 (14A283o) - Yosemite
Preview Update 3 - and Xcode Version 6.0 (6A254o) - Xcode 6 Beta 3.

The first set was from OS X 10.9.3 with Xcode 6 Beta 2.

On Tue, Jul 8, 2014 at 10:42 AM, Todd Fiala <todd.fiala at gmail.com> wrote:

> Here's how I run the tests:
>
> Make sure your Xcode project "locations" setting is set to "relative", so
> that the DerivedData is stored relative to your lldb source directory.
>  (Above, that will be /my/workspace-root/lldb/DerivedData).
>
> I have the canonical MacOSX Xcode source layout:
>
> /my/workspace-root/
>   |- lldb/
>        |- llvm/
>             |- tools/clang/
>
> $ cd /my/workspace-root
> $ mkdir test-run
> $ cd test-run
> # {save attached do-tests.sh file here}
> $ chmod a+x do-tests.sh
>
> # Now run it - this will run the tests using one parallel test per core.
>  Tests are chunked up in units of directories, so all tests in a given
> directory are serialized.
>
> $ ./do-tests.sh
>
> The script currently only runs x86_64 - just adjust the -A value if you
> want to run more/different architectures in the tests.  It also assumes
> that it will put test output into the test-run/lldb-test-results directory.
>
> The behavior is no different from the previous multithreaded behavior on
> Linux/FreeBSD test runs.
>
> Let me know if you run into any issues with that.
>
>
> On Tue, Jul 8, 2014 at 9:50 AM, Todd Fiala <tfiala at google.com> wrote:
>
>> For the record, these are the set of tests that currently fail for me on
>> MacOSX 10.9.3 as of svn r212548 on MacOSX:
>>
>> Ran 297 tests.
>> Failing Tests (11)
>> FAIL: LLDB (suite) :: TestDataFormatterObjC.py (Darwin
>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>> i386)
>> FAIL: LLDB (suite) :: TestDataFormatterStdMap.py (Darwin
>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>> i386)
>> FAIL: LLDB (suite) :: TestDataFormatterStdVector.py (Darwin
>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>> i386)
>> FAIL: LLDB (suite) :: TestProcessLaunch.py (Darwin
>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>> i386)
>> FAIL: LLDB (suite) :: TestRegisterVariables.py (Darwin
>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>> i386)
>> FAIL: LLDB (suite) :: TestObjCMethods.py (Darwin
>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>> i386)
>> FAIL: LLDB (suite) :: TestHiddenIvars.py (Darwin
>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>> i386)
>> FAIL: LLDB (suite) :: TestObjCDynamicSBType.py (Darwin
>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>> i386)
>> FAIL: LLDB (suite) :: TestObjCDynamicValue.py (Darwin
>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>> i386)
>> FAIL: LLDB (suite) :: TestRealDefinition.py (Darwin
>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>> i386)
>> FAIL: LLDB (suite) :: TestTargetAPI.py (Darwin
>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>> i386)
>>
>>
>> Also, there were no performance regressions detected on Linux
>> multithreaded test run for the change in implementation of the multi-core
>> testing.  What I changed was implementing the worker threads via Python's
>> multiprocessing.Pool class instead of using the threading module's threads.
>>  All the MacOSX threads were locked on the Python global interpreter lock.
>>  Not entirely sure what the diff was that prevented that in Linux and
>> FreeBSD, but no matter.  The Pool implementation is slightly cleaner anyway.
>>
>> -Todd
>>
>>
>> On Tue, Jul 8, 2014 at 9:46 AM, Todd Fiala <tfiala at google.com> wrote:
>>
>>> Hi all,
>>>
>>> I've updated the test runner for lldb so it now runs multithreaded on
>>> MacOSX with similar performance gains to the Linux/FreeBSD test speedups.
>>>
>>> On my MacBookPro (mid-2012 Retina), test runs when from ~28 minutes to
>>> 7.5 minutes.
>>> On my MacPro (late-2013), test runs went from ~25 minutes to 3.5 minutes.
>>>
>>> It's totally dependent on the number of cores, so YMMV, but it is faster.
>>>
>>> Also, I've been discussing ways of mitigating tests that are
>>> load-sensitive (which multithreaded test running exposes).  I'm looking at
>>> adding a new test declaration that lists a test as load sensitive
>>> (@load_sensitive_test), and somehow allowing that to fail in the
>>> multithreaded pass without failing the test run.  Then, run all the
>>> load-sensitive tests in a follow-up single-threaded pass if they failed
>>> under load.  Then, only mark them as failed if they fail under the
>>> single-threaded pass.  I'll have more to say on that when I get some time
>>> behind it.  If we do something like that, it should eliminate the tests
>>> that are hard to simplify and work under load, without forcing us to run
>>> all tests in a *much longer* single-threaded manner.
>>>
>>> --
>>> Todd Fiala | Software Engineer |  tfiala at google.com |  650-943-3180
>>>
>>
>>
>>
>> --
>> Todd Fiala | Software Engineer |  tfiala at google.com |  650-943-3180
>>
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
>>
>>
>
>
> --
> -Todd
>

-- 
Todd Fiala | Software Engineer | tfiala at google.com | 650-943-3180
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20140708/e7a424f0/attachment.html>