[lldb-dev] MacOSX tests now run multi-threaded

Tue Jul 8 10:42:54 PDT 2014

Here's how I run the tests:

Make sure your Xcode project "locations" setting is set to "relative", so
that the DerivedData is stored relative to your lldb source directory.
 (Above, that will be /my/workspace-root/lldb/DerivedData).

I have the canonical MacOSX Xcode source layout:

/my/workspace-root/
  |- lldb/
       |- llvm/
            |- tools/clang/

$ cd /my/workspace-root
$ mkdir test-run
$ cd test-run
# {save attached do-tests.sh file here}
$ chmod a+x do-tests.sh

# Now run it - this will run the tests using one parallel test per core.
 Tests are chunked up in units of directories, so all tests in a given
directory are serialized.

$ ./do-tests.sh

The script currently only runs x86_64 - just adjust the -A value if you
want to run more/different architectures in the tests.  It also assumes
that it will put test output into the test-run/lldb-test-results directory.

The behavior is no different from the previous multithreaded behavior on
Linux/FreeBSD test runs.

Let me know if you run into any issues with that.

On Tue, Jul 8, 2014 at 9:50 AM, Todd Fiala <tfiala at google.com> wrote:

> For the record, these are the set of tests that currently fail for me on
> MacOSX 10.9.3 as of svn r212548 on MacOSX:
>
> Ran 297 tests.
> Failing Tests (11)
> FAIL: LLDB (suite) :: TestDataFormatterObjC.py (Darwin
> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
> i386)
> FAIL: LLDB (suite) :: TestDataFormatterStdMap.py (Darwin
> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
> i386)
> FAIL: LLDB (suite) :: TestDataFormatterStdVector.py (Darwin
> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
> i386)
> FAIL: LLDB (suite) :: TestProcessLaunch.py (Darwin
> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
> i386)
> FAIL: LLDB (suite) :: TestRegisterVariables.py (Darwin
> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
> i386)
> FAIL: LLDB (suite) :: TestObjCMethods.py (Darwin
> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
> i386)
> FAIL: LLDB (suite) :: TestHiddenIvars.py (Darwin
> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
> i386)
> FAIL: LLDB (suite) :: TestObjCDynamicSBType.py (Darwin
> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
> i386)
> FAIL: LLDB (suite) :: TestObjCDynamicValue.py (Darwin
> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
> i386)
> FAIL: LLDB (suite) :: TestRealDefinition.py (Darwin
> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
> i386)
> FAIL: LLDB (suite) :: TestTargetAPI.py (Darwin
> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
> i386)
>
>
> Also, there were no performance regressions detected on Linux
> multithreaded test run for the change in implementation of the multi-core
> testing.  What I changed was implementing the worker threads via Python's
> multiprocessing.Pool class instead of using the threading module's threads.
>  All the MacOSX threads were locked on the Python global interpreter lock.
>  Not entirely sure what the diff was that prevented that in Linux and
> FreeBSD, but no matter.  The Pool implementation is slightly cleaner anyway.
>
> -Todd
>
>
> On Tue, Jul 8, 2014 at 9:46 AM, Todd Fiala <tfiala at google.com> wrote:
>
>> Hi all,
>>
>> I've updated the test runner for lldb so it now runs multithreaded on
>> MacOSX with similar performance gains to the Linux/FreeBSD test speedups.
>>
>> On my MacBookPro (mid-2012 Retina), test runs when from ~28 minutes to
>> 7.5 minutes.
>> On my MacPro (late-2013), test runs went from ~25 minutes to 3.5 minutes.
>>
>> It's totally dependent on the number of cores, so YMMV, but it is faster.
>>
>> Also, I've been discussing ways of mitigating tests that are
>> load-sensitive (which multithreaded test running exposes).  I'm looking at
>> adding a new test declaration that lists a test as load sensitive
>> (@load_sensitive_test), and somehow allowing that to fail in the
>> multithreaded pass without failing the test run.  Then, run all the
>> load-sensitive tests in a follow-up single-threaded pass if they failed
>> under load.  Then, only mark them as failed if they fail under the
>> single-threaded pass.  I'll have more to say on that when I get some time
>> behind it.  If we do something like that, it should eliminate the tests
>> that are hard to simplify and work under load, without forcing us to run
>> all tests in a *much longer* single-threaded manner.
>>
>> --
>> Todd Fiala | Software Engineer |  tfiala at google.com |  650-943-3180
>>
>
>
>
> --
> Todd Fiala | Software Engineer |  tfiala at google.com |  650-943-3180
>
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
>
>

-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20140708/4953fcc5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: do-tests.sh
Type: application/x-sh
Size: 389 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20140708/4953fcc5/attachment.sh>