[lldb-dev] MacOSX tests now run multi-threaded

Wed Jul 9 15:16:36 PDT 2014

As of r212660, lldb tests run cleanly on MacOSX 10.9.4 for x86_64 (i.e. -A
x86_64).  I've marked as XFAIL all tests I found that consistently fail,
and marked the two tests I found intermittent there as skip on MacOSX.  I
filed bugs for all tests that were marked XFAIL or skip.

Tests also run reasonably fast on MacOSX.  So - if you are on OSX, please
run the tests, and if any of them fail, you can be reasonably assured that
there is a decent chance that the pending change may be introducing it.

I'm about to make one more pass on the Linux side for the same goal: so
that test failures mean new breakages.  (Note we still need to go back and
address all the bugs filed against XFAIL/skipped tests, but at least new
breakages can be detected much more readily --- any test failure should
raise suspicion).

Happy days!

-Todd

On Tue, Jul 8, 2014 at 1:50 PM, Todd Fiala <tfiala at google.com> wrote:

> FWIW I got the same set of 11 errors on MacOSX 10.10 (14A283o) - Yosemite
> Preview Update 3 - and Xcode Version 6.0 (6A254o) - Xcode 6 Beta 3.
>
> The first set was from OS X 10.9.3 with Xcode 6 Beta 2.
>
>
> On Tue, Jul 8, 2014 at 10:42 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> Here's how I run the tests:
>>
>> Make sure your Xcode project "locations" setting is set to "relative", so
>> that the DerivedData is stored relative to your lldb source directory.
>>  (Above, that will be /my/workspace-root/lldb/DerivedData).
>>
>> I have the canonical MacOSX Xcode source layout:
>>
>> /my/workspace-root/
>>   |- lldb/
>>        |- llvm/
>>             |- tools/clang/
>>
>> $ cd /my/workspace-root
>> $ mkdir test-run
>> $ cd test-run
>> # {save attached do-tests.sh file here}
>> $ chmod a+x do-tests.sh
>>
>> # Now run it - this will run the tests using one parallel test per core.
>>  Tests are chunked up in units of directories, so all tests in a given
>> directory are serialized.
>>
>> $ ./do-tests.sh
>>
>> The script currently only runs x86_64 - just adjust the -A value if you
>> want to run more/different architectures in the tests.  It also assumes
>> that it will put test output into the test-run/lldb-test-results directory.
>>
>> The behavior is no different from the previous multithreaded behavior on
>> Linux/FreeBSD test runs.
>>
>> Let me know if you run into any issues with that.
>>
>>
>> On Tue, Jul 8, 2014 at 9:50 AM, Todd Fiala <tfiala at google.com> wrote:
>>
>>> For the record, these are the set of tests that currently fail for me on
>>> MacOSX 10.9.3 as of svn r212548 on MacOSX:
>>>
>>> Ran 297 tests.
>>> Failing Tests (11)
>>> FAIL: LLDB (suite) :: TestDataFormatterObjC.py (Darwin
>>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>>> i386)
>>> FAIL: LLDB (suite) :: TestDataFormatterStdMap.py (Darwin
>>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>>> i386)
>>> FAIL: LLDB (suite) :: TestDataFormatterStdVector.py (Darwin
>>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>>> i386)
>>> FAIL: LLDB (suite) :: TestProcessLaunch.py (Darwin
>>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>>> i386)
>>> FAIL: LLDB (suite) :: TestRegisterVariables.py (Darwin
>>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>>> i386)
>>> FAIL: LLDB (suite) :: TestObjCMethods.py (Darwin
>>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>>> i386)
>>> FAIL: LLDB (suite) :: TestHiddenIvars.py (Darwin
>>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>>> i386)
>>> FAIL: LLDB (suite) :: TestObjCDynamicSBType.py (Darwin
>>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>>> i386)
>>> FAIL: LLDB (suite) :: TestObjCDynamicValue.py (Darwin
>>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>>> i386)
>>> FAIL: LLDB (suite) :: TestRealDefinition.py (Darwin
>>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>>> i386)
>>> FAIL: LLDB (suite) :: TestTargetAPI.py (Darwin
>>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0:
>>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64
>>> i386)
>>>
>>>
>>> Also, there were no performance regressions detected on Linux
>>> multithreaded test run for the change in implementation of the multi-core
>>> testing.  What I changed was implementing the worker threads via Python's
>>> multiprocessing.Pool class instead of using the threading module's threads.
>>>  All the MacOSX threads were locked on the Python global interpreter lock.
>>>  Not entirely sure what the diff was that prevented that in Linux and
>>> FreeBSD, but no matter.  The Pool implementation is slightly cleaner anyway.
>>>
>>> -Todd
>>>
>>>
>>> On Tue, Jul 8, 2014 at 9:46 AM, Todd Fiala <tfiala at google.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I've updated the test runner for lldb so it now runs multithreaded on
>>>> MacOSX with similar performance gains to the Linux/FreeBSD test speedups.
>>>>
>>>> On my MacBookPro (mid-2012 Retina), test runs when from ~28 minutes to
>>>> 7.5 minutes.
>>>> On my MacPro (late-2013), test runs went from ~25 minutes to 3.5
>>>> minutes.
>>>>
>>>> It's totally dependent on the number of cores, so YMMV, but it is
>>>> faster.
>>>>
>>>> Also, I've been discussing ways of mitigating tests that are
>>>> load-sensitive (which multithreaded test running exposes).  I'm looking at
>>>> adding a new test declaration that lists a test as load sensitive
>>>> (@load_sensitive_test), and somehow allowing that to fail in the
>>>> multithreaded pass without failing the test run.  Then, run all the
>>>> load-sensitive tests in a follow-up single-threaded pass if they failed
>>>> under load.  Then, only mark them as failed if they fail under the
>>>> single-threaded pass.  I'll have more to say on that when I get some time
>>>> behind it.  If we do something like that, it should eliminate the tests
>>>> that are hard to simplify and work under load, without forcing us to run
>>>> all tests in a *much longer* single-threaded manner.
>>>>
>>>> --
>>>> Todd Fiala | Software Engineer |  tfiala at google.com |  650-943-3180
>>>>
>>>
>>>
>>>
>>> --
>>> Todd Fiala | Software Engineer |  tfiala at google.com |  650-943-3180
>>>
>>> _______________________________________________
>>> lldb-dev mailing list
>>> lldb-dev at cs.uiuc.edu
>>> http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
>>>
>>>
>>
>>
>> --
>> -Todd
>>
>
>
>
> --
> Todd Fiala | Software Engineer |  tfiala at google.com |  650-943-3180
>

-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20140709/a8429ce2/attachment.html>