[Lldb-commits] [lldb] r238467 - Refactor test runner to print sub-test-case pass/fail rate.

[dawn at burble.org]

This commit appears to have caused 548 (=1342-794) tests to no longer run on
OSX when run using dosep.py:
    Commit 0fc10989a7da571e96a43c3d8a24dd515fecd45b by zturner
    Refactor test runner to print sub-test-case pass/fail rate.
    git-svn-id: https://llvm.org/svn/llvm-project/lldb/trunk@238467 91177308-0d34-0410-b5e6-96231b3b80d8

The lldb build/test commands from Jenkins script which run on Yosemite slave
are as follows:
    cmake -G Ninja .. "-DLLVM_TARGETS_TO_BUILD=ARM;X86;AArch64" -DCMAKE_CXX_FLAGS="-std=c++11 -stdlib=libc++" -DCMAKE_BUILD_TYPE=Release
    [...]
    ninja install
    [...]
    security unlock-keychain -p testuser /Users/testuser/Library/Keychains/login.keychain
    ./dosep.py --options "-v --executable $INSTALLDIR/bin/lldb" 2>&1 | tee $INSTALLDIR/lldb_test_out.log || true

Test output from Jenkins build after above commit
(at commit svn 238512/git 3c8acff997166aaaa6bcdad491b55fdd6c6446e8):
    Ran 378 test suites (1 failed) (0.264550%)
    Ran 794 test cases (5 failed) (0.629723%)
    FAIL: LLDB (suite) :: TestMultithreaded.py (Darwin ngnu.embarcadero.com 14.3.0 Darwin Kernel Version 14.3.0: Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 x86_64 i386)
results in totals of:
    lldb_failures=5
    lldb_errors=0
    lldb_total=794

Test output from previous build
(at commit svn 238455/git 7e22c8e9b6b4ac7225fe1740cc6fe5221c27533d):
    [...]
    Ran 1 test in 0.008s
    [...]
    Ran 1 test in 0.007s
    [...]
    Ran 1 test in 0.069s
    [...]
using the following script commands to check results:
    lldb_failures=`grep -E "^FAILED" lldb_test_out.log | grep failures | sed 's/.*(failures=\([0-9]*\).*/\1/' | awk '{count+=$1} END {print count}'` || true
    lldb_errors=`grep -E "^FAILED" lldb_test_out.log | grep errors | sed 's/.*errors=\([0-9]*\).*/\1/' | awk '{count+=$1} END {print count}'` || true
    lldb_total=`grep -E "^Ran [0-9]+ tests? in" lldb_test_out.log | awk '{count+=$2} END {print count}'`
results in totals of:
    lldb_failures=0
    lldb_errors=0
    lldb_total=1342

Trying to run the tests using dotest.py
(via ./dotest.py -v --executable $INSTALLDIR/bin/lldb) now hangs in OSX on:
    464: test_create_after_attach_with_dsym (TestCreateAfterAttach.CreateAfterAttachTestCase)
I suspect this is a regression which went unnoticed due to the new dosep
output, but it means we have to use dosep.py for running our tests until the
hang is fixed.

Can you have a look please?

Thanks,
-Dawn