[lldb-dev] test results look typical?

[Todd Fiala via lldb-dev]

Hi all,

I'm just trying to get a handle on current lldb test failures across
different platforms.

On Linux on non-virtualized hardware, I currently see the failures below on
Ubuntu 14.04.2 using a setup like this:
* stock linker (ld.bfd),
* g++ 4.9.2
* cmake
* ninja
* libstdc++

ninja check-lldb output:

Ran 394 test suites (15 failed) (3.807107%)
Ran 474 test cases (17 failed) (3.586498%)
Failing Tests (15)
FAIL: LLDB (suite) :: TestCPPThis.py (Linux rad 3.13.0-57-generic
#95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterLibccIterator.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterLibccMap.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterLibccMultiMap.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterLibcxxMultiSet.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterLibcxxSet.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterLibcxxString.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterSkipSummary.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterUnordered.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestMiGdbSetShowPrint.py (Linux rad 3.13.0-57-generic
#95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestRegisterVariables.py (Linux rad 3.13.0-57-generic
#95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestStaticVariables.py (Linux rad 3.13.0-57-generic
#95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestStepNoDebug.py (Linux rad 3.13.0-57-generic
#95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestTypedefArray.py (Linux rad 3.13.0-57-generic
#95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestVectorTypesFormatting.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)

Unexpected Successes (10)
UNEXPECTED SUCCESS: LLDB (suite) :: TestBatchMode.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestEvents.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestExitDuringStep.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestFdLeak.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestInferiorAssert.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestMiGdbSetShow.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestMiInterpreterExec.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestMiSyntax.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestRaise.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestStubSetSID.py (Linux rad
3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
ninja: build stopped: subcommand failed.



On a similar setup, although bumped up to Ubuntu 14.04.3 and now on a
VMWare VM, everything else the same, I see a similar report from 'ninja
check-lldb':

Ran 394 test suites (17 failed) (4.314721%)
Ran 474 test cases (19 failed) (4.008439%)
Failing Tests (17)
FAIL: LLDB (suite) :: TestAttachResume.py (Linux lldb 3.19.0-26-generic
#28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestCPPThis.py (Linux lldb 3.19.0-26-generic
#28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterLibccIterator.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterLibccMap.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterLibccMultiMap.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterLibcxxMultiSet.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterLibcxxSet.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterLibcxxString.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterSkipSummary.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterUnordered.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
FAIL: LLDB (suite) :: TestMiGdbSetShowPrint.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
FAIL: LLDB (suite) :: TestRegisterVariables.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
FAIL: LLDB (suite) :: TestStaticVariables.py (Linux lldb 3.19.0-26-generic
#28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestStepNoDebug.py (Linux lldb 3.19.0-26-generic
#28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestStepOverWatchpoint.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
FAIL: LLDB (suite) :: TestTypedefArray.py (Linux lldb 3.19.0-26-generic
#28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015 x86_64 x86_64)
FAIL: LLDB (suite) :: TestVectorTypesFormatting.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)

Unexpected Successes (11)
UNEXPECTED SUCCESS: LLDB (suite) :: TestBatchMode.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestEvents.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestExitDuringStep.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestFdLeak.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestInferiorAssert.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestMiGdbSetShow.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestMiInterpreterExec.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestMiSyntax.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestRaise.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestStubSetSID.py (Linux lldb
3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
UNEXPECTED SUCCESS: LLDB (suite) :: TestWatchedVarHitWhenInScope.py (Linux
lldb 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
x86_64 x86_64)
ninja: build stopped: subcommand failed.


Do these more or less match the current state of affairs for Linux tests?
If not, are there any suggestions on best practices for upgrades?  A while
back I had tried things like clang and libc++ but I had arrived at the
above setup (albeit typically with the gold linker) simply due to what
worked, what was easy to debug and what built fast.

Thanks for any confirmation on that!

-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150824/48856e25/attachment.html>