[lldb-dev] test results look typical?

Todd Fiala via lldb-dev lldb-dev at lists.llvm.org
Mon Aug 24 16:14:43 PDT 2015 

On Mon, Aug 24, 2015 at 4:03 PM, Greg Clayton <gclayton at apple.com> wrote:

> We should have a decorator like:
>
> @skipLinuxUnlessInstalled("/usr/lib/libc++.so")
>
> or something that tells us to install this library and fails the test
> suite before you run anything.
>
>
Yeah, I like that idea, Greg.  We shouldn't report failures on scenarios
that we know cannot work due to missing dependencies.

I'll give them a run with libc++ to see how that goes.


> > On Aug 24, 2015, at 4:01 PM, Chaoren Lin via lldb-dev <
> lldb-dev at lists.llvm.org> wrote:
> >
> > The TestDataFormatterLibcc* tests require libc++-dev:
> >
> > $ sudo apt-get install libc++-dev
> >
> > On Mon, Aug 24, 2015 at 3:42 PM, Todd Fiala via lldb-dev <
> lldb-dev at lists.llvm.org> wrote:
> >
> > On Mon, Aug 24, 2015 at 3:39 PM, Zachary Turner <zturner at google.com>
> wrote:
> > Can't comment on the failures for Linux, but I don't think we have a
> good handle on the unexpected successes.  I only added that information to
> the output about a week ago, before that unexpected successes were actually
> going unnoticed.
> >
> > Okay, thanks Zachary.   A while back we had some flapping tests that
> would oscillate between unexpected success and failure on Linux.  Some of
> those might still be in that state but maybe (!) are fixed.
> >
> > Anyone on the Linux end who happens to know if the fails in particular
> look normal, that'd be good to know.
> >
> > Thanks!
> >
> >
> > It's likely that someone could just go in there and remove the XFAIL
> from those tests.
> >
> > On Mon, Aug 24, 2015 at 3:37 PM Todd Fiala via lldb-dev <
> lldb-dev at lists.llvm.org> wrote:
> > Hi all,
> >
> > I'm just trying to get a handle on current lldb test failures across
> different platforms.
> >
> > On Linux on non-virtualized hardware, I currently see the failures below
> on Ubuntu 14.04.2 using a setup like this:
> > * stock linker (ld.bfd),
> > * g++ 4.9.2
> > * cmake
> > * ninja
> > * libstdc++
> >
> > ninja check-lldb output:
> >
> > Ran 394 test suites (15 failed) (3.807107%)
> > Ran 474 test cases (17 failed) (3.586498%)
> > Failing Tests (15)
> > FAIL: LLDB (suite) :: TestCPPThis.py (Linux rad 3.13.0-57-generic
> #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestDataFormatterLibccIterator.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestDataFormatterLibccMap.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestDataFormatterLibccMultiMap.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestDataFormatterLibcxxMultiSet.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestDataFormatterLibcxxSet.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestDataFormatterLibcxxString.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestDataFormatterSkipSummary.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestDataFormatterUnordered.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestMiGdbSetShowPrint.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestRegisterVariables.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestStaticVariables.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestStepNoDebug.py (Linux rad 3.13.0-57-generic
> #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestTypedefArray.py (Linux rad 3.13.0-57-generic
> #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestVectorTypesFormatting.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> >
> > Unexpected Successes (10)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestBatchMode.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestEvents.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestExitDuringStep.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestFdLeak.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestInferiorAssert.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestMiGdbSetShow.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestMiInterpreterExec.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestMiSyntax.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestRaise.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestStubSetSID.py (Linux rad
> 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64)
> > ninja: build stopped: subcommand failed.
> >
> >
> >
> > On a similar setup, although bumped up to Ubuntu 14.04.3 and now on a
> VMWare VM, everything else the same, I see a similar report from 'ninja
> check-lldb':
> >
> > Ran 394 test suites (17 failed) (4.314721%)
> > Ran 474 test cases (19 failed) (4.008439%)
> > Failing Tests (17)
> > FAIL: LLDB (suite) :: TestAttachResume.py (Linux lldb 3.19.0-26-generic
> #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestCPPThis.py (Linux lldb 3.19.0-26-generic
> #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestDataFormatterLibccIterator.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestDataFormatterLibccMap.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestDataFormatterLibccMultiMap.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestDataFormatterLibcxxMultiSet.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestDataFormatterLibcxxSet.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestDataFormatterLibcxxString.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestDataFormatterSkipSummary.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestDataFormatterUnordered.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestMiGdbSetShowPrint.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestRegisterVariables.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestStaticVariables.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestStepNoDebug.py (Linux lldb 3.19.0-26-generic
> #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestStepOverWatchpoint.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestTypedefArray.py (Linux lldb 3.19.0-26-generic
> #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015 x86_64 x86_64)
> > FAIL: LLDB (suite) :: TestVectorTypesFormatting.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> >
> > Unexpected Successes (11)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestBatchMode.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestEvents.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestExitDuringStep.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestFdLeak.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestInferiorAssert.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestMiGdbSetShow.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestMiInterpreterExec.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestMiSyntax.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestRaise.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestStubSetSID.py (Linux lldb
> 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17 UTC 2015
> x86_64 x86_64)
> > UNEXPECTED SUCCESS: LLDB (suite) :: TestWatchedVarHitWhenInScope.py
> (Linux lldb 3.19.0-26-generic #28~14.04.1-Ubuntu SMP Wed Aug 12 14:09:17
> UTC 2015 x86_64 x86_64)
> > ninja: build stopped: subcommand failed.
> >
> >
> > Do these more or less match the current state of affairs for Linux
> tests?  If not, are there any suggestions on best practices for upgrades?
> A while back I had tried things like clang and libc++ but I had arrived at
> the above setup (albeit typically with the gold linker) simply due to what
> worked, what was easy to debug and what built fast.
> >
> > Thanks for any confirmation on that!
> >
> > --
> > -Todd
> > _______________________________________________
> > lldb-dev mailing list
> > lldb-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> >
> >
> >
> > --
> > -Todd
> >
> > _______________________________________________
> > lldb-dev mailing list
> > lldb-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> >
> >
> > _______________________________________________
> > lldb-dev mailing list
> > lldb-dev at lists.llvm.org
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.llvm.org_cgi-2Dbin_mailman_listinfo_lldb-2Ddev&d=BQIGaQ&c=eEvniauFctOgLOKGJOplqw&r=e494ZqshU04UaasD49FjA8X8-41XrPe2DEAUCC4uIBA&m=cTc4WjYxftac8AXg_mviAV8dhnyfI2hSALiTt1pv0IY&s=DmPx7m6miGyvTf_HhNkbLk5SLKVyLcv7n06Lm585yww&e=
>
>


-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150824/cb9f4422/attachment.html>

More information about the lldb-dev mailing list