[lldb-dev] Heads up - test runner fix

[Todd Fiala]

For example, here is the set of test failures I'm getting on MacOSX 10.10
GM Candidate 3, with the latest Xcode 6.1 beta:

Ran 313 tests.

Failing Tests (6)

FAIL: LLDB (suite) :: TestFormatters.py (Darwin snappy-2.local 14.0.0
Darwin Kernel Version 14.0.0: Tue Sep 23 23:01:41 PDT 2014;
root:xnu-2782.1.97~6/RELEASE_X86_64 x86_64 i386)

FAIL: LLDB (suite) :: TestCallWithTimeout.py (Darwin snappy-2.local 14.0.0
Darwin Kernel Version 14.0.0: Tue Sep 23 23:01:41 PDT 2014;
root:xnu-2782.1.97~6/RELEASE_X86_64 x86_64 i386)

FAIL: LLDB (suite) :: TestMemoryHistory.py (Darwin snappy-2.local 14.0.0
Darwin Kernel Version 14.0.0: Tue Sep 23 23:01:41 PDT 2014;
root:xnu-2782.1.97~6/RELEASE_X86_64 x86_64 i386)

FAIL: LLDB (suite) :: TestReportData.py (Darwin snappy-2.local 14.0.0
Darwin Kernel Version 14.0.0: Tue Sep 23 23:01:41 PDT 2014;
root:xnu-2782.1.97~6/RELEASE_X86_64 x86_64 i386)

FAIL: LLDB (suite) :: TestDataFormatterStdMap.py (Darwin snappy-2.local
14.0.0 Darwin Kernel Version 14.0.0: Tue Sep 23 23:01:41 PDT 2014;
root:xnu-2782.1.97~6/RELEASE_X86_64 x86_64 i386)

FAIL: LLDB (suite) :: TestDataFormatterStdVector.py (Darwin snappy-2.local
14.0.0 Darwin Kernel Version 14.0.0: Tue Sep 23 23:01:41 PDT 2014;
root:xnu-2782.1.97~6/RELEASE_X86_64 x86_64 i386)

On Tue, Oct 14, 2014 at 9:33 AM, Todd Fiala <todd.fiala at gmail.com> wrote:

> > This is the accumulation of all build breaks
>
> test breaks, rather...
>
> On Tue, Oct 14, 2014 at 9:32 AM, Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> Hey all,
>>
>> I have good news and bad news.
>>
>> Good: our test runners now report failures correctly.
>> Bad news: our test runners have not been reporting failures since Aug 8
>> 2014.
>>
>> As of svn r219689, dotest.py will once again report errors/failures via
>> non-zero return value, which will then propagate correctly through test run
>> results as reported by 'make test', 'ninja check-lldb', etc.
>>
>> You will almost certainly see test failures show up after r219689 that
>> you were not seeing before.  This is the accumulation of all build breaks
>> we essentially made starting with r215256.  We will need to fix the errors,
>> not revert r219689.  (That would be akin to shooting the messenger).
>>
>> The lesson here is make sure we don't break our test runner, and ensure
>> return values from the test runner are preserved on exit.  That is a hugely
>> critical facility.  I will look into adding a test runner test so we don't
>> break that again.
>> --
>> -Todd
>>
>
>
>
> --
> -Todd
>



-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20141014/f5f5cd74/attachment.html>