[lldb-dev] RFC: Making unit tests run by default on ninja check-lldb

[Todd Fiala via lldb-dev]

Oh haha okay.  :-)

Thanks for explaining, Ying!

-Todd

On Wed, Oct 21, 2015 at 10:01 AM, Ying Chen <chying at google.com> wrote:

> Yes, the output of dotest.py goes through LitTestCommand parse.
> The parser is matching for "XPASS", but dotest output is using "UNEXPECTED
> SUCCESS". :)
>
> Thanks,
> Ying
>
> On Tue, Oct 20, 2015 at 6:34 PM, Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> Hi Ying,
>>
>> Our dotest.py lldb test results go through that lit test parser system?
>> I see XPASS happen frequently (and in fact is my whole reason for starting
>> a thread on getting rid of flakey tests, or making them run enough times so
>> that their output can be a useful signal rather than useless).  According
>> to this script, an XPASS would be listed as failure.  I'm not seeing us
>> treat XPASS as failures AFAICT.
>>
>> Are we just saying that our gtests get processed by that?
>>
>> -Todd
>>
>> On Tue, Oct 20, 2015 at 4:51 PM, Ying Chen via lldb-dev <
>> lldb-dev at lists.llvm.org> wrote:
>>
>>> Hi Zachary,
>>>
>>> The big unknown here is how to make the buildbots understand unit test
>>>> failures and trigger a failure when ninja check-lldb-unit fails.
>>>>
>>>
>>> There're two conditions buildbot will identity a test step as failure.
>>> One is that the command has non-zero return code.
>>> The other is that there're failing codes in stdout message. (Refer to
>>> LitTestCommand::evaluateCommand in this file
>>> <https://github.com/llvm-mirror/zorg/blob/master/zorg/buildbot/commands/LitTestCommand.py>
>>> .)
>>> Failing codes are defined as:
>>>     failingCodes = set(['FAIL', 'XPASS', 'KPASS', 'UNRESOLVED',
>>> 'TIMEOUT'])
>>>
>>> So if the failures are print out as '^FAIL: (.*) \(.*\)', buildbot will
>>> understand it's failing even if ninja check-lldb-unit returns 0.
>>> Or we could add some logic to the above file to handle the output of
>>> unit test.
>>>
>>> Thanks,
>>> Ying
>>>
>>> _______________________________________________
>>> lldb-dev mailing list
>>> lldb-dev at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>
>>>
>>
>>
>> --
>> -Todd
>>
>
>


-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151021/2d54bb02/attachment-0001.html>