[lldb-dev] RFC: Making unit tests run by default on ninja check-lldb

[Ying Chen via lldb-dev]

Yes, the output of dotest.py goes through LitTestCommand parse.
The parser is matching for "XPASS", but dotest output is using "UNEXPECTED
SUCCESS". :)

Thanks,
Ying

On Tue, Oct 20, 2015 at 6:34 PM, Todd Fiala <todd.fiala at gmail.com> wrote:

> Hi Ying,
>
> Our dotest.py lldb test results go through that lit test parser system?  I
> see XPASS happen frequently (and in fact is my whole reason for starting a
> thread on getting rid of flakey tests, or making them run enough times so
> that their output can be a useful signal rather than useless).  According
> to this script, an XPASS would be listed as failure.  I'm not seeing us
> treat XPASS as failures AFAICT.
>
> Are we just saying that our gtests get processed by that?
>
> -Todd
>
> On Tue, Oct 20, 2015 at 4:51 PM, Ying Chen via lldb-dev <
> lldb-dev at lists.llvm.org> wrote:
>
>> Hi Zachary,
>>
>> The big unknown here is how to make the buildbots understand unit test
>>> failures and trigger a failure when ninja check-lldb-unit fails.
>>>
>>
>> There're two conditions buildbot will identity a test step as failure.
>> One is that the command has non-zero return code.
>> The other is that there're failing codes in stdout message. (Refer to
>> LitTestCommand::evaluateCommand in this file
>> <https://github.com/llvm-mirror/zorg/blob/master/zorg/buildbot/commands/LitTestCommand.py>
>> .)
>> Failing codes are defined as:
>>     failingCodes = set(['FAIL', 'XPASS', 'KPASS', 'UNRESOLVED',
>> 'TIMEOUT'])
>>
>> So if the failures are print out as '^FAIL: (.*) \(.*\)', buildbot will
>> understand it's failing even if ninja check-lldb-unit returns 0.
>> Or we could add some logic to the above file to handle the output of unit
>> test.
>>
>> Thanks,
>> Ying
>>
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>>
>
>
> --
> -Todd
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151021/20603416/attachment-0001.html>