[lldb-dev] BasicResultsFormatter - new test results summary

[Pavel Labath via lldb-dev]

On 14 December 2015 at 16:19, Todd Fiala <todd.fiala at gmail.com> wrote:
>> We would lose the ability to individually expect "failures" and
>> "timeouts", but I don't think that is really necessary, and I think it
>> will be worth the extra maintainability we get from the fact of having
>> fewer test decorators.
>>
>
> OTOH, the piece we then lose is the ability to have an XFAIL mean "Hey this
> test really should fail, we haven't implemented feature XYZ (correctly or
> otherwise), so this better fail."  In that semantic meaning, an unexpected
> success would truly be an actionable signal --- either the test is now
> passing because the feature now works (actionable signal option A: the XFAIL
> should come off after verifying), or or the test is passing because it is
> not testing what it thought it was, and the test needs to be modified to
> more tightly bound the expected fail condition (actionable item option B).
>
> So it eliminates the definiteness of an XFAIL ideally meaning "this really
> should fail," turning it into "it is permissible for this to fail."
>
> All that said, our Python test suite is so far away from that ideal right
> now.  The highest level output of our test suite that I care about is "if
> tests run green, this is a good build", and if "tests run red, this is a bad
> build."  I don't see the timeout being rolled into XFAIL as hurting that.
> It seems reasonable to roll them together at this time.  And the test output
> will list and count the timeouts.

I'd say that the root cause here is something different, namely the
fact that our tests do not behave deterministically. If they were
always ending with the same result, then all you said is above would
be true, regardless of whether that result was "failure" or "timeout"
- having a test consistently failing would give the same kind of
signal as a test consistently timing out (although I hope we never
have the latter kind). I am really only interested in hanging tests
here, using this to handle tests that were just slightly too slow is
quite a bad idea. In fact, I think using a uniform decorator would
discourage this, as then you will not have the option of saying "I
want this test to succeed, but if it takes too long, then don't worry
about that" (which is actually what we do right now, and we needed to
do that as we had a lot of tests hanging in the past, but I think
that's gotten better now).

>
> So I'd be okay with that at this time in the sake of simplifying markup for
> tests.

Ok, I'll get on it then.

pl