[lldb-dev] intermittent tests - moving to XFAIL rather than skip

[Todd Fiala]

Done, r212762.


On Thu, Jul 10, 2014 at 1:02 PM, Todd Fiala <todd.fiala at gmail.com> wrote:

> Hey all,
>
> I had been operating under the assumption that a test marked XFAIL
> (expected failure) that passes (which gets categorized as an "unexpected
> success") would fail the test run.  I have recently learned in the last few
> days that this is not true - the unexpected success does get captured in
> the logs and does show up as a 'u' in the test run, but it doesn't cause
> the entire test process exit code to fail in the way that one or more
> failed tests would.
>
> Given that, I'm going to flip the tests that were intermittent to be XFAIL
> instead of skip.  This allows the tests to still run, allowing the code to
> run, and enabling us to catch (as an "Error") if the test actually seg
> faults or something.  All things being equal, I'd prefer to get notice this
> way over hiding the unexpected success.  (Note we have a different issue in
> the test runner where we don't always get something useful when we
> segfault, but I've already filed a bug on that).
>
> I'd be in favor of doing more to track the unexpected successes --- if we
> have tests marked as XFAIL that have since been fixed and always succeed,
> it would be good to flip them from XFAIL to normal test status that is
> expected to pass.  But we can address that later, maybe with buildbots that
> can track those over time and start generating "potentially passing XFAILS"
> reports if they pass all of the last x runs (with some largish x).
>
> I'll go ahead and flip the intermittent skipped tests that I changed
> recently to XFAIL (generally on MacOSX and Linux) to adhere to this idea.
> --
> -Todd
>



-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20140710/856c1c34/attachment.html>