[lldb-dev] intermittent tests - moving to XFAIL rather than skip

[Todd Fiala]

Hey all,

I had been operating under the assumption that a test marked XFAIL
(expected failure) that passes (which gets categorized as an "unexpected
success") would fail the test run.  I have recently learned in the last few
days that this is not true - the unexpected success does get captured in
the logs and does show up as a 'u' in the test run, but it doesn't cause
the entire test process exit code to fail in the way that one or more
failed tests would.

Given that, I'm going to flip the tests that were intermittent to be XFAIL
instead of skip.  This allows the tests to still run, allowing the code to
run, and enabling us to catch (as an "Error") if the test actually seg
faults or something.  All things being equal, I'd prefer to get notice this
way over hiding the unexpected success.  (Note we have a different issue in
the test runner where we don't always get something useful when we
segfault, but I've already filed a bug on that).

I'd be in favor of doing more to track the unexpected successes --- if we
have tests marked as XFAIL that have since been fixed and always succeed,
it would be good to flip them from XFAIL to normal test status that is
expected to pass.  But we can address that later, maybe with buildbots that
can track those over time and start generating "potentially passing XFAILS"
reports if they pass all of the last x runs (with some largish x).

I'll go ahead and flip the intermittent skipped tests that I changed
recently to XFAIL (generally on MacOSX and Linux) to adhere to this idea.
-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20140710/ae6ddef2/attachment.html>