[Lldb-commits] [PATCH] D12831: Add JUnit/XUnit-formatted output to the lldb test run system

[Todd Fiala via lldb-commits]

tfiala added a comment.

In http://reviews.llvm.org/D12831#244935, @zturner wrote:

> I can't check this until Monday.


No worries, that is fine.

  But drive by comment: Why do we need the

> MAPMODE?  Can't the JUnitFormatter just treat xpasses as whatever it wants

>  without the help of this option?


There are several CI solutions that have support for JUnit-style test results.  It is more or less the ubiquitous test result format for many commercial and open source CI tools.  JUnit output is XML-based.  The CI front-ends for builds that read the JUnit/XUnit output will display and call out the number of passes and fails.

With using defaults, I have unexpected successes being marked as failures in the JUnit report, and expected failures ignored.  These I think are valid (although opinionated) defaults.  They satisfy the property that we XFAILs are expected, don't indicate successes (so don't get counted as such), and don't generate actionable "red" in the JUnit output visualization tools (such as in Jenkins, Bamboo, TeamBuilder and other build systems that read in JUnit output).  The options allow somebody to choose to map those values a different way.  For example, I think it is reasonable to ignore unexpected successes once the truly stable passing ones have been weeded out and you're left with flaky tests that pass a high percentage of the time.  (I am a proponent of adding the actual flaky category so we can keep those running but make no hard assumptions about pass/fail rates other than tracking them to see trends).

> Also, I'm not really familiar with JUnit as a general tool.  Is there some

>  reason a one-size-fits-all JSON formatter is insufficient?


We can absolutely add a JSON formatter.

For existing CI solutions, though, JUnit is already exceptionally well supported, and having that be the format we write to as an option gives us immediate ability to report test results meaningfully in just about every CI build system on the planet.

> Last question: Is the current stdout/stderr formatter re-written as a

>  second formatter plugin, or not?


Not yet.  I see that as a future goal for us to take on, though.  And when we solve that, we'll be able to properly capture per-test-method stdout/stderr in the test event stream.  Which will again make the JUnit/XUnit consumption better, as well as any other output format we care to create/adapt.


http://reviews.llvm.org/D12831