[llvm-dev] [RFC] Require PRs for XFAILing tests

[Krzysztof Parzyszek via llvm-dev]

On 10/3/2016 12:55 PM, Robinson, Paul wrote:
>
> If the XFAIL-linked PRs are content-free, that is worse than useless.
> The point is to have those PRs say something useful about the specific
> XFAIL case, which is a vast improvement over what we have today (i.e.,
> almost nothing).

They will be content-free.

Following of processes tends to degrade to meet the required minimum, 
especially when large groups of people are involved. In this proposal 
there is no way to enforce meaningful content, only that the PR exists.

To make sure that the process works, it needs a motivation that goes 
beyond being solely a requirement. If the PR indicated a problem that 
will need to be fixed, adding content would become natural. If the PR is 
there only to keep a test from failing, adding information to it is just 
extra work.

All that aside, what would be "meaningful information" in case of an 
xfailed test? Doing enough analysis to find out why it fails is often 
good enough to just fix it. A common thing for PRs is that they only 
serve as a record that a problem exists, ideally with information on how 
to reproduce it (they usually come from users, not developers). In case 
of xfailed testcases we already know that they fail and how to reproduce 
them.

-Krzysztof

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, 
hosted by The Linux Foundation