[llvm-dev] buildbot failure in LLVM on clang-native-arm-cortex-a9

[Renato Golin via llvm-dev]

On 26 August 2015 at 17:01, Mehdi Amini <mehdi.amini at apple.com> wrote:
> The situation does not seem that black and white to me here. In the end, it seems to me that is is about a threshold: if a bot is crashing 90% of the time, does it really contributes to increase the quality of testing or on the opposite it is just adding noise?

That question is not alone, and per se, it's meaningless. Your next
question, however, is the key.


> Another way of considering in general the impact of a bot on the quality is: “how many legit failures were found by this bot in the last x years that weren’t covered by another bot”.

In this criteria, which was my point, those bots haven't added much
after I added some faster ones. So, if we want to shut them down,
let's do so because they don't add value, not because they are
unstable.

However, that is the *only* bot running on an A9. As an example, this
year, I spent 2 whole weeks during the release to bisect and fix an
issue because I disabled, for two months, one bot that I thought was
already covered by another.

That headache was real. I've wasted two whole weeks, maybe more or my
time. I've wasted time from other people waiting to do the release
validation. I have delayed the release and all that it entangles. All
because I thought that bot was noisy. It's ratio was about 20 passes
to 1 failure.

The A9 bots are more than that, so on my monitor[1], I currently
ignore their results. I still keep them there to see what's going on,
and when my bots fail, I look at that, too, to see if the problem is
the same. Sometimes, they do provide useful insight on the other bot's
breakages.

So, for me, disabling the A9 bots would be a loss. But as I said
before, that's up to Galina, as she's the bot owner. If she's ok with
finally putting them to rest, I'll respect the community's decision
and remove it from my monitor. But we can't transform this in to a
which hunt. It's not about thresholds, it's about cost and value,
which may be different for you than it is for me. We have to consider
the whole community, not just our opinions.

For every broken bot that someone want to get rid of, I propose
consult the bot owner first, and then a vote on llvm-dev@ / cfe-dev@
if the owner is not in agreement. After all, you can always have an
internal buildmaster (like I have) for unstable bots.

cheers,
--renato
[1] http://llvm.tcwglab.linaro.org/monitor/