[llvm-dev] [cfe-dev] RFC: End-to-end testing

[Renato Golin via llvm-dev]

Hi David,

You answer some of your own questions down below, so I'll try to
collate the responses and shorten my reply.

On Fri, 11 Oct 2019 at 15:20, David Greene <dag at cray.com> wrote:
> How are regressions reported?  On llvm-dev?

They're buildbots, exactly like any other. Direct email, llvm-commits,
irc, bugzilla. There is no distinction, broken bots need to be fixed.

llvm-dev is not the place to report bugs.

> I'm confused.  On the one hand you say you don't want to put e2e tests
> in a dark corner, but here you speculate they could be easily removed.
> Presumably a test was added because there was some failure that other
> tests did not catch.  It's true that once a test is fixed it's very
> likely it will never break again.  Is that a reason to remove tests?

Sorry, my point is about the dynamics between number of tests, their
coverage, time to run, frequency of *unrelated* breakage, etc.

There are no set rules, but there is a back-pressure as developers and
bot owners tend to breakages.

> What do you mean by "annoy?"  Taking too long to run?

Tests that break more often are looked at more often, and if their
breakages overlap with other simpler tests, than developers will begin
to question their importance. Tests that take too long to run will be
looked into, and if they don't add much, they can be asked for
removal. That pressure is higher in the LIT side than in the
test-suite.

I'm trying to find a place where we can put the tests that will run at
the appropriate frequency and have the lowest probability of upsetting
CI and developers, so we can evolve them into what they *need* to be,
not cap it from the start and end up with something sub-par.

> Would it be possible to keep them in the monorepo but have bots that
> exercise those tests at the test-suite frequency?  I suspect that if e2e
> tests live in test-suite very few people will ever run them before
> merging to master.

Bots are pretty dumb: either they run something or they don't.

But more importantly, if we split the e2e tests in LIT, then people
won't run them before merging to master anyway.

Truth is, we don't *need* to. That's the whole point of having a fast
CI and the community really respects that.

As long as we have the tests running every few dozen commits, and bot
owner and developers work to fix them in time, we're good.

Furthermore, the test-suite already has e2e tests in there, so it is
the right place to add more. We can have more control of which tools
and libraries to use, how to check for quality, etc.

> I still think the kinds of e2e tests I'm thinking of are much closer to
> the existing LIT tests in the monorepo than things in test-suite.  I
> expect them to be quite small.

Adding tests to LIT means all fast bots will be slower. Adding them to
the test-suite means all test-suite bots will still take the about
same time.

If the tests only need to be ran once ever few dozen commits, then
having them on LIT is clearly not the right place.

> They wouldn't necessarily need to run as
> part of check-all (and indeed, I've been told that no one runs check-all
> anyway because it's too fragile).

check-all doesn't need to check everything that is in the repo, but
everything that is built.

So if you build llvm+clang, then you should *definitely* check both.
"make check" doesn't do that.

With the monorepo this may change slightly, but we still need a way to
test everything that our patches touch, including clang, rt, and
others.

I always ran check-all before every patch, FWIW.