[llvm-dev] [Openmp-dev] [cfe-dev] RFC: End-to-end testing

[Renato Golin via llvm-dev]

On Fri, 18 Oct 2019 at 16:30, David Greene <greened at obbligato.org> wrote:
> I have been viewing test-suite as a kind of second-level/backup testing
> that catches things not flagged by "make check-all."  Is that a
> reasonable interpretation?  I was hoping to get some end-to-end tests
> under "make check-all" because that's easier for developers to run in
> their workflows.

It is a common understanding, which makes the test-suite less useful,
but that's not really relevant.

No one needs to the test-suite as part of their development processes,
because we have bots for that.

If you have decent piece-wise tests in Clang/LLVM, you really don't
need end-to-end tests in Clang/LLVM, because the test-suite will run
on bots and you will be told if you break them.

Most errors will be picked up by piece-wise tests, and the minority
where e2e make a difference can be reactive, rather than pro-active
fixing.

> Is there a proposal somewhere of what companies would be expected to do?
> It's difficult for us engineers to talk to management without a concrete
> set of expectations, resource requirements, etc.

There were talks about upgrading Buildbot (the service), moving to
Jenkins or something else (Travis?). None of them have the majority of
the community behind, and that's the main problem.

IIRC, the arguments (definitely outdated, probably wrong) were:

Buildbot:
 - Pros: we already have a big infra based on it, it's passable, an
upgrade could ameliorate a lot of problems without creating many new
ones.
 - Cons: it's old tech and requires extensive coding to make it work

Jenkins:
 - Pros: Most companies use that already, it's more modetn than
Apple's GreenBot is based on that, lots of plugins and expertise in
the community
 - Cons: It requires Java running on the client, which not all targets
like. Alternatives require a separate server to run as a slave and
connect to targets.

Travis:
 - Pros: It's natively compatible with Github (is is still the case?)
and it could be the easiest to connect with our new repo for CI
 - Cons: less expertise, I guess, and other things I don't really know.


> Nice!  That's much better.  Yes, it won't scale but it's much clearer
> about what is being run.

Perhaps adding a new column as to what components we test in each one
would be nice.

--renato