[llvm-dev] [Openmp-dev] [cfe-dev] RFC: End-to-end testing

[Mehdi AMINI via llvm-dev]

On Fri, Oct 18, 2019 at 8:53 AM Renato Golin via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> On Fri, 18 Oct 2019 at 16:30, David Greene <greened at obbligato.org> wrote:
> > I have been viewing test-suite as a kind of second-level/backup testing
> > that catches things not flagged by "make check-all."  Is that a
> > reasonable interpretation?  I was hoping to get some end-to-end tests
> > under "make check-all" because that's easier for developers to run in
> > their workflows.
>
> It is a common understanding, which makes the test-suite less useful,
> but that's not really relevant.
>
> No one needs to the test-suite as part of their development processes,
> because we have bots for that.
>
> If you have decent piece-wise tests in Clang/LLVM, you really don't
> need end-to-end tests in Clang/LLVM, because the test-suite will run
> on bots and you will be told if you break them.
>
> Most errors will be picked up by piece-wise tests, and the minority
> where e2e make a difference can be reactive, rather than pro-active
> fixing.
>
> > Is there a proposal somewhere of what companies would be expected to do?
> > It's difficult for us engineers to talk to management without a concrete
> > set of expectations, resource requirements, etc.
>
> There were talks about upgrading Buildbot (the service), moving to
> Jenkins or something else (Travis?). None of them have the majority of
> the community behind, and that's the main problem.
>
> IIRC, the arguments (definitely outdated, probably wrong) were:
>
> Buildbot:
>  - Pros: we already have a big infra based on it, it's passable, an
> upgrade could ameliorate a lot of problems without creating many new
> ones.
>  - Cons: it's old tech and requires extensive coding to make it work
>
> Jenkins:
>  - Pros: Most companies use that already, it's more modetn than
> Apple's GreenBot is based on that, lots of plugins and expertise in
> the community
>  - Cons: It requires Java running on the client, which not all targets
> like. Alternatives require a separate server to run as a slave and
> connect to targets.
>
> Travis:
>  - Pros: It's natively compatible with Github (is is still the case?)
> and it could be the easiest to connect with our new repo for CI
>  - Cons: less expertise, I guess, and other things I don't really know.
>

This is good summary! I just want to put some new options like BuildKite
(we've been experimenting with this on MLIR: https://buildkite.com/mlir/ ),
and GitHub Actions (Still in Beta: https://github.com/features/actions , I
haven't had time to play with it). There are also paying options like
CircleCI and TeamCity, but they seem out-of-scope for us I think?
It'd be interesting to collect a more complete list of CI tools
free-for-open-source :)

-- 
Mehdi



>
>
> > Nice!  That's much better.  Yes, it won't scale but it's much clearer
> > about what is being run.
>
> Perhaps adding a new column as to what components we test in each one
> would be nice.
>
> --renato
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191019/3814c063/attachment.html>