[llvm-dev] [Openmp-dev] [cfe-dev] RFC: End-to-end testing

[Renato Golin via llvm-dev]

On Thu, 17 Oct 2019 at 18:10, David Greene <greened at obbligato.org> wrote:
> From other discussion, it sounds like at least some people are open to
> asm tests under clang.  I think that should be fine.  But there are
> probably other kinds of end-to-end tests that should not live under
> clang.

That is my position as well. Some tests, especially similar to
existing ones, are fine.

But if we really want to do do complete tests and stress more than
just grepping a couple of instructions, should be in a better suited
place.

> How often would such tests be run as part of test-suite?

Every time the TS is executed. Some good work has been put on it to
run with CMake etc, so it should be trivial to to run that before
commits, but it *does* require more than just "make check-all".

On CI, a number of bots run those as often as they can, non-stop.

> Honestly, it's not really clear to me exactly which bots cover what, how
> often they run and so on.  Is there a document somewhere describing the
> setup?

Not really. The main Buildbot page is a mess and the system is very
old. There is a round table at the dev meeting to discuss the path
forward.

This is not the first, though. We have been discussing this for a
number of years, but getting people / companies to commit to testing
is not trivial.

I created a page for the Arm bots (after many incarnations, it ended
up here: http://ex40-01.tcwglab.linaro.org/) to make that simpler. But
that wouldn't scale, nor it fixes the real problems.