[llvm-dev] [Openmp-dev] [cfe-dev] RFC: End-to-end testing

[David Greene via llvm-dev]

Renato Golin <rengolin at gmail.com> writes:

> On Thu, 17 Oct 2019 at 18:10, David Greene <greened at obbligato.org> wrote:
>> From other discussion, it sounds like at least some people are open to
>> asm tests under clang.  I think that should be fine.  But there are
>> probably other kinds of end-to-end tests that should not live under
>> clang.
>
> That is my position as well. Some tests, especially similar to
> existing ones, are fine.

Ok.

> But if we really want to do do complete tests and stress more than
> just grepping a couple of instructions, should be in a better suited
> place.

That's probably true.

>> How often would such tests be run as part of test-suite?
>
> Every time the TS is executed. Some good work has been put on it to
> run with CMake etc, so it should be trivial to to run that before
> commits, but it *does* require more than just "make check-all".

I have been viewing test-suite as a kind of second-level/backup testing
that catches things not flagged by "make check-all."  Is that a
reasonable interpretation?  I was hoping to get some end-to-end tests
under "make check-all" because that's easier for developers to run in
their workflows.

> On CI, a number of bots run those as often as they can, non-stop.
>
>> Honestly, it's not really clear to me exactly which bots cover what, how
>> often they run and so on.  Is there a document somewhere describing the
>> setup?
>
> Not really. The main Buildbot page is a mess and the system is very
> old. There is a round table at the dev meeting to discuss the path
> forward.

Yeah, I saw that.  I will see if I can attend.  There are some conflicts
we have to work out here.

> This is not the first, though. We have been discussing this for a
> number of years, but getting people / companies to commit to testing
> is not trivial.

Is there a proposal somewhere of what companies would be expected to do?
It's difficult for us engineers to talk to management without a concrete
set of expectations, resource requirements, etc.

> I created a page for the Arm bots (after many incarnations, it ended
> up here: http://ex40-01.tcwglab.linaro.org/) to make that simpler. But
> that wouldn't scale, nor it fixes the real problems.

Nice!  That's much better.  Yes, it won't scale but it's much clearer
about what is being run.

                         -David