[llvm-dev] [Openmp-dev] [cfe-dev] RFC: End-to-end testing

David Greene via llvm-dev llvm-dev at lists.llvm.org
Fri Oct 18 08:30:28 PDT 2019

Renato Golin <rengolin at gmail.com> writes:

> On Thu, 17 Oct 2019 at 18:10, David Greene <greened at obbligato.org> wrote:
>> From other discussion, it sounds like at least some people are open to
>> asm tests under clang.  I think that should be fine.  But there are
>> probably other kinds of end-to-end tests that should not live under
>> clang.
> That is my position as well. Some tests, especially similar to
> existing ones, are fine.


> But if we really want to do do complete tests and stress more than
> just grepping a couple of instructions, should be in a better suited
> place.

That's probably true.

>> How often would such tests be run as part of test-suite?
> Every time the TS is executed. Some good work has been put on it to
> run with CMake etc, so it should be trivial to to run that before
> commits, but it *does* require more than just "make check-all".

I have been viewing test-suite as a kind of second-level/backup testing
that catches things not flagged by "make check-all."  Is that a
reasonable interpretation?  I was hoping to get some end-to-end tests
under "make check-all" because that's easier for developers to run in
their workflows.

> On CI, a number of bots run those as often as they can, non-stop.
>> Honestly, it's not really clear to me exactly which bots cover what, how
>> often they run and so on.  Is there a document somewhere describing the
>> setup?
> Not really. The main Buildbot page is a mess and the system is very
> old. There is a round table at the dev meeting to discuss the path
> forward.

Yeah, I saw that.  I will see if I can attend.  There are some conflicts
we have to work out here.

> This is not the first, though. We have been discussing this for a
> number of years, but getting people / companies to commit to testing
> is not trivial.

Is there a proposal somewhere of what companies would be expected to do?
It's difficult for us engineers to talk to management without a concrete
set of expectations, resource requirements, etc.

> I created a page for the Arm bots (after many incarnations, it ended
> up here: http://ex40-01.tcwglab.linaro.org/) to make that simpler. But
> that wouldn't scale, nor it fixes the real problems.

Nice!  That's much better.  Yes, it won't scale but it's much clearer
about what is being run.


More information about the llvm-dev mailing list