[llvm-dev] [Openmp-dev] [cfe-dev] RFC: End-to-end testing

[Renato Golin via llvm-dev]

On Thu, 17 Oct 2019 at 16:28, Robinson, Paul <paul.robinson at sony.com> wrote:
> This is no different than today. Many tests in Clang require a specific
> target to exist. Grep clang/test for "registered-target" for example;
> I get 577 hits.  Integration tests (here called "end-to-end" tests)
> clearly need to specify their REQUIRES conditions correctly.

Right, why I wrote in the beginning that Clang already has tests like that.

So, if all David wants is to extend those tests, than I think this
thread was a heck of a time wasting exercise. :)

It's nothing new, nothing deeply controversial and it's in the list of
things we know are not great, but accept anyway.

I personally don't think it's a good idea (for reasons already
expressed in this thread), and that has brought me trouble when I was
setting up the Arm bots. I had to build the x86 target, even though I
never used it, just because of some tests.

Today, Arm bots are faster, so it doesn't matter much, but new
hardware will still have that problem. I would like, long term, to
have the right tests on the right places.

> Monorepo isn't the relevant thing.  It's all about the build config.

I didn't mean it would, per se. Yes, it's about build config, but
setting up CI with SVN means you have to actively checkout repos,
while in monorepo, they all come together, so it's easier to forget
they are tangled, or to hack around build issues (like I did when I
marked the x86 to build) and never look back (that was 7 years ago).

> I have to say, it's highly unusual for me to make a commit that
> does *not* produce blame mail from some bot running lit tests.
> Thankfully it's rare to get one that is actually my fault.

I was hoping to reduce that. :)

> I can't remember *ever* getting blame mail related to test-suite.
> Do they actually run?  Do they ever catch anything?  Do they ever
> send blame mail?  I have to wonder about that.

They do run, on both x86 and Arm at least, in different
configurations, including correctness and benchmark mode, on anything
between 5 and 100 commits, continuously.

They rarely catch much nowadays because the toolchain is stable and no
new tests are being added. They work very well, though, for external
system tests and benchmarks, and people use it downstream a lot.

They do send blame mail occasionally, but only after all the others,
and people generally ignore them. Bot owners usually have to pressure
people, create bugs, revert patches or just fix the issues themselves.