[llvm-dev] [cfe-dev] RFC: End-to-end testing

[David Greene via llvm-dev]

Mehdi AMINI via cfe-dev <cfe-dev at lists.llvm.org> writes:

> I don't think these particular tests are the most controversial though, and
> it is really still fairly "focused" testing. I'm much more curious about
> larger end-to-end scope: for instance since you mention debug info and
> LLDB, what about a test that would verify that LLDB can print a particular
> variable content from a test that would come as a source program for
> instance. Such test are valuable in the absolute, it isn't clear to me that
> we could in practice block any commit that would break such test though:
> this is because a bug fix or an improvement in one of the pass may be
> perfectly correct in isolation but make the test fail by exposing a bug
> where we are already losing some debug info precision in a totally
> unrelated part of the codebase.
> I wonder how you see this managed in practice: would you gate any change on
> InstCombine (or other mid-level pass) on not regressing any of the
> debug-info quality test on any of the backend, and from any frontend (not
> only clang)? Or worse: a middle-end change that would end-up with a
> slightly different Dwarf construct on this particular test, which would
> trip LLDB but not GDB (basically expose a bug in LLDB). Should we require
> the contributor of inst-combine to debug LLDB and fix it first?

Good questions!  I think for situations like this I would tend toward
allowing the change and the test would alert us that something else is
wrong.  At that point it's probably a case-by-case decision.  Maybe we
XFAIL the test.  Maybe the fix is easy enough that we just do it and the
test starts passing again.  What's the policy for breaking current tests
when the change itself is fine but exposes a problem elsewhere (adding
an assert, for example)?

                       -David