[llvm-dev] [cfe-dev] RFC: End-to-end testing

[David Greene via llvm-dev]

Mehdi AMINI via llvm-dev <llvm-dev at lists.llvm.org> writes:

>> I absolutely disagree about vectorization tests.  We have seen
>> vectorization loss in clang even though related LLVM lit tests pass,
>> because something else in the clang pipeline changed that caused the
>> vectorizer to not do its job.
>
> Of course, and as I mentioned I tried to add these tests (probably 4 or 5
> years ago), but someone (I think Chandler?) was asking me at the time: does
> it affect a benchmark performance? If so why isn't it tracked there? And if
> not does it matter?
> The benchmark was presented as the actual way to check this invariant
> (because you're only vectoring to get performance, not for the sake of it).
> So I never pursued, even if I'm a bit puzzled that we don't have such tests.

Thanks for explaining.

Our experience is that relying solely on performance tests to uncover
such issues is problematic for several reasons:

- Performance varies from implementation to implementation.  It is
  difficult to keep tests up-to-date for all possible targets and
  subtargets.
  
- Partially as a result, but also for other reasons, performance tests
  tend to be complicated, either in code size or in the numerous code
  paths tested.  This makes such tests hard to debug when there is a
  regression.

- Performance tests don't focus on the why/how of vectorization.  They
  just check, "did it run fast enough?"  Maybe the test ran fast enough
  for some other reason but we still lost desired vectorization and
  could have run even faster.

With a small asm test one can documented why vectorization is desired
and how it comes about right in the test.  Then when it breaks it's
usually pretty obvious what the problem is.

They don't replace performance tests, they complement each other, just
as end-to-end and component tests complement each other.

>> Debugging and PGO involve other components, no?
>
> I don't think that you need anything else than LLVM core (which is a
> dependency of clang) itself?

What about testing that what clang produces is debuggable with lldb?
debuginfo tests do that now but AFAIK they are not end-to-end.

> Things like PGO (unless you're using frontend instrumentation) don't even
> have anything to do with clang, so we may get into the situation David
> mentioned where we would rely on clang to test LLVM features, which I find
> non-desirable.

We would still expect component-level tests.  This would be additional
end-to-end testing, not replacing existing testing methodology.  I agree
the concern is valid but shouldn't code review discover such problems?

>> > Actually the first one seems more of a pure LLVM test.
>>
>> Definitely not.  It would test the pipeline as constructed by clang,
>> which is very different from the default pipeline constructed by
>> opt/llc.
>
> I am not convinced it is "very" difference (they are using the
> PassManagerBuilder AFAIK), I am only aware of very subtle difference.

I don't think clang exclusively uses PassManagerBuilder but it's been a
while since I looked at that code.

> But more fundamentally: *should* they be different? I would want `opt -O3`
> to be able to reproduce 1-1 the clang pipeline.

Which clang pipeline?  -O3?  -Ofast?  opt currently can't do -Ofast.  I
don't *think* -Ofast affects the pipeline itself but I am not 100%
certain.

> Isn't it the role of LLVM PassManagerBuilder to expose what is the "-O3"
> pipeline?

Ideally, yes.  In practice, it's not.

> If we see the PassManagerBuilder as the abstraction for the pipeline, then
> I don't see what testing belongs to clang here, this seems like a layering
> violation (and maintaining the PassManagerBuilder in LLVM I wouldn't want
> to have to update the tests of all the subproject using it because they
> retest the same feature).

If nothing else, end-to-end testing of the pipeline would uncover
layering violations.  :)

>> The old and new pass managers also construct different
>> pipelines.  As we have seen with various mailing list messages, this is
>> surprising to users.  Best to document and check it with testing.
>>
>
> Yes: both old and new pass managers are LLVM components, so hopefully that
> are documented and tested in LLVM :)

But we have nothing to guarantee that what clang does matches what opt
does.  Currently they do different things.

                         -David