[llvm-dev] [RFC] Compiled regression tests.

[Michael Kruse via llvm-dev]

Am Mi., 1. Juli 2020 um 15:07 Uhr schrieb Chris Lattner <clattner at nondot.org>:
>
> On Jun 30, 2020, at 10:42 PM, Michael Kruse <llvmdev at meinersbur.de> wrote:
> >
> > Am Di., 30. Juni 2020 um 15:58 Uhr schrieb Chris Lattner <clattner at nondot.org>:
> >> One of the major and important pieces of the LLVM design is how its testing infrastructure works.  The choice to use a small number of tools (llc, opt, etc) is important for multiple reasons:
> >>
> >> 1) Link time of executables is a significant problem, particularly in large scale builds.
> >
> > You can use dynamic linking. Unfortunately, an LLVM dynamic library is
> > not (yet?) supported on Windows, so we need a static linking fallback.
> > If this is the only issue, I'd work on a solution (at least with
> > gtest) that works on Windows as well.
>
> Agreed - this is something I investigated early on in LLVM’s evolution (and one of the reasons the libraries are all dynamically linkable).
>
> The problem with this is that you shift time from the static linker to the dynamic linker - specifically making the launch time of the tests really slow.  This is even on Linux, which has a pretty efficient loader.
>
> >> 2) This encourages investing in testing tools (see, e.g. the recent improvements to FileCheck etc)
> >
> > Google test also is a testing tool that is worth investing into, such
> > as more expressive ASSERT macros as in the RFC.
>
> Yes, I’m very familiar with it, we use gtest in LLVM already.
>
> >> 3) It reduces/factors the number of users of API surface area, making it easier to do large scale refactoring etc.
> >
> > FileCheck makes string output (including llvm::dbgs()) part of the
> > interface, which becomes not only harder to change, but also harder to
> > extend (new lines/tags to not match existing CHECK lines).
>
> Yes, agreed.  The reason this is ok is that a change to a DBGS string breaks tests of (e.g.) the pass that it lives in.  API changes affect everything touching the API, which unit tests generally expose much more broadly.
>
> >> Unit tests are very widely used across the industry, and it is certainly true that they are fully general and more flexible.  This makes them attractive, but it is a trap.  I’d really rather we don’t go down this route, and maintain the approach of only using unit tests for very low level things like apis in ADT etc.
> >
> > Note that we already have unittests for non low-level APIs such as
> > passes (VPlan, LICM, Unrolling, ...)
> >
> > Can you elaborate on what the trap is?
>
> I am also opposed to those - I think they have exactly the same problems described above, and we should invest in better testing infra to support them.

Except for the linking issue, I don't see them being problems or even
being advantages.

You seem to acknowledge that these kind of unit tests are widely used
for supposedly good reasons. Why is LLVM special?

Michael