[llvm-dev] [RFC] Compiled regression tests.

[Chris Lattner via llvm-dev]

On Jun 24, 2020, at 9:50 AM, Michael Kruse via llvm-dev <llvm-dev at lists.llvm.org> wrote:
>>> 
>>> I propose to add an additional kind of test, which I call "compiled
>>> regression test", combining the advantages of the two.
>> 
>> 
>> You expand below on the mechanism you'd like to implement, but I am a bit puzzled about the motivation right now?
> 
> See https://reviews.llvm.org/D82426 and
> http://lists.llvm.org/pipermail/llvm-dev/2020-June/142706.html for
> more motivation.

Hi Michael,

I’m sorry I’m late to this thread, but I would really rather not go this direction.  Unit tests in general (and this sort of extension to the idea) come at very high cost to testing flows, particularly with large scale builds.

One of the major and important pieces of the LLVM design is how its testing infrastructure works.  The choice to use a small number of tools (llc, opt, etc) is important for multiple reasons:

1) Link time of executables is a significant problem, particularly in large scale builds.
2) This encourages investing in testing tools (see, e.g. the recent improvements to FileCheck etc)
3) It reduces/factors the number of users of API surface area, making it easier to do large scale refactoring etc.
4) It encourages the development of textual interfaces to libraries, which aids with understandability and helps reinforce stronger interfaces (llvm-mc is one example of this, LLVM IR text syntax is another).
5) Depending on the details, this can make the build dependence graph more serialized.

Unit tests are very widely used across the industry, and it is certainly true that they are fully general and more flexible.  This makes them attractive, but it is a trap.  I’d really rather we don’t go down this route, and maintain the approach of only using unit tests for very low level things like apis in ADT etc.

-Chris