[llvm-dev] [RFC] Compiled regression tests.

[Chris Lattner via llvm-dev]

> On Jul 1, 2020, at 10:52 PM, Michael Kruse <llvmdev at meinersbur.de> wrote:
> 
>>>> Unit tests are very widely used across the industry, and it is certainly true that they are fully general and more flexible.  This makes them attractive, but it is a trap.  I’d really rather we don’t go down this route, and maintain the approach of only using unit tests for very low level things like apis in ADT etc.
>>> 
>>> Note that we already have unittests for non low-level APIs such as
>>> passes (VPlan, LICM, Unrolling, ...)
>>> 
>>> Can you elaborate on what the trap is?
>> 
>> I am also opposed to those - I think they have exactly the same problems described above, and we should invest in better testing infra to support them.
> 
> Except for the linking issue, I don't see them being problems or even
> being advantages.
> 
> You seem to acknowledge that these kind of unit tests are widely used
> for supposedly good reasons.

To be clear, I am not saying that unit tests are bad, I’m advocating we use them for things like ADT tests.  I am just saying there is something better for many purposes and that we should use the better thing instead of the worse thing.

> Why is LLVM special?


It depends on what you are comparing it against.  A short answer is that LLVM is ahead of most of the rest of the industry in this respect.

If you are comparing it against other compilers, one of the founding principles of LLVM is that the IR is round triable to a textual form that does not lose information.  I wouldn’t go so far to say that this was novel at the time, but it was certainly unusual: most compilers used lossy dump formats.  Even MachineInstr was this way until recent work.  This is even mentioned in section 3.6 of this thesis <http://llvm.org/pubs/2002-12-LattnerMSThesis.pdf>.

If you are comparing it to other application level software, you’ll find that testing is much more important for LLVM than for most (not all) app software, and LLVM has a LOT of tests.  Individual .ll files end up containing dozens (or sometimes hundreds) of tests, and making this efficient to run, efficient to maintain, and scalable across a growing community is not a trivial matter.

LLVM is also unusual in that it has a very modular design, this design has many benefits including reuse, but also testability.


-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200703/cb7ea5cd/attachment.html>