[llvm-dev] [RFC] Compiled regression tests.

[Michael Kruse via llvm-dev]

Am Mi., 1. Juli 2020 um 11:37 Uhr schrieb Hal Finkel <hfinkel at anl.gov>:
> We can have different printing modes. There can be a more-human-friendly mode and a more-FileCheck-friendly mode. Or modes customized for different kinds of tests. I agree, however, that this does not solve the fragility problems with CHECK-NOT.

This would be similar to git's porcelain and plumbing modes. However,
even with git which had this from the beginning, often scripts use
porcelain output.

Another example are commands such as that change output on whether
stdout is a terminal or a pipe. However, I find such distinction
between modes more confusing than helpful.


> That's the interesting question... it does add to the maintenance burden. However, having textual outputs are also quite convenient when debugging things (because I can change the input and see the output quickly, without needing to create and compile another program). Obviously, at some point, this becomes ridiculous. How much is too much? I don't know. But it's also not clear to me that we're at that point yet. We could add more textual analysis outputs and still have that be a net benefit in many places.
>
> In cases where the requisite output would just be too specific, we do have unit tests. Should we just add more? Maybe we're too happy to add lit tests instead of unit tests for some of these cases.

The RFC is not about replacing all uses of FileCheck, there are
certainly cases where it is straightforward, simple and robust, but
for some things it would be nice to have another tool in the toolbox.
The more workarounds, FileCheck features, test generators, etc are
needed to author appropriate tests, the more I get the impression
FileCheck is the wrong tool.

As an example, take `clang -verify` tests. It is certainly possible to
check diagnostic output using FileCheck, so why does clang have a
-verify option?


Michael