[llvm-dev] [RFC] Compiled regression tests.

Sun Jun 28 11:42:28 PDT 2020

Hi Michael,

You propose using the preprocessor for mixing C++ test code with its input
code, such as LLVM IR.  Of course, LIT, FileCheck, `clang -verify`, and
unit tests all enable mixing their various forms of test code with input
code.  To my eyes, a difference between unit tests vs. LIT, FileCheck, and
`clang -verify` tests is that the latter tend to make the input code more
prominent (I prefer that) and make it easier to clarify which test code is
associated with which input code.  So far, I think the preprocessor
approach is better in this regard than unit tests, but what about a more
familiar syntax, like the following?

```
// RUN-CXX: #include "compiledtestboilerplate.h"
// RUN-CXX: unique_ptr<Module> Output = run_opt("%s",
"-passes=loop-vectorize");

// RUN-CXX: /* Check func1 Output */
define void @func1() {
entry:
  ret
}

// RUN-CXX: /* Check func2 Output */
define void @func2() {
entry:
  ret
}
```

It seems it should be feasible to automate extraction of the `RUN-CXX:`
code for compilation and for analysis by clang-format and clang-tidy.
Perhaps there would be a new script that extracts at build time for all
such uses.  But there are other possibilities that could be considered: a
LIT extension for `RUN-CXX:`, C++ JIT compilation, a clang-format-diff.py
extension that greps modified files for `RUN-CXX:`, etc.

LIT, FileCheck, and `RUN-CXX:` directives should then be able to co-exist
in a single test file.  Thus, you might incrementally add `RUN-CXX:`
directives to test files that already contain LIT and FileCheck directives
to handle cases where FileCheck directives are difficult to use.  You could
keep the FileCheck directives when they are more reasonable, or you could
eventually replace them.  You might run `opt` once with a `RUN:` directive
and then check its output `.ll` file with both `RUN-CXX:` and FileCheck
directives (maybe all `RUN:` directives execute before all `RUN-CXX:`
directives, or maybe C++ JIT compilation would permit them to execute in
the order specified).

It's not clear to me whether the above idea is worth the trouble, but I
think I'd at least prefer the syntax to the preprocessor approach.

I also have a vague feeling that something like this has been discussed
before.  If so, please just point me to the discussion.

In any case, thanks for working to improve LLVM's testing infrastructure.

Joel

On Tue, Jun 23, 2020 at 9:33 PM Michael Kruse via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> Hello LLVM community,
>
> For testing IR passes, LLVM currently has two kinds of tests:
>  1. regression tests (in llvm/test); .ll files invoking opt, and
> matching its text output using FileCheck.
>  2. unittests (in llvm/unittests); Google tests containing the IR as a
> string, constructing a pass pipeline, and inspecting the output using
> code.
>
> I propose to add an additional kind of test, which I call "compiled
> regression test", combining the advantages of the two. A test is a
> single .cxx file of the general structure below that can be dumped
> into the llvm/test directory. I am not proposing to replace FileCheck,
> but in a lot of cases, domain-specific verifiers can be more powerful
> (e.g. verify-uselistorder or `clang -verify`).
>
>     #ifdef IR
>       define void @func() {
>       entry:
>         ret
>       }
>     #else /* IR */
>       #include "compiledtestboilerplate.h"
>       TEST(TestSuiteName, TestName) {
>         unique_ptr<Module> Output = run_opt(__FILE__, "IR",
> "-passes=loop-vectorize");
>         /* Check Output */
>       }
>     #endif /* IR */
>
> That is, input IR and check code are in the same file. The run_opt
> command is a replica of main() from the opt tool, so any command line
> arguments (passes with legacy or new passmanager, cl::opt options,
> etc.) can be passed. It also makes converting existing tests simpler.
>
> The top-level structure is C++ (i.e. the LLVM-IR is removed by the
> preprocessor) and compiled with cmake. This allows a
> compile_commands.json to be created such that refactoring tools,
> clang-tidy, and clang-format can easily be applied on the code. The
> second argument to run_opt is the preprocessor directive for the IR
> such that multiple IR modules can be embedded into the file.
>
> Such tests can be compiled in two modes: Either within the LLVM
> project, or as an external subproject using llvm_ExternalProject_Add.
> The former has the disadvantage that new .cxx files dumped into the
> test folder are not recognized until the next cmake run, unless the
> CONFIGURE_DEPENDS option is used. I found this adds seconds to each
> invocation of ninja which I considered a dealbreaker. The external
> project searched for tests every time, but is only invoked in a
> check-llvm run, no different than llvm-lit. It uses CMake's
> find_package to build against the main project's results (which
> currently we do not have tests for) and could also be compiled in
> debug mode while LLVM itself is compiled in release mode.
>
> The checks themselves can be any of gtest's ASSERT/EXPECT macros, but
> for common test idioms I suggest to add custom macros, such as
>
>     ASSERT_ALL_OF(InstList, !isa<VectorType>(I->getType()));
>
> which on failure prints the instruction that does not return a vector.
> Try that with FileCheck. PattenMatch.h from InstCombine can be used as
> well. Structural comparison with a reference output could also be
> possible (like clang-diff,
> [llvm-canon](http://llvm.org/devmtg/2019-10/talk-abstracts.html#tech12),
> https://reviews.llvm.org/D80916).
>
> Some additional tooling could be helpful:
>
>  * A test file creator, taking some IR, wrapping it into the above
> structure, and write it into the test directory.
>  * A tool for extracting and updating (running opt) the IR inside the
> #ifdef, if not even add this functionality to opt itself. This is the
> main reason to not just the IR inside a string.
>
> A Work-in-Progress differential and what it improves over FileCheck
> and unittests is available here: https://reviews.llvm.org/D82426
>
> Any kind of feedback welcome.
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200628/0d8fd6cc/attachment-0001.html>