[llvm-dev] [RFC] Compiled regression tests.

[Roman Lebedev via llvm-dev]

On Wed, Jun 24, 2020 at 12:35 PM David Blaikie via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
>
> I'm pretty change averse - and am in this case (but doesn't mean other folks won't be in favor and doesn't mean it isn't worth trying, etc - but if it were up to me, at the moment, I'd decline)
>
> To take one point from the phab review (to try to keep the discussion here rather than there - might be worth a note on the phab review to discourage commenting there so the discussion doesn't get spread through different threads):
>
> > Because of all of the above, maintenance of all the regression tests is a nightmare. I expect it to be a serious issue for introducing opaque pointers. My prediction is that we will have a typed-pointer command line flag to not have to require updating all the write-only regression tests.
>
> Actually I already did a lot of work with the initial patches years ago for opaque pointers (that added explicit types to gep/store/etc) and used (& provided in the commit messages) python scripts to migrate all the tests, both the IR itself and the CHECK text. This is probably more readily automatable than a more free-form C++ based checking proposed here.

+1, i'm not sure what kind of scalability issues with updating
existing tests there is.

> That said, it sounds like the proposal is a lot closer to the GUnit tests, and if this testing strategy is valuable, it seems like it could be mostly achieved by adding some API utilities (like the one you proposed in the patch) to make it more convenient to run optimization passes in a GUnit test. It doesn't seem to me like an #ifdef based approach to embedding IR in C++ would result in particularly more manageable/formattable code than a raw string. Perhaps the proposed improvements could be used to reduce/remove the cost of adding new GUnit tests/the need to touch CMake files/etc. (though I'd worry about the divergence of where optimization tests are written - as unit tests or as lit/FileCheck tests - that doesn't mean experimentation isn't worthwhile, but I think it'd require a pretty compelling justification to propose a replacement to the FileCheck approach (& perhaps a timeline for an experiment before either removing it, or deciding that it is the intended future state) - and if it's not a replacement, then I think we'd need to discuss what sort of situations this new thing is suitable and what FileCheck should be used for going forward)
>
> - Dave
Roman

> On Tue, Jun 23, 2020 at 6:34 PM Michael Kruse via llvm-dev <llvm-dev at lists.llvm.org> wrote:
>>
>> Hello LLVM community,
>>
>> For testing IR passes, LLVM currently has two kinds of tests:
>>  1. regression tests (in llvm/test); .ll files invoking opt, and
>> matching its text output using FileCheck.
>>  2. unittests (in llvm/unittests); Google tests containing the IR as a
>> string, constructing a pass pipeline, and inspecting the output using
>> code.
>>
>> I propose to add an additional kind of test, which I call "compiled
>> regression test", combining the advantages of the two. A test is a
>> single .cxx file of the general structure below that can be dumped
>> into the llvm/test directory. I am not proposing to replace FileCheck,
>> but in a lot of cases, domain-specific verifiers can be more powerful
>> (e.g. verify-uselistorder or `clang -verify`).
>>
>>     #ifdef IR
>>       define void @func() {
>>       entry:
>>         ret
>>       }
>>     #else /* IR */
>>       #include "compiledtestboilerplate.h"
>>       TEST(TestSuiteName, TestName) {
>>         unique_ptr<Module> Output = run_opt(__FILE__, "IR",
>> "-passes=loop-vectorize");
>>         /* Check Output */
>>       }
>>     #endif /* IR */
>>
>> That is, input IR and check code are in the same file. The run_opt
>> command is a replica of main() from the opt tool, so any command line
>> arguments (passes with legacy or new passmanager, cl::opt options,
>> etc.) can be passed. It also makes converting existing tests simpler.
>>
>> The top-level structure is C++ (i.e. the LLVM-IR is removed by the
>> preprocessor) and compiled with cmake. This allows a
>> compile_commands.json to be created such that refactoring tools,
>> clang-tidy, and clang-format can easily be applied on the code. The
>> second argument to run_opt is the preprocessor directive for the IR
>> such that multiple IR modules can be embedded into the file.
>>
>> Such tests can be compiled in two modes: Either within the LLVM
>> project, or as an external subproject using llvm_ExternalProject_Add.
>> The former has the disadvantage that new .cxx files dumped into the
>> test folder are not recognized until the next cmake run, unless the
>> CONFIGURE_DEPENDS option is used. I found this adds seconds to each
>> invocation of ninja which I considered a dealbreaker. The external
>> project searched for tests every time, but is only invoked in a
>> check-llvm run, no different than llvm-lit. It uses CMake's
>> find_package to build against the main project's results (which
>> currently we do not have tests for) and could also be compiled in
>> debug mode while LLVM itself is compiled in release mode.
>>
>> The checks themselves can be any of gtest's ASSERT/EXPECT macros, but
>> for common test idioms I suggest to add custom macros, such as
>>
>>     ASSERT_ALL_OF(InstList, !isa<VectorType>(I->getType()));
>>
>> which on failure prints the instruction that does not return a vector.
>> Try that with FileCheck. PattenMatch.h from InstCombine can be used as
>> well. Structural comparison with a reference output could also be
>> possible (like clang-diff,
>> [llvm-canon](http://llvm.org/devmtg/2019-10/talk-abstracts.html#tech12),
>> https://reviews.llvm.org/D80916).
>>
>> Some additional tooling could be helpful:
>>
>>  * A test file creator, taking some IR, wrapping it into the above
>> structure, and write it into the test directory.
>>  * A tool for extracting and updating (running opt) the IR inside the
>> #ifdef, if not even add this functionality to opt itself. This is the
>> main reason to not just the IR inside a string.
>>
>> A Work-in-Progress differential and what it improves over FileCheck
>> and unittests is available here: https://reviews.llvm.org/D82426
>>
>> Any kind of feedback welcome.
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev