[PATCH] D30521: Introduce llc/ExecuteTestCommands pass

[Chandler Carruth via Phabricator via llvm-commits]

chandlerc added a comment.

In https://reviews.llvm.org/D30521#698089, @MatzeB wrote:

> In https://reviews.llvm.org/D30521#698048, @chandlerc wrote:
>
> > Currently, I really don't understand why this is the right approach...
> >
> > > We sometimes want to test a specific codegen API rather than a whole pass.
> > > 
> > > This adds a special pass to llc that features a minimalistic scripting language to
> > >  call some predefined API functions so we can test the API with the usual lit+FileCheck tools.
> >
> > If you're actually trying to test a specific API, why aren't unittests the correct approach?
> >
> > If the problem is that the unittests are hard to write, why not write libraries to make authoring the unittest easier?
>
>
> I think there is something fundamental about how we write tests in llvm, and why we do not write pass tests in a unit testing framework because hey, who stops us implementing llc/opt as some functions in a unit testing library, they mostly have trivial APIs:
>
> I think there is a beauty and we are hitting a sweet spot in the way most of our testing works in llvm: Forcing tests to be a simple human readable text file keeps things approachable and understandable. Understanding the basics of lit, llc, opt and FileCheck is easy. Having the full power of C++ in a unit test library on the other hand makes it way too easy do "fancy" stuff that is harder to understand because it happens to abstracted away in generic code and multiple layers of APIs that you have to learn and dig through when you actually have to understand a bug.


I don't see how things being abstracted away in a scripting language and complex fake "pass" infrastructure is going to be substantially better here.

> 
> 
>> Essentially, I totally understand prefering FileCheck and friends when testing something that is reasonably cohesive as a *pass*. But I feel like the point at which a scripting language or other things are being used to handle the fact that we're actually testing a specific API, I would *much rather* just write a unittest against that API directly.
> 
> I don't see why wanting FileCheck for things that aren't a pass isn't valid.

I don't think that pass vs. non-pass is the right distinction.

I think FileCheck makes sense when there is already a fundamental reason to have textual output that is sufficiently stable to use in tests. This can be everything from having an IR to having a clear and precise printing structure for an analysis. Those have utility *outside* of tests and also are great tools for testing.

But when the thing we are testing, fundamentally, is *an API*, I think we should test it *by calling that API*.

> Admittedly I am not happy to introduce a scripting language here, but it seemed like the pragmatic choice compared with the alternative of building up a pile of small passes for every test or putting .mir code into C++ string literals in a unittest and still lacking all the llc mechanics for bringing up a target or having the nice matching capabilities of FileCheck.

Again, I am very much in favor of having the facilities from 'llc' or 'FileCheck' in useful APIs for unittests to leverage. But I don't think we should be adding a scripting language in order to write tests in a language other than C++ to exercise C++ API calls. I think this is significantly more burdensome than just writing unittests.

Sorry for the slow response, I think we may just fundamentally disagree about the right approach here. Currently, the approach you are proposing would make the resulting tests nearly incomprehensible to me... On the other hand, I think unittests (which are "just" C++ code even if sometimes non-obvious C++ code) *when used well* are much more friendly to people outside of the particular area than a MIR-specific scripting framework.


Repository:
  rL LLVM

https://reviews.llvm.org/D30521