[PATCH] D30521: Introduce llc/ExecuteTestCommands pass

[Matthias Braun via Phabricator via llvm-commits]

MatzeB added a comment.

In https://reviews.llvm.org/D30521#698048, @chandlerc wrote:

> Currently, I really don't understand why this is the right approach...
>
> > We sometimes want to test a specific codegen API rather than a whole pass.
> > 
> > This adds a special pass to llc that features a minimalistic scripting language to
> >  call some predefined API functions so we can test the API with the usual lit+FileCheck tools.
>
> If you're actually trying to test a specific API, why aren't unittests the correct approach?
>
> If the problem is that the unittests are hard to write, why not write libraries to make authoring the unittest easier?


I think there is something fundamental about how we write tests in llvm, and why we do not write pass tests in a unit testing framework because hey, who stops us implementing llc/opt as some functions in a unit testing library, they mostly have trivial APIs:

I think there is a beauty and we are hitting a sweet spot in the way most of our testing works in llvm: Forcing tests to be a simple human readable text file keeps things approachable and understandable. Understanding the basics of lit, llc, opt and FileCheck is easy. Having the full power of C++ in a unit test library on the other hand makes it way too easy do "fancy" stuff that is harder to understand because it happens to abstracted away in generic code and multiple layers of APIs that you have to learn and dig through when you actually have to understand a bug.

> Essentially, I totally understand prefering FileCheck and friends when testing something that is reasonably cohesive as a *pass*. But I feel like the point at which a scripting language or other things are being used to handle the fact that we're actually testing a specific API, I would *much rather* just write a unittest against that API directly.

I don't see why wanting FileCheck for things that aren't a pass isn't valid. Admittedly I am not happy to introduce a scripting language here, but it seemed like the pragmatic choice compared with the alternative of building up a pile of small passes for every test or putting .mir code into C++ string literals in a unittest and still lacking all the llc mechanics for bringing up a target or having the nice matching capabilities of FileCheck.


Repository:
  rL LLVM

https://reviews.llvm.org/D30521