[PATCH] D150856: [lit] Add %{for-each-file} substitution

[Louis Dionne via Phabricator via llvm-commits]

ldionne added a comment.

In D150856#4421565 <https://reviews.llvm.org/D150856#4421565>, @aaron.ballman wrote:

> The reason I favor the current `%{for-each-file}` approach is that it works well for more complex scenarios and it's extensible.

I actually had the exact opposite reaction: I was not able to see how this would compose. For example, can you nest a command that uses `%{for-each-file}` within a command that uses `%{for-each-file}`?

In D150856#4436879 <https://reviews.llvm.org/D150856#4436879>, @jdenny wrote:

> In other words, instead of incrementally designing, debating, and maintaining more and more lit features to try to address every use case in the most ideal manner, eventually ending up with a full blown scripting language in lit, why don't we invest that effort in connecting to an existing scripting language? Python seems like the obvious choice.  This would open so many more doors than one more lit control structure.

I think this should be at the top of our minds.

I just noticed this thread and skimmed through it. I'd like to offer libc++'s experience to solve a similar issue. We have several tests that are pretty repetitive and we started generating them automatically using python scripts. To do this, we added a new kind of test called `.gen.py` to libc++'s own custom test format. This kind of test is basically a Python script that generates other tests based on arbitrary Python code. So what we do is something like this:

  # Test that we can include each header in a TU while using modules.
  
  # RUN: %{python} %s
  
  for header in SOME-LIST-OF-HEADERS:
    print(f"""
  //--- {header}.pass.cpp
  // RUN: %{cxx} %s %{flags} %{compile_flags} -fmodules -fcxx-modules -fmodules-cache-path=%t -fsyntax-only
  #include <{header}>
  """)

The real thing is actually more complicated than this, but this is the gist of it. The result of this `.gen.py` test is actually many sub-tests, one for each `header.pass.cpp` file output by this script. Each test looks like this (e.g. for `<thread>`):

  //--- thread.pass.cpp
  // RUN: %{cxx} %s %{flags} %{compile_flags} -fmodules -fcxx-modules -fmodules-cache-path=%t -fsyntax-only
  #include <thread>

The benefits of this approach are:

1. This is extremely extensible. We're writing arbitrary Python code to generate the tests, so if somebody needs additional custom logic when generating their test, they can go as crazy as they want without impacting Lit at all.
2. This interacts nicely with the rest of Lit because each file is then a self-contained test with its own `RUN` command. When it fails, you can see exactly what failed in that test and in that test only, as-if you never actually had all the tests written in the same file in the first place.

The downside is that (in this case at least) there's a bit more upfront complexity required for writing the test, since you have to write it in Python. I think this would probably be easy to improve with something like the `PYTHON` directive (which I wasn't aware of until recently) or something else -- libc++'s test suite can't use most of the tools that Clang can use, like `split-files`. What you folks are trying to achieve is slightly different because most of your DR test is not generated, it's just normal C++, and the only part that gets generated is the `RUN` lines. However, I think a similar approach might work nicely if tweaked to your needs. In particular, I would argue that if the solution to this problem requires adding anything **specific** to Lit (as opposed to a general capability), then the design of that solution might be slightly off.

Anyway, I don't have a large stake in Lit or in the Clang test suite, but I wanted to share libc++'s positive experience with solving a similar problem using a general approach.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D150856/new/

https://reviews.llvm.org/D150856