[PATCH] D150856: [lit] Add %{for-each-file} substitution

[Vlad Serebrennikov via Phabricator via llvm-commits]

Endill added a comment.

In D150856#4428635 <https://reviews.llvm.org/D150856#4428635>, @jdenny wrote:

> In D150856#4428523 <https://reviews.llvm.org/D150856#4428523>, @Endill wrote:
>
>> We're also can't chain compiler invocations via `&&`: `RUN: %{run98} && %{run11} && %{run14} && %{run17} && %{run20} && %{run23}`, because having dedicated stdout log per each language mode makes DR test debugging workflow much better.
>
> Can you explain that a bit more or point to a reference?  I find lit -vv makes it pretty easy to understand what commands are executed and what output goes with each.

One of the most common pitfalls is to forget to guard a test with `#if __cplusplus`. I commented one of those in `dr13xx.cpp`, and changed order of language modes to avoid special case of first command failing. That's what I currently get from `ninja check-clang-cxx-drs` (and would get from a buildbot, for that matter):

  ******************** TEST 'Clang :: CXX/drs/dr13xx.cpp' FAILED ********************
  Script:
  --
  : 'RUN: at line 1';   /home/user/endill/llvm-project/build-linux/bin/clang -cc1 -internal-isystem /home/user/endill/llvm-project/build-linux/lib/clang/17/include -nostdsysteminc -std=c++17 /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp -verify -fexceptions -fcxx-exceptions -pedantic-errors
  : 'RUN: at line 2';   /home/user/endill/llvm-project/build-linux/bin/clang -cc1 -internal-isystem /home/user/endill/llvm-project/build-linux/lib/clang/17/include -nostdsysteminc -std=c++14 /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp -verify -fexceptions -fcxx-exceptions -pedantic-errors
  : 'RUN: at line 3';   /home/user/endill/llvm-project/build-linux/bin/clang -cc1 -internal-isystem /home/user/endill/llvm-project/build-linux/lib/clang/17/include -nostdsysteminc -std=c++11 /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp -verify -fexceptions -fcxx-exceptions -pedantic-errors
  : 'RUN: at line 4';   /home/user/endill/llvm-project/build-linux/bin/clang -cc1 -internal-isystem /home/user/endill/llvm-project/build-linux/lib/clang/17/include -nostdsysteminc -std=c++98 /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp -verify -fexceptions -fcxx-exceptions -pedantic-errors
  --
  Exit Code: 1
  
  Command Output (stderr):
  --
  error: 'error' diagnostics seen but not expected: 
    File /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp Line 212: a type specifier is required for all declarations
    File /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp Line 212: no member named 'g' in 'dr1330::B<dr1330::P>'
    File /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp Line 214: a type specifier is required for all declarations
    File /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp Line 214: no member named 'g' in 'dr1330::B<dr1330::Q>'
  4 errors generated.
  
  --
  
  ********************

Thank you for mentioning `-vv`, because it resolves that for local development at least. Surprisingly, it helps even with chained compiler invocations:

  ******************** TEST 'Clang :: CXX/drs/dr13xx.cpp' FAILED ********************
  Script:
  --
  : 'RUN: at line 1';   /home/user/endill/llvm-project/build-linux/bin/clang -cc1 -internal-isystem /home/user/endill/llvm-project/build-linux/lib/clang/17/include -nostdsysteminc -std=c++17 /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp -verify -fexceptions -fcxx-exceptions -pedantic-errors && /home/user/endill/llvm-project/build-linux/bin/clang -cc1 -internal-isystem /home/user/endill/llvm-project/build-linux/lib/clang/17/include -nostdsysteminc -std=c++14 /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp -verify -fexceptions -fcxx-exceptions -pedantic-errors && /home/user/endill/llvm-project/build-linux/bin/clang -cc1 -internal-isystem /home/user/endill/llvm-project/build-linux/lib/clang/17/include -nostdsysteminc -std=c++98 /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp -verify -fexceptions -fcxx-exceptions -pedantic-errors && /home/user/endill/llvm-project/build-linux/bin/clang -cc1 -internal-isystem /home/user/endill/llvm-project/build-linux/lib/clang/17/include -nostdsysteminc -std=c++11 /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp -verify -fexceptions -fcxx-exceptions -pedantic-errors
  --
  Exit Code: 1
  
  Command Output (stderr):
  --
  + : 'RUN: at line 1'
  + /home/user/endill/llvm-project/build-linux/bin/clang -cc1 -internal-isystem /home/user/endill/llvm-project/build-linux/lib/clang/17/include -nostdsysteminc -std=c++17 /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp -verify -fexceptions -fcxx-exceptions -pedantic-errors
  + /home/user/endill/llvm-project/build-linux/bin/clang -cc1 -internal-isystem /home/user/endill/llvm-project/build-linux/lib/clang/17/include -nostdsysteminc -std=c++14 /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp -verify -fexceptions -fcxx-exceptions -pedantic-errors
  + /home/user/endill/llvm-project/build-linux/bin/clang -cc1 -internal-isystem /home/user/endill/llvm-project/build-linux/lib/clang/17/include -nostdsysteminc -std=c++98 /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp -verify -fexceptions -fcxx-exceptions -pedantic-errors
  error: 'error' diagnostics seen but not expected: 
    File /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp Line 209: a type specifier is required for all declarations
    File /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp Line 209: no member named 'g' in 'dr1330::B<dr1330::P>'
    File /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp Line 211: a type specifier is required for all declarations
    File /home/user/endill/llvm-project/clang/test/CXX/drs/dr13xx.cpp Line 211: no member named 'g' in 'dr1330::B<dr1330::Q>'
  4 errors generated.
  
  --
  
  ********************

Irrespective of what happens to this patch, I'd like failed DR test to be diagnosed in `-vv` mode by default. It's very helpful, and lowers the bar for newcomers (we advertise DR tests as an activity for them).

> https://gist.github.com/jdenny-ornl/82aae9c3706c88fedebc8c44ac2a6694

So we're looking at 20% more noise in a large test suite. I'm not happy with that either.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D150856/new/

https://reviews.llvm.org/D150856