[llvm] 19b44c2 - [lit] Echo full RUN lines in case of external shells (#65267)

[via llvm-commits]

Author: Joel E. Denny
Date: 2023-09-05T10:48:18-04:00
New Revision: 19b44c2bdf1726acd380f76d26673a27ecf826ba

URL: https://github.com/llvm/llvm-project/commit/19b44c2bdf1726acd380f76d26673a27ecf826ba
DIFF: https://github.com/llvm/llvm-project/commit/19b44c2bdf1726acd380f76d26673a27ecf826ba.diff

LOG: [lit] Echo full RUN lines in case of external shells (#65267)

Before <https://reviews.llvm.org/D154984> and
<https://reviews.llvm.org/D156954>, lit reported full RUN lines in a
`Script:` section. Now, in the case of lit's internal shell, it's the
execution trace that includes them. However, if lit is configured to use
an external shell (e.g., bash, windows `cmd`), they aren't reported at
all.

A fix was requested at the following:

* <https://reviews.llvm.org/D154984#4627605>
*
<https://discourse.llvm.org/t/rfc-improving-lits-debug-output/72839/35?u=jdenny-ornl>

This patch does not correctly address the case when the external shell
is windows `cmd`. As discussed at
<https://github.com/llvm/llvm-project/pull/65242>, it's not clear
whether that's a use case that people still care about, and it seems to
be generally broken anyway.

Added: 
    llvm/utils/lit/tests/Inputs/shtest-run-at-line/external-shell/empty-run-line.txt
    llvm/utils/lit/tests/Inputs/shtest-run-at-line/internal-shell/empty-run-line.txt

Modified: 
    llvm/utils/lit/lit/TestRunner.py
    llvm/utils/lit/tests/shtest-run-at-line.py

Removed: 
    


################################################################################
diff  --git a/llvm/utils/lit/lit/TestRunner.py b/llvm/utils/lit/lit/TestRunner.py
index 461cf63d6b9685f..6af7d0ccd0faf91 100644
--- a/llvm/utils/lit/lit/TestRunner.py
+++ b/llvm/utils/lit/lit/TestRunner.py
@@ -1153,8 +1153,14 @@ def executeScript(test, litConfig, tmpBase, commands, cwd):
     for j, ln in enumerate(commands):
         match = re.match(kPdbgRegex, ln)
         if match:
+            dbg = match.group(1)
             command = match.group(2)
-            commands[j] = match.expand(": '\\1'; \\2" if command else ": '\\1'")
+            commands[j] = f"echo '{dbg}'"
+            if command:
+                commands[j] += f": {shlex.quote(command.lstrip())} >&2 " \
+                               f"&& {command}"
+            else:
+                commands[j] += " has no command after substitutions >&2"
             if litConfig.per_test_coverage:
                 # Extract the test case name from the test object
                 test_case_name = test.path_in_suite[-1]

diff  --git a/llvm/utils/lit/tests/Inputs/shtest-run-at-line/external-shell/empty-run-line.txt b/llvm/utils/lit/tests/Inputs/shtest-run-at-line/external-shell/empty-run-line.txt
new file mode 100644
index 000000000000000..40a5a7d6e7cce0a
--- /dev/null
+++ b/llvm/utils/lit/tests/Inputs/shtest-run-at-line/external-shell/empty-run-line.txt
@@ -0,0 +1,3 @@
+# DEFINE: %{empty} =
+# RUN: %{empty}
+# RUN: false

diff  --git a/llvm/utils/lit/tests/Inputs/shtest-run-at-line/internal-shell/empty-run-line.txt b/llvm/utils/lit/tests/Inputs/shtest-run-at-line/internal-shell/empty-run-line.txt
new file mode 100644
index 000000000000000..40a5a7d6e7cce0a
--- /dev/null
+++ b/llvm/utils/lit/tests/Inputs/shtest-run-at-line/internal-shell/empty-run-line.txt
@@ -0,0 +1,3 @@
+# DEFINE: %{empty} =
+# RUN: %{empty}
+# RUN: false

diff  --git a/llvm/utils/lit/tests/shtest-run-at-line.py b/llvm/utils/lit/tests/shtest-run-at-line.py
index a0626f872c4c9e0..397ae20ea4879b8 100644
--- a/llvm/utils/lit/tests/shtest-run-at-line.py
+++ b/llvm/utils/lit/tests/shtest-run-at-line.py
@@ -6,7 +6,7 @@
 # END.
 
 
-# CHECK: Testing: 4 tests
+# CHECK: Testing: 6 tests
 
 
 # In the case of the external shell, we check for only RUN lines in stderr in
@@ -14,15 +14,27 @@
 
 # CHECK-LABEL: FAIL: shtest-run-at-line :: external-shell/basic.txt
 
-# CHECK:     RUN: at line 4
-# CHECK:     RUN: at line 5
-# CHECK-NOT: RUN
+#      CHECK: Command Output (stderr)
+# CHECK-NEXT: --
+#      CHECK: {{^}}RUN: at line 4: true
+#      CHECK: {{^}}RUN: at line 5: false
+#  CHECK-NOT: RUN
+
+# CHECK-LABEL: FAIL: shtest-run-at-line :: external-shell/empty-run-line.txt
+
+#      CHECK: Command Output (stderr)
+# CHECK-NEXT: --
+#      CHECK: {{^}}RUN: at line 2 has no command after substitutions
+#      CHECK: {{^}}RUN: at line 3: false
+#  CHECK-NOT: RUN
 
 # CHECK-LABEL: FAIL: shtest-run-at-line :: external-shell/line-continuation.txt
 
-# CHECK:     RUN: at line 4
-# CHECK:     RUN: at line 6
-# CHECK-NOT: RUN
+#      CHECK: Command Output (stderr)
+# CHECK-NEXT: --
+#      CHECK: {{^}}RUN: at line 4: echo 'foo bar' | FileCheck
+#      CHECK: {{^}}RUN: at line 6: echo 'foo baz' | FileCheck
+#  CHECK-NOT: RUN
 
 
 # CHECK-LABEL: FAIL: shtest-run-at-line :: internal-shell/basic.txt
@@ -37,6 +49,16 @@
 # CHECK-NEXT: # executed command: false
 # CHECK-NOT:  RUN
 
+# CHECK-LABEL: FAIL: shtest-run-at-line :: internal-shell/empty-run-line.txt
+
+#      CHECK: Command Output (stdout)
+# CHECK-NEXT: --
+# CHECK-NEXT: # RUN: at line 2 has no command after substitutions
+# CHECK-NEXT: # RUN: at line 3
+# CHECK-NEXT: false
+# CHECK-NEXT: # executed command: false
+#  CHECK-NOT: RUN
+
 # CHECK-LABEL: FAIL: shtest-run-at-line :: internal-shell/line-continuation.txt
 
 # CHECK:      Command Output (stdout)