[PATCH] D156954: [lit] Improve test output from lit's internal shell

[Joel E. Denny via Phabricator via llvm-commits]

jdenny created this revision.
jdenny added reviewers: Endill, jhenderson, yln, ldionne, aaron.ballman, awarzynski, MaskRay, rnk, mstorsjo, asavonic.
Herald added a subscriber: delcypher.
Herald added a project: All.
jdenny requested review of this revision.
Herald added a reviewer: jdoerfert.
Herald added subscribers: llvm-commits, jplehr, sstefan1.
Herald added a project: LLVM.

Motivation
----------

D154984 <https://reviews.llvm.org/D154984> removes the "Script:" section that lit prints along with a test's output, and it makes -v and -a imply -vv.  For example, after D154984 <https://reviews.llvm.org/D154984>, the "Script:" section below is never shown, but -v is enough to produce the execution trace following it:

  Script:
  --
  : 'RUN: at line 1'; echo hello | FileCheck bogus.txt && echo success
  --
  Exit Code: 2
  
  Command Output (stdout):
  --
  $ ":" "RUN: at line 1"
  $ "echo" "hello"
  # command output:
  hello
  
  $ "FileCheck" "bogus.txt"
  # command stderr:
  Could not open check file 'bogus.txt': No such file or directory
  
  error: command failed with exit status: 2
  
  --

In the D154984 <https://reviews.llvm.org/D154984> review, some reviewers point out that they have been using the "Script:" section for copying and pasting a test's shell commands to a terminal window.  The shell commands as printed in the execution trace can be harder to copy and paste for the following reasons:

- They drop redirections and break apart RUN lines at `&&`, `|`, etc.
- They add `$` at the start of every command, which makes it hard to copy and paste multiple commands in bulk.
- Command stdout, stderr, etc. are interleaved with the commands and are not clearly delineated.
- They don't always use proper shell quoting.  Instead, they blindly enclose all command-line arguments in double quotes.

Changes
-------

D154984 <https://reviews.llvm.org/D154984> plus this patch converts the above example into:

  Exit Code: 2
  
  Command Output (stdout):
  --
  # RUN: at line 1
  echo hello | FileCheck bogus-file.txt && echo success
  # executed command: echo hello
  # .---command stdout------------
  # | hello
  # `-----------------------------
  # executed command: FileCheck bogus-file.txt
  # .---command stderr------------
  # | Could not open check file 'bogus-file.txt': No such file or directory
  # `-----------------------------
  # error: command failed with exit status: 2
  
  --

Thus, this patch addresses the above issues as follows:

- The entire execution trace can be copied and pasted in bulk to a terminal for correct execution of the RUN lines, which are printed intact as they appeared in original the RUN lines except lit substitutions are expanded.  Everything else in the execution trace appears in shell comments so it has no effect in a terminal.
- Each of the RUN line's commands is repeated (in shell comments) as it executes to show (1) that the command actually executed (e.g., `echo success` above didn't) and (2) what stdout, stderr, non-zero exit status, and output files are associated with the command, if any.  Shell quoting in the command is now correct and minimal but is not necessarily the original shell quoting from the RUN line.
- The start and end of the contents of stdout, stderr, or an output file is now delineated clearly in the trace.

TODO: While this patch updates lit's tests, new tests for the new features would be worthwhile.  I'll wait for some acceptance of the changes in the patch before doing that.

Caveat
------

This patch only makes the above changes for lit's internal shell.  In most cases, we do not know how to force external shells (e.g., bash, sh, window's `cmd`) to produce execution traces in the manner we want.

To configure a test suite to use lit's internal shell (which is usually better for test portability than external shells anyway), add this to the test suite's `lit.cfg` or other configuration file:

  config.test_format = lit.formats.ShTest(execute_external=False)


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D156954

Files:
  llvm/utils/lit/lit/TestRunner.py
  llvm/utils/lit/tests/Inputs/lit-opts/lit.cfg
  llvm/utils/lit/tests/Inputs/shtest-define/examples/param-subst.txt
  llvm/utils/lit/tests/Inputs/shtest-define/expansion-order.txt
  llvm/utils/lit/tests/Inputs/shtest-define/line-number-substitutions.txt
  llvm/utils/lit/tests/Inputs/shtest-define/name-chars.txt
  llvm/utils/lit/tests/Inputs/shtest-define/recursiveExpansionLimit.txt
  llvm/utils/lit/tests/Inputs/shtest-define/value-equals.txt
  llvm/utils/lit/tests/Inputs/shtest-define/value-escaped.txt
  llvm/utils/lit/tests/Inputs/shtest-define/ws-and-continuations.txt
  llvm/utils/lit/tests/Inputs/shtest-if-else/test.txt
  llvm/utils/lit/tests/lit-opts.py
  llvm/utils/lit/tests/shtest-define.py
  llvm/utils/lit/tests/shtest-env.py
  llvm/utils/lit/tests/shtest-format.py
  llvm/utils/lit/tests/shtest-if-else.py
  llvm/utils/lit/tests/shtest-not.py
  llvm/utils/lit/tests/shtest-output-printing.py
  llvm/utils/lit/tests/shtest-pushd-popd.py
  llvm/utils/lit/tests/shtest-recursive-substitution.py
  llvm/utils/lit/tests/shtest-run-at-line.py
  llvm/utils/lit/tests/shtest-shell.py

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D156954.546653.patch
Type: text/x-patch
Size: 90468 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20230803/441ff39a/attachment.bin>