[all-commits] [llvm/llvm-project] 9e739f: [lit] Fix some issues from --per-test-coverage (#6...

[Joel E. Denny via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 9e739fdb85ac672f3e25e971d96e71823e07ebda
      https://github.com/llvm/llvm-project/commit/9e739fdb85ac672f3e25e971d96e71823e07ebda
  Author: Joel E. Denny <jdenny.ornl at gmail.com>
  Date:   2023-09-14 (Thu, 14 Sep 2023)

  Changed paths:
    M llvm/utils/lit/lit/TestRunner.py
    M llvm/utils/lit/tests/Inputs/per-test-coverage-by-lit-cfg/lit.cfg
    M llvm/utils/lit/tests/Inputs/per-test-coverage-by-lit-cfg/per-test-coverage-by-lit-cfg.py
    M llvm/utils/lit/tests/Inputs/per-test-coverage/lit.cfg
    M llvm/utils/lit/tests/Inputs/per-test-coverage/per-test-coverage.py
    M llvm/utils/lit/tests/allow-retries.py
    M llvm/utils/lit/tests/per-test-coverage-by-lit-cfg.py
    M llvm/utils/lit/tests/per-test-coverage.py

  Log Message:
  -----------
  [lit] Fix some issues from --per-test-coverage (#65242)

D154280 (landed in 64d19542e78a in July, 2023) implements
`--per-test-coverage` (which can also be specified via 
`lit_config.per_test_coverage`).  However, it has a few issues, which
the current patch addresses:

1. D154280 implements `--per-test-coverage` only for the case that lit 
   is configured to use an external shell.  The current patch extends
   the implementation to lit's internal shell.

2. In the case that lit is configured to use an external shell,
   regardless of whether `--per-test-coverage` is actually specified,
   D154280 causes `%dbg(RUN: at line N)` to be expanded in RUN lines
   early and in a manner that is specific to sh-like shells.  As a
   result, later code in lit that expands it in a shell-specific
   manner is useless as there's nothing left to expand.  The current
   patch cleans up the implementation to avoid useless code.

3. Because of issue 2, D154280 corrupts support for windows `cmd` as
   an external shell (effectively comments out all RUN lines with
   `:`).  The current patch happens to fix that particular corruption
   by addressing issue 2.  However, D122569 (landed in 1041a9642ba0 in
   April, 2022) had already broken support for windows `cmd` as an
   external shell (discards RUN lines when expanding `%dbg(RUN: at
   line N)`).  The current patch does not attempt to fix that bug.
   For further details, see the PR discussion of the current patch.

The current patch addresses the above issues by implementing
`--per-test-coverage` before selecting the shell (internal or
external) and by leaving `%dbg(RUN: at line N)` unexpanded there.
Thus, it is expanded later in a shell-specific manner, as before
D154280.

This patch introduces `buildPdbgCommand` into lit's implementation to
encapsulate the process of building (or rebuilding in the case of the 
`--per-test-coverage` implementation) a full `%dbg(RUN: at line N)
cmd` line and asserting that the result matches `kPdbgRegex`.  It also
cleans up that and all other uses of `kPdbgRegex` to operate on the 
full line with `re.fullmatch` not `re.match`.  This change better
reflects the intention in every case, but it is expected to be NFC 
because `kPdbgRegex` ends in `.*` and thus avoids the difference
between `re.fullmatch` and `re.match`.  The only caveat is that `.*`
does not match newlines, but RUN lines cannot contain newlines
currently, so this caveat currently shouldn't matter in practice.

The original `--per-test-coverage` implementation avoided accumulating
`export LLVM_PROFILE_FILE={profile}` insertions across retries (due to
`ALLOW_RETRIES`) by skipping the insertion if `%dbg(RUN: at line N)` 
was not present and thus had already been expanded.  However, the 
current patch makes sure the insertions also happen for commands
without `%dbg(RUN: at line N)`, such as preamble commands or some
commands from other lit test formats.  Thus, the current patch
implements a different mechanism to avoid accumulating those
insertions (see code comments).