[llvm] 6f6af49 - Clarify `lit`'s definition of failure and conditions when it exits with exit code 1 (#136190)

[via llvm-commits]

Author: i-ky
Date: 2025-04-28T07:06:42+01:00
New Revision: 6f6af49938bfd0b1649f25064fd9619552f82b23

URL: https://github.com/llvm/llvm-project/commit/6f6af49938bfd0b1649f25064fd9619552f82b23
DIFF: https://github.com/llvm/llvm-project/commit/6f6af49938bfd0b1649f25064fd9619552f82b23.diff

LOG: Clarify `lit`'s definition of failure and conditions when it exits with exit code 1 (#136190)

Documentation of when `lit` exits with code 1 is out of date.

It is no longer just "FAIL or XPASS", there are more failure types:

https://github.com/llvm/llvm-project/blob/b30100b87f24847afd6407b4939a184ebcf16ef9/llvm/utils/lit/lit/Test.py#L51-L55

Exit code can also be affected by `--ignore-fail` option:

https://github.com/llvm/llvm-project/blob/b30100b87f24847afd6407b4939a184ebcf16ef9/llvm/utils/lit/lit/main.py#L154-L162

This PR extracts a clear definition of "failure" from the description of
`--report-failures-only` option:

https://github.com/llvm/llvm-project/blob/b30100b87f24847afd6407b4939a184ebcf16ef9/llvm/docs/CommandGuide/lit.rst?plain=1#L194-L196
...puts it into "Test Status Results" section and references it when
describing exit codes and `--ignore-fail` option.

Added: 
    

Modified: 
    llvm/docs/CommandGuide/lit.rst

Removed: 
    


################################################################################
diff  --git a/llvm/docs/CommandGuide/lit.rst b/llvm/docs/CommandGuide/lit.rst
index 812e1d819e9c8..2a0ddd0ea04b4 100644
--- a/llvm/docs/CommandGuide/lit.rst
+++ b/llvm/docs/CommandGuide/lit.rst
@@ -193,7 +193,7 @@ EXECUTION OPTIONS
 
 .. option:: --report-failures-only
 
- Only include unresolved, timed out, failed and unexpectedly passed tests in the report.
+ Only include failures (see :ref:`test-status-results`) in the report.
 
 .. option:: --resultdb-output RESULTDB_OUTPUT
 
@@ -384,8 +384,9 @@ ADDITIONAL OPTIONS
 EXIT STATUS
 -----------
 
-:program:`lit` will exit with an exit code of 1 if there are any FAIL or XPASS
-results.  Otherwise, it will exit with the status 0.  Other exit codes are used
+:program:`lit` will exit with an exit code of 1 if there are any failures
+(see :ref:`test-status-results`) and :option:`--ignore-fail` has not been
+passed.  Otherwise, it will exit with the status 0.  Other exit codes are used
 for non-test related failures (for example a user error or an internal program
 error).
 
@@ -461,8 +462,10 @@ Each test ultimately produces one of the following eight results:
 
 **TIMEOUT**
 
- The test was run, but it timed out before it was able to complete. This is
- considered a failure.
+ The test was run, but it timed out before it was able to complete.
+
+Unresolved (**UNRESOLVED**), timed out (**TIMEOUT**), failed (**FAIL**) and
+unexpectedly passed (**XPASS**) tests are considered failures.
 
 Depending on the test format tests may produce additional information about
 their status (generally only for failures).  See the :ref:`output-options`