[llvm-dev] Small improvements for the lit tool
Julian Lettner via llvm-dev
llvm-dev at lists.llvm.org
Fri Oct 4 14:34:00 PDT 2019
I am interested in upstreaming a few small improvements for the LLVM Integrated Tester. The main goal of this effort is improving the existing code base and features to make them more approachable for future contributors and a bit more useful for developers. It is a non-goal to change the existing behavior in any significant way. Ideally, most people will not even notice.
If there are no general objections then I will start creating review-sized patches from the changes that I have accumulated .
Rough list of improvements:
* Use "single process mode" when executing with `-j 1` or when executing a
single test. In this mode we do not create additional processes, which can
help with debugging. Make sure timeouts are supported in this mode. In this
mode CTRL+C prints a run summary instead of a Python process termination
* Greatly improve argument validation. Prevent incompatible options from being
used together. Add argument aliases that improve consistency of existing
options which have accumulated over time. All existing CL arguments remain
functional! Deprecating/retiring any existing options is a non-goal of this
* Make it an error when no tests are discovered or all discovered tests are
* In "no progress bar" mode test results are now printed in order. This should
help with things like diffing two CI runs.
* Test result categories (e.g., PASS, FAIL, XFAIL, ...). Not all "discovered"
tests are accounted for. Add "unexecuted" (e.g., due to REQUIRES in test) to
SKIPPED category. Add new category, FILTERED that contain filtered-out tests
(due to `--filter <regex>` or sharding feature). This means each test starts
out as "discovered" and will in one of the categories at the end of the run.
Now all tests are accounted for.
* In the run summary tests that belong to a "failing" category (e.g., FAIL,
XPASS, ...) are always printed. In addition, we already have
`--show-unsupported` and `--show-xfail` to show those specific "success"
categories. Introduced a consistent `--show=<list of categories>` option that
works with all categories, to configure which tests will be printed. Default
is `--show=<all failing>`, i.e., no change in behavior.
* Make run summary more consistent. Remove redundant word "Test" from
categories. Print summary counts in the same order as test categories.
* Neatly align test counts as decimal numbers. Align counts to longest label in
Passing With Retry: 18
Failing : 8
* Progress bar now changes color to red on first test failure.
* Fix bug related to `--max-failures`.
* Improve formatting of error messages.
* Use smart defaults for command line arguments to streamline code/avoid
checking for None.
* Factor out main.py into smaller, separate source files, e.g., cl_arguments.py.
* A few additional tests for new and refactored features.
* Lots of smaller refactorings to make codebase more approachable to future
contributions: extract small functions, remove unnecessary code, remove "out
parameters", try to assign good names. Use Python-style function and variable
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the llvm-dev