[llvm-dev] Small improvements for the lit tool

[Julian Lettner via llvm-dev]

Hi All,

I am interested in upstreaming a few small improvements for the LLVM Integrated Tester.  The main goal of this effort is improving the existing code base and features to make them more approachable for future contributors and a bit more useful for developers.  It is a non-goal to change the existing behavior in any significant way.  Ideally, most people will not even notice.

If there are no general objections then I will start creating review-sized patches from the changes that I have accumulated [1].

[1] https://github.com/yln/llvm-project/commits/lit-refactor


Rough list of improvements:

* Use "single process mode" when executing with `-j 1` or when executing a
  single test.  In this mode we do not create additional processes, which can
  help with debugging.  Make sure timeouts are supported in this mode.  In this
  mode CTRL+C prints a run summary instead of a Python process termination
  exception.

* Greatly improve argument validation.  Prevent incompatible options from being
  used together.  Add argument aliases that improve consistency of existing
  options which have accumulated over time.  All existing CL arguments remain
  functional!  Deprecating/retiring any existing options is a non-goal of this
  effort.

* Make it an error when no tests are discovered or all discovered tests are
  filtered out.

* In "no progress bar" mode test results are now printed in order.  This should
  help with things like diffing two CI runs.

* Test result categories (e.g., PASS, FAIL, XFAIL, ...). Not all "discovered"
  tests are accounted for.  Add "unexecuted" (e.g., due to REQUIRES in test) to
  SKIPPED category. Add new category, FILTERED that contain filtered-out tests
  (due to `--filter <regex>` or sharding feature).  This means each test starts
  out as "discovered" and will in one of the categories at the end of the run.
  Now all tests are accounted for.

* In the run summary tests that belong to a "failing" category (e.g., FAIL,
  XPASS, ...) are always printed. In addition,  we already have
  `--show-unsupported` and `--show-xfail` to show those specific "success"
  categories.  Introduced a consistent `--show=<list of categories>` option that
  works with all categories, to configure which tests will be printed.  Default
  is `--show=<all failing>`, i.e., no change in behavior.

* Make run summary more consistent.  Remove redundant word "Test" from
  categories. Print summary counts in the same order as test categories.

* Neatly align test counts as decimal numbers.  Align counts to longest label in
  run summary.
```
    Passing With Retry: 18
    Failing           :  8
```

* Progress bar now changes color to red on first test failure.

* Fix bug related to `--max-failures`.

* Improve formatting of error messages.

* Use smart defaults for command line arguments to streamline code/avoid
  checking for None.

* Factor out main.py into smaller, separate source files, e.g., cl_arguments.py.

* A few additional tests for new and refactored features.

* Lots of smaller refactorings to make codebase more approachable to future
  contributions: extract small functions, remove unnecessary code, remove "out
  parameters", try to assign good names.  Use Python-style function and variable
  names.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191004/a1da73a1/attachment.html>