[llvm-dev] Small improvements for the lit tool

[Jordan Rose via llvm-dev]

> On Oct 4, 2019, at 14:34, Julian Lettner <julian.lettner at apple.com> wrote:
> 
> Hi All,
> 
> I am interested in upstreaming a few small improvements for the LLVM Integrated Tester.  The main goal of this effort is improving the existing code base and features to make them more approachable for future contributors and a bit more useful for developers.  It is a non-goal to change the existing behavior in any significant way.  Ideally, most people will not even notice.
> 
> If there are no general objections then I will start creating review-sized patches from the changes that I have accumulated [1].
> 
> [1] https://github.com/yln/llvm-project/commits/lit-refactor <https://github.com/yln/llvm-project/commits/lit-refactor>
> 
> 
> Rough list of improvements:
> 
> * Use "single process mode" when executing with `-j 1` or when executing a
>   single test.  In this mode we do not create additional processes, which can
>   help with debugging.  Make sure timeouts are supported in this mode.  In this
>   mode CTRL+C prints a run summary instead of a Python process termination
>   exception.

Mild preference for -j1 to not mean "single-process mode". -j1 is a nice way to say "multiprocess mode but deterministic".

> 
> * Greatly improve argument validation.  Prevent incompatible options from being
>   used together.  Add argument aliases that improve consistency of existing
>   options which have accumulated over time.  All existing CL arguments remain
>   functional!  Deprecating/retiring any existing options is a non-goal of this
>   effort.
> 
> * Make it an error when no tests are discovered or all discovered tests are
>   filtered out.
> 
> * In "no progress bar" mode test results are now printed in order.  This should
>   help with things like diffing two CI runs.

Minorly against this one. I can watch the tests pass or fail on a bot right now, and waiting for them all to finish so they can be printed sorted doesn't seem worth it over `pbpaste | sort`.

> 
> * Test result categories (e.g., PASS, FAIL, XFAIL, ...). Not all "discovered"
>   tests are accounted for.  Add "unexecuted" (e.g., due to REQUIRES in test) to
>   SKIPPED category. Add new category, FILTERED that contain filtered-out tests
>   (due to `--filter <regex>` or sharding feature).  This means each test starts
>   out as "discovered" and will in one of the categories at the end of the run.
>   Now all tests are accounted for.
> 
> * In the run summary tests that belong to a "failing" category (e.g., FAIL,
>   XPASS, ...) are always printed. In addition,  we already have
>   `--show-unsupported` and `--show-xfail` to show those specific "success"
>   categories.  Introduced a consistent `--show=<list of categories>` option that
>   works with all categories, to configure which tests will be printed.  Default
>   is `--show=<all failing>`, i.e., no change in behavior.
> 
> * Make run summary more consistent.  Remove redundant word "Test" from
>   categories. Print summary counts in the same order as test categories.
> 
> * Neatly align test counts as decimal numbers.  Align counts to longest label in
>   run summary.
> ```
>     Passing With Retry: 18
>     Failing           :  8
> ```
> 
> * Progress bar now changes color to red on first test failure.

Cute. :-)

> 
> * Fix bug related to `--max-failures`.
> 
> * Improve formatting of error messages.
> 
> * Use smart defaults for command line arguments to streamline code/avoid
>   checking for None.
> 
> * Factor out main.py into smaller, separate source files, e.g., cl_arguments.py.
> 
> * A few additional tests for new and refactored features.
> 
> * Lots of smaller refactorings to make codebase more approachable to future
>   contributions: extract small functions, remove unnecessary code, remove "out
>   parameters", try to assign good names.  Use Python-style function and variable
>   names.
> 

Jordan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191004/8431d6c7/attachment.html>