[all-commits] [llvm/llvm-project] 54a9b4: [lit] Better/earlier errors for empty runs

[Julian Lettner via All-commits]

  Branch: refs/heads/master
  Home:   https://github.com/llvm/llvm-project
  Commit: 54a9b4c02ff57e9847e0c501578e51db6f73d3be
      https://github.com/llvm/llvm-project/commit/54a9b4c02ff57e9847e0c501578e51db6f73d3be
  Author: Julian Lettner <julian.lettner at apple.com>
  Date:   2019-11-12 (Tue, 12 Nov 2019)

  Changed paths:
    M llvm/utils/lit/lit/cl_arguments.py
    M llvm/utils/lit/lit/main.py
    M llvm/utils/lit/lit/run.py
    M llvm/utils/lit/tests/selecting.py

  Log Message:
  -----------
  [lit] Better/earlier errors for empty runs

Fail early, when we discover no tests at all, or filter out all of them.

There is also `--allow-empty-runs` to disable test to allow workflows
like `LIT_FILTER=abc ninja check-all`.  Apparently `check-all` invokes
lit multiple times if certain projects are enabled, which would produce
unwanted "empty runs". Specify via `LIT_OPTS=--allow-empty-runs`.

There are 3 causes for empty runs:
1) No tests discovered.  This is always an error.  Fix test suite config
   or command line.
2) All tests filtered out.  This is an error by default, but can be
   suppressed via `--alow-empty-runs`.  Should prevent accidentally
   passing empty runs, but allow the workflow above.
3) The number of shards is greater than the number of tests.  Currently,
   this is never an error.  Personally, I think we should consider
   making this an error by default; if this happens, you are doing
   something wrong. I added a warning but did not change the behavior,
   since this warrants more discussion.

Reviewed By: atrick, jdenny

Differential Revision: https://reviews.llvm.org/D70105