[llvm] d8f2bff - [lit] Better/earlier errors when no tests are executed
Joel E. Denny via llvm-commits
llvm-commits at lists.llvm.org
Thu Nov 7 13:22:25 PST 2019
Hi Julian,
It looks like your check-all is running lit only once. What subprojects do
you have enabled? I have most of them enabled.
Thanks.
Joel
On Thu, Nov 7, 2019 at 1:50 PM Julian Lettner <julian.lettner at apple.com>
wrote:
> Hi Joel,
>
> I can’t reproduce your issue. Can you describe what happened?
> When I run `env LIT_FILTER=xxx ninja check-all` I get the following:
>
> ➤ env LIT_FILTER=deadlock ninja check-all
> [3/6] cd
> /Users/yln/work/llvm-upstream/llvm-project/clang/binding...work/llvm-upstream/build/lib
> /usr/bin/python -m unittest discover
> <snip>
> [4/6] Checking TSan Go runtime...
> <snip>
> [5/6] Running lint check for sanitizer sources...
> <lots of linter output>
> Total errors found: 0
> [5/6] Running all regression tests
> <lots of lit notes and warnings>
>
> Testing Time: 50.04s
> Expected Passes : 23
> Expected Failures : 2
> Unsupported Tests : 5
>
> 40 warning(s) in tests.
>
> ➤ echo $status
> 0
>
> ➤ env LIT_FILTER=does_not_exist ninja check-all
> <same as above>
> [5/6] Running all regression tests
> <lots of lit notes and warnings>
> Filter did not match any tests (of 62432 discovered).
> FAILED: CMakeFiles/check-all
> cd /Users/yln/work/llvm-upstream/build && /usr/bin/python
> /Users/yln/work/llvm-upstream/build/./bin/llvm-lit -sv --param
> clang_site_config=/Users/yln/work/llvm-upstream/build/tools/clang/test/lit.site.cfg
> --param USE_Z3_SOLVER=0 <long list of test suites>
> ninja: build stopped: subcommand failed.
>
> ➤ echo $status
> 1
>
>
> On Nov 5, 2019, at 12:32 PM, Joel E. Denny <jdenny.ornl at gmail.com> wrote:
>
> Hi Julian,
>
> I don't know that I would have thought of the check-all issue during a
> patch review, but someone might have, and my understanding is that patch
> reviews are required in LLVM.
>
> Thanks for the fast response. I do appreciate your effort to address
> empty lit runs.
>
> Joel
>
> On Tue, Nov 5, 2019 at 3:20 PM Julian Lettner <jlettner at apple.com> wrote:
> Hi Joel,
>
> Thanks for letting me know. I reverted the change until I have time to
> investigate.
> I didn’t create a review for this change, but I have learned my lesson and
> will definitely include you in the review for the re-land. Sorry for the
> churn,
>
> Thanks,
> Julian
>
> On Nov 5, 2019, at 10:54 AM, Joel E. Denny <jdenny.ornl at gmail.com> wrote:
>
> Hi Julian,
>
> This patch breaks LIT_FILTER for check-all, which runs lit multiple
> times. For such use cases, maybe there should be a command-line option to
> suppress this error.
>
> Also, was there a review for this patch? I don't see one referenced in
> the commit log.
>
> Thanks.
>
> Joel
>
> On Mon, Nov 4, 2019 at 1:17 PM Julian Lettner via llvm-commits <
> llvm-commits at lists.llvm.org> wrote:
>
> Author: Julian Lettner
> Date: 2019-11-04T10:16:24-08:00
> New Revision: d8f2bff75126c6dde694ad245f9807fa12ad5630
>
> URL:
> https://github.com/llvm/llvm-project/commit/d8f2bff75126c6dde694ad245f9807fa12ad5630
> DIFF:
> https://github.com/llvm/llvm-project/commit/d8f2bff75126c6dde694ad245f9807fa12ad5630.diff
>
> LOG: [lit] Better/earlier errors when no tests are executed
>
> Fail early, when we discover no tests at all, or filter out all of them.
>
> Added:
>
>
> Modified:
> llvm/utils/lit/lit/cl_arguments.py
> llvm/utils/lit/lit/main.py
> llvm/utils/lit/lit/run.py
> llvm/utils/lit/tests/selecting.py
>
> Removed:
>
>
>
>
> ################################################################################
> diff --git a/llvm/utils/lit/lit/cl_arguments.py
> b/llvm/utils/lit/lit/cl_arguments.py
> index 7b15041d1905..c17f6c7824e1 100644
> --- a/llvm/utils/lit/lit/cl_arguments.py
> +++ b/llvm/utils/lit/lit/cl_arguments.py
> @@ -5,6 +5,7 @@
>
> import lit.util
>
> +# TODO(yln): scan dest, to see if is redundant
> def parse_args():
> parser = argparse.ArgumentParser()
> parser.add_argument('test_paths',
>
> diff --git a/llvm/utils/lit/lit/main.py b/llvm/utils/lit/lit/main.py
> index 337d154a560d..af36b07f1fef 100755
> --- a/llvm/utils/lit/lit/main.py
> +++ b/llvm/utils/lit/lit/main.py
> @@ -42,6 +42,9 @@ def main(builtin_params = {}):
> echo_all_commands = opts.echoAllCommands)
>
> tests = lit.discovery.find_tests_for_inputs(litConfig,
> opts.test_paths)
> + if not tests:
> + sys.stderr.write('Did not disover any tests for provided
> path(s).\n')
> + sys.exit(2)
>
> # Command line overrides configuration for maxIndividualTestTime.
> if opts.maxIndividualTestTime is not None: # `not None` is important
> (default: 0)
> @@ -62,12 +65,20 @@ def main(builtin_params = {}):
>
> if opts.filter:
> tests = [t for t in tests if opts.filter.search(t.getFullName())]
> + if not tests:
> + sys.stderr.write('Filter did not match any tests '
> + '(of %d discovered).\n' % numTotalTests)
> + sys.exit(2)
>
> determine_order(tests, opts.order)
>
> if opts.shard:
> (run, shards) = opts.shard
> tests = filter_by_shard(tests, run, shards, litConfig)
> + if not tests:
> + sys.stderr.write('Shard does not contain any tests. Consider
> '
> + 'decreasing the shard count.\n')
> + sys.exit(0)
>
> if opts.max_tests:
> tests = tests[:opts.max_tests]
> @@ -84,7 +95,7 @@ def main(builtin_params = {}):
> write_test_results_xunit(tests, opts)
>
> if litConfig.numErrors:
> - sys.stderr.write('\n%d error(s), exiting.\n' %
> litConfig.numErrors)
> + sys.stderr.write('\n%d error(s) in tests.\n' %
> litConfig.numErrors)
> sys.exit(2)
>
> if litConfig.numWarnings:
>
> diff --git a/llvm/utils/lit/lit/run.py b/llvm/utils/lit/lit/run.py
> index 4d16399fec47..11992c8073d2 100644
> --- a/llvm/utils/lit/lit/run.py
> +++ b/llvm/utils/lit/lit/run.py
> @@ -13,7 +13,7 @@ def acquire(self): pass
> def release(self): pass
>
> def create_run(tests, lit_config, workers, progress_callback,
> timeout=None):
> - # TODO(yln) assert workers > 0
> + assert workers > 0
> if workers == 1:
> return SerialRun(tests, lit_config, progress_callback, timeout)
> return ParallelRun(tests, lit_config, progress_callback, timeout,
> workers)
> @@ -45,9 +45,6 @@ def execute(self):
> computed. Tests which were not actually executed (for any reason)
> will
> be given an UNRESOLVED result.
> """
> - if not self.tests:
> - return 0.0
> -
> self.failure_count = 0
> self.hit_max_failures = False
>
>
> diff --git a/llvm/utils/lit/tests/selecting.py
> b/llvm/utils/lit/tests/selecting.py
> index 0921cdd31ac9..cfbb9d19b6dc 100644
> --- a/llvm/utils/lit/tests/selecting.py
> +++ b/llvm/utils/lit/tests/selecting.py
> @@ -1,6 +1,18 @@
> # RUN: %{lit} %{inputs}/discovery | FileCheck --check-prefix=CHECK-BASIC
> %s
> # CHECK-BASIC: Testing: 5 tests
>
> +
> +# Check that we exit with an error if we do not discover any tests.
> +#
> +# RUN: not %{lit} %{inputs}/nonexistent 2>&1 | FileCheck
> --check-prefix=CHECK-BAD-PATH %s
> +# CHECK-BAD-PATH: Did not disover any tests for provided path(s).
> +
> +# Check that we exit with an error if we filter out all tests.
> +#
> +# RUN: not %{lit} --filter 'nonexistent' %{inputs}/discovery 2>&1 |
> FileCheck --check-prefix=CHECK-BAD-FILTER %s
> +# CHECK-BAD-FILTER: Filter did not match any tests (of 5 discovered).
> +
> +
> # Check that regex-filtering works, is case-insensitive, and can be
> configured via env var.
> #
> # RUN: %{lit} --filter 'o[a-z]e' %{inputs}/discovery | FileCheck
> --check-prefix=CHECK-FILTER %s
> @@ -8,6 +20,7 @@
> # RUN: env LIT_FILTER='o[a-z]e' %{lit} %{inputs}/discovery | FileCheck
> --check-prefix=CHECK-FILTER %s
> # CHECK-FILTER: Testing: 2 of 5 tests
>
> +
> # Check that maximum counts work
> #
> # RUN: %{lit} --max-tests 3 %{inputs}/discovery | FileCheck
> --check-prefix=CHECK-MAX %s
> @@ -68,15 +81,13 @@
> #
> # RUN: %{lit} --num-shards 100 --run-shard 6 %{inputs}/discovery >%t.out
> 2>%t.err
> # RUN: FileCheck --check-prefix=CHECK-SHARD-BIG-ERR2 < %t.err %s
> -# RUN: FileCheck --check-prefix=CHECK-SHARD-BIG-OUT2 < %t.out %s
> # CHECK-SHARD-BIG-ERR2: note: Selecting shard 6/100 = size 0/5 = tests
> #(100*k)+6 = []
> -# CHECK-SHARD-BIG-OUT2: Testing: 0 of 5 tests
> +# CHECK-SHARD-BIG-ERR2: Shard does not contain any tests. Consider
> decreasing the shard count.
> #
> # RUN: %{lit} --num-shards 100 --run-shard 50 %{inputs}/discovery >%t.out
> 2>%t.err
> # RUN: FileCheck --check-prefix=CHECK-SHARD-BIG-ERR3 < %t.err %s
> -# RUN: FileCheck --check-prefix=CHECK-SHARD-BIG-OUT3 < %t.out %s
> # CHECK-SHARD-BIG-ERR3: note: Selecting shard 50/100 = size 0/5 = tests
> #(100*k)+50 = []
> -# CHECK-SHARD-BIG-OUT3: Testing: 0 of 5 tests
> +# CHECK-SHARD-BIG-ERR3: Shard does not contain any tests. Consider
> decreasing the shard count.
>
>
> # Check that range constraints are enforced
>
>
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20191107/6bffb0f6/attachment.html>
More information about the llvm-commits
mailing list