[llvm] d8f2bff - [lit] Better/earlier errors when no tests are executed
Joel E. Denny via llvm-commits
llvm-commits at lists.llvm.org
Thu Nov 7 14:30:19 PST 2019
Here's what I have:
LLVM_ENABLE_PROJECTS:STRING=clang;clang-tools-extra;compiler-rt;libclc;libcxx;libcxxabi;libunwind;lld;lldb;openmp;parallel-libs;polly;pstl
$ LIT_FILTER=uar_signals.cpp ninja check-all
[15/28] Running GWP-ASan unit tests
Filter did not match any tests (of 34 discovered).
FAILED: projects/compiler-rt/test/gwp_asan/CMakeFiles/check-gwp_asan-unit
cd
/home/jdenny/ornl/llvm/llvm-mono-git-build/projects/compiler-rt/test/gwp_asan
&& /usr/bin/python
/home/jdenny/ornl/llvm/llvm-mono-git-build/./bin/llvm-lit -sv
/home/jdenny/ornl/llvm/llvm-mono-git-build/projects/compiler-rt/test/gwp_asan/unit
Joel
On Thu, Nov 7, 2019 at 4:22 PM Joel E. Denny <jdenny.ornl at gmail.com> wrote:
> Hi Julian,
>
> It looks like your check-all is running lit only once. What subprojects
> do you have enabled? I have most of them enabled.
>
> Thanks.
>
> Joel
>
> On Thu, Nov 7, 2019 at 1:50 PM Julian Lettner <julian.lettner at apple.com>
> wrote:
>
>> Hi Joel,
>>
>> I can’t reproduce your issue. Can you describe what happened?
>> When I run `env LIT_FILTER=xxx ninja check-all` I get the following:
>>
>> ➤ env LIT_FILTER=deadlock ninja check-all
>> [3/6] cd
>> /Users/yln/work/llvm-upstream/llvm-project/clang/binding...work/llvm-upstream/build/lib
>> /usr/bin/python -m unittest discover
>> <snip>
>> [4/6] Checking TSan Go runtime...
>> <snip>
>> [5/6] Running lint check for sanitizer sources...
>> <lots of linter output>
>> Total errors found: 0
>> [5/6] Running all regression tests
>> <lots of lit notes and warnings>
>>
>> Testing Time: 50.04s
>> Expected Passes : 23
>> Expected Failures : 2
>> Unsupported Tests : 5
>>
>> 40 warning(s) in tests.
>>
>> ➤ echo $status
>> 0
>>
>> ➤ env LIT_FILTER=does_not_exist ninja check-all
>> <same as above>
>> [5/6] Running all regression tests
>> <lots of lit notes and warnings>
>> Filter did not match any tests (of 62432 discovered).
>> FAILED: CMakeFiles/check-all
>> cd /Users/yln/work/llvm-upstream/build && /usr/bin/python
>> /Users/yln/work/llvm-upstream/build/./bin/llvm-lit -sv --param
>> clang_site_config=/Users/yln/work/llvm-upstream/build/tools/clang/test/lit.site.cfg
>> --param USE_Z3_SOLVER=0 <long list of test suites>
>> ninja: build stopped: subcommand failed.
>>
>> ➤ echo $status
>> 1
>>
>>
>> On Nov 5, 2019, at 12:32 PM, Joel E. Denny <jdenny.ornl at gmail.com> wrote:
>>
>> Hi Julian,
>>
>> I don't know that I would have thought of the check-all issue during a
>> patch review, but someone might have, and my understanding is that patch
>> reviews are required in LLVM.
>>
>> Thanks for the fast response. I do appreciate your effort to address
>> empty lit runs.
>>
>> Joel
>>
>> On Tue, Nov 5, 2019 at 3:20 PM Julian Lettner <jlettner at apple.com> wrote:
>> Hi Joel,
>>
>> Thanks for letting me know. I reverted the change until I have time to
>> investigate.
>> I didn’t create a review for this change, but I have learned my lesson
>> and will definitely include you in the review for the re-land. Sorry for
>> the churn,
>>
>> Thanks,
>> Julian
>>
>> On Nov 5, 2019, at 10:54 AM, Joel E. Denny <jdenny.ornl at gmail.com> wrote:
>>
>> Hi Julian,
>>
>> This patch breaks LIT_FILTER for check-all, which runs lit multiple
>> times. For such use cases, maybe there should be a command-line option to
>> suppress this error.
>>
>> Also, was there a review for this patch? I don't see one referenced in
>> the commit log.
>>
>> Thanks.
>>
>> Joel
>>
>> On Mon, Nov 4, 2019 at 1:17 PM Julian Lettner via llvm-commits <
>> llvm-commits at lists.llvm.org> wrote:
>>
>> Author: Julian Lettner
>> Date: 2019-11-04T10:16:24-08:00
>> New Revision: d8f2bff75126c6dde694ad245f9807fa12ad5630
>>
>> URL:
>> https://github.com/llvm/llvm-project/commit/d8f2bff75126c6dde694ad245f9807fa12ad5630
>> DIFF:
>> https://github.com/llvm/llvm-project/commit/d8f2bff75126c6dde694ad245f9807fa12ad5630.diff
>>
>> LOG: [lit] Better/earlier errors when no tests are executed
>>
>> Fail early, when we discover no tests at all, or filter out all of them.
>>
>> Added:
>>
>>
>> Modified:
>> llvm/utils/lit/lit/cl_arguments.py
>> llvm/utils/lit/lit/main.py
>> llvm/utils/lit/lit/run.py
>> llvm/utils/lit/tests/selecting.py
>>
>> Removed:
>>
>>
>>
>>
>> ################################################################################
>> diff --git a/llvm/utils/lit/lit/cl_arguments.py
>> b/llvm/utils/lit/lit/cl_arguments.py
>> index 7b15041d1905..c17f6c7824e1 100644
>> --- a/llvm/utils/lit/lit/cl_arguments.py
>> +++ b/llvm/utils/lit/lit/cl_arguments.py
>> @@ -5,6 +5,7 @@
>>
>> import lit.util
>>
>> +# TODO(yln): scan dest, to see if is redundant
>> def parse_args():
>> parser = argparse.ArgumentParser()
>> parser.add_argument('test_paths',
>>
>> diff --git a/llvm/utils/lit/lit/main.py b/llvm/utils/lit/lit/main.py
>> index 337d154a560d..af36b07f1fef 100755
>> --- a/llvm/utils/lit/lit/main.py
>> +++ b/llvm/utils/lit/lit/main.py
>> @@ -42,6 +42,9 @@ def main(builtin_params = {}):
>> echo_all_commands = opts.echoAllCommands)
>>
>> tests = lit.discovery.find_tests_for_inputs(litConfig,
>> opts.test_paths)
>> + if not tests:
>> + sys.stderr.write('Did not disover any tests for provided
>> path(s).\n')
>> + sys.exit(2)
>>
>> # Command line overrides configuration for maxIndividualTestTime.
>> if opts.maxIndividualTestTime is not None: # `not None` is
>> important (default: 0)
>> @@ -62,12 +65,20 @@ def main(builtin_params = {}):
>>
>> if opts.filter:
>> tests = [t for t in tests if opts.filter.search(t.getFullName())]
>> + if not tests:
>> + sys.stderr.write('Filter did not match any tests '
>> + '(of %d discovered).\n' % numTotalTests)
>> + sys.exit(2)
>>
>> determine_order(tests, opts.order)
>>
>> if opts.shard:
>> (run, shards) = opts.shard
>> tests = filter_by_shard(tests, run, shards, litConfig)
>> + if not tests:
>> + sys.stderr.write('Shard does not contain any tests.
>> Consider '
>> + 'decreasing the shard count.\n')
>> + sys.exit(0)
>>
>> if opts.max_tests:
>> tests = tests[:opts.max_tests]
>> @@ -84,7 +95,7 @@ def main(builtin_params = {}):
>> write_test_results_xunit(tests, opts)
>>
>> if litConfig.numErrors:
>> - sys.stderr.write('\n%d error(s), exiting.\n' %
>> litConfig.numErrors)
>> + sys.stderr.write('\n%d error(s) in tests.\n' %
>> litConfig.numErrors)
>> sys.exit(2)
>>
>> if litConfig.numWarnings:
>>
>> diff --git a/llvm/utils/lit/lit/run.py b/llvm/utils/lit/lit/run.py
>> index 4d16399fec47..11992c8073d2 100644
>> --- a/llvm/utils/lit/lit/run.py
>> +++ b/llvm/utils/lit/lit/run.py
>> @@ -13,7 +13,7 @@ def acquire(self): pass
>> def release(self): pass
>>
>> def create_run(tests, lit_config, workers, progress_callback,
>> timeout=None):
>> - # TODO(yln) assert workers > 0
>> + assert workers > 0
>> if workers == 1:
>> return SerialRun(tests, lit_config, progress_callback, timeout)
>> return ParallelRun(tests, lit_config, progress_callback, timeout,
>> workers)
>> @@ -45,9 +45,6 @@ def execute(self):
>> computed. Tests which were not actually executed (for any
>> reason) will
>> be given an UNRESOLVED result.
>> """
>> - if not self.tests:
>> - return 0.0
>> -
>> self.failure_count = 0
>> self.hit_max_failures = False
>>
>>
>> diff --git a/llvm/utils/lit/tests/selecting.py
>> b/llvm/utils/lit/tests/selecting.py
>> index 0921cdd31ac9..cfbb9d19b6dc 100644
>> --- a/llvm/utils/lit/tests/selecting.py
>> +++ b/llvm/utils/lit/tests/selecting.py
>> @@ -1,6 +1,18 @@
>> # RUN: %{lit} %{inputs}/discovery | FileCheck --check-prefix=CHECK-BASIC
>> %s
>> # CHECK-BASIC: Testing: 5 tests
>>
>> +
>> +# Check that we exit with an error if we do not discover any tests.
>> +#
>> +# RUN: not %{lit} %{inputs}/nonexistent 2>&1 | FileCheck
>> --check-prefix=CHECK-BAD-PATH %s
>> +# CHECK-BAD-PATH: Did not disover any tests for provided path(s).
>> +
>> +# Check that we exit with an error if we filter out all tests.
>> +#
>> +# RUN: not %{lit} --filter 'nonexistent' %{inputs}/discovery 2>&1 |
>> FileCheck --check-prefix=CHECK-BAD-FILTER %s
>> +# CHECK-BAD-FILTER: Filter did not match any tests (of 5 discovered).
>> +
>> +
>> # Check that regex-filtering works, is case-insensitive, and can be
>> configured via env var.
>> #
>> # RUN: %{lit} --filter 'o[a-z]e' %{inputs}/discovery | FileCheck
>> --check-prefix=CHECK-FILTER %s
>> @@ -8,6 +20,7 @@
>> # RUN: env LIT_FILTER='o[a-z]e' %{lit} %{inputs}/discovery | FileCheck
>> --check-prefix=CHECK-FILTER %s
>> # CHECK-FILTER: Testing: 2 of 5 tests
>>
>> +
>> # Check that maximum counts work
>> #
>> # RUN: %{lit} --max-tests 3 %{inputs}/discovery | FileCheck
>> --check-prefix=CHECK-MAX %s
>> @@ -68,15 +81,13 @@
>> #
>> # RUN: %{lit} --num-shards 100 --run-shard 6 %{inputs}/discovery >%t.out
>> 2>%t.err
>> # RUN: FileCheck --check-prefix=CHECK-SHARD-BIG-ERR2 < %t.err %s
>> -# RUN: FileCheck --check-prefix=CHECK-SHARD-BIG-OUT2 < %t.out %s
>> # CHECK-SHARD-BIG-ERR2: note: Selecting shard 6/100 = size 0/5 = tests
>> #(100*k)+6 = []
>> -# CHECK-SHARD-BIG-OUT2: Testing: 0 of 5 tests
>> +# CHECK-SHARD-BIG-ERR2: Shard does not contain any tests. Consider
>> decreasing the shard count.
>> #
>> # RUN: %{lit} --num-shards 100 --run-shard 50 %{inputs}/discovery
>> >%t.out 2>%t.err
>> # RUN: FileCheck --check-prefix=CHECK-SHARD-BIG-ERR3 < %t.err %s
>> -# RUN: FileCheck --check-prefix=CHECK-SHARD-BIG-OUT3 < %t.out %s
>> # CHECK-SHARD-BIG-ERR3: note: Selecting shard 50/100 = size 0/5 = tests
>> #(100*k)+50 = []
>> -# CHECK-SHARD-BIG-OUT3: Testing: 0 of 5 tests
>> +# CHECK-SHARD-BIG-ERR3: Shard does not contain any tests. Consider
>> decreasing the shard count.
>>
>>
>> # Check that range constraints are enforced
>>
>>
>>
>> _______________________________________________
>> llvm-commits mailing list
>> llvm-commits at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20191107/7e0e5a22/attachment.html>
More information about the llvm-commits
mailing list