[llvm-dev] RFC: FileCheck debugging changes and CI configs
Joel E. Denny via llvm-dev
llvm-dev at lists.llvm.org
Fri Jul 10 12:05:16 PDT 2020
Hi,
# TLDR #
FileCheck just became less noisy on failures by default. However, for
some cases, it now provides too little debug info, so new
configuration options have landed. When debugging CI failures, many
LLVM committers have no feasible way to adjust these options, so I
think LLVM CI configs should include the following environment setting
to maximize the debug info:
```
FILECHECK_OPTS='-dump-input-filter=all -vv -color'
```
# Background: FileCheck Input Dumps by Default #
D81422 recently landed to enable `-dump-input=fail` by default in
FileCheck. As a result, when FileCheck fails, it by default produces
an annotated input dump like the following (which is intended to be
viewed in a monospaced font):
```
<<<<<<
1: foo
2: foo
3: foo
4: foo
5: foo
6: foo
7: foo
8: foo
9: foo
10: foo
11: foo
12: foo
13: foo
14: foo
15: hello world
16: foo
check:2'0 X~~ error: no match found
17: foo
check:2'0 ~~~
18: foo
check:2'0 ~~~
19: foo
check:2'0 ~~~
20: foo
check:2'0 ~~~
21: foo
check:2'0 ~~~
22: foo
check:2'0 ~~~
23: foo
check:2'0 ~~~
24: foo
check:2'0 ~~~
25: foo
check:2'0 ~~~
26: foo
check:2'0 ~~~
27: foo
check:2'0 ~~~
28: foo
check:2'0 ~~~
29: foo
check:2'0 ~~~
30: goodbye word
check:2'0 ~~~~~~~~~~~~
check:2'1 ? possible intended match
31: foo
check:2'0 ~~~
32: foo
check:2'0 ~~~
33: foo
check:2'0 ~~~
34: foo
check:2'0 ~~~
35: foo
check:2'0 ~~~
36: foo
check:2'0 ~~~
37: foo
check:2'0 ~~~
38: foo
check:2'0 ~~~
39: foo
check:2'0 ~~~
40: foo
check:2'0 ~~~
>>>>>>
```
LLVM developers' views on this change vary. See the following
discussions:
* <https://reviews.llvm.org/D81422>
* Thread starting at: <
http://lists.llvm.org/pipermail/llvm-dev/2020-June/142369.html>
## The Good ##
Some people feel this change is a significant improvement for
debugging FileCheck errors. It's especially helpful for those who
aren't aware of the `-dump-input=fail` option or the `FILECHECK_OPTS`
environment variable, which can be used to set such debugging options.
This change is perhaps even more helpful in the case of CI
configurations that don't set `-dump-input=fail`. In that case, upon
a test failure, it is not feasible for many LLVM committers to set
`-dump-input=fail` and run a failing test again on precisely the same
platform in order to get the debugging info they need. Now they don't
have to.
## The Bad ##
Others feel this dump produces far too much noise to be useful,
especially when the FileCheck input is large and the test suite is
being run in a terminal window. Sometimes the actual error message is
buried thousands of lines earlier in a terminal's scrollback or may
have even overflowed the scrollback buffer.
# Fix 1: Filtered Input Dumps #
In an attempt to satisfy both views, D82203 just landed to filter the
input dump to just error diagnostics plus some context. Thus, the
previous example now looks like the following by default:
```
<<<<<<
.
.
.
11: foo
12: foo
13: foo
14: foo
15: hello world
16: foo
check:2'0 X~~ error: no match found
17: foo
check:2'0 ~~~
18: foo
check:2'0 ~~~
19: foo
check:2'0 ~~~
20: foo
check:2'0 ~~~
21: foo
check:2'0 ~~~
.
.
.
25: foo
check:2'0 ~~~
26: foo
check:2'0 ~~~
27: foo
check:2'0 ~~~
28: foo
check:2'0 ~~~
29: foo
check:2'0 ~~~
30: goodbye word
check:2'0 ~~~~~~~~~~~~
check:2'1 ? possible intended match
31: foo
check:2'0 ~~~
32: foo
check:2'0 ~~~
33: foo
check:2'0 ~~~
34: foo
check:2'0 ~~~
35: foo
check:2'0 ~~~
.
.
.
>>>>>>
```
## The Good ##
Unless there are many FileCheck errors (e.g., one FileCheck run can
report an error per `CHECK-LABEL`), the input dump is now relatively
small. Thus, this change supports the original motivation to enable
`-dump-input=fail` by default but avoids being overwhelming when the
full input is large.
## The Bad ##
I find that it is *often* true that the default amount of context, as
shown in the example above, is far from sufficient. For example,
often there is no `possible intended match` or it's a bad guess, and
the intended match is hundreds of lines away from any error
diagnostic. Even worse, the actual bug is often at a different
directive than the directive for which FileCheck reports an error. In
other words, the information I need is often filtered out, so I need
to be able to configure the input dump filter.
# Fix 2: Filter Configuration #
D83097 and D82203 have landed to implement `-dump-input-filter` and
`-dump-input-context` for configuring the filtered input lines (that
is, include more than just errors) and their lines of context. See
`FileCheck -help` for details.
## Where to Specify Them ##
Like other FileCheck debugging options, these can be set via
`FILECHECK_OPTS` either globally (e.g., in a `~/.profile`) or per test
suite run (e.g., `FILECHECK_OPTS=-dump-input-context=100 ninja
check`). They can even be set as command-line options in specific
FileCheck calls within specific tests in case those calls have very
specific debugging needs. In that case, they won't conflict with
options in `FILECHECK_OPTS` because multiple occurrences of an option
are resolved in favor of the option requesting the most debugging
info.
## Is This Too Configurable? ##
It has been suggested that the configuration space provided by
`-dump-input-filter` and `-dump-input-context` is too large. We could
have instead reduced it to a simple boolean: filter as shown above by
default, or show the full input if requested. However, I anticipate
that the default amount of context, no matter what it is, will be too
frequently wrong for some LLVM developers' use cases. On the other
hand, a full input dump can be overwhelming even when enabled for a
single test. I think the ability to select levels in between these
two extremes is important. Finally, in my opinion, this extra
configurability adds minimal complexity to the FileCheck
implementation. Most of the implementation needed is already needed
to support just the default configuration.
# Fix 3: Filter Configuration in CI #
As I mentioned above, upon a test failure in CI, it is not feasible
for many LLVM committers to set `FILECHECK_OPTS` and run the failing
test again on precisely the same platform in order to configure the
precise information they need for debugging. In my experience, the
inability to request sufficient debugging information is a much more
severe problem than encountering too much debugging information. For
these reasons, I think LLVM CI configs should maximize the debugging
info upon FileCheck failures by setting:
```
FILECHECK_OPTS='-dump-input-filter=all -vv -color'
```
I use this successfully in my local Gitlab CI configs. Hopefully
others will adopt it too.
Thanks.
Joel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200710/d343bfe9/attachment.html>
More information about the llvm-dev
mailing list