[PATCH] D141508: [utils][filecheck-lint] Add filecheck-lint, a linter for FileCheck directives.

[Benjamin Chetioui via Phabricator via llvm-commits]

bchetioui created this revision.
bchetioui added a reviewer: sammccall.
Herald added subscribers: kosarev, thopre.
Herald added a project: All.
bchetioui requested review of this revision.
Herald added a project: LLVM.
Herald added a subscriber: llvm-commits.

FileCheck is a versatile tool for testing compiler output regardless of syntax.
FileCheck achieves this versatility by processing only input lines in which it
detects a valid CHECK directive, and ignoring everything else.

This comes at a price: if a directive is not typed properly, it is not
processed by FileCheck, and the test code it precedes is effectively disabled.
This results in the illusion that the code is tested, while the test may have
actually never run.

This scenario is not hypothetical, see the fixes introduced in, e.g.
https://github.com/tensorflow/tensorflow/commit/48cacf049f3d6ed3f289ccc48ec50491b6d8d9a8,
https://reviews.llvm.org/D139698, https://reviews.llvm.org/D139636,
https://github.com/iree-org/iree/pull/11693.

The findings corrected in the above changes originate in filecheck-lint.
In a given test file, filecheck-lint uses the edit distance between anything
that looks like a FileCheck directive and the set of locally valid directives
to detect likely misspelled directives.

The tool is not yet feature complete---e.g. it does not parse custom comment
prefixes to exclude them from reporting---but it already yields useful results,
as demonstrated above.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D141508

Files:
  llvm/utils/filecheck_lint/README.md
  llvm/utils/filecheck_lint/filecheck_lint.py
  llvm/utils/filecheck_lint/filecheck_lint/filecheck_lint.py
  llvm/utils/filecheck_lint/tests/filecheck_lint_test.py

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D141508.488229.patch
Type: text/x-patch
Size: 16852 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20230111/11c40894/attachment.bin>