[all-commits] [llvm/llvm-project] f702c7: [utils][filecheck-lint] Add filecheck-lint, a lint...

[Benjamin Chetioui via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: f702c75995db9ea3978d39e3f3885b2113356f68
      https://github.com/llvm/llvm-project/commit/f702c75995db9ea3978d39e3f3885b2113356f68
  Author: Benjamin Chetioui <bchetioui at google.com>
  Date:   2023-01-12 (Thu, 12 Jan 2023)

  Changed paths:
    A llvm/utils/filecheck_lint/filecheck_lint.py
    A llvm/utils/filecheck_lint/filecheck_lint_test.py

  Log Message:
  -----------
  [utils][filecheck-lint] Add filecheck-lint, a linter for FileCheck directives.

FileCheck is a versatile tool for testing compiler output regardless of syntax.
FileCheck achieves this versatility by processing only input lines in which it
detects a valid CHECK directive, and ignoring everything else.

This comes at a price: if a directive is not typed properly, it is not
processed by FileCheck, and the test code it precedes is effectively disabled.
This results in the illusion that the code is tested, while the test may have
actually never run.

This scenario is not hypothetical, see the fixes introduced in, e.g.
https://github.com/tensorflow/tensorflow/commit/48cacf049f3d6ed3f289ccc48ec50491b6d8d9a8,
https://reviews.llvm.org/D139698, https://reviews.llvm.org/D139636,
https://github.com/iree-org/iree/pull/11693.

The findings corrected in the above changes originate in filecheck-lint.
In a given test file, filecheck-lint uses the edit distance between anything
that looks like a FileCheck directive and the set of locally valid directives
to detect likely misspelled directives.

The tool is not yet feature complete---e.g. it does not parse custom comment
prefixes to exclude them from reporting---but it already yields useful results,
as demonstrated above.

Differential Revision: https://reviews.llvm.org/D141508