[llvm] [FileCheck] Add a diff output option for FileCheck (PR #187120)

[Fangrui Song via llvm-commits]

MaskRay wrote:

Claude opus 4.7 test gap analysis
```
● Test gap review — commit 40333cde2bbd

  The patch adds 14 lit tests but all follow the pattern %ProtectFileCheckOutput not FileCheck ... --diff=unidiff. The resulting coverage has material holes:

  Biggest gaps (any one would have caught our bug)

  1. No positive test. Every test uses not FileCheck, so the suite only asserts failure output. It never runs --diff against input that should match to verify the run exits 0 with no diff noise. A single RUN: FileCheck %s --input-file=%s --diff=unidiff with valid CHECK-NEXT would have
  caught the MatchPos > 0 false-positive regression immediately. This is the single most important missing shape.
  2. No FILECHECK_OPTS=--diff path. The documented consumption model (FILECHECK_OPTS='…' ninja check-*) flips the flag on for every existing test in-tree, which is how the bug surfaced for you. llvm/test/FileCheck/envvar-opts.txt already exercises the env var — a diff analog is
  missing.
  3. No default-value test. Docs promise --diff without =value defaults to unidiff (the cl::values(clEnumValN(Unified, "", "")) line). Not covered.

  Directive coverage holes

  - CHECK-SAME, CHECK-NOT, CHECK-DAG, CHECK-COUNT — none tested under --diff. CHECK-NOT failure semantics are particularly non-obvious (the "failure line" isn't a mismatch of expected-vs-actual but an unwanted match), and handleDiffFailure has no carve-out for it.

  Feature-interaction holes

  - Variables / substitutions. PR explicitly says "only have unified with no substitution," but printDiff still renders check lines that may contain [[VAR:re]], [[VAR]], [[#NUM+1]] — behavior is unspecified and untested. The one regex test (diff-regex-escaping.txt) only exercises a
  literal {{U}}.
  - --dump-input suppression. Commit changes utils/FileCheck/FileCheck.cpp:869 so dump-input fires only when DiffFormat == None. No test covers --diff + --dump-input=always / =fail.
  - --strict-whitespace, --match-full-lines, multiple --check-prefixes, --implicit-check-not — all untested interactions.
  - --diff=none — the explicit opt-out isn't exercised directly.

  Minor

  - Color codes are emitted by handleDiffFailure/the trailer but never asserted on; %ProtectFileCheckOutput strips TTY, so there's no coverage of the ANSI path.
  - No stdin-input case (- / omitted --input-file), no CRLF input, no very-first-CHECK-fails-before-any-CHECK-LABEL variant (current tests all start after a CHECK or CHECK-LABEL).

  Bottom line

  Gap #1 (no positive/success test) is the decisive one: had the PR included a single "passing CHECK-NEXT under --diff" case, the MatchPos > 0 heuristic would never have merged. Gaps #2–#3 would catch regressions in the env-var and default-value surfaces, and the substitution +
  CHECK-NOT/DAG gaps are where the next class of bugs is most likely to hide.
```

https://github.com/llvm/llvm-project/pull/187120