[clang] [clang][Diagnostics] Highlight code snippets (PR #66514)

[Aaron Ballman via cfe-commits]

Timm =?utf-8?q?Bäder?= <tbaeder at redhat.com>,
Timm =?utf-8?q?Bäder?= <tbaeder at redhat.com>,
Timm =?utf-8?q?Bäder?= <tbaeder at redhat.com>,
Timm =?utf-8?q?Bäder?= <tbaeder at redhat.com>,
Timm =?utf-8?q?Bäder?= <tbaeder at redhat.com>,
Timm =?utf-8?q?Bäder?= <tbaeder at redhat.com>,
Timm =?utf-8?q?Bäder?= <tbaeder at redhat.com>,
Timm =?utf-8?q?Bäder?= <tbaeder at redhat.com>
Message-ID:
In-Reply-To: <llvm/llvm-project/pull/66514/clang at github.com>


AaronBallman wrote:

> Of course it's more expensive, I was just trying to make a point :) I expect this to be only visible in situations where very few errors are ever shown, i.e. when an actual human compiles a project. Build servers redirect the output to a file anyway, which disables any sort of colored output.

I don't think relexing the entire file is an approach we should take; for example, generated code can be *huge* and users may not even care about the diagnostics that come from it. Or the developer might be doing a unity build, etc.

Also, redirecting output to a file doesn't always disable color output. From our precommit CI pipeline: https://buildkite.com/llvm-project/clang-ci/builds/4334#018af546-9a63-45f1-a8f9-6262f3d45b75 -- click on the raw log and you'll see it has contents like:
```
_bk;t=1696332172990~~~ Running global pre-checkout hook
_bk;t=1696332172991$ /etc/buildkite-agent/hooks/pre-checkout
_bk;t=1696332173007BUILDKITE_REPO: https://github.com/llvm-premerge-tests/llvm-project.git

_bk;t=1696332173008current remote URL: https://github.com/llvm-premerge-tests/llvm-project.git

_bk;t=1696332173010GC counter 16/20

_bk;t=1696332173026$ cd /var/lib/buildkite-agent/builds/linux-56-7f758798dd-v9p4k-1/llvm-project/clang-ci
_bk;t=1696332173026# OLDPWD added
_bk;t=1696332173026# PWD changed
```

> TBH I was fine with only lexing the current line. Yes it's not 100% correct but probably 99.5%. We could also just use the current line and go back 10 lines or 100 or whatever and will probably _never_ get a bug report for wrong highlighting.

I don't think 99.5% is what we should aim for here; Richard's earlier comment is spot-on:

> So I suppose part of what we need to decide here is how much imperfection we're OK with. I think this highlighting will become important feedback to developers to help them see how Clang is interpreting their code, so I think it's important that the highlighting is reliable. If the highlighting is weird / wrong, the developer will assume that Clang is interpreting the code in that weird / wrong way.

Did you investigate what Richard was suggesting with caching "we should be back to normal lexing here" information and re-lexing from that point? That would make me feel a lot more comfortable than re-lexing the entire file.

https://github.com/llvm/llvm-project/pull/66514