[clang] clang: Relax LangOpts checks when lexing quoted numbers during preprocessing (PR #95798)

[Aaron Ballman via cfe-commits]

================
@@ -2068,7 +2068,8 @@ bool Lexer::LexNumericConstant(Token &Result, const char *CurPtr) {
   }
 
   // If we have a digit separator, continue.
-  if (C == '\'' && (LangOpts.CPlusPlus14 || LangOpts.C23)) {
+  if (C == '\'' &&
+      (LangOpts.CPlusPlus14 || LangOpts.C23 || ParsingPreprocessorDirective)) {
----------------
AaronBallman wrote:

> We've been trying to avoid specializing Clang itself for dependency scanning. Instead, we have introduced generic feature flags that the scanner uses to tweak the compiler behavior.

That's still specializing Clang for dependency scanning though. We don't typically *want* generic feature flags that suggest an a la carte approach to the language features, so those are only introduced for dependency scanning unless we happen to have a feature flag allowing the feature to be enabled or disabled by the user.

>  People working on Clang don't have to wonder what's dependency scanning and why it needs to lex numeric literals differently; they just see there's a feature flag named `AllowLiteralDigitSeparator` that just makes sense in that context. You can still dive deeper and find that it's the scanner that enables that feature of course.

That's actually the problem though -- folks are telling you that we feel the need to dive deeper when seeing those sort of flags because we need to figure out under what conditions the flag is or is not set in order to determine whether we have appropriate test coverage for changes. Using your example, `AllowLiteralDigitSeparators` instead of looking at the language standard means that reviewers have to ask questions like "so what happens when SYCL requires C++20 but `AllowLiteralDigitSeparators` is set to false?" or similar. Tying language features to language modes makes this far easier for Clang developers to reason about -- C++20 mode implies C++14 mode and so digit separators always work.

https://github.com/llvm/llvm-project/pull/95798