[clang] [clang-tools-extra] [clang] Don't add documentation comments to the AST if not requested (PR #206363)

[Corentin Jabot via cfe-commits]

================
@@ -2703,11 +2705,37 @@ LambdaScopeInfo *Sema::getCurGenericLambda() {
   return nullptr;
 }
 
+bool Sema::shouldRetainCommentsFromLexer(SourceLocation Loc) const {
+  if (LangOpts.CommentOpts.ParseAllComments)
+    return true;
+
+  if (PP.getPreprocessorOpts().RetainComments)
+    return true;
+
+  // When building a PCH the comments are serialized into the AST file
+  // so downstream consumers like clangd) can retrieve documentation, and the
+  // incremental/REPL front end may query them interactively.
+  if (TUKind != TU_Complete)
+    return true;
+
+  if (PP.isCodeCompletionEnabled())
+    return true;
+
+  // Keep the comment if -Wdocumentation is enabled at its location (checking
+  // the location handles warnings turned on by `#pragma clang diagnostic`).
+  if (!Diags.isIgnored(diag::warn_doc_param_not_found, Loc) ||
+      !Diags.isIgnored(diag::warn_unknown_comment_command_name, Loc))
+    return true;
+
+  return false;
+}
 
 void Sema::ActOnComment(SourceRange Comment) {
   if (!LangOpts.RetainCommentsFromSystemHeaders &&
       SourceMgr.isInSystemHeader(Comment.getBegin()))
     return;
+  if (!shouldRetainCommentsFromLexer(Comment.getBegin()))
+    return;
----------------
cor3ntin wrote:

We have -

RetainCommentsFromSystemHeaders in LangOpts
ParseAllComments in LangOpts.CommentOpts
and RetainComments in PreprocessorOptions


Can we maybe clean that up by having the new option in `CommentOpts` and potentially move 
`RetainCommentsFromSystemHeaders` ?

Lets also make sure these things are handle at the same places

https://github.com/llvm/llvm-project/pull/206363