[clang] [clang-tools-extra] [clangd] Handle C++20 annot_module_name token and don't discard successfully built modules when on fail (PR #187858)

[Yihan Wang via cfe-commits]

================
@@ -681,16 +681,25 @@ class TokenCollector::CollectPPExpansions : public PPCallbacks {
 TokenCollector::TokenCollector(Preprocessor &PP) : PP(PP) {
   // Collect the expanded token stream during preprocessing.
   PP.setTokenWatcher([this](const clang::Token &T) {
-    if (T.isAnnotation())
+    if (T.is(tok::annot_module_name)) {
+      auto &SM = this->PP.getSourceManager();
+      StringRef Text = Lexer::getSourceText(
+          CharSourceRange::getTokenRange(T.getAnnotationRange()), SM,
+          this->PP.getLangOpts());
+      Expanded.push_back(
+          syntax::Token(T.getLocation(), Text.size(), tok::annot_module_name));
+    } else if (T.isAnnotation()) {
       return;
+    } else {
+      Expanded.push_back(syntax::Token(T));
+    }
----------------
yronglin wrote:

This because the token collector will ignore all the annotation tokens, but IIUC, the annot_module_name shoulde not be ignored. 

Maybe we should have a separate PR to handling this.

https://github.com/llvm/llvm-project/pull/187858