[clang] 3ab238a - [AST Matchers] Update dump_ast_matchers.py to query only class index page.

[Yitzhak Mandelbaum via cfe-commits]

Author: James King
Date: 2021-10-08T17:45:10Z
New Revision: 3ab238af4fa44a12ff3d479dc7fac50332ad2875

URL: https://github.com/llvm/llvm-project/commit/3ab238af4fa44a12ff3d479dc7fac50332ad2875
DIFF: https://github.com/llvm/llvm-project/commit/3ab238af4fa44a12ff3d479dc7fac50332ad2875.diff

LOG: [AST Matchers] Update dump_ast_matchers.py to query only class index page.

It may be possible to avoid relying on accessing many individual class pages,
by instead scanning the class index page at
https://clang.llvm.org/doxygen/classes.html. This updates the script to do so,
and includes updates to `LibASTMatchersReference.html` generated by the
modified script.

Reviewed By: aaron.ballman, sammccall

Differential Revision: https://reviews.llvm.org/D111332

Added: 
    

Modified: 
    clang/docs/tools/dump_ast_matchers.py

Removed: 
    


################################################################################
diff  --git a/clang/docs/tools/dump_ast_matchers.py b/clang/docs/tools/dump_ast_matchers.py
index 2a26d10f7a04d..2ac0af1f38bca 100755
--- a/clang/docs/tools/dump_ast_matchers.py
+++ b/clang/docs/tools/dump_ast_matchers.py
@@ -10,6 +10,12 @@
 except ImportError:
     from urllib2 import urlopen
 
+CLASS_INDEX_PAGE_URL = 'https://clang.llvm.org/doxygen/classes.html'
+try:
+  CLASS_INDEX_PAGE = urlopen(CLASS_INDEX_PAGE_URL).read()
+except Exception as e:
+  raise Exception('Unable to get %s: %s' % (CLASS_INDEX_PAGE_URL, e))
+
 MATCHERS_FILE = '../../include/clang/ASTMatchers/ASTMatchers.h'
 
 # Each matcher is documented in one row of the form:
@@ -40,15 +46,18 @@ def esc(text):
   text = re.sub(r'<', '<', text)
   text = re.sub(r'>', '>', text)
   def link_if_exists(m):
+    """Wrap a likely AST node name in a link to its clang docs.
+
+       We want to do this only if the page exists, in which case it will be
+       referenced from the class index page.
+    """
     name = m.group(1)
     url = 'https://clang.llvm.org/doxygen/classclang_1_1%s.html' % name
     if url not in doxygen_probes:
-      try:
-        print('Probing %s...' % url)
-        urlopen(url)
-        doxygen_probes[url] = True
-      except:
-        doxygen_probes[url] = False
+      search_str = 'href="classclang_1_1%s.html"' % name
+      doxygen_probes[url] = search_str in CLASS_INDEX_PAGE
+      if not doxygen_probes[url]:
+        print('Did not find %s in class index page' % name)
     if doxygen_probes[url]:
       return r'Matcher<<a href="%s">%s</a>>' % (url, name)
     else: