[clang] [clang][doc] Add documentation for the ASTs used to represent C++ templates (PR #66436)

[Scott McPeak via cfe-commits]

smcpeak wrote:

# On the utility of the diagrams

I'm surprised this is a point of contention.  Could someone elaborate on why the diagrams are troublesome?  To me, saying that the diagrams are not useful is sort of like saying the `-ast-dump` output is not useful.  Isn't it better to be able to see all of the relevant AST objects in play at once?

Perhaps the problem is simply that the diagram notation is not adequately explained.  I see that I never did explain it; that was an oversight.  I therefore propose to add the following text right after the first diagram, in the section "Diagram: Function template: Definition".  Would this addition resolve objections to the diagrams?  Should I add even more detail about the notation, for example, discussing each of the objects, pointers, and fields that is shown individually?

## Proposed additional text below ft-defn.ded

In this diagram, each box (except "Source Code") represents a single object in the AST that was built by the Clang parser and semantic analyzer, each arrow represents a single pointer from one AST object to another, and each item inside a box represents a single data field of the containing object.  For example, the "TU" box represents the `TranslationUnitDecl` object.  That object has a pointer called `DeclContext::LastDecl` that points at the `FunctionTemplateDecl 14` object.  The `FunctionTemplateDecl 14` object has a field called `Name` with value `"identity"` when rendered as a string.

Object names include their type, such as `FunctionTemplateDecl`, and a unique numeric identifier (such as `14`) to distinguish them from other objects with the same type and facilitate naming them in the prose.

The diagram can be understood as a complement to the `-ast-dump` output, which instead has one line per object.  The diagram does not show all of the AST objects because that would be too much clutter, and the point is to focus on a few key interactions.  But for the nodes that are shown, the diagram includes details that `-ast-dump` omits, especially the pointer relations that are not part of the tree backbone.

# On generating the diagrams during the build

I think it is preferable to check in the PNG files than to regenerate them on every build.  Regenerating them would mean all developers who change documentation would need the `ded` tool, which is written in Java.  If it's checked in to Clang, that means adding a JDK dependency (as Clang currently has no Java code).  If not, the editor itself is then an extra (documentation) build-time dependency for people to obtain and install.

Regenerating the PNGs avoids a potential issue where the DED and PNG are out of sync, but this is unlikely to happen in practice because the `ded` tool always saves both at the same time.

Regarding size, the PNG files are usually not larger than the DED files, so removing them will not save a lot of space in the repo.

Also, if space is a concern, both the DED and PNG files can be "trimmed" somewhat, removing elements of the underlying AST graph that are not shown in the diagram, thereby reducing file size (by about half for the larger DEDs, and maybe 10% for the PNGs).  I didn't do that in this initial submission to make it easier for others to add information to a diagram if they thought something was missing.

Furthermore, it is possible to remove the DED files entirely, since `ded` can directly edit the PNG files without loss of information (the contents of the DED file are embedded as a compressed comment in the PNG).  The reason to keep the DED files is it's possible to see what has changed when reviewing changes in `git` and, in some cases, merge conflicting changes automatically (and even when they cannot be merged automatically, it's helpful to see the diffs in order to perform a manual merge).


https://github.com/llvm/llvm-project/pull/66436