<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Henry,<div><br></div><div>it is true that the separate ASTContexts for each symbol file can cause redundancies between different symbol files. However, we avoid rendering types from the debug information into the relevant AST context until these types are needed (either by direct request or through a deep copy) so although there is some memory bloat (from indexes, etc. in the actual DWARF parsing code, not in the ASTContext) we won't necessarily actually create those redundancies.</div><div><br></div><div>However, if for example you hit a breakpoint in code in shared library A, access a local variable that has a type that's defined in that library, and then continue and hit a breakpoint in shared library B, and access a local variable of the same type, it's likely that we will use the definition from B.</div><div><br></div><div>A very nice feature we would like to see is the ability to generate hashes of types that would allow us to recognize that there is a type we have already loaded elsewhere that exactly matches what we're about to load. That would thoroughly eliminate these redundancies and probably even allow us to get rid of the indexing bloat.</div><div><br></div><div>Sean</div><div><br></div><div><br></div><div><div><div>On Aug 16, 2012, at 11:23 AM, Henry Abbey <<a href="mailto:hsabbey@gmail.com">hsabbey@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Sean,<br><br>This is promising news, as a follow up question to your response I was wondering how/if redundancies are handled in the ASTContext? Since, if i understand correctly, the same AST structures can easily be in many different ASTContexts associated with the different symbol files for the different executables. Is this handled in the ClangASTImporter or is this not a memory concern because of your other mitigation factors (not always using deep copies).<br>
<br>--henry abbey<br><br><div class="gmail_quote">On Thu, Aug 16, 2012 at 12:53 PM, Sean Callanan <span dir="ltr"><<a href="mailto:scallanan@apple.com" target="_blank">scallanan@apple.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Henry,<br>
<br>
I'd be happy to give you a quick overview.<br>
<br>
–<br>
<br>
In order to optimize compilation time and memory usage, Clang keeps Types, Decls, and Stmts in ASTContexts. ASTContexts overload the placement new operator with a custom bump allocator that allows these objects to be associated tightly with them, and allow a single free to destroy everything associated with that ASTContext.<br>
<br>
As a side effect of this structure, ASTContexts are insert-only. This means that they will grow in size over time, and if you ever want to have "temporary" Types/Decls, they will need to live in their own ASTContext and deleted as a unit.<br>
<br>
The Clang parser assumes that all declarations and types it deals with are in a single ASTContext.<br>
<br>
LLDB loads debug information from many files and wants to be able to unload it as well. That means that we create an independent AST context for each symbol file, and create/delete them at will. This is great for memory usage, and our own type handling functions all know how to deal with types from different AST contexts.<br>
<br>
We embed Clang into LLDB to do our expression parsing, though, and when an expression uses a Decl or a Type that lives in a symbol file, that Decl or Type needs to be moved to the ASTContext that Clang is using for parsing. This is a "deep copy" – everything that the Type or Decl refers to needs to be potentially moved into the parser's ASTContext.<br>
<br>
That's where the ASTImporter comes in. The ASTImporter knows how to "deep copy" entities between ASTContexts. In addition, it has a "minimal" mode where it allows us to provide particular chunks of the abstract syntax graph on demand rather than deep copying everything. LLDB relies heavily on this "minimal" mode and when it doesn't work the Clang parser can crash or produce bizarre parse errors.<br>
<br>
LLDB actually has several types of ASTContexts:<br>
<br>
(1) the ASTContexts for entities in symbol files, one per symbol file;<br>
(2) the ASTContexts to parse an expression, one per expression; and<br>
(3) the scratch ASTContexts, which contains types and Decls that were created by the user (for example, by declaring a type in an expression) but need to persist beyond the lifetime of the expression (for example, if the user created an expression result variable of that type).<br>
<br>
LLDB's use of AST importers is managed by the ClangASTImporter class. It maintains origin information for all Decls in ASTContexts it manages, and dynamically allocates Minions to handle copying between specific ASTContexts. Inside the ClangASTImporter class is a custom subclass of ASTImporter called Minion. ClangASTImporter uses Import(Decl *) and Import(QualType) directly on its Minions, which call through to much of the rest of the ASTImporter interface. Each minion also overloads Imported(), which is a callback that indicates what portions of the AST have been copied. When they receive this callback, they update the ClangASTImporter keep the origin information up to date.<br>
<br>
–<br>
<span><font color="#888888"><br>
Sean<br>
</font></span><div><br>
On Aug 16, 2012, at 10:15 AM, <a href="mailto:habbey@cs.wisc.edu" target="_blank">habbey@cs.wisc.edu</a> wrote:<br>
<br>
</div><div>> Hello,<br>
><br>
> I'm a graduate student at the University of Wisconsin Madison and am<br>
> beginning work on a static analyzer using clang. I've been looking at the<br>
> ASTImporter class of clang and noticed that it doesn't seem to be complete<br>
> and isn't used directly by any part of clang, but that LLDB does use it.<br>
> I was wondering if anyone could help me understand what parts of the<br>
> ASTImporter class LLDB uses and for what overall purpose. If anyone has<br>
> even a little bit of insight on this topic it'd be much appreciated.<br>
><br>
> Thank you<br>
><br>
> --henry abbey<br>
><br>
> _______________________________________________<br>
> lldb-dev mailing list<br>
> <a href="mailto:lldb-dev@cs.uiuc.edu" target="_blank">lldb-dev@cs.uiuc.edu</a><br>
> <a href="http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev</a><br>
<br>
</div></blockquote></div><br>
</blockquote></div><br></div></body></html>