[lldb-dev] Trying to understand symbol importing and its relationship to ASTs

[Greg Clayton]

> On Jul 22, 2014, at 11:41 AM, Randy Smith <rdsmith at chromium.org> wrote:
> 
> 
> I'm chasing a crash in lldb, and my current "that doesn't seem right" has to do with a conflict between a decl and its origin decl (the transformation done at the beginning of tools/lldb/source/Expression/ClangASTSource.cpp:ClangASTSource::layoutRecordType()).  So I'm trying to understand how decls and origin decls get setup during the symbol import process.  Can anyone give me a sketch/hand?  Specific questions include: 
> * There are multiple ASTContexts involved (e.g. the src and dst contexts in the signature of tools/lldb/source/Symbol/ClangASTImporter.cpp:ClangASTImporter::CopyType); do those map to compilation units, or to shared library modules?  Is there a simple way to tell what CU/.so an ASTContext maps to?

Every executable file is represented by a lldb_private::Module (this includes both executables and shared libraries) and each lldb_private::Module has its own ASTContext (one per module, and all compilation units are all represented in one big ASTContext). The DWARF debug info is parsed and it creates types in the ASTContext in the corresponding lldb_private::Module.

> * Does a decl always have an origin decl, even if it was loaded from an ASTContext (?) that has a complete definition?

Origin decl is so we know where a decl originally came from because the definition might not yet be complete (think "class Foo;") and might need to be completed. A little background on how we lazily parse classes.

When someone needs a type, we parse the type (SymbolFileDWARF::ParseType). If that type is a class we always just parse a forward decl to the class ("class Foo;"). The DWARF parser (SymbolFileDWARF) implements clang::ExternalASTSource so it can complete a type only when the compiler needs to know more. When the compiler or ClangASTType needs to know more about a type it asks the type to get a complete version of itself and SymbolFileDWARF::CompleteTagDecl is called to complete the type. We then parse all ivars, methods, and everything else about a type. We also assist in laying out the CXXRecordDecl by another callback SymbolFileDWARF::LayoutRecordType (which is part of the clang::ExternalASTSource). We need to assist in laying things out because the DWARF debug info doesn't always include all required attributes or #pragma information in order for us to create the types correctly. So this SymbolFileDWARF::LayoutRecordType allows us to tell the compiler about the offsets of ivars so they are always correct.

Back to origin decls: When running an expression we create a new ASTContext that is for the expression only. decls are copied from the ASTContext for the lldb_private::Module over into the ASTContext for the expression. When they are copied, only a forward decls are copied, and they may need to be completed. When this happens we might need to ask the type in the original ASTContext to complete itself so that we can copy a complete definition over into the expression ASTContext. This is the reason we track the origin decls. Sometimes you have a type that is only a forward decl, and that is ok as we don't always have the full definition of a class.

> * When an origin decl is looked up, should all the types in it be completed, or might it have incomplete types?  It seems as if there is code assuming that these types will always be complete.

There are two forms of incomplete types:
1 - incomplete types that have full definitions and just haven't been completed (and might have to find the original decl, ask it to complete itself, then copy the origin decl when the current decl needs to be copied from one AST to another)
2 - types that are actually forward declarations and will be told they are just forward decls

So we sometimes do run into cases where we don't have the debug info for something because the compiler pulled it out trying to minimize the debug info.

> 
> Context (warning, gets detailed, possibly with irrelevant details because newbie): lldb is crashing in clang::ASTContext::getASTRecordLayout with the assertion "Cannot get layout of forward declarations!".  The type in question is an incomplete type (string16, aka. basic_string<unsigned short, ...>).  Normally clang::ASTContext::getASTRecordLayout() would call getExternalSource()->CompleteType() to complete the type, but in this case it isn't because the type is marked as !hasExternalLexicalStorage().  

That mean the type was not complete in the DWARF for the lldb_private::Module it originates from.
> 
> The *weird* thing is that the type has previously been completed, further up the stack, but in a different AST node (same name).  In more detail: Class A contains an instance of class B contains an instance of class C (==string16).  I'm seeing getASTRecordLayout called on class A, which then calls it (indirectly, though the EmptySubobjectMap construtor) on class B, which then calls it (ditto) on class C (all works).  Then the stack unwinds up to the B call, which proceeds to the Builder.Layout() line in that function.  It ends up (through the transformation mentioned above in clang::ClangASTSource::LayoutRecordType()) calling getASTRecordLayout() on the origin decl.  When it recurses down to class C, that node isn't complete, isn't completed, and causes an assertion.  So I'm trying to figure out whether the problem is that any decl hanging off an origin_decl should be complete, or that that node shouldn't be marked as !hasExternalLexicalStorage().  (Or something else; I've already gone through several twists and turns debugging this problem :-}.)

We have a problem in the compiler currently where for classes like:

class A : public B
{
    ...
}

The compiler says "ahh, you didn't use class B so I am not going to emit debug info for it.". This really can hose us up because we now create a ASTContext for the expression and we want a definition for "A" and the user wants to call a method that is in class "B", but we can't because the compiler removed the definition. What we currently do is figure out that we have a forward declaration to "B" only, and when we create type "A" in the module's ASTContext, we say "B" is an empty class with no ivars and no methods. To fix this, you can specify "-fstandalone-debug" to the clang compiler to tell it not to do this removal of debug info for things that are inherited from.

The other problem we have is say you two modules "foo.dylib" and "bar.dylib", both have debug info, and "foo.dylib" has debug info with a complete "A" and complete "B" definition, but "bar.dylib" has a complete "A" definition, but only a forward "B" definition. The ASTContext for foo.dylib believes class "A" to look like it really is, and "bar.dylib" has a definition for "A" that believe it inherits from an empty class with no ivars and no methods. Now we write and expression that uses a variable in "foo.dylib" whose type is "A" and one from "bar.dylib" whose type is "A" and we try to copy the definitions for "A" from the source ASTContext in "foo.dylib" over into the expression AST (this works) and then we try to copy the version from "bar.dylib" into the expression context and the AST copying code notices that the definitions for class "A" don't match. The copy would have worked in the copies of "A" are the same and nothing would have been copied, but it fails when they are different. This is a know limitation of using the clang ASTContext classes to represent our types and is also the reason the "-fstandalone-debug" is the default setting for clang or Darwin, and probably should be for anyone else wanting to use lldb to debug.

> The crash is reproducible, but one of the reproduction steps is "Build chrome", so I figured I'd work on it some myself to teach myself lldb rather than try to file a bug on it.   The wisdom of that choice in question :-}.
> 
> Any thoughts anyone has would be welcome.

So try things out with -fstandalone-debug and see if that fixes your problems. If it does it gives us a work around for now, but we should really be fixing any crashing bugs that occur due to this kind of issue in LLDB in the long run.

I hope this helps you understand a bit more and gives you enough to go on.

Greg