[lldb-dev] Trying to understand symbol importing and its relationship to ASTs

[Randy Smith]

Greg: Thanks very much for the detailed explanation!  As I mentioned in my
response to Reid, this does indeed seem related to the known
-fstandalone-debug issue.  I'd still like to dig down to the floor (i.e. to
the point where I understand this specific issue), with a vague hope that
it may be a reasonable thing for me to try and fix.  So I'd like to ask you
a couple of questions on your summary.

First question: Is there a tool to probe for symbol information (forward
decl vs. full information) in a shared library?  I see llvm-dwarfdump, but
it looks to be just dumping symbols rather than interpreting them.

On Tue, Jul 22, 2014 at 4:09 PM, Greg Clayton <gclayton at apple.com> wrote:

>
> > On Jul 22, 2014, at 11:41 AM, Randy Smith <rdsmith at chromium.org> wrote:
> >
> >
> > I'm chasing a crash in lldb, and my current "that doesn't seem right"
> has to do with a conflict between a decl and its origin decl (the
> transformation done at the beginning of
> tools/lldb/source/Expression/ClangASTSource.cpp:ClangASTSource::layoutRecordType()).
>  So I'm trying to understand how decls and origin decls get setup during
> the symbol import process.  Can anyone give me a sketch/hand?  Specific
> questions include:
> > * There are multiple ASTContexts involved (e.g. the src and dst contexts
> in the signature of
> tools/lldb/source/Symbol/ClangASTImporter.cpp:ClangASTImporter::CopyType);
> do those map to compilation units, or to shared library modules?  Is there
> a simple way to tell what CU/.so an ASTContext maps to?
>
> Every executable file is represented by a lldb_private::Module (this
> includes both executables and shared libraries) and each
> lldb_private::Module has its own ASTContext (one per module, and all
> compilation units are all represented in one big ASTContext). The DWARF
> debug info is parsed and it creates types in the ASTContext in the
> corresponding lldb_private::Module.
>
> > * Does a decl always have an origin decl, even if it was loaded from an
> ASTContext (?) that has a complete definition?
>
> Origin decl is so we know where a decl originally came from because the
> definition might not yet be complete (think "class Foo;") and might need to
> be completed. A little background on how we lazily parse classes.
>
> When someone needs a type, we parse the type (SymbolFileDWARF::ParseType).
> If that type is a class we always just parse a forward decl to the class
> ("class Foo;"). The DWARF parser (SymbolFileDWARF) implements
> clang::ExternalASTSource so it can complete a type only when the compiler
> needs to know more. When the compiler or ClangASTType needs to know more
> about a type it asks the type to get a complete version of itself and
> SymbolFileDWARF::CompleteTagDecl is called to complete the type. We then
> parse all ivars, methods, and everything else about a type. We also assist
> in laying out the CXXRecordDecl by another callback
> SymbolFileDWARF::LayoutRecordType (which is part of the
> clang::ExternalASTSource). We need to assist in laying things out because
> the DWARF debug info doesn't always include all required attributes or
> #pragma information in order for us to create the types correctly. So this
> SymbolFileDWARF::LayoutRecordType allows us to tell the compiler about the
> offsets of ivars so they are always correct.
>
> Back to origin decls: When running an expression we create a new
> ASTContext that is for the expression only. decls are copied from the
> ASTContext for the lldb_private::Module over into the ASTContext for the
> expression. When they are copied, only a forward decls are copied, and they
> may need to be completed. When this happens we might need to ask the type
> in the original ASTContext to complete itself so that we can copy a
> complete definition over into the expression ASTContext. This is the reason
> we track the origin decls. Sometimes you have a type that is only a forward
> decl, and that is ok as we don't always have the full definition of a class.
>
> > * When an origin decl is looked up, should all the types in it be
> completed, or might it have incomplete types?  It seems as if there is code
> assuming that these types will always be complete.
>
> There are two forms of incomplete types:
> 1 - incomplete types that have full definitions and just haven't been
> completed (and might have to find the original decl, ask it to complete
> itself, then copy the origin decl when the current decl needs to be copied
> from one AST to another)
> 2 - types that are actually forward declarations and will be told they are
> just forward decls
>
> So we sometimes do run into cases where we don't have the debug info for
> something because the compiler pulled it out trying to minimize the debug
> info.
>
> >
> > Context (warning, gets detailed, possibly with irrelevant details
> because newbie): lldb is crashing in clang::ASTContext::getASTRecordLayout
> with the assertion "Cannot get layout of forward declarations!".  The type
> in question is an incomplete type (string16, aka. basic_string<unsigned
> short, ...>).  Normally clang::ASTContext::getASTRecordLayout() would call
> getExternalSource()->CompleteType() to complete the type, but in this case
> it isn't because the type is marked as !hasExternalLexicalStorage().
>
> That mean the type was not complete in the DWARF for the
> lldb_private::Module it originates from.
> >
> > The *weird* thing is that the type has previously been completed,
> further up the stack, but in a different AST node (same name).  In more
> detail: Class A contains an instance of class B contains an instance of
> class C (==string16).  I'm seeing getASTRecordLayout called on class A,
> which then calls it (indirectly, though the EmptySubobjectMap construtor)
> on class B, which then calls it (ditto) on class C (all works).  Then the
> stack unwinds up to the B call, which proceeds to the Builder.Layout() line
> in that function.  It ends up (through the transformation mentioned above
> in clang::ClangASTSource::LayoutRecordType()) calling getASTRecordLayout()
> on the origin decl.  When it recurses down to class C, that node isn't
> complete, isn't completed, and causes an assertion.  So I'm trying to
> figure out whether the problem is that any decl hanging off an origin_decl
> should be complete, or that that node shouldn't be marked as
> !hasExternalLexicalStorage().  (Or something else; I've already gone
> through several twists and turns debugging this problem :-}.)
>
> We have a problem in the compiler currently where for classes like:
>
> class A : public B
> {
>     ...
> }
>
> The compiler says "ahh, you didn't use class B so I am not going to emit
> debug info for it.". This really can hose us up because we now create a
> ASTContext for the expression and we want a definition for "A" and the user
> wants to call a method that is in class "B", but we can't because the
> compiler removed the definition. What we currently do is figure out that we
> have a forward declaration to "B" only, and when we create type "A" in the
> module's ASTContext, we say "B" is an empty class with no ivars and no
> methods. To fix this, you can specify "-fstandalone-debug" to the clang
> compiler to tell it not to do this removal of debug info for things that
> are inherited from.
>


> The other problem we have is say you two modules "foo.dylib" and
> "bar.dylib", both have debug info, and "foo.dylib" has debug info with a
> complete "A" and complete "B" definition, but "bar.dylib" has a complete
> "A" definition, but only a forward "B" definition. The ASTContext for
> foo.dylib believes class "A" to look like it really is, and "bar.dylib" has
> a definition for "A" that believe it inherits from an empty class with no
> ivars and no methods. Now we write and expression that uses a variable in
> "foo.dylib" whose type is "A" and one from "bar.dylib" whose type is "A"
> and we try to copy the definitions for "A" from the source ASTContext in
> "foo.dylib" over into the expression AST (this works) and then we try to
> copy the version from "bar.dylib" into the expression context and the AST
> copying code notices that the definitions for class "A" don't match. The
> copy would have worked in the copies of "A" are the same and nothing would
> have been copied, but it fails when they are different. This is a know
> limitation of using the clang ASTContext classes to represent our types and
> is also the reason the "-fstandalone-debug" is the default setting for
> clang or Darwin, and probably should be for anyone else wanting to use lldb
> to debug.
>

So that sounds like it could be my situation (with A (defined in liba)
containing B (defined in libb) rather than inheriting from it, but I'd
think that'd be identical from a layout perspective).   But I'm not quite
seeing how that maps to the execution flow I'm seeing in my debugging.  If
I understand your description above correctly, what I was seeing was
CompleteType called on the forward decl of my A, and called successfully;
both A & B were fully populated.  But then later we got the origin decl for
A, and CompleteType was called on it, and B was not filled out in that.  Is
it that the first CompleteType was done in the expression ASTContext (which
presumably has access to search all the library ASTContexts) and the second
one was done in the context of the liba ASTContext, and so didn't have
access to the libb information?  And if so, why isn't the first one
strictly better?


>
> > The crash is reproducible, but one of the reproduction steps is "Build
> chrome", so I figured I'd work on it some myself to teach myself lldb
> rather than try to file a bug on it.   The wisdom of that choice in
> question :-}.
> >
> > Any thoughts anyone has would be welcome.
>
> So try things out with -fstandalone-debug and see if that fixes your
> problems. If it does it gives us a work around for now, but we should
> really be fixing any crashing bugs that occur due to this kind of issue in
> LLDB in the long run.
>

Do you have a sense of what the proper fix would be?  In the previous
thread I think you indicated that the compiler should emit debug
information a la' -fstandalone-debug, and the linker should collapse the
information back down, but in this case it seems like the debugger should
be able to find the information in the other shared library (though I do
understand that there's a more general problem that doesn't solve, when the
debugging information isn't emitted anywhere for a particular class).

Thanks again for the detailed explanation above!

-- Randy


>
> I hope this helps you understand a bit more and gives you enough to go on.
>
> Greg
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20140722/7838d052/attachment.html>