[cfe-dev] stable layout bug for imported record decls.
Gábor Márton via cfe-dev
cfe-dev at lists.llvm.org
Tue Aug 14 06:49:33 PDT 2018
> It might be interesting to consider how this is addressed by the ExternalASTSource interface. There, we have separate "import the lexical contents of this AST node (including populating the lexical declarations list with all members, in the right order)", "import the lookup results for this name in this context (and cache them but don't add them to the list of lexical members)" and "import this specific declaration member (but don't add it to the list of lexical members)" queries. One could imagine doing something similar for the AST importer: when you're performing a minimal import but you want to import a method declaration, don't insert it into the list of lexical members of the enclosing record. Instead, defer doing that until a complete type is required (at that point, you'd need to import all the relevant members anyway, including the base classes and virtual functions in the right order).
Hi Richard, maybe my understanding is not correct, but I believe the
situation is somewhat twisted and reversed.
Seems like LLDB already exercises the ExternalASTSource interface. The
purpose of using it is exactly to load the external lexical contents
of an AST node (as you suggested). However, the load is implemented by
the means of the ASTImporter (in minimal mode). Consider the attached
function call trace:
When the user instructs LLDB to evaluate an expression then LLDB
exercises the parser `clang::ParseAST`, which in turn does a lookup
`clang::DeclContext::lookup`. During the lookup,
`lldb_private::ClangASTSource::FindExternalVisibleDeclsByName` is
called (this function overloads
`ExternalASTSource::FindExternalVisibleDeclsByName`).
And finally, through `lldb_private::ClangASTImporter::CopyType` we end
up in `clang::ASTImporter::Import`.
Gabor
```
#4 0x00007ffff41a72ed in clang::ASTNodeImporter::ImportDeclParts
(this=this at entry=0x7fffffff8be0, D=D at entry=0x632038,
DC=@0x7fffffff8ac8: 0x0,
LexicalDC=@0x7fffffff8ad0: 0x0, Name=..., ToD=@0x7fffffff8ad8: 0x0, Loc=...)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/AST/ASTImporter.cpp:1062
#5 0x00007ffff41a8d55 in clang::ASTNodeImporter::VisitRecordDecl
(this=0x7fffffff8be0, D=0x632038)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/AST/ASTImporter.cpp:2126
#6 0x00007ffff419377b in clang::ASTImporter::Import (this=0x70cc90,
FromD=0x632038)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/AST/ASTImporter.cpp:7054
#7 0x00007ffff41939f7 in clang::ASTNodeImporter::VisitRecordType
(this=0x7fffffff8c40, T=<optimized out>)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/AST/ASTImporter.cpp:851
#8 0x00007ffff4196f65 in clang::TypeVisitor<clang::ASTNodeImporter,
clang::QualType>::Visit (this=this at entry=0x7fffffff8c40, T=<optimized
out>)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/include/clang/AST/TypeNodes.def:92
#9 0x00007ffff4197187 in clang::ASTImporter::Import (this=0x70cc90,
FromT=FromT at entry=...)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/AST/ASTImporter.cpp:6983
#10 0x00007ffff7351c92 in lldb_private::ClangASTImporter::CopyType
(this=<optimized out>, dst_ast=<optimized out>, src_ast=<optimized
out>, type=type at entry=...)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Symbol/ClangASTImporter.cpp:61
#11 0x00007ffff74bdb95 in
lldb_private::ClangASTSource::GuardedCopyType
(this=this at entry=0x611e20, src_type=...)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Plugins/ExpressionParser/Clang/ClangASTSource.cpp:2040
#12 0x00007ffff74e9ffe in
lldb_private::ClangExpressionDeclMap::GetVariableValue
(this=this at entry=0x611e20, var=std::shared_ptr (count 6, weak 1)
0x7fffe80020f0,
var_location=..., user_type=user_type at entry=0x7fffffff8fc0,
parser_type=parser_type at entry=0x7fffffff8fe0)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Plugins/ExpressionParser/Clang/ClangExpressionDeclMap.cpp:1627
#13 0x00007ffff74ea3a7 in
lldb_private::ClangExpressionDeclMap::AddOneVariable
(this=this at entry=0x611e20, context=...,
var=std::shared_ptr (count 6, weak 1) 0x7fffe80020f0, valobj=...,
current_id=current_id at entry=2)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Plugins/ExpressionParser/Clang/ClangExpressionDeclMap.cpp:1679
#14 0x00007ffff74ec50b in
lldb_private::ClangExpressionDeclMap::FindExternalVisibleDecls
(this=this at entry=0x611e20, context=..., module_sp=
std::shared_ptr (empty) 0x0, namespace_decl=...,
current_id=current_id at entry=2)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Plugins/ExpressionParser/Clang/ClangExpressionDeclMap.cpp:1240
#15 0x00007ffff74ef4ce in
lldb_private::ClangExpressionDeclMap::FindExternalVisibleDecls
(this=0x611e20, context=...)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Plugins/ExpressionParser/Clang/ClangExpressionDeclMap.cpp:843
#16 0x00007ffff74c71cc in
lldb_private::ClangASTSource::FindExternalVisibleDeclsByName
(this=0x611e20, decl_ctx=0x71d8b8, clang_decl_name=...)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Plugins/ExpressionParser/Clang/ClangASTSource.cpp:261
#17 0x00007ffff421d92d in clang::DeclContext::lookup
(this=this at entry=0x71d8b8, Name=...)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/AST/DeclBase.cpp:1577
#18 0x00007ffff4d4bb42 in LookupDirect (S=..., R=..., DC=DC at entry=0x71d8b8)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Sema/SemaLookup.cpp:843
#19 0x00007ffff4d52f50 in CppNamespaceLookup (S=..., R=...,
NS=NS at entry=0x71d8b8, UDirs=..., Context=...)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Sema/SemaLookup.cpp:942
#20 0x00007ffff4d53925 in clang::Sema::CppLookupName
(this=this at entry=0x7245b0, R=..., S=0x72cad0)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Sema/SemaLookup.cpp:1322
#21 0x00007ffff4d53c8a in clang::Sema::LookupName (this=0x7245b0,
R=..., S=0x7314f0, AllowBuiltinCreation=<optimized out>)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Sema/SemaLookup.cpp:1827
#22 0x00007ffff4b6b945 in clang::Sema::ClassifyName (this=0x7245b0,
S=0x7314f0, SS=..., Name=@0x7fffffffa3a8: 0x750dc8, NameLoc=...,
NameLoc at entry=...,
NextToken=..., IsAddressOfOperand=false,
CCC=std::unique_ptr<clang::CorrectionCandidateCallback> containing
0x63c250)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Sema/SemaDecl.cpp:868
#23 0x00007ffff52c6a9f in clang::Parser::TryAnnotateName
(this=this at entry=0x728c60,
IsAddressOfOperand=IsAddressOfOperand at entry=false,
CCC=std::unique_ptr<clang::CorrectionCandidateCallback> containing
0x0) at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Parse/Parser.cpp:1531
#24 0x00007ffff52aa56b in
clang::Parser::ParseStatementOrDeclarationAfterAttributes
(this=this at entry=0x728c60, Stmts=...,
Allowed=Allowed at entry=clang::Parser::ACK_Any,
TrailingElseLoc=<optimized out>, Attrs=...)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Parse/ParseStmt.cpp:189
#25 0x00007ffff52aa843 in clang::Parser::ParseStatementOrDeclaration
(this=this at entry=0x728c60, Stmts=...,
Allowed=Allowed at entry=clang::Parser::ACK_Any,
TrailingElseLoc=TrailingElseLoc at entry=0x0) at
/home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Parse/ParseStmt.cpp:111
#26 0x00007ffff52ae577 in clang::Parser::ParseCompoundStatementBody
(this=this at entry=0x728c60, isStmtExpr=isStmtExpr at entry=false)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Parse/ParseStmt.cpp:1003
#27 0x00007ffff52b0a49 in clang::Parser::ParseFunctionStatementBody
(this=this at entry=0x728c60, Decl=Decl at entry=0x72dbd0, BodyScope=...)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Parse/ParseStmt.cpp:1972
#28 0x00007ffff52cc6e0 in clang::Parser::ParseFunctionDefinition
(this=this at entry=0x728c60, D=..., TemplateInfo=...,
LateParsedAttrs=LateParsedAttrs at entry=0x7fffffffabe0) at
/home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Parse/Parser.cpp:1248
#29 0x00007ffff524aa56 in clang::Parser::ParseDeclGroup
(this=this at entry=0x728c60, DS=...,
Context=Context at entry=clang::DeclaratorContext::FileContext,
DeclEnd=DeclEnd at entry=0x0, FRI=FRI at entry=0x0) at
/home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Parse/ParseDecl.cpp:1968
#30 0x00007ffff52c7f90 in
clang::Parser::ParseDeclOrFunctionDefInternal
(this=this at entry=0x728c60, attrs=..., DS=...,
AS=AS at entry=clang::AS_none)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Parse/Parser.cpp:1012
#31 0x00007ffff52c8761 in
clang::Parser::ParseDeclarationOrFunctionDefinition (this=0x728c60,
attrs=..., AS=clang::AS_none, DS=0x0)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Parse/Parser.cpp:1028
#32 0x00007ffff52c87bf in
clang::Parser::ParseDeclarationOrFunctionDefinition (this=<optimized
out>, attrs=..., DS=<optimized out>, AS=<optimized out>)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Parse/Parser.cpp:1030
#33 0x00007ffff52cd05d in clang::Parser::ParseExternalDeclaration
(this=this at entry=0x728c60, attrs=..., DS=DS at entry=0x0)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Parse/Parser.cpp:853
#34 0x00007ffff52ce05c in clang::Parser::ParseTopLevelDecl
(this=this at entry=0x728c60, Result=...)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Parse/Parser.cpp:609
#35 0x00007ffff522ce1b in clang::ParseAST (S=...,
PrintStats=PrintStats at entry=false,
SkipFunctionBodies=SkipFunctionBodies at entry=false)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Parse/ParseAST.cpp:146
#36 0x00007ffff522d0e0 in clang::ParseAST (PP=..., Consumer=0x535f70,
Ctx=..., PrintStats=<optimized out>, TUKind=clang::TU_Complete,
CompletionConsumer=0x0,
SkipFunctionBodies=false) at
/home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/clang/lib/Parse/ParseAST.cpp:110
#37 0x00007ffff74f0059 in lldb_private::ClangExpressionParser::Parse
(this=this at entry=0x7fffffffcae0, diagnostic_manager=...)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Plugins/ExpressionParser/Clang/ClangExpressionParser.cpp:619
#38 0x00007ffff74d23ad in lldb_private::ClangUserExpression::Parse
(this=0x6d23c0, diagnostic_manager=..., exe_ctx=...,
execution_policy=lldb_private::eExecutionPolicyOnlyWhenNeeded,
keep_result_in_memory=<optimized out>, generate_debug_info=<optimized
out>)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Plugins/ExpressionParser/Clang/ClangUserExpression.cpp:472
---Type <return> to continue, or q <return> to quit---
#39 0x00007ffff72cea9e in lldb_private::UserExpression::Evaluate
(exe_ctx=..., options=..., expr=..., prefix=..., result_valobj_sp=...,
error=..., line_offset=0,
fixed_expression=0x4bced8, jit_module_sp_ptr=0x0) at
/home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Expression/UserExpression.cpp:239
#40 0x00007ffff7401996 in lldb_private::Target::EvaluateExpression
(this=this at entry=0x5b8d10, expr=..., exe_scope=<optimized out>,
result_valobj_sp=...,
options=..., fixed_expression=0x4bced8) at
/home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Target/Target.cpp:2321
#41 0x00007ffff77912dd in
lldb_private::CommandObjectExpression::EvaluateExpression
(this=this at entry=0x4bcb50, expr=expr at entry=0x7fffffffd4c0 "val.v",
output_stream=output_stream at entry=0x7fffffffd5f0,
error_stream=error_stream at entry=0x7fffffffd648,
result=result at entry=0x7fffffffd5f0)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Commands/CommandObjectExpression.cpp:377
#42 0x00007ffff7792018 in
lldb_private::CommandObjectExpression::DoExecute (this=0x4bcb50,
command=0x7fffffffd4c0 "val.v", result=...)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Commands/CommandObjectExpression.cpp:625
#43 0x00007ffff7305baa in lldb_private::CommandObjectRaw::Execute
(this=0x4bcb50, args_string=0x7fffffffd4c0 "val.v", result=...)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Interpreter/CommandObject.cpp:1013
#44 0x00007ffff7302e1b in
lldb_private::CommandInterpreter::HandleCommand
(this=this at entry=0x4a8740, command_line=<optimized out>,
lazy_add_to_history=lazy_add_to_history at entry=lldb_private::eLazyBoolCalculate,
result=..., override_context=override_context at entry=0x0,
repeat_on_empty_command=repeat_on_empty_command at entry=true,
no_context_switching=false)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Interpreter/CommandInterpreter.cpp:1683
#45 0x00007ffff73047c8 in
lldb_private::CommandInterpreter::IOHandlerInputComplete
(this=0x4a8740, io_handler=..., line="expr val.v")
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Interpreter/CommandInterpreter.cpp:2771
#46 0x00007ffff723e241 in lldb_private::IOHandlerEditline::Run (this=0x5b4da0)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Core/IOHandler.cpp:573
#47 0x00007ffff720a820 in lldb_private::Debugger::ExecuteIOHandlers
(this=0x4a6f90)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Core/Debugger.cpp:961
#48 0x00007ffff72f731f in
lldb_private::CommandInterpreter::RunCommandInterpreter
(this=0x4a8740, auto_handle_events=auto_handle_events at entry=true,
spawn_thread=spawn_thread at entry=false, options=...) at
/home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/Interpreter/CommandInterpreter.cpp:2971
#49 0x00007ffff7094776 in lldb::SBDebugger::RunCommandInterpreter
(this=this at entry=0x7fffffffda60,
auto_handle_events=auto_handle_events at entry=true,
spawn_thread=spawn_thread at entry=false) at
/home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/source/API/SBDebugger.cpp:892
#50 0x0000000000404a76 in Driver::MainLoop (this=this at entry=0x7fffffffda40)
at /home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/tools/driver/Driver.cpp:1156
#51 0x0000000000403724 in main (argc=2, argv=0x7fffffffdcb8) at
/home/gerazo/repos/codechecker_dev_env/CTU/llvm/tools/lldb/tools/driver/Driver.cpp:1253
```
On Tue, Aug 14, 2018 at 2:34 AM Richard Smith via cfe-dev
<cfe-dev at lists.llvm.org> wrote:
>
> On Mon, 13 Aug 2018 at 17:08, Lang Hames via cfe-dev <cfe-dev at lists.llvm.org> wrote:
>>
>> Hi Richard,
>>
>>> Perhaps a better approach would be to make the "minimal" mode for the ASTImporter provide an ExternalASTSource to lazily complete the AST as needed (thereby avoiding violating the invariant, because you would populate the lexical declarations list whenever anyone actually asks for it).
>>
>>
>> This seems worth investigating. I wonder if it might also fix another bug that I know of involving virtual methods with covariant return types. (You and I actually discussed it at one of the socials a while back, but I have not had time to dig into it further.)
>>
>> The reproducer for the bug is:
>>
>> class Foo {};
>> class Bar : public Foo {};
>>
>> class Base {
>> public:
>> virtual Foo* foo() { return nullptr; }
>> };
>>
>> class Derived : public Base {
>> public:
>> virtual Bar* foo() { return nullptr; }
>> };
>>
>> int main() {
>> Derived D;
>> D.foo(); // Evaluate 'D.foo()' here, crash LLDB.
>> }
>>
>> The issue is that since Bar's definition is not used its bases are never imported, and so the call to Bar::bases() crashes in CodeGen. If we provided an ExternalASTSource, would that be able to lazily complete the bases?
>
>
> Yes, such an approach should be able to solve that problem too: when CodeGen (or indeed anything) queries any property of the definition of Foo / Bar (including whether a definition exists), you'll get a callback and a chance to provide a complete type.
>
>>
>> Cheers,
>> Lang.
>>
>>
>> On Mon, Aug 13, 2018 at 3:30 PM Richard Smith <richard at metafoo.co.uk> wrote:
>>>
>>> On Thu, 9 Aug 2018 at 10:47, Lang Hames via cfe-dev <cfe-dev at lists.llvm.org> wrote:
>>>>
>>>> Hi clang-dev, lldb-dev,
>>>>
>>>> It looks like my clang commit r305850, which modified ASTImporter to import method override tables from an external context, introduced a new bug which manifests as incorrect vtable layouts for LLDB expression code.
>>>>
>>>> The bug itself is fairly straightforward. In r305850 I introduced the following method, which is called from ASTNodeImporter::VisitFunctionDecl:
>>>>
>>>> void ASTNodeImporter::ImportOverrides(CXXMethodDecl *ToMethod,
>>>> CXXMethodDecl *FromMethod) {
>>>> for (auto *FromOverriddenMethod : FromMethod->overridden_methods())
>>>> ToMethod->addOverriddenMethod(
>>>> cast<CXXMethodDecl>(Importer.Import(const_cast<CXXMethodDecl*>(
>>>> FromOverriddenMethod))));
>>>> }
>>>>
>>>> This will produce the correct override table, but can also cause methods in the Base class to be visited in the wrong order. Consider:
>>>>
>>>> class Base {
>>>> public:
>>>> virtual void bar() {}
>>>> virtual void foo() {}
>>>> };
>>>>
>>>> class Derived : public Base {
>>>> public:
>>>> void foo() override {}
>>>> };
>>>>
>>>> If Derived is imported into a new context before Base, then the importer will visit Derived::foo, and (via ImportOverrides) immediately import Base::foo, but this happens before Base::bar is imported. As a consequence, the decl order on the imported Base class will end up being [ foo, bar ], instead of [ bar, foo ]. In LLDB expression evaluation this manifests as an incorrect vtable layout for Base, with foo occupying the first slot.
>>>>
>>>> I am looking for suggestions on the right way to fix this. A brute force solution might be to always have ASTNodeImporter::VisitRecordDecl visit all base classes, then all virtual methods, which would ensure they are visited in the original decl order. However I do not know whether this covers all paths by which a CXXRecordDecl might be imported, nor whether the performance of this solution would be acceptable (it seems like it would preclude a lot of laziness).
>>>>
>>>> Alternatively we might be able to associate an index with each imported decl and sort on that when we complete the type, but that would leave imported decls in the wrong order until the type was complete, and since I do not know all the use cases for the importer I'm concerned people may rely on the decl order before type is complete.
>>>>
>>>> Any insight from ASTImporter experts would be greatly appreciated. :)
>>>
>>>
>>> Hi Lang,
>>>
>>> It might be interesting to consider how this is addressed by the ExternalASTSource interface. There, we have separate "import the lexical contents of this AST node (including populating the lexical declarations list with all members, in the right order)", "import the lookup results for this name in this context (and cache them but don't add them to the list of lexical members)" and "import this specific declaration member (but don't add it to the list of lexical members)" queries. One could imagine doing something similar for the AST importer: when you're performing a minimal import but you want to import a method declaration, don't insert it into the list of lexical members of the enclosing record. Instead, defer doing that until a complete type is required (at that point, you'd need to import all the relevant members anyway, including the base classes and virtual functions in the right order).
>>>
>>> The above approach would violate AST invariants (you'd have a declaration whose lexical parent doesn't believe it lexically contains the child), but I don't know off-hand whether that would be problematic. Perhaps a better approach would be to make the "minimal" mode for the ASTImporter provide an ExternalASTSource to lazily complete the AST as needed (thereby avoiding violating the invariant, because you would populate the lexical declarations list whenever anyone actually asks for it).
>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
More information about the cfe-dev
mailing list