<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-tab-span
{mso-style-name:apple-tab-span;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Sean,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks for this nice write-up. I observed that when I evaluate an expression like ‘expr variable’, LLDB runs the target as doing an inferior function call and
then run the interpreter too. I was wondering if this is expected behaviour or I am missing something.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Abid<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> lldb-dev-bounces@cs.uiuc.edu [mailto:lldb-dev-bounces@cs.uiuc.edu]
<b>On Behalf Of </b>Sean Callanan<br>
<b>Sent:</b> 17 August 2013 02:13<br>
<b>To:</b> lldb-dev@cs.uiuc.edu<br>
<b>Subject:</b> [lldb-dev] LLDB expression parser presentation<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This is the outline of a brief presentation I gave on the LLDB expression parser.<o:p></o:p></p>
<div>
<p class="MsoNormal">I’ve included some “thorny issues;” if we resolve these, the expression parser will get a lot better.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Please let me know if you have any questions.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><b>Class layout</b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>The master - ClangExpressionParser manages Clang and LLVM to compile a single expression<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>Its minions:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>ClangExpression - a unit of parseable code<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span">
</span>ClangUserExpression - specialized for the case where we’re using the “expr” command<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>ExpressionSourceCode - handles wrapping<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>ClangASTSource - resolves external variables<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span">
</span>ClangExpressionDeclMap - specialized for the current frame (if stopped at a particular location in the program being debugged)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>IRForTarget - rewrites IR<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>ASTResultSynthesizer - makes the result<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>IRMemoryMap - manages memory that may or may be in the program being debugged, or may be simulated by LLDB<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span">
</span>IRExecutionUnit - specialized to be able to interact with the JIT<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><b>Basic Expression Flow</b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>User enters the expression: (lldb) expr a + 2<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>We wrap the expression: void expr(arg *) { a + 2; }<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>We wrap differently based on expression context.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>If stopped in a C++ instance method, we wrap as $__lldb_class::$__lldb_expr(void *)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>If stopped in an Objective-C instance method, we wrap as an Objective-C category<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>If stopped in regular C code, we wrap as $__lldb_expr(void*)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>But we always parse in Objective-C++ mode.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>Typical wrapped expression:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span">
</span>#define … // custom definitions provided by LLDB or the user<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span">
</span>void<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span">
</span>$__lldb_class::$__lldb_expr // __lldb_class resolves to the type of *this in the current frame<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span">
</span>(void *$__lldb_arg)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span">
</span>{<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span">
</span>// expression text goes here<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span">
</span>}<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>We resolve externals: “a” => int &a;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>This happens via a question-and-answer process with the Clang compiler through the clang::ExternalASTSource interface<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>FindExternalVisibleDeclsByName searches for “globals” (globals from the perspective of the expression; these may be locals in the current stack frame)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>FindExternalLexicalDecls searches a single struct for all entities of a particular type<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>CompleteType ensures that a single struct has all of its contents<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>(These are useful because we lazily complete structs, providing a forward declaration first and only filling it in when needed)<span class="apple-tab-span">
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>clang::ASTImporter is responsible for transferring Decls from one ASTContext (e.g., the ASTContext for a DWARF file) to another (e.g., the AST context for an expression)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>Our ClangASTImporter manages many of these (“Minions"), because there are many separate DWARF files containing debug information.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>We need to be able to remember where things came from.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>We add the result: static int ret = a + 2;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>This happens at the Clang AST level<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>We handle Lvalues and Rvalues differently.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>For Lvalues, we store a pointer to them: T *$__result_ptr = …<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>For Rvalues, we store the value itself: static T $__result = … // static ensures the expression doesn’t try to use a register or something silly like that<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>We also store persistent types at this stage, e.g. struct $my_foo { int a; int b; }<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>We rewrite the IR: *(arg+0) = *(arg+8)+2<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>The IR as emitted by Clang’s CodeGen expects all external variables to be in symbols<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>This is inconvenient if they are e.g. in registers, since you can’t link against a register<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>This is also inconvenient for expression re-use, for example as a breakpoint condition… we’d have to re-link each time<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>Our solution is to indirect variables through a struct passed into the expression (void *$__lldb_arg)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>Materializer’s job is to put all variables that aren’t referred to by symbols into this struct<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>It will create temporary storage as necessary (e.g., to hold a variable value that was in a register)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>After the expression runs, a Dematerializer takes down all temporary storage, and ensures that variables are updated to reflect the expression’s side effects<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>The IRForTarget class does various cleanup to help RTDyldMemoryManager (ideally much of this shouldn’t be necessary)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>It resolves all external symbols to avoid forcing RTDyldMemoryManager to resolve symbols<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>It creates a string and float literal pool so RTDyldMemoryManager doesn’t have to relocate the constant pool<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>It strips off nasty Objective-C metadata so RTDyldMemoryManager doesn’t have to look at it<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>We interpret or execute the result: (int)$0 = 6<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>IRExecutionUnit contains a module and the (real or simulated) memory it uses<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>IRInterpreter can interpret a module without ever running the underlying process<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>It emulates IR instructions one by one<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>It uses lldb_private::Scalar to hold intermediate values, which is kinda limiting (no vectors, no FP math)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>IRExecutionUnit simulates memory allocation etc. so we can do a lot of pointer magic<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>If the IRInterpreter can’t run, the MCJIT produces machine code and LLDB runs it<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>IRExecutionUnit vends a custom JITMemoryManager implementation<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>It remembers memory allocations and where functions were placed<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>After JIT, all sections are placed into the target and we report their new locations with mapSectionAddress<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><b>Selected Thorny Issues (concentrating on JIT-related issues)</b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>Make the MCJIT more robust so we can rely on it more<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>Support all Mach-O and ELF relocation types<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>Don’t assume resolved symbols are in the current process<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>Don’t assume addresses fit into void*s<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>Make the IRInterpreter support all data types and instructions<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>Completely replace the LLVM interpreter!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">Sean<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>