<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">This is the outline of a brief presentation I gave on the LLDB expression parser.<div>I’ve included some “thorny issues;” if we resolve these, the expression parser will get a lot better.</div><div>Please let me know if you have any questions.</div><div><br></div><div><div><b>Class layout</b></div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>The master - ClangExpressionParser manages Clang and LLVM to compile a single expression</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>Its minions:</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>ClangExpression - a unit of parseable code</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>ClangUserExpression - specialized for the case where we’re using the “expr” command</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>ExpressionSourceCode - handles wrapping</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>ClangASTSource - resolves external variables</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>ClangExpressionDeclMap - specialized for the current frame (if stopped at a particular location in the program being debugged)</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>IRForTarget - rewrites IR</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>ASTResultSynthesizer - makes the result</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>IRMemoryMap - manages memory that may or may be in the program being debugged, or may be simulated by LLDB</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>IRExecutionUnit - specialized to be able to interact with the JIT</div></div><div><br></div><div><b>Basic Expression Flow</b></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>User enters the expression: (lldb) expr a + 2</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>We wrap the expression: void expr(arg *) { a + 2; }</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>We wrap differently based on expression context.</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>If stopped in a C++ instance method, we wrap as $__lldb_class::$__lldb_expr(void *)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>If stopped in an Objective-C instance method, we wrap as an Objective-C category</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>If stopped in regular C code, we wrap as $__lldb_expr(void*)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>But we always parse in Objective-C++ mode.</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Typical wrapped expression:</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>#define … // custom definitions provided by LLDB or the user</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>void</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>$__lldb_class::$__lldb_expr // __lldb_class resolves to the type of *this in the current frame</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>(void *$__lldb_arg)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>{</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>// expression text goes here</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>}</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>We resolve externals: “a” => int &a;</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>This happens via a question-and-answer process with the Clang compiler through the clang::ExternalASTSource interface</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>FindExternalVisibleDeclsByName searches for “globals” (globals from the perspective of the expression; these may be locals in the current stack frame)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>FindExternalLexicalDecls searches a single struct for all entities of a particular type</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>CompleteType ensures that a single struct has all of its contents</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>(These are useful because we lazily complete structs, providing a forward declaration first and only filling it in when needed)<span class="Apple-tab-span" style="white-space: pre;"> </span></div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>clang::ASTImporter is responsible for transferring Decls from one ASTContext (e.g., the ASTContext for a DWARF file) to another (e.g., the AST context for an expression)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Our ClangASTImporter manages many of these (“Minions"), because there are many separate DWARF files containing debug information.</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>We need to be able to remember where things came from.</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>We add the result: static int ret = a + 2;</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>This happens at the Clang AST level</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>We handle Lvalues and Rvalues differently.</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>For Lvalues, we store a pointer to them: T *$__result_ptr = …</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>For Rvalues, we store the value itself: static T $__result = … // static ensures the expression doesn’t try to use a register or something silly like that</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>We also store persistent types at this stage, e.g. struct $my_foo { int a; int b; }</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>We rewrite the IR: *(arg+0) = *(arg+8)+2</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>The IR as emitted by Clang’s CodeGen expects all external variables to be in symbols</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>This is inconvenient if they are e.g. in registers, since you can’t link against a register</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>This is also inconvenient for expression re-use, for example as a breakpoint condition… we’d have to re-link each time</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Our solution is to indirect variables through a struct passed into the expression (void *$__lldb_arg)</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Materializer’s job is to put all variables that aren’t referred to by symbols into this struct</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>It will create temporary storage as necessary (e.g., to hold a variable value that was in a register)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>After the expression runs, a Dematerializer takes down all temporary storage, and ensures that variables are updated to reflect the expression’s side effects</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>The IRForTarget class does various cleanup to help RTDyldMemoryManager (ideally much of this shouldn’t be necessary)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>It resolves all external symbols to avoid forcing RTDyldMemoryManager to resolve symbols</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>It creates a string and float literal pool so RTDyldMemoryManager doesn’t have to relocate the constant pool</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>It strips off nasty Objective-C metadata so RTDyldMemoryManager doesn’t have to look at it</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>We interpret or execute the result: (int)$0 = 6</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>IRExecutionUnit contains a module and the (real or simulated) memory it uses</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>IRInterpreter can interpret a module without ever running the underlying process</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>It emulates IR instructions one by one</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>It uses lldb_private::Scalar to hold intermediate values, which is kinda limiting (no vectors, no FP math)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>IRExecutionUnit simulates memory allocation etc. so we can do a lot of pointer magic</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>If the IRInterpreter can’t run, the MCJIT produces machine code and LLDB runs it</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>IRExecutionUnit vends a custom JITMemoryManager implementation</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>It remembers memory allocations and where functions were placed</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>After JIT, all sections are placed into the target and we report their new locations with mapSectionAddress</div><div><br></div><div><b>Selected Thorny Issues (concentrating on JIT-related issues)</b></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Make the MCJIT more robust so we can rely on it more</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Support all Mach-O and ELF relocation types</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Don’t assume resolved symbols are in the current process</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Don’t assume addresses fit into void*s</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Make the IRInterpreter support all data types and instructions</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Completely replace the LLVM interpreter!</div><div><br></div><div><div>
<div>Sean</div>
</div>
<br></div></body></html>