<div dir="ltr">Hi Walter,<div><br></div><div>I've only done a brief scan of the document but, in general, I'm favorable of the goals, aim, and approach. Something I think would be good would be to compare/contrast against rr as an "exploring alternatives" section of the document. I think the document should also be made available/adapted to be part of the documentation on "why lldb is implementing this feature/what it can be used for/why".</div><div><br></div><div>Thanks so much for starting this and looking forward to the work and collaboration.</div><div><br></div><div>-eric</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Sep 17, 2020 at 8:28 PM Walter via lldb-dev <<a href="mailto:lldb-dev@lists.llvm.org">lldb-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Hi all,</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Here I propose, along with Greg Clayton, Processor Trace support for LLDB. I’m attaching a link to the document that contains this proposal if that’s easier to read for you: <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1cOVTGp1sL-5FHBXjP9eB7qjVtDNr5xnuZvUUtv43G5eVI_edit-23heading-3Dh.t5mblb9ugv8f&d=DwMGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=erxV6KMIZvIQjyWYW8YpOiKz-WqJt4giKQA34YMHsRY&m=DuuwXHUQJpW4TcCay4hPsBund-eBI2uVaVimqEPsp5k&s=o6vqoYYbn-Tz_d34hoLJvWhEnnhracOO6yDsMzq8wR0&e=" style="color:rgb(5,99,193)" target="_blank">https://docs.google.com/document/d/1cOVTGp1sL_HBXjP9eB7qjVtDNr5xnuZvUUtv43G5eVI/edit#heading=h.t5mblb9ugv8f</a>. Please make any comments in this mail list.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">If you want to quickly know what Processor Trace can do, you can read this <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__easyperf.net_blog_2019_08_23_Intel-2DProcessor-2DTrace&d=DwMGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=erxV6KMIZvIQjyWYW8YpOiKz-WqJt4giKQA34YMHsRY&m=DuuwXHUQJpW4TcCay4hPsBund-eBI2uVaVimqEPsp5k&s=iaErHaf8byXlZb1YFUk0BpQ-duMhNouUUMyktLm3soQ&e=" style="color:rgb(5,99,193)" target="_blank">https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace</a>.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Any comments are appreciated, especially the ones regarding the commands the user will interact with. </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Thanks,</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Walter Erquinigo.</pre><div style="color:rgb(0,0,0);font-family:-webkit-standard;border-style:none none solid;border-bottom-width:1pt;border-bottom-color:windowtext;padding:0in 0in 1pt"><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";border:none;padding:0in"> </pre></div><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"># RFC: Processor Trace Support in LLDB</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"># What is processor tracing?</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Processor tracing works by capturing information about the execution of a process so that the control flow of the program can be reconstructed later. Implementations of this are Intel Processor Trace for X86, x86_64 ([<a href="https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html%5D(https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html)" style="color:rgb(5,99,193)" target="_blank">https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html](https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html)</a>) and ARM CoreSight for some ARM devices ([<a href="https://developer.arm.com/ip-products/system-ip/coresight-debug-and-trace%5D(https://developer.arm.com/ip-products/system-ip/coresight-debug-and-trace)" style="color:rgb(5,99,193)" target="_blank">https://developer.arm.com/ip-products/system-ip/coresight-debug-and-trace](https://developer.arm.com/ip-products/system-ip/coresight-debug-and-trace)</a>). </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">As a clarifying example, with these technologies it’s possible to trace all the threads of a process, and after the process has finished, reconstruct every single instruction address each thread has executed. This could include some additional information like timestamps, async CPU events, kernel instructions, bus clock ratio changes, etc. On the other hand, memory and registers are not traced as a way to limit the size of the trace.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"># Intel Processor Trace as the first implementation</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">We’ll focus on Intel Processor Trace (Intel PT), but in a generic way so that in the future similar technologies can be onboarded in LLDB.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Intel PT has the following features:</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Control flow tracing in a highly encoded format</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* 3% to 5% slowdown when capturing</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* No memory nor registers captured</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Kernel tracing support</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Timestamps of branches are produced, which can be used for profiling</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Adjustable size of trace buffer</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Supported on most Intel CPUs since 2015</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* X86 and x86_64 only</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Official support only on Linux</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Basic support on Windows</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Decoding/analysis can be done on any operating system </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">A very nice introduction to Intel PT can be found [<a href="https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html%5D(https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html)" style="color:rgb(5,99,193)" target="_blank">https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html](https://software.intel.com/content/www/us/en/develop/blogs/processor-tracing.html)</a> and [<a href="https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace%5D(https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace)" style="color:rgb(5,99,193)" target="_blank">https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace](https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace)</a>. Totally recommended to fully grasp the impact of this project. </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">More technical details are in [<a href="https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/perf-intel-pt.txt%5D(https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/perf-intel-pt.txt)" style="color:rgb(5,99,193)" target="_blank">https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/perf-intel-pt.txt](https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/perf-intel-pt.txt)</a>. </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Even more technical details are in the processor manual [<a href="https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-3c-part-3-manual.pdf%5D(https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-3c-part-3-manual.pdf)" style="color:rgb(5,99,193)" target="_blank">https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-3c-part-3-manual.pdf](https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-3c-part-3-manual.pdf)</a> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"># Basic Definitions</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Trace file: A trace file basically contains the information of the target addresses of each branch or jump within the program execution in a highly encoded format.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Capturing: The act of tracing a process and producing a trace file.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Decoding: Decoding outputs a sequential list of instructions given a trace file and the images of a process. Decoding is generally an offline step as it’s expensive.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Trace buffer: In order to limit the size of the trace, an on-memory circular buffer can be used, keeping the most recent branching information. The trace file is a snapshot of this.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Gap: Sporadically some branching information can be lost or be impossible to decode, which creates a gap in the reconstructed control flow.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"># New LLDB features</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Loading traces: We want to load traces potentially from other computers, and have LLDB symbolicating it. A flow like the following should be possible \</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ```</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> $ trace load /path/to/trace</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> $ trace dump --instructions</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> pid: '1234', tid: '1981309'</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> a.out`main</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [57] 0x400549 <+13>: movl %eax, -0x4(%rbp)</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> a.out`bar()</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [56] 0x40053b <+46>: retq</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [55] 0x40053a <+45>: leave</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [54] 0x400537 <+42>: movl -0x4(%rbp), %eax</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [53] 0x400535 <+40>: jle 0x400525 ; <+24> at main.cpp:7</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [52] 0x400531 <+36>: cmpl $0x3, -0x8(%rbp)</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [51] 0x40052d <+32>: addl $0x1, -0x8(%rbp)</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [50] 0x40052a <+29>: addl %eax, -0x4(%rbp)</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> a.out`foo()</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [49] 0x400567 <+15>: retq</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [48] 0x400566 <+14>: popq %rbp</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [47] 0x400563 <+11>: movl -0x4(%rbp), %eax</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [46] 0x40055c <+4>: movl $0x2a, -0x4(%rbp)</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ...</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [1] 0x400559 <+1>: movq %rsp, %rbp</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [0] 0x400558 <+0>: pushq %rbp</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> // Format:</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ```</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ` // [instruction index] <instruction disassembly> \</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">`Notice the resemblance to loading a core file, but in this case we can get the control flow, printed in reverse order in this example.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Decoding: LLDB can use libipt ([<a href="https://github.com/intel/libipt%5D(https://github.com/intel/libipt)" style="color:rgb(5,99,193)" target="_blank">https://github.com/intel/libipt](https://github.com/intel/libipt)</a>), which is the low level Intel PT decoding library, to convert trace files into instructions.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Showing instructions: LLDB can output the list of instructions of the control flow, as shown above</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Showing function calls: Similarly, LLDB can print a hierarchical view of the function calls. A flow like this should be possible: \</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ```</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> $ trace load /path/to/trace</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> $ trace dump --function-calls</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> pid: '1234', tid: '1981309'</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [50] a.out`bar() 0x40052a</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [45] a.out`zaz() 0x400558 </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [40] a.out`baz() 0x400559 </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [30] a.out`foo() 0x400567</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ```</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ` [0] a.out`main 0x400000 \</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> \</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">`This functionality allows LLDB to reconstruct the call stack at any point and potentially do reverse debugging.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Capturing: LLDB can also do the Intel PT capturing of a live process, so that at any stop the user can do reverse stepping or simply inspect the trace. A possible flow is:</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ```</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> $ <stopped at main></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> $ b main.cpp:50</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> $ trace start intel-pt // this initiates the tracing</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> $ continue</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> $ <stopped at main.cpp:50></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> $ trace dump --instructions</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">pid: '1234', tid: '1981309'</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> a.out`main</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [57] 0x400549 <+13>: movl %eax, -0x4(%rbp)</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> a.out`bar()</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [56] 0x40053b <+46>: retq</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [55] 0x40053a <+45>: leave</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ```</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> Displaying time information: If the trace contains timing information, we could also display it along with each instruction, e.g.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ```</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> a.out`bar()</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [56: 1600284226]: 0x40053b <+46>: retq</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ...</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [4: 1600284200]: 0x40053a <+45>: leave</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> // Format:</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> // [instruction index: unix timestamp] <instruction disassembly></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ```</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> Furthermore, we could display the time spent in each function.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"># Future LLDB features</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Reverse Stepping: With the hierarchical reconstruction of the function calls, along with the individual instructions, LLDB can offer reverse stepping. Operations like reverse-next, reverse-step-out, reverse-continue could work by traversing the trace. We plan to work on this once the features presented above are in place.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Trace-based profiling</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* SB API of the mentioned features</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"># Why is this useful?</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Bug root-causing:</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> * For example, a crash in a production Release build ends up being analyzed with logs, a coredump, and a stack trace. Logs are not comprehensive, and a stack trace only contains the final state of the program. Providing the user with the control flow of the last milliseconds gives a tremendous amount of information that is game-changing in root-causing issues. It could be said that the user goes from a single stack trace to a list of stack traces.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> * Reverse stepping enables more efficient debugging, as it reduces the number of iterations to efficiently root-cause bugs. More often than not, reproducing a bug takes a considerable amount of time, and the user needs to reproduce it several times until the correct breakpoints are hit. This takes a considerable amount of time. Giving the user the information of what has been executed so far can help them figuring out where’s the location to place a breakpoint, or to very easily figure out what went wrong.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Low cost: unlike other similar technologies, Intel PT has an almost negligible performance cost regardless of whether the build is optimized or not, making it appealing to a wide range of scenarios.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* This infrastructure can be used for enabling other tools like non-sample-based profilers with instruction-level accuracy, security analyzers that check if certain memory regions are executed, and trace comparators, which could find bugs by comparing similar traces.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"># Goals of this document:</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Gather feedback on the basic Trace implementation, which would include the following basic operations: loading, decoding, and dumping.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Create awareness about this work.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Get a green light on the current set of patches implementing this feature starting with <a href="https://reviews.llvm.org/D85705" style="color:rgb(5,99,193)" target="_blank">https://reviews.llvm.org/D85705</a>.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"># Non-Goals:</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Discuss how reverse-stepping will be implemented. This can be left for another discussion. Once the Trace architecture is in place and robust, reverse-stepping can then be discussed, as it’s a more controversial change than this one.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Explain thoroughly Intel PT. </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"># Existing Tool Support</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* GDB has a basic implementation of the features above ([<a href="https://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-Replay.html%5D(https://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-Replay.html)" style="color:rgb(5,99,193)" target="_blank">https://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-Replay.html](https://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-Replay.html)</a>) and some ideas are taken from there.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Perf is a standalone tool that can do capturing and decoding.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* The Linux kernel has full support for doing capturing at thread, logical cpu or cgroup level. </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">* Intel developed a basic version of Intel PT support in LLDB as an external plugin. [<a href="https://reviews.llvm.org/D33674%5D(https://reviews.llvm.org/D33674)" style="color:rgb(5,99,193)" target="_blank">https://reviews.llvm.org/D33674](https://reviews.llvm.org/D33674)</a>, [<a href="https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b%5D(https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b)" style="color:rgb(5,99,193)" target="_blank">https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b](https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b)</a>. </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"># New Trace Commands</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Based on this patch [<a href="https://reviews.llvm.org/D85705%5D(https://reviews.llvm.org/D85705)" style="color:rgb(5,99,193)" target="_blank">https://reviews.llvm.org/D85705](https://reviews.llvm.org/D85705)</a>, there would be a common Trace class along with plug-in implementations.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">## Trace loading</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">### $ trace load /path/to/trace/settings/file.json</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">As decoding a trace requires the images of the object files, the trace files and some CPU information, it’s convenient to have a JSON file that describes an entire trace session. The following JSON schema could be used.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">```</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">{</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">"trace": { </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> … // plug-in specific information </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> },</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "processes": [ // process information common to all trace plug-ins</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> {</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "pid": integer,</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "triple": string, // llvm-triple</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "threads": [</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> {</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "tid": integer,</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "traceFile": string</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> }</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ],</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "modules": [</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> {</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "systemPath": string, // original path of the module at runtime</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "file"?: string, // copy of the file if not available at "systemPath"</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "loadAddress": string, // string address in hex or decimal form</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "uuid"?: string,</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> }</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ]</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> }</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ]</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">}</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">// Notes:</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">// All paths are either absolute or relative to the settings file.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">```</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">**Corefiles:**</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">We plan to extend this schema to support corefiles, but we would leave it out of this discussion, as can be easily seen as an extension of this basic schema.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">**Implementation details:**</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">To make our first implementation easier, we’ll ask for an individual trace file per thread. This is the simpler collection mode for Intel PT. </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">The entire json file will be translated into a Trace object, which contains the trace information of each thread and process in it. </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Each process in the json file will be represented as a new Target. Similarly, threads and modules for each target will be created following the json file. This is very similar to what loading a minidump or coredump does.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Each Target will be associated with a Trace, and multiple targets can share the same Trace. The contract is that Trace is assumed to end at the current PC of each thread of the target.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">### $ trace schema <plug-in></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">This command prints the JSON schema of the trace settings file for the provided plug-in. It would output something similar to this</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">```</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">{</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">"trace": { </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "type": "intel-pt",</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "pt_cpu": {</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "vendor": "intel" | "unknown",</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "family": integer,</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "model": integer,</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "stepping": integer</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> }</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> },</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "processes": [ </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> {</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "pid": integer,</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "triple": string, // llvm-triple</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "threads": [</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> {</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "tid": integer,</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "traceFile": string</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> }</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ],</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "modules": [</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> {</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "systemPath": string, // original path of the module at runtime</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "file"?: string, // copy of the file if not available at "systemPath"</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "loadAddress": string, // string address in hex or decimal form</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> "uuid"?: string,</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> }</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ]</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> }</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> ]</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">}</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">// Notes:</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">// All paths are either absolute or relative to the settings file.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">```</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">### $ trace dump [--verbose] [-t tid1] [-t tid2] ...</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Print the trace information corresponding to the provided thread ids of the currently selected target, which would mainly include the same information as the trace settings file. If no tid is provided, the currently selected thread is used. This would be useful for debugging. The information would be like</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> Modules: </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> <module info like systemPath, file, load address, uuid, size></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> Threads:</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> <thread info like location of trace file, number of instructions (if already decoded), number of function calls (if already decoded)></pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">If <--verbose> is passed, the original settings.json file is printed as well.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">## Decoder-based commands</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">The following commands require decoding the trace and are of the form. “trace dump <action> [-t <tid>]”. If tids are not specified, then the current thread or the current target will be used.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">### $ trace dump --instructions [-t <tid>] [-c <count> = 10] [-o <offset> = 0]</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">This command would print the last <count> instructions starting at the given offset from the last instruction in the trace. The output would be similar to that of the “disassembly” command and would include timing information if available.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">```</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> $ trace dump --instructions -c 5</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> pid: '1234', tid: '1981309'</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> a.out`main</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [57] 0x400549 <+13>: movl %eax, -0x4(%rbp)</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> a.out`bar()</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [56] 0x40053b <+46>: retq</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [55] 0x40053a <+45>: leave</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [54] error -13. 'no memory mapped at this address'</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> a.out`foo()</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [53] 0x400567 <+15>: retq</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">```</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Repeating the command would continue printing where it was left off in the last run.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">**Implementation details:**</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Each instruction output by the decoder is either an actual instruction or an error. An error can be caused due to a collection error (e.g. internal CPU buffer overflow error) or a decoding error (e.g. the image of an object file is missing while decoding). These errors represent gaps in the trace and the user should know about them, so we print them accordingly in this dump.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Each instruction (including errors) has an index in the decoded trace, and serves as a checkpoint.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">### $ trace dump --function-calls [-t <tid>] [-c <count> = 10] [-o <offset> = 0] [--flat]</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">This command would print the hierarchical list of function calls. Similar to the “--instructions” command, it would show the last <count> function calls with the given offset from the last instructions. Timing information would be included if available.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">```</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> $ trace dump --function-calls</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> pid: '1234', tid: '1981309'</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [50] a.out`bar() 0x40052a</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [45] a.out`zaz() 0x400558 </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [40] a.out`baz() 0x400559 </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [30] a.out`foo() 0x400567</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> [0] a.out`main 0x400000</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">```</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Repeating the command would continue printing where it was left off in the last run.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">If <--flat> is passed, then instead of a hierarchical view, a flat list would be produced.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">## Capturing command</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">### $ trace start <plugin_name> [-t <tid>] [--all] [-b <buffer_size_in_KB>]</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">This command will start tracing the given thread of the currently selected target, or all the threads of that target if “--all” is passed. If “--all” is passed, any thread created after this command will also be traced automatically.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Besides, the optional -b parameter can define the size of each trace buffer to be created. I haven’t yet decided a default one, but 1M might be acceptable, as it traces around 1 million instructions on average according to Intel, and that’s more than enough for a useful analysis.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">For an initial implementation, the plugin_name parameter will be required (e.g. intel-pt). Later a more automated mechanism for finding the right plugin can be implemented.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">**Implementation notes:**</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">There’s already a basic implementation in lldb as an external plugin. It’s in [<a href="https://reviews.llvm.org/source/llvm-github/browse/master/lldb/tools/intel-features/intel-pt/%5D(https://reviews.llvm.org/source/llvm-github/browse/master/lldb/tools/intel-features/intel-pt/)" style="color:rgb(5,99,193)" target="_blank">https://reviews.llvm.org/source/llvm-github/browse/master/lldb/tools/intel-features/intel-pt/](https://reviews.llvm.org/source/llvm-github/browse/master/lldb/tools/intel-features/intel-pt/)</a> created by [<a href="https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b%5D(https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b)" style="color:rgb(5,99,193)" target="_blank">https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b](https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b)</a>. It hasn’t received much attention and has been mostly unmaintained since it was created. It’s already capable of tracing a given thread and collecting the trace buffer. We plan to reuse that logic, which is already working.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">A Trace object will be created and will be associated with the current Target.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Any interaction with trace, like dumping instructions, will trigger a fetch of the most recent trace buffer, unless it hasn’t changed. </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">When multiple threads are traced, each one will have its own trace buffer, as sharing one buffer in multiple threads requires knowing when each context switch happened so that the decoded trace can be split correctly among threads. This is beyond the scope of the initial version of this project.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">### $ trace save /path/to/file.json [--copy-images]</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">This creates a bundle trace with settings saved in the given json file for the current process. By default, it doesn’t create any copy of the images loaded on the process, unless the “--copy-images” parameter is specified. That parameter is useful for analyzing the trace in a machine other than where it was captured. </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"># Remote Protocol Changes</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">No remote protocol changes are required, as [<a href="https://reviews.llvm.org/D33674%5D(https://reviews.llvm.org/D33674)" style="color:rgb(5,99,193)" target="_blank">https://reviews.llvm.org/D33674](https://reviews.llvm.org/D33674)</a> and [<a href="https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b%5D(https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b)" style="color:rgb(5,99,193)" target="_blank">https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b](https://reviews.llvm.org/rG307db0f8974d1b28d7b237cb0d50895efc7f6e6b)</a> already created them some years ago.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"># Build Requirements</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">In order to build LLDB with this support, it has to be linked with a build of libipt [<a href="https://github.com/intel/libipt%5D(https://github.com/intel/libipt)" style="color:rgb(5,99,193)" target="_blank">https://github.com/intel/libipt](https://github.com/intel/libipt)</a>, which is the decoder.</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"># Operating System Requirements for Collection/Tracing</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">Collection can only be done on linux if the file /sys/bus/event_source/devices/intel_pt/type is defined. The logic gating this feature is already checked in and defined in [<a href="https://reviews.llvm.org/D33674%5D(https://reviews.llvm.org/D33674)" style="color:rgb(5,99,193)" target="_blank">https://reviews.llvm.org/D33674](https://reviews.llvm.org/D33674)</a>. </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"># Testing</pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)"> </pre><pre style="margin:0in 0in 0.0001pt;font-size:10pt;font-family:"Courier New";color:rgb(0,0,0)">It’s fortunately straightforward to test this feature. It’s possible to capture traces with perf or with the future “trace start” / ”trace save” commands and create trace bundles with their corresponding settings .json file. Analyzing those traces should give the same results on any machine, making testing deterministic. [<a href="https://reviews.llvm.org/D85705%5D(https://reviews.llvm.org/D85705)" style="color:rgb(5,99,193)" target="_blank">https://reviews.llvm.org/D85705](https://reviews.llvm.org/D85705)</a> and descendents already implement some deterministic tests.</pre></div></div></div></div>
_______________________________________________<br>
lldb-dev mailing list<br>
<a href="mailto:lldb-dev@lists.llvm.org" target="_blank">lldb-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev</a><br>
</blockquote></div>