[lldb-dev] RFC: Processor Trace Support in LLDB

[Walter via lldb-dev]

Hi Pavel, thanks for the comments. I'll reply inline

> The part that's not clear to me is what is the story with multi-process
traces. The file format enables those, but it's not clear how are they
going be created or used. Can you elaborate more on what you intend to
use those for?

Something we are doing at Facebook is having a global Intel PT collector
that can trace all processes of a given machine for some seconds. This can
produce a multi-process trace. I imagine these traces won't ever be
generated by LLDB, though. Having one single json trace file for this is
going to be useful for sharing the trace more easily. Multi-process tracing
is also something you can do with the perf tool, so It's not uncommon.

There are some technical details that are worth mentioning as well. Intel
PT offers two main modes of tracing: single thread tracing and logical CPU
tracing.
- The first one is the easiest to implement, but it requires having a
dedicated buffer per thread, which can consume too much RAM if there are
thousands of threads traced. It also adds a little bit of performance cost,
as the kernel disables and enables tracing whenever there's a context
switch.
- The other mode, logical CPU tracing, traces all the activity in one
logical core and uses one single buffer. Also it is more performant as the
kernel doesn't disable tracing intermittently. Sadly, that trace contains
no information regarding context switches, so a separated context switch
trace is used for splitting this big trace into per-thread subtraces. The
decoder we are implementing eventually will be able to do this splitting,
and it will require being fed with the information of all processes. This
is also a reason why allowing multi-process traces is important.

Regarding the commands structure, I'd prefer to keep it under "trace" for
now, because of the multi-process case and because we still have no users
that can report feedback. Very soon we'll start building some tools around
this feature, so we'll have more concrete experiences to share. Then it'll
be good to sync up and revisit the structure.

Btw, the gdb implementation of this kind of tracing is under the "record"
main command (
https://sourceware.org/gdb/current/onlinedocs/gdb/Process-Record-and-Replay.html).
I think this allows for some flexibility, as each trace plugin has
different characteristics.

> I'm not sure how Cap'n Proto comes into play here. The way I understand
it, the real data is contained in a separate file in the specialized intel
format and the json is just for the metadata. I'd expect the metadata file
to be small even for enormous traces, so I'm not sure what's to be gained
by optimizing it.

I didn't mention it in that email, but there is some additional information
that we'll eventually include in the traces, like the context-switch trace
I mentioned above. I think that we could probably use Cap'n Proto for cases
like this. We might also not use it at all as well, but it was nice to
learn about it and keep it in mind.


Thanks,
Walter

Il giorno gio 1 ott 2020 alle ore 07:08 Pavel Labath <pavel at labath.sk> ha
scritto:

> Thank you for writing this Walter. I think this document will be a
> useful reference both now and in the future.
>
> The part that's not clear to me is what is the story with multi-process
> traces. The file format enables those, but it's not clear how are they
> going be created or used. Can you elaborate more on what you intend to
> use those for?
>
> The main reason I am asking that is because I am thinking about the
> proposed command structure. I'm wondering if it would not be better to
> fit this into the existing target/process/thread commands instead of
> adding a new top-level command. For example, one could imagine the
> following set of commands:
>
> - "process trace start" + "thread trace start" instead of "thread trace
> [tid]". That would be similar to "process continue" + "thread continue".
> - "thread trace dump [tid]" instead of "trace dump [-t tid]". That would
> be similar to "thread continue" and other thread control commands.
> - "target create --trace" instead of "trace load". (analogous to target
> create --core).
> - "process trace save" instead of "trace save" -- (mostly) analogous to
> "process save-core"
>
> I am thinking this composition may fit in better into the existing lldb
> command landscape, though I also see the appeal in grouping everything
> trace-related under a single top-level command. What do you think?
>
> The main place where this idea breaks down is the multi-process traces.
> While we could certainly make "target create --trace" create multiple
> targets, that would be fairly unusual. OTOH, the whole concept of having
> multiple targets share something is a pretty unusual thing for lldb.
> That's why I'd like to hear more about where you want to go with this idea.
>
>
> On 21/09/2020 22:17, Walter via lldb-dev wrote:
> > Thanks for your feedback Fangrui, I've just been checking Capn' Proto
> > and it looks really good. I'll keep it in mind in the design and see how
> > it can optimize the overall data transfer.
>
> I'm not sure how Cap'n Proto comes into play here. The way I understand
> it, the real data is contained in a separate file in the specialized
> intel format and the json is just for the metadata. I'd expect the
> metadata file to be small even for enormous traces, so I'm not sure
> what's to be gained by optimizing it.
>
> pl
>
>

-- 
- Walter Erquínigo Pezo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20201001/40407681/attachment.html>