[PATCH] D24376: [XRay] Implement `llvm-xray convert` -- trace file conversion

[Dean Michael Berris via llvm-commits]

dberris added a comment.

In https://reviews.llvm.org/D24376#604850, @dblaikie wrote:

> In https://reviews.llvm.org/D24376#603715, @dberris wrote:
>
> > In https://reviews.llvm.org/D24376#602808, @dblaikie wrote:
> >
> > > (I was thinking more going the other direction: removing the dead code, rather than adding a use of it)
> > >
> > > What's the purpose of the binary/raw output format? (I think it was mentioned that the binary output format would be for testing - so we could write yaml, then generate raw files - then feed those into other tools for testing? But if the yaml input format is supported everywhere, what's the purpose of that? We'll want to have a couple of binary tests, but I imagine they would test checked in binary -> yaml output (if they test yaml -> binary -> yaml, then I'm not sure they achieve much because it's just roundtripping so we have as much code under test (binary -> yaml) as we do in the test harness (yaml -> binary), effectively)
> > >
> > > That's my mental model at least.
> >
> >
> > Other things going on concurrently (work on the FDR mode for the runtime library) is making me need to be able to turn versions of the binary log from one form to another. I've found that working with the binary versions is much more convenient from a tool perspective, and that this current simple format is more amenable to analysis than the condensed format I'm working on.
>
>
> I'm not sure I'm following here - could you describe this in more detail?


Sure, there's a patch I've uploaded recently that has a lot more about this different format (see https://reviews.llvm.org/D27038).

> My understanding was that tools would likely use these LLVM APIs for data handling (or they'd use YAML input) - so there wouldn't be any benefit to any particular format (except YAML) since these LLVM APIs could handle all the formats. This would make the binary format(s) fairly niche - just convenient for runtime emission, but not desirable for any other use.
> 
> Where have I gone wrong here?

That's one half of the story.

The other half is the new log format (still binary) is not conducive to this simple approach of loading up records with full information. The FDR mode log format is substantially different, in that per-thread buffers are chunked up into fixed-sized blocks and each record might be a metadata record (16 bytes) or a function record (8 bytes). That log has all sorts of information consolidated in very different ways (e.g. the thread ID is stored once, as a metadata record at the beginning of a buffer, CPU id's only show up when the CPU is first encountered, we store TSC deltas instead of full TSCs in function records), and is not conducive to just "load in memory, read records one by one".

So then, the conversion tool ought to be able to convert the FDR mode logs into this simpler log format that we already support in this tool -- then let all the other tools upcoming (accounting, timeline visualisation, etc.) just use the simpler log format.

The closest analogy I can make for existing tools that do something similar would be 'dot' and friends.

>> So effectively, version 2 of the log format is turning out to have both non-fixed-size records and more interleaving happening. This tool, that's able to take that complex format into something simpler/different doesn't make sense for generating just YAML files.
>> 
>>> (this commit seems to add several .bin test files - but I don't see where those are used?)
>> 
>> Some of the tests are looking at symbolization of the function id's, and associating functions with debug info, etc. for coverage of the various modes by which the YAML output could be generated.
> 
> Not quite following, more practically: which tests are these files used in?

convert-with-debug-syms.txt
convert-with-standalone-instrmap.txt
convert-with-yaml-instrmap.txt


https://reviews.llvm.org/D24376