[cfe-dev] [Templight] Templight "v2" with gdb-style debugger

[Nick Lewycky]

On 11 March 2014 18:28, Mikael Persson <mikael.s.persson at gmail.com> wrote:

> Hi all,
>
> I recently picked up the Templight code (see
> http://plc.inf.elte.hu/templight/) because I love the idea of having
> proper debugging and profiling tools for template meta-programming in C++.
> As admitted by the original authors (Zoltán Borók-Nagy, Zoltán Porkoláb
> and József Mihalicza), the code was rather crude, but a great first step.
>
> -- Templight "version 2" --  (patch: templight_v2_clang35.diff)
>
> So, I refactored the code to make it a bit more modular and less intrusive
> on the Sema class. Here is an outline of the main changes included in the
> attached patch (from Clang r202671):
>

Hi Mikael,

Sorry to lead with this, but it's important. The llvm project requires that
patches be submitted by their author (or authors). If you grabbed the
original templight code and refactored it, then it may be coauthored
between you and the original authors. Our policy requires that an author
submit their own code, not mail in someone else's, see
http://llvm.org/docs/DeveloperPolicy.html#attribution-of-changes for
specifics.

Changes to existing Clang code-base:
>
>  (1) Moved the "ActiveTemplateInstantiation" class out of the Sema class
> scope, and into the clang namespace-scope. This was done to make the
> templight code independent from the Sema class by allowing a
> forward-declaration of ActiveTemplateInstantiation (can't be done if it is
> nested in Sema).
>

Great! I think that ActiveTemplateInstantiation should be changed to a form
of InstantiationRecord and held from the relevant AST nodes.

  (2) Minor updates to reflect the above change in scope of
> the ActiveTemplateInstantiation class.
>

The changes to ATI to pull it out of Sema and make it part of the AST are
complicated enough, I think you should send that out as a first patch. It
may not be much code, but for review we'll want to make sure that the
representation is accurate to what the standard says, efficient in memory
usage and has fast accessors.

There's also template instantiation for different reasons which are not
currently recorded (ie., due to overload resolution, due to virtual
methods). Since that's a change to ATI, that would logically go next.

 (3) Created a "TemplateInstantiationObserver" base-class which bundles
> callback functions that are called during the template instantiations (and
> when leaving them). This class also supports linked-list-style chaining of
> additional observers.
>

I think we usually use "Callbacks" instead of "Observer", similar to
PPCallbacks?

  (4) Added a OwningPtr<TemplateInstantiationObserver> in the Sema class to
> hold the chain of template instantiation observers.
>
>  (5) Added callbacks for begin / end of template instantiations. These
> were added in all the places where the original Templight code had
> templight calls. This replaces the original templight tracing calls.
>

At a high level, I think that what we currently do in Sema by tracking
template instantiations as a stack (Sema::ActiveTemplateInstantiations)
should be available in the AST after instantiation is complete. Then we
could query the AST to ask why something was instantiated instead of
tracing the act of instantiation. Is there any use case for templight where
this wouldn't work?

Nick

 (6) Added calls to initialize and finalize the template instantiation
> observers in the ParseAST code (see issues below).
>
>  (7) Added front-end options for Templight (mostly the same as in the
> original templight patch).
>
> Changes to the Templight implementation: (unless noted here, the behavior
> is the same)
> N.B.: The output (yaml, xml, txt) is *identical* to the original Templight
> code.
>
>  (8) All the templight code was moved out of the Sema class and into its
> own cpp-file. This was done mainly to avoid additional pollution in the
> Sema class, and to isolate the templight code's implementation dependencies
> (llvm support libs, yaml output code, etc.).
>
>  (9) Created the TemplightTracer class (derived from
> TemplateInstantiationObserver) to expose the functionality of Templight.
> This class is PImpl'd.
>
>  (10) Removed the "TemplightFlag" in favor of having created the tracer or
> not.
>
>  (11) Removed the "-templight-capacity" option because I changed the
> non-safe-mode to record the instantiation traces up to when the
> instantiations come back to the top-level context, at which point, I dump
> the traces. This will not affect the profiling because at the top-level
> context there is no pending timing or memory to be recorded. With this
> change, the number of traces recorded become proportional to the maximum
> instantiation depth (which is limited by the compiler), therefore making
> the templight-capacity redundant.
>
>  (12) Changed the recording of the traces to be stored in a std::vector,
> as opposed to a bare dynamic array.
>
>  (13) Added the "-templight-ignore-system" option to ignore (not traced)
> any template instantiation coming from a system-include header file.
>
>  (14) Updated the "creation" code where front-end options are pulled and
> used to create the tracer.
>
>
> -- Templight GDB-style Debugger --  (patch:
> templight_v2_clang35_with_debugger.diff)
>
> Here is the really sweet part. When I realized how easy it would be to
> create a gdb-style debugger with the new layout that I described above, I
> just had to give it a go, and the result is pretty nice, albeit still
> crude. I welcome you to try it!
>
> I added the option "-templight-debugger" which turns on an interactive
> debugging session during the template instantiations. This is implemented
> in the TemplightDebugger class (also PImpl'd, and also deriving
> from TemplateInstantiationObserver). The debugger respects the relevant
> templight options (memory and ignore-system) and can be used in parallel
> with the templight tracer (although timing values will obviously be
> useless).
>
> The debugger implements most of the basic gdb commands and reproduces its
> behavior as much as possible. You can set breakpoints (by complete name of
> template instantiation), you can "step", "next", "run", "kill", etc.. with
> the same behavior as gdb. You can print back-traces. And each time a
> template instantiation is entered or left, there is a structured print out
> similar to gdb, making it usable (I hope) in a GUI front-end similar to the
> various GUI front-ends to gdb.
>
> This is really just a first draft of it, it's quite crude. I use C-style
> functions for the console input to avoid pulling in the C++
> iostream monster (as required in coding standards of LLVM). And, as usual
> with interactive programs, it's hard to make sure that it is robust to
> people entering random nonsense.
>
> Here are a few issues I'd like some feedback on:
>
>  (15) I feel that the XML output of the templight tracer could easily be
> nested, as opposed to having matched "Begin" and "End" blocks.
>
>  (16) I feel a bit uneasy about the place where the tracer / debugger get
> created, initialized and finalized. Currently, this is done in the
> "createSema" function and in the ParseAST code. It works this way, but it
> feels out-of-place. I don't know the Clang code-base well enough (in fact,
> barely at all) to know where this should go. Ideally, I would also like to
> get rid of the initialize / finalize functions in
> the TemplateInstantiationObserver class, but that is impossible as long as
> the Sema object and most of Clang's main structures are left to leak
> (always-on "-delete-free" option), as RAII is no longer available as a
> mechanism for handling the clean up. If anyone that understands Clang's
> main structures could weight in on this, I'd be glad to hear suggestions.
>
>  (17) Is there anything in LLVM/Clang that can help in terms of doing
> console input? Currently, I use <cstdio> functions and hand-written
> "tokenizing".
>
>  (18) I feel like the template debugger should be a separate entity (not
> an option to the main clang compiler), but at the same time, it needs the
> entire compilation process to be running, and so, if separate, it would
> have to be duplicate of clang with only small additions (in terms of code).
>
>  (19) Finally, I'd like to have ideas about what kind of diagnostic
> information that could be added to the debugger. Currently, it only prints
> out the name of the template instantiation, the location, and total memory
> usage. It would be nice to have features similar to GDB's "print" commands,
> like being able to print the list of template arguments and their values
> within the current context, and things like that. I simply don't have
> enough knowledge of Clang's code to know how to tap into that kind of
> information.
>
>
> I know this was a long email, but I hope you had the interest to read up
> to here, and I hope to hear back.
>
> Cheers,
> Mikael.
>
>
> --
> Sven Mikael Persson, M.Sc.(Tech.)
> PhD Candidate, McGill University,
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20150106/a4effd96/attachment.html>