<div dir="ltr"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">> Do you plan to include the backends in the clang (tools) repository?<br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">> I think it would be great to have at least one reference backend next to the frontend. <span> </span><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span><br></span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span>Yes, there's currently YAML and Markdown generators in the works (see <a href="https://reviews.llvm.org/D43667">https://reviews.llvm.org/D43667</a> and <a href="https://reviews.llvm.org/D43424">https://reviews.llvm.org/D43424</a> for rough details, the markdown one in particular is still in flux).</span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span><br></span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">> Building an AST can be quite expensive. Do you plan to support generating documentation as a by-product of building the code? Similar to how indexing while building</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">> was proposed by Apple.<br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">A good idea -- for right now, that's not supported, but I'd definitely be interested in extending it to be a by-product.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">> Do you plant the intermediate representation to be self-contained, or the backends will need access to the original files (available at the original paths)?</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">The representation is self-contained.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Please let me know if you have more questions!</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Julie</div><br></div><br><br><div class="gmail_quote"><div dir="ltr">On Mon, Mar 26, 2018 at 11:40 AM Gábor Horváth <<a href="mailto:xazax.hun@gmail.com">xazax.hun@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Julie,<br><br></div><div>I saw that the first bits already landed. Congratulations, great work! :)<br>I have a few questions.<br></div><div>Do you plan to include the backends in the clang (tools) repository?<br></div><div>I think it would be great to have at least one reference backend next to the frontend.  <br></div><div>Building an AST can be quite expensive. Do you plan to support generating documentation as a by-product of building the code? Similar to how indexing while building was proposed by Apple.<br></div><div>Do you plant the intermediate representation to be self-contained, or the backends will need access to the original files (available at the original paths)?<br></div><div><br></div><div>Regards,<br></div><div>Gábor<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 4 December 2017 at 21:21, Julie Hockett via cfe-dev <span dir="ltr"><<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span id="m_-5850506797740018462m_-2731000931905328505gmail-docs-internal-guid-4ab54f72-232a-73d9-2464-0a2aec30f523"><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent:36pt"><span style="text-indent:36pt;background-color:transparent;font-size:11pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap">This proposal is to build a new Clang tool for generating C/C++ documentation, similar to other modern documentation generators such as rustdoc</span><span style="text-indent:36pt;background-color:transparent;font-size:11pt;font-family:Arial;color:rgb(0,0,0);vertical-align:baseline;white-space:pre-wrap">.  This tool would be a modular and extensible documentation generator for C/C++ code. It also would introduce a more elegant way of documenting C/C++ code, as well as maintaining backwards-compatibility with Doxygen-style markup to generate documentation.</span><br></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent:36pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Today, Doxygen</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> is a de-facto standard for generating C/C++ documentation. While widely used, the tool itself is a bit cumbersome, its output is both aesthetically and functionally lacking, and the non-permissive license combined with outdated codebase make any improvements difficult. This new tool would aim to simplify the overhead of generating documentation, integrating it into a Clang tool as well as allowing existing comments to continue to be used. It would also allow for relatively easy adaptation to new language features, as it would be built on the Clang parser and would use the Clang AST to generate documentation.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">Proposed Tool</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent:36pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">The proposed tool would consist of two parts. First, it would have a frontend that consumes the Clang-generated AST and generates an intermediate representation of the code and documentation structure, including additional Markdown files. Second, it would have a set of backend modules that consume that representation and output documentation in a particular format (e.g. Markdown, HTML/website, etc.).</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent:36pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">The frontend would be a new tool that uses the Clang parser, which can already parse C/C++ documentation comments (using </span><span style="font-size:11pt;font-family:Inconsolata;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">-</span><span style="font-size:11pt;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font face="monospace, monospace">Wdocumentation</font></span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> option). It can be easily used through the LibTooling interface</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">, similarly to other Clang tools such as clang-check or clang-format</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">. The initial steps in this project would be to build this tool using Clang's documentation parser. This tool would be able to attach comments to both functions, types, and macros and resolve declaration references, both of which will be useful in generating effective documentation. Since a good deal of existing C/C++ code uses the Doxygen documentation comment style, which is also supported by Clang's parser (and Doxygen itself can use Clang to parse these comments</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">), this is the syntax we are going to support as well. In the future, we would also like to support Markdown-style comments, akin to Apple Swift Markup</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">. </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent:36pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">For implementation, this tool will use the JSON Compilation Database format</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> to integrate with existing build systems. It would also have subcommands to choose which parts of the code will be documented (e.g. all code, all public signatures, all comment-documented signatures). Once the code is processed, the tool will write out the internal representation of the the documentation in an intermediate representation, encapsulating the necessary information about the code, comments, and structure. This will allow backend tools to take the output and transform it as necessary.</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent:36pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">The backend modules would cover different possible outputs for the defined intermediate representation. Each module will consume the representation and output documentation in a specific format. Initially, we propose to focus on a module that generates Markdown files, in order to make the first version as simple as possible. Markdown files are automatically rendered on a number of sites and systems, as well as being clear and uncluttered in raw text form. It is also relatively easy to convert Markdown files into other formats, making it a good starting target. An additional module would target HTML/website output.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">Intermediate Format</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent:36pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">The frontend would process the code and comments into an output, to be consumed by the backend. This representation would be internally represented as a set of classes and structs. Once the frontend has finished, it would write this representation to a file. While existing tools like Doxygen emit XML, XML is somewhat restrictive and bulky. Also, in order to fully use XML, the tool would need to define the representation twice (once for the internal classes/structs, once in the XML schema). So, we are instead considering two possible formats for this intermediate step: LLVM bitstream and JSON/YAML. </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent:36pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">LLVM bitstream format is space-efficient, and is natively written out by the Clang parser. It has the benefit of being similar to existing clang functionality, as the compiler frontend writes out its AST into the bitstream format to pass along to the LLVM backend. Using this format would allow the tool to emit the representation with minimal manipulation or additional parsing. </span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent:36pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Alternatively, JSON/YAML, while less space-efficient than bitstream, are human-readable and widely extensible. Neither has formal grammar or namespacing support, so if the tool needed rules of the sort it would need to define them itself on the frontend and require that the backend modules know them. While this would require a bit more parsing to emit on the frontend and load on the backend, the representation would be able to stand separately from the tool, and the backend modules would not necessarily need an understanding of the LLVM bitstream to load it.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">Extensions</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent:36pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">In addition to generating documentation from comments, a future extension would be to automatically generate and insert boilerplate comments into the code on demand. As the tool would have access to the AST, it could insert comments into the code similar to how tools like clang-tidy and clang-format adjust the code. Such generated comments would follow the documentation style for comments, and so would generate basic, if not wholly described, documentation, including information about parameters, return types, class members, etc. For example, the following would be generated for the below function:</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font face="monospace, monospace"><span style="font-size:11pt;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><span class="m_-5850506797740018462m_-2731000931905328505gmail-Apple-tab-span" style="white-space:pre-wrap">        </span></span><span style="font-size:11pt;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">/// Do Things</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font face="monospace, monospace"><span style="font-size:11pt;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><span class="m_-5850506797740018462m_-2731000931905328505gmail-Apple-tab-span" style="white-space:pre-wrap">    </span></span><span style="font-size:11pt;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">///</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font face="monospace, monospace"><span style="font-size:11pt;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><span class="m_-5850506797740018462m_-2731000931905328505gmail-Apple-tab-span" style="white-space:pre-wrap">      </span></span><span style="font-size:11pt;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">/// TODO: Write detailed description</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font face="monospace, monospace"><span style="font-size:11pt;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><span class="m_-5850506797740018462m_-2731000931905328505gmail-Apple-tab-span" style="white-space:pre-wrap">     </span></span><span style="font-size:11pt;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">///</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font face="monospace, monospace"><span style="font-size:11pt;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><span class="m_-5850506797740018462m_-2731000931905328505gmail-Apple-tab-span" style="white-space:pre-wrap">      </span></span><span style="font-size:11pt;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">/// \param value</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font face="monospace, monospace"><span style="font-size:11pt;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><span class="m_-5850506797740018462m_-2731000931905328505gmail-Apple-tab-span" style="white-space:pre-wrap"> </span></span><span style="font-size:11pt;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">/// \return int</span></font></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;text-indent:36pt"><span style="font-size:11pt;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><font face="monospace, monospace">  int doThings(int value) { return value; }</font></span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">In addition, the parsing tool could also be expanded to also parse Markdown-style comments, using the Apple Swift Markup style as a reference</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">.</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><br></span></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Please let us know if you have comments or concerns about this proposal.</span></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Thanks!</span></p><span class="m_-5850506797740018462HOEnZb"><font color="#888888"><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Julie</span></p><div><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><br></span></div></font></span></span></div>
<br>_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
<br></blockquote></div><br></div>
</blockquote></div>