[cfe-dev] [RFC] New ClangDebuggerSupport Library

[David Blaikie via cfe-dev]

On Tue, Dec 13, 2016 at 11:19 AM Chris Bieneman <cbieneman at apple.com> wrote:

> On Dec 13, 2016, at 10:52 AM, David Blaikie via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
> (actually, one extra wrinkle: What if we could just create the DWARF
> parsing API results directly for the Clang tests - that'd actually be
> better. Then we wouldn't be needlessly exercising all the object and DWARF
> parsing code in the Clang tests (in the same way we don't run Clang tests
> that go all the way through to LLVM IR - if we can avoid it we should be
> doing similarly here))
>
>
> Because there is no "one true way" to encode DWARF I think exercising the
> parsing code is actually valuable.
>

I agree it's valuable - but that it should be exercised directly in the
DWARF parsing tests in LLVM.


>
> -Chris
>
>
> On Tue, Dec 13, 2016 at 10:49 AM David Blaikie <dblaikie at gmail.com> wrote:
>
> On Tue, Dec 13, 2016 at 10:42 AM Chris Bieneman <cbieneman at apple.com>
> wrote:
>
> On Dec 13, 2016, at 8:59 AM, David Blaikie via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
>
>
> On Mon, Dec 12, 2016 at 4:59 PM Chris Bieneman <cbieneman at apple.com>
> wrote:
>
> On Dec 12, 2016, at 4:40 PM, David Blaikie via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
>
>
> On Mon, Dec 12, 2016 at 4:23 PM Chris Bieneman <cbieneman at apple.com>
> wrote:
>
> On Dec 12, 2016, at 4:13 PM, David Blaikie <dblaikie at gmail.com> wrote:
>
>
>
> On Mon, Dec 12, 2016 at 4:09 PM Chris Bieneman <cbieneman at apple.com>
> wrote:
>
> David, the two approaches address very different problems.
>
> The YAML tools are focused on a bit-for-bit identical round trip path for
> DWARF into and out of YAML. The goal with that work is to be able to
> generate a test suite from the output of many different versions of many
> different compilers. This is specifically with the goal of creating
> LIT-style tests that read DWARF and operate on it.
>
>
> Ah, thanks for explaining.
>
> These tests wouldn't appear in LLVM/Clang's test suite, then, right? So
> normal regression tests for the ClangDebuggerSupport library would be
> written as unit tests using Greg's DWARF-generation library?
>
>
> My goal is actually to have reduced test cases based on the YAML tools in
> the clang test suite. LLDB's use of clang APIs with dwarf generated by
> mismatched compilers is the source of many issues for the debugger, so
> having basic testing of DWARF generated by alternate compilers in Clang is
> highly desirable.
>
>
> Well, having DWARF that's representative of that generated by alternate
> compilers is important - and it seems like Greg's work on the unit test API
> for creating DWARF should still allow that. Seems reasonable to continue to
> enhance that to produce any DWARF we care about (since we'll need to
> generate it to test the DWARF parsing APIs - so that's a prerequisite
> before we worry about whether the ClangDebuggerSupport library can do
> something sensible with it, right?)
>
>
> I haven't dug too deep into Greg's work (although I certainly will). Where
> it makes sense I may even try and leverage his APIs in the YAML tools (as I
> have been leveraging the existing DWARF parser).
>
> In my (limited) discussions with Greg, it didn't seem like creating
> bit-for-bit identical DWARF was something his APIs were suited to.
>
>
> This seems strange to me that it would not be a need for his work, yet be
> a need for yours. Your work would presumably layer on top of his (got to
> parse the DWARF first, before you can build Clang ASTs from it).
>
>
> The dwarfgen APIs are designed around being able to create a DIE and have
> the APIs create the accompanying abbreviations. That means you don't have
> direct control over the bit layouts.
>
> One of the complications here is that in DWARF there is no "one true way"
> to encode to binary. Since my test need to test the outputs of other
> compilers, I really need bit-for-bit identical translations.
>
>
> Any irregular DWARF is going to be /more/ interesting (weird bit
> twiddling, etc) for the DWARF parsing API testing (because most of it would
> just error out, and that which doesn't would need to have positive test
> coverage as well) than for the Clang AST generation part that happens after
> that.
>
>
> Bit twiddling of individual fields should be possible in the dwarfgen
> APIs, but there is no API for explicit specification of DWARF bytes other
> than just encoding a byte array (like we do in the disassembler tests).
>
>
> It seems backwards to need more fidelity/control when creating inputs for
> testing the Clang AST generation, than for the API below it for parsing
> DWARF.
>
>
> I don't disagree with your assertion here. I need bit-for-bit identical
> data encodings, so I'm writing a solution that provides that.
>
>
> OK - so you disagree with Greg's approach to testing the underlying APIs
> your tests rely on?
>
> Perhaps you two could hash that out, since you're working in the same
> space/same project/company/etc. This seems inconsistent and I think it's
> reasonable to ask for the strategy here to become consistent in some way.
>
> I think that way would probably be to make the dwarfgen APIs sufficiently
> descriptive to be able to generate the inputs you require (& to convince
> Greg that he needs to test those inputs too/instead - many of them should
> just be error cases in LLVM's DWARF parsing APIs, before it even reaches
> your ClangDebuggerSupport code - and those that aren't, should probably
> still have full fidelity testing in the DWARF parsing APIs before reaching
> the Clang side) and just use that.
>
> We don't generally generate uneditable test cases for LLVM projects (yeah,
> I've done some of that - we have binaries checked in to test
> llvm-dwarfdump, but that's being addressed by this whole discussion)
>
> Here's what I'd suggest/have in mind: starting with dwarfgen, and seeing
> how it works for creating the sort of test cases you have in mind. When
> there are particular difficulties, then it might be good to look at the
> concrete examples and decide how best to approach them. I'm sure there's
> lots of improvements, higher level APIs, etc, we could make around the
> dwarfgen code (creating DWARF is verbose - in any format, I think - helpers
> and utilities could be really handy, for sure).
>
> - Dave
>
>
>
> -Chris
>
>
>
>
> In YAML I've made the textual representation mirror the binary
> representation to a degree that the translation from YAML to binary has
> very little logic to it. As a point of context the YAML->DWARF
> implementation for dumping debug_abbrev, debug_str, and debug_aranges is
> under 100 lines of code.
>
>
>
>
>
> Large tests generated from other compilers on raw source I would expect to
> appear in something like the test-suite, rather than in an LLVM project's
> regression or unit test suite.
>
>
> Large tests will certainly not be included in the clang test suite. YAML
> representations of DWARF should enable us to make reduced test cases in
> many situations, and where we cannot we will put the test in an external
> suite.
>
>
> Why the need for round tripping, then? Would it be sufficient for the
> test-suite to have binaries checked in next to info about what compiler
> generated them?
>
>
> The benefit of supporting round tripping in and out of a text-based format
> is that we may be able to reduce the test cases to things that we can
> include in the Clang test suite.
>
> (& why not just have the source checked in & run a variety of buildbot
> configurations (or one meta-configuration that could enumerate a variety of
> compilers) with different host compilers to test the behavior? That's how
> GDB's test suite works (for better and worse, don't get me wrong - there
> are things that could be improved from that position))
>
>
> This is actually basically how the LLDB test suite works. There is one
> huge drawback to this. Not everyone has access to every compiler we want to
> support, and certainly most people don't have them all installed. As a
> result having source-based tests means that many people may not be able to
> reproduce test failures locally. Using YAML encodings to generate the
> binary DWARF removes the compiler from the picture, and allows everyone to
> test every compiler's output.
>
>
> Fair - so why YAML rather than something more like the unit tests Greg's
> working on in LLVM?
>
>
> I mostly gravitated to YAML because I have experience using YAML-based
> tests for libObject code, and have found it very useful to be able to
> translate binaries in and out of YAML for testing.
>
>
> (this is clearly my preference - to use the unit test type API, since in
> both Greg and your case, you're testing an API, not a tool, so it seems
> cool/fine/reasonable to have an API for generating the input.
>
>
> I actually expect in my use case that I'll be testing both APIs and one or
> more tools. My intention is to write a tool that reads dwarf and dumps
> Clang ASTs. For that purpose having a YAML->DWARF generator is ideal.
>
> Also for my use case YAML has an added advantage that when a user reports
> an issue I can either take a binary or YAML file from the user, and
> textually reduce that down to a test case which could live in-tree.
>
>
> But the alternative question would be: Why not test the LLVM DWARF parsing
> API Greg's testing, with this yaml input instead of the unit test API?)
>
>
> Personally, I think having both types of tests are valuable. Unit tests of
> APIs are particularly valuable for writing small-grained tests, with
> limited input sizes. When I start running down the path of constructing
> Clang ASTs from complex C++ programs the code required to generate that
> DWARF in a unit test could be substantial, and that would make it a lot
> harder to write tests.
>
> Converting a binary to a YAML file is easy, hand crafting DWARF from APIs
> might not be.
>
> -Chris
>
>
>
>
> -Chris
>
>
> - Dave
>
>
>
> -Chris
>
> On Dec 12, 2016, at 3:57 PM, David Blaikie via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
> I realize work is already underway/being committed here, but figured
> discussing the following in this thread rather than on some random commit
> email might be better.
>
> We now have two ways of generating DWARF, both committed in relation to a
> similar effort to integrate LLDB better with teh rest of the LLVM project.
>
> There's this YAML effort, to help test the library that will allow the
> generation of Clang ASTs from DWARF. (currently such code resides in LLDB,
> and it's proposing to be rolled up into Clang here)
>
> Then there's Greg's effort to provide a unit test API for generating DWARF
> for unit testing LLVM's DWARF parsing APIs for use in LLDB (currently what
> LLVM has was a fork of LLDB's, and Greg's working on reconciling that,
> rolling in LLDB's post-fork features, then migrating LLDB to use the fully
> featured LLVM version)
>
> Why are these done in two different ways? They seem like really similar
> use cases - generating DWARF for the purpose of testing some (LLVM or
> Clang) API that consumes DWARF bytes.
>
> Could we resolve this in favor of one approach or the other - I'm somewhat
> partial to the API approach & writing unit tests against the
> ClangDebuggerSupport library, myself.
>
> - David
>
> On Wed, Nov 9, 2016 at 2:26 PM Chris Bieneman via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
> Hello cfe-dev,
>
> I would like to propose a new Clang library for implementing functionality
> that is used by LLDB. I see this as the first step in a long process of
> refactoring the language interfaces for LLDB.
>
> The short-term goal is for this library is to be a place for us to rebuild
> functionality that exists in LLDB today and relies heavily on the
> implementation of Clang. As we rebuild the functionality we will build a
> suite of testing tools in Clang that exercise this library and more general
> Clang functionality in the same ways that LLDB will.
>
> As bits of functionality become fully implemented and tested, we will
> migrate LLDB to using the Clang implementations, allowing LLDB to remove
> its own copies. This will provide the Clang community with a higher
> confidence that changes in Clang do not break LLDB, and it will provide
> LLDB with better test coverage of the Clang functionality.
>
> The long-term goal of this library is to provide the implementation for
> what could some day become a defined debugger<->frontend interface for
> providing modularized (maybe even plugin-based) language debugging support
> in LLDB. In the distant future I could see us being able to tell people
> building new frontends that we have a defined interface they need to
> implement for the debugger, and once implemented the debugger should “Just
> Work”.
>
> The first bit of functionality that I would like to build up into the
> ClangDebuggerSupport library is materialization of Clang AST types from
> DWARF. To support this development I intend to add a new tool in Clang that
> reads DWARF types, generates a Clang AST, and prints the AST. I will also
> add DWARF support to obj2yaml and yaml2obj, so we will be able to write
> YAML LIT tests for the functionality.
>
> If people are in favor of this general approach I’ll begin working in this
> direction, and I’ll probably add the new library sometime next month.
>
> Thoughts?
> -Chris
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20161213/b018c13d/attachment.html>