<div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Fri, Sep 8, 2017 at 10:46 AM Robinson, Paul <<a href="mailto:paul.robinson@sony.com" target="_blank">paul.robinson@sony.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Let me say up front that I sympathize deeply with the problem; debug<br>
info is an interface, and it is frequently unclear whether the goal of<br>
some bit of work is to test the producer or test the consumer of the<br>
interface. In fact we end up using the producer to test the consumer,<br>
and (in the case at hand) using the consumer to test the producer.<br>
There are distinct analogies to testing compilers by seeing what the<br>
linker thinks, and testing linkers by seeing whether they can handle<br>
what the compiler produces.<br>
<br>
> I understand the desire to keep them as similar as possible, but I'm<br>
> still not really sold that massaging fickle text output into a different<br>
> text format is going to make things more scalable. I'd like there to be<br>
> as few layers of text processing as possible.<br>
<br>
If the text output is fickle, then I'd think hiding the fickleness behind<br>
a wrapper we control would be preferable to updating dozens of tests when<br>
something changes. Or if a debugger changes its presentation in version<br>
N+1, but people are still running tests with version N, persuading the<br>
wrapper to handle both would be less overall work than making every test<br>
accommodate both formats.<br></blockquote></div><div dir="ltr"><div class="gmail_quote"><div>But that's just my point. There are clearly going to be tests where both formats don't even make sense because it's testing something specific to one debugger. What if I want to test that we output correct exception information, so I send a .exr command to the debugger and get back this:</div><div><br></div><div><pre><span style="color:rgb(30,30,30)">0:000> .exr -1</span>
<span style="color:rgb(30,30,30)">ExceptionAddress: 77a6db8b (ntdll!LdrpDoDebuggerBreak+0x0000002b)</span>
<span style="color:rgb(30,30,30)"> ExceptionCode: 80000003 (Break instruction exception)</span>
<span style="color:rgb(30,30,30)"> ExceptionFlags: 00000000</span>
<span style="color:rgb(30,30,30)">NumberParameters: 1</span>
<span style="color:rgb(30,30,30)"> Parameter[0]: 00000000</span></pre><pre><div style="font-family:sans-serif;white-space:normal"> What if I want to test that that the debugger can print a valid stack trace, so I send a kv command and get back this?</div><div style="font-family:sans-serif;white-space:normal"><br></div><div style="font-family:sans-serif;white-space:normal"><pre><span style="color:rgb(30,30,30)"> # ChildEBP RetAddr Args to Child </span>
<u><span style="color:rgb(0,102,204)">00</span></u><span style="color:rgb(30,30,30)"> </span><span style="color:rgb(30,30,30)">0198fa4c 77a2f5ca 55fe0b87 00000000 00000000 ntdll!LdrpDoDebuggerBreak+0x2b (FPO: [Non-Fpo])</span>
<u><span style="color:rgb(0,102,204)">01</span></u><span style="color:rgb(30,30,30)"> </span><span style="color:rgb(30,30,30)">0198fc8c 77a18a42 55fe0bef 00000000 00000000 ntdll!LdrpInitializeProcess+0x1967 (FPO: [Non-Fpo])</span>
<u><span style="color:rgb(0,102,204)">02</span></u><span style="color:rgb(30,30,30)"> </span><span style="color:rgb(30,30,30)">0198fce4 77a1886c 00000000 bad81aba 00000000 ntdll!_LdrpInitialize+0x180 (FPO: [Non-Fpo])</span>
<u><span style="color:rgb(0,102,204)">03</span></u><span style="color:rgb(30,30,30)"> </span><span style="color:rgb(30,30,30)">0198fcf4 00000000 0198fd08 779c0000 00000000 ntdll!LdrInitializeThunk+0x1c (FPO: [Non-Fpo])</span></pre></div><pre><div style="font-family:sans-serif;white-space:normal">Whereas GDB would print something like</div></pre><pre>#0 m4_traceon (obs=0x24eb0, argc=1, argv=0x2b8c8)
at builtin.c:993
#1 0x6e38 in expand_macro (sym=0x2b600) at macro.c:242
#2 0x6840 in expand_token (obs=0x0, t=177664, td=0xf7fffb08)
at macro.c:71
(More stack frames follow...)</pre><span style="font-family:sans-serif;white-space:normal">I really don't want to get in the business of trying to convert the first format into the second format. Not only is it a recipe for disaster, but it leads to worse diagnostics. When my CHECK statement fails, I can't even see the original stack trace anymore, only a generic error message like "could not parse stack trace"</span>
<span style="font-family:sans-serif"> </span>
</pre></div></div></div><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> I also expect that on Windows we will end up having far more debug info<br>
> tests than other platforms, specifically because we don't have the<br>
> ability to write tests against the debugger (as it's proprietary /<br>
> closed source).<br>
<br>
I don't see how that follows. Sony runs the GDB test suite using clang as<br>
the compiler, and while that is certainly perverting a debugger test suite<br>
into being a compiler test suite, it has value in being a body of tests<br>
that exercise a variety of debug-info features. GDB being open source is<br>
completely irrelevant to this use-case. We treat it as closed. We have<br>
local changes to the test suite, but not to GDB. The expected results<br>
from the suite are based on GDB+GCC, which we treat as an oracle; then we<br>
don't bother with tests of debugger features that clearly don't depend on<br>
debug info, such as thread handling.<br></blockquote><div>GDB being open source is very relevant to this case, because it means you *have* GDB's test suite. We don't have WinDbg or Visual Studio debugger's test suite.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Whatever CodeView/PDB tests you want to write, you can use MS tools as<br>
your oracle. Maybe you can't leverage an existing test suite, but it<br>
doesn't mean you can't write tests.<br></blockquote><div>Right, it just means we will end up writing plenty of tests that test specific features of the debugger, something that would normally be handled in a debugger test suite, which we don't have.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> I don't see a useful abstraction that glosses over these differences<br>
> that isn't a ton of work for minimal gain, given the frequency with<br>
> which we'd need to fall back to a custom test anyway.<br>
<br>
As I mentioned above, you seem to be heading in the direction of a<br>
completely separate project, rather than being able to usefully<br>
leverage anything from debuginfo-tests other than the basic idea.<br>
--paulr<br>
<br></blockquote><div><br></div></div></div><div dir="ltr"><div class="gmail_quote"><div>I don't entirely disagree with this assessment. On the other hand, I don't see any reason to call it something other than "debuginfo-tests" or to put it somewhere else, since conceptually both things are the same.</div><div><br></div><div>Even in this case though, reusing the source code of the tests seems like a clear win since the high level ideas behind a test case often transcend consumer boundaries, even when the implementation doesn't.</div><div><br></div><div>Plus, there is more to be gained from sharing than just the tests themselves. For example, I'm trying to get debuginfo-tests working properly with CMake in more idiomatic LLVM style. If I go off and fork the tests into an entirely separate project, we wouldn't have that shared benefit. </div></div></div></div>