<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 26, 2021 at 6:10 AM <<a href="mailto:stephen.tozer@sony.com">stephen.tozer@sony.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
As David has said, the coverage % is not an especially meaningful number in general because we do not have a general method of determining the true upper bound of coverage for an optimised program. To answer your question as best I can though, here are the
coverage numbers:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Consolas,Courier,monospace">Project Variable availability PC ranges covered </span>
<div><span style="font-family:Consolas,Courier,monospace"> Old New Delta Old New Delta</span></div>
<div><span style="font-family:Consolas,Courier,monospace">7zip 79.48% 80.01% 0.53% 60.55% 60.73% 0.18%</span></div>
<div><span style="font-family:Consolas,Courier,monospace">bullet 44.57% 45.21% 0.65% 55.55% 56.00% 0.46%</span></div>
<div><span style="font-family:Consolas,Courier,monospace">ClamAV 88.89% 89.37% 0.48% 53.48% 53.57% 0.09%</span></div>
<div><span style="font-family:Consolas,Courier,monospace">consumer-typeset 91.62% 91.44% -0.19% 32.48% 32.48% 0.00%</span></div>
<div><span style="font-family:Consolas,Courier,monospace">kimwitu++ 68.34% 68.75% 0.41% 69.12% 69.82% 0.70%</span></div>
<div><span style="font-family:Consolas,Courier,monospace">lencod 89.86% 90.77% 0.91% 48.41% 48.83% 0.42%</span></div>
<div><span style="font-family:Consolas,Courier,monospace">mafft 89.26% 89.14% -0.12% 57.89% 57.89% 0.00%</span></div>
<div><span style="font-family:Consolas,Courier,monospace">SPASS 83.23% 83.26% 0.03% 52.61% 52.66% 0.05%</span></div>
<div><span style="font-family:Consolas,Courier,monospace">sqlite3 73.55% 75.62% 2.07% 51.59% 52.01% 0.43%</span></div>
<div><span style="font-family:Consolas,Courier,monospace">tramp3d-v4 54.77% 63.04% 8.27% 66.67% 68.16% 1.49%</span></div>
<div></div>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
These numbers are not high resolution - the change is simply the difference of the rounded "old" and "new" numbers. Notably, the variable availability for some of the projects has actually gone down, as we have more variables being emitted to DWARF with 0%
coverage (the DWARF emission of variables with 0% coverage is an issue in itself, but not one introduced or fixed by this patch). The PC bytes numbers are also slightly misleading, as the % is calculated as "the sum of PC bytes covered for each variable" divided
by "the sum of PC bytes in the parent scope for each variable". This means that if, for example, we doubled the number of variables covered by the program but all of the new variables had slightly lower average coverage than the old variables, we would see
this number decrease despite the clear increase in actual coverage. <br></div></div></blockquote><div><br>Hmm, that seems like a somewhat unhelpful statistic - when you say "more variables being emitted to DWARF with 0% coverage" - what do you mean by that? Are we counting a variable with no location attribute as being 100% covered, because it isn't partially covered? Could we instead count such variables as 0% covered?<br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
As you can see, these numbers aren't as helpful as we'd like - for example, we could easily hit 100% coverage by choosing not to emit any variables that don't have a location for their entire scope, but this would not translate to a better debug experience.
We could compare the number of available variables with the program at O0, but this also does not work out as it might first seem, because optimizations can
<i>increase</i> the number of variables by inlining functions; for all of these projects, the number of variables at O2 is several times larger than the number at O0. </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hopefully this summarizes why comparing the raw variable counts and PC bytes covered is, as far as I can tell, the best way of comparing the actual change in debug quality between the two patches.<br>
</div>
<div id="gmail-m_-1731775646871690829appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-1731775646871690829divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> David Blaikie <<a href="mailto:dblaikie@gmail.com" target="_blank">dblaikie@gmail.com</a>><br>
<b>Sent:</b> 22 January 2021 19:45<br>
<b>To:</b> Owen Anderson <<a href="mailto:resistor@mac.com" target="_blank">resistor@mac.com</a>><br>
<b>Cc:</b> Tozer, Stephen <<a href="mailto:stephen.tozer@sony.com" target="_blank">stephen.tozer@sony.com</a>>; <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a> <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br>
<b>Subject:</b> Re: [llvm-dev] [DebugInfo] The current status of debug values using multiple machine locations</font>
<div> </div>
</div>
<div>
<div dir="ltr">It's hard to know the upper bound on what's possible.<br>
<br>
eg: code like this:<br>
<br>
int x = f1();<br>
f2(x);
<div>f2(4);<br>
<br>
With optimized code, there's no way to recover the value of 'x' during the second f2 call. We can compute an absolute upper bound that's certainly unreachable - by looking at the scope of variables (assuming our scope tracking is perfect - which, it's not bad,
but can get weird under optimizations) and comparing total scope bytes of variables compared to the bytes for which a location is described. We do have those two stats in llvm-dwarfdump --statistics. But generally we don't bother looking at that because it's
a fair way off and the limitations of any such measurement as I've described here. (we also don't currently track where a variable's scope starts - so the upper bound for "x" in "{ f1(); int x = ...; f1(); }" includes both calls to f1, even though the location
shouldn't ever extend to cover the first f1 call)</div>
</div>
<br>
<div>
<div dir="ltr">On Fri, Jan 22, 2021 at 11:31 AM Owen Anderson via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>Hi Stephen,
<div><br>
</div>
<div>Is it possible to quantify this coverage in absolute terms, at least the PC bytes portion? It would be helpful to understand how close this is bringing us to 100% coverage, for example.</div>
<div><br>
</div>
<div>—Owen<br>
<div><br>
<blockquote type="cite">
<div>On Jan 22, 2021, at 7:23 AM, via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:</div>
<br>
<div>
<div style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt">
<div style="margin-top:0px;margin-bottom:0px">Following the previous discussion on the mailing list[0], I have been writing a series of patches that implement the proposed instructions[1], enabling multi-location debug values (debug value lists) in LLVM. Although
the patches are still in review, the basic implementation is finished (except for the removal and replacement of the old DBG_VALUE instruction, as discussed on the mailing list[2]).</div>
<div style="margin-top:0px;margin-bottom:0px"><br>
</div>
<div style="margin-top:0px;margin-bottom:0px">Given below is the change in debug info output for the LLVM test-suite CTMark, from before and after the debug value list patch:</div>
<pre><code>Project Available variables PC bytes covered
Old New Change Old New Change
7zip 40252 40501 0.62% 7112336 7142255 0.42%
bullet 32655 33296 1.96% 6272034 6323049 0.81%
ClamAV 8795 8842 0.53% 5090634 5099634 0.18%
consumer-typeset 4354 4356 0.05% 3171498 3171605 0.00%
kimwitu++ 30006 30177 0.57% 1736826 1755152 1.06%
lencod 14176 14319 1.01% 6123957 6177106 0.87%
mafft 6854 6859 0.07% 12045196 12046744 0.01%
SPASS 38477 38492 0.04% 3396246 3399668 0.10%
sqlite3 29479 30301 2.79% 7964547 8024747 0.76%
tramp3d-v4 91732 105588 15.10% 7925131 8106167 2.28%
</code></pre>
<div style="margin-top:0px;margin-bottom:0px">As most of the patches have been approved, I am hopeful that the full set of patches will be merged into main in the near future. Part of the purpose of this email is to give notice of the upcoming merge, as the
changes are significant and may conflict with any private changes concerning debug info. In terms of output, the patch should not change any existing variable locations; it should only add new locations for some variables. This may break tests that expect
certain variables to be missing or optimized out, but should not be disruptive otherwise. If you want to test this patch, either to benchmark compiler performance, gather DWARF statistics, or test its merging with private changes, there is a single patch comprising
the entirety of the current work on Phabricator[3].</div>
<div style="margin-top:0px;margin-bottom:0px"><br>
</div>
<div style="margin-top:0px;margin-bottom:0px">The other purpose of this email is to request further reviews on the patches, as all but 5 have been accepted and most of the remaining patches have been well-reviewed by now. Due to the size of the patches, there
will likely be conflicts between any in-development debug-info work and these patches, creating extra work for any developers that need to update their patches to handle the new instruction. It will also allow current and future work to take advantage of the
new functionality to preserve more debug information.</div>
<div style="margin-top:0px;margin-bottom:0px"><br>
</div>
<hr>
<div style="margin-top:0px;margin-bottom:0px">With the patch implementations essentially complete, we can see more precisely the effect of these patches. With respect to the results above, it is important to note that although this patch is functional it does
not cover all the potential ground of this feature. The only direct improvement added by this patch is enabling the salvage of non-constant binary operator and GetElementPtr instructions within the existing salvage function during opt passes. This is a significant
improvement, but more may come: follow-up patches can enable this improved salvaging during instruction selection, and enable the salvage of<span> </span><code>cmp</code><span> </span>and<span> </span><code>select</code><span> </span>instructions; there are
also some gaps in the instruction selection implementation, such as resolving dangling debug value lists. Some of these are easy wins that can be implemented immediately after landing the patch, and some are longer term projects that can progress when this
work has merged.</div>
<div style="margin-top:0px;margin-bottom:0px"><br>
</div>
<div style="margin-top:0px;margin-bottom:0px">The patch currently leads to a medium-sized improvement for most cases, with some very small and one very large improvement. As mentioned previously, this set of patches primarily lays the groundwork for more complex
variable locations; once it has landed there will be further work to salvage from more places, as well as improving the handling of list dbg.values to better preserve variable locations through optimizations. Compilation times do not appear to be significantly
affected; I've been measuring compile times for the CTMark projects in the llvm test-suite (using the best practices from the benchmarking guide[4]), but so far any change in compile time is much smaller than the measurement noise, so I don't have an exact
number to give. I estimate from the results so far that the increase will be no more than 1% in the worst case, but it could be smaller - I'm testing further to verify.<br>
</div>
<div style="margin-top:0px;margin-bottom:0px"><u><br>
</u></div>
<div style="margin-top:0px;margin-bottom:0px">[0]<span> </span><a href="https://lists.llvm.org/pipermail/llvm-dev/2020-February/139376.html" rel="nofollow" target="_blank">https://lists.llvm.org/pipermail/llvm-dev/2020-February/139376.html</a><br>
[1]<span> </span><a href="https://reviews.llvm.org/D82363" rel="nofollow" target="_blank">https://reviews.llvm.org/D82363</a><br>
[2]<span> </span><a href="https://lists.llvm.org/pipermail/llvm-dev/2020-August/144589.html" rel="nofollow" target="_blank">https://lists.llvm.org/pipermail/llvm-dev/2020-August/144589.html</a><br>
[3]<span> </span><a href="https://reviews.llvm.org/D94631" rel="nofollow" target="_blank">https://reviews.llvm.org/D94631</a></div>
<div style="margin-top:0px;margin-bottom:0px">[4]<span> </span><a href="https://llvm.org/docs/Benchmarking.html" id="gmail-m_-1731775646871690829x_gmail-m_-3715548265144252763LPlnk" target="_blank">https://llvm.org/docs/Benchmarking.html</a><br>
</div>
<div></div>
<br>
<br>
</div>
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline">_______________________________________________</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline">LLVM
Developers mailing list</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<a href="mailto:llvm-dev@lists.llvm.org" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">llvm-dev@lists.llvm.org</a><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a></div>
</blockquote>
</div>
<br>
</div>
</div>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote>
</div>
</div>
</div>
</blockquote></div></div>