<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">I like that idea!<div class=""><br class=""></div><div class=""><br class=""><div class=""><div><blockquote type="cite" class=""><div class="">On Jun 2, 2015, at 12:00 PM, Smith, Kevin B <<a href="mailto:kevin.b.smith@intel.com" class="">kevin.b.smith@intel.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252" class="">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)" class="">
<style class=""><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Arial","sans-serif";
color:#1F497D;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Arial","sans-serif";
color:#1F497D;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Arial","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div lang="EN-US" link="blue" vlink="purple" class="">
<div class="WordSection1"><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">The code for cmpimage and getdep consists of five source files, with the following sizes<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""><o:p class=""> </o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">$ wc *<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""> 5912 20353 191869 cmpimage.cpp<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""> 290 1328 10668 elf.h<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""> 1496 5006 41691 getdep.cpp<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""> 233 959 7692 macho.h<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""> 403 1831 18394 pecoff.h<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""> 8334 29477 270314 total<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""><o:p class=""> </o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">to build each of them is just a simple compilation for whatever C++ compiler you happen to be using (clang, icc, cl, g++)<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""><o:p class=""> </o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">$(CXX) –o cmpimage –O2 cmpimage.cpp<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">$(CXX) –o getdep –O2 getdep.cpp<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""><o:p class=""> </o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">This seems like it would fit rather easily into test-suite/tools, which already exists and has a Makefile that the commands to build<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">these could be integrated into.<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""><o:p class=""> </o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">This is my best guess/opinion based on a cursory look over the test-suite directory structure.<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""><o:p class=""> </o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">Kevin<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""><o:p class=""> </o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""><o:p class=""> </o:p></span></p>
<div class="">
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in" class=""><p class="MsoNormal"><b class=""><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class="">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class=""> Chris Matthews [<a href="mailto:chris.matthews@apple.com" class="">mailto:chris.matthews@apple.com</a>]
<br class="">
<b class="">Sent:</b> Thursday, May 28, 2015 1:02 PM<br class="">
<b class="">To:</b> Smith, Kevin B<br class="">
<b class="">Cc:</b> Philip Reames; Sean Silva; LLVM Developers Mailing List<br class="">
<b class="">Subject:</b> Re: [LLVMdev] Proposal: change LNT’s regression detection algorithm and how it is used to reduce false positives<o:p class=""></o:p></span></p>
</div>
</div><p class="MsoNormal"><o:p class=""> </o:p></p><p class="MsoNormal">Where is the best place to keep this? <o:p class=""></o:p></p>
<div class=""><p class="MsoNormal"><o:p class=""> </o:p></p>
</div>
<div class=""><p class="MsoNormal">- As third party tool we all use?<o:p class=""></o:p></p>
</div>
<div class=""><p class="MsoNormal">- Contribute as new project?<o:p class=""></o:p></p>
</div>
<div class=""><p class="MsoNormal">- Lives in test-suite/utils?<o:p class=""></o:p></p>
</div>
<div class=""><p class="MsoNormal">- Lives in llvm/utils?<o:p class=""></o:p></p>
<div class=""><p class="MsoNormal"><o:p class=""> </o:p></p>
<div class="">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt" class="">
<div class=""><p class="MsoNormal">On May 28, 2015, at 11:22 AM, Smith, Kevin B <<a href="mailto:kevin.b.smith@intel.com" class="">kevin.b.smith@intel.com</a>> wrote:<o:p class=""></o:p></p>
</div><p class="MsoNormal"><o:p class=""> </o:p></p>
<div class="">
<div class=""><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">OK, there is interest from at least a couple of people. What should next steps be?</span><o:p class=""></o:p></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""> </span><o:p class=""></o:p></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">Kevin</span><o:p class=""></o:p></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""> </span><o:p class=""></o:p></p>
<div class="">
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in" class=""><p class="MsoNormal"><b class=""><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class="">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class=""> Chris Matthews [<a href="mailto:chris.matthews@apple.com" class="">mailto:chris.matthews@apple.com</a>]
<br class="">
<b class="">Sent:</b> Thursday, May 28, 2015 10:57 AM<br class="">
<b class="">To:</b> Philip Reames<br class="">
<b class="">Cc:</b> Smith, Kevin B; Sean Silva; LLVM Developers Mailing List<br class="">
<b class="">Subject:</b> Re: [LLVMdev] Proposal: change LNT’s regression detection algorithm and how it is used to reduce false positives</span><o:p class=""></o:p></p>
</div>
</div><p class="MsoNormal"> <o:p class=""></o:p></p><p class="MsoNormal">I agree. I think there are a lot of exciting uses for this tool. A stage 3 build bot would be another one.<o:p class=""></o:p></p>
<div class=""><p class="MsoNormal"> <o:p class=""></o:p></p>
</div>
<div class="">
<div class="">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt" class="">
<div class=""><p class="MsoNormal">On May 28, 2015, at 10:14 AM, Philip Reames <<a href="mailto:listmail@philipreames.com" class="">listmail@philipreames.com</a>> wrote:<o:p class=""></o:p></p>
</div><p class="MsoNormal"> <o:p class=""></o:p></p>
<div class="">
<div class=""><p class="MsoNormal" style="margin-bottom:12.0pt">I'd love to see this tool contributed, even it isn't used for regression detection work. I've got a couple of hacked up scripts which do similar things and having a robust tool available for this would be very
useful. <br class="">
<br class="">
Philip<o:p class=""></o:p></p>
<div class=""><p class="MsoNormal">On 05/26/2015 09:53 AM, Smith, Kevin B wrote:<o:p class=""></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt" class=""><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">Intel has a binary comparator tool that we have been using for several years for comparing output binaries</span><o:p class=""></o:p></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">to see if the code within them is considered identical. We use it to eliminate runs (and therefore some performance noise)</span><o:p class=""></o:p></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">from our own performance tracking tools.</span><o:p class=""></o:p></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""> </span><o:p class=""></o:p></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">We are willing to contribute the source code for this to the LLVM community if there is interest.</span><o:p class=""></o:p></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""> </span><o:p class=""></o:p></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">There are two programs involved: getdep, which displays the list of DLL/.so dependencies of the image in question, and cmpimage itself, which does the comparison
ignoring the parts not contributed by the compiler. The cmpimage program is also almost completely derived from the published object format descriptions.</span><o:p class=""></o:p></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""> </span><o:p class=""></o:p></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">Let me know if there is interest in these pieces of tooling, and if so, what you think next steps should be.</span><o:p class=""></o:p></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""> </span><o:p class=""></o:p></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">Kevin B. Smith</span><o:p class=""></o:p></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""> </span><o:p class=""></o:p></p><p class="MsoNormal"><b class=""><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class="">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class="">
<a href="mailto:llvmdev-bounces@cs.uiuc.edu" class="">llvmdev-bounces@cs.uiuc.edu</a> [<a href="mailto:llvmdev-bounces@cs.uiuc.edu" class="">mailto:llvmdev-bounces@cs.uiuc.edu</a>]
<b class="">On Behalf Of </b>Sean Silva<br class="">
<b class="">Sent:</b> Thursday, May 21, 2015 2:14 PM<br class="">
<b class="">To:</b> Chris Matthews<br class="">
<b class="">Cc:</b> LLVM Developers Mailing List<br class="">
<b class="">Subject:</b> Re: [LLVMdev] Proposal: change LNT’s regression detection algorithm and how it is used to reduce false positives</span><o:p class=""></o:p></p><p class="MsoNormal"> <o:p class=""></o:p></p>
<div class=""><p class="MsoNormal"> <o:p class=""></o:p></p>
<div class=""><p class="MsoNormal"> <o:p class=""></o:p></p>
<div class=""><p class="MsoNormal">On Thu, May 21, 2015 at 11:24 AM, Chris Matthews <<a href="mailto:chris.matthews@apple.com" target="_blank" class="">chris.matthews@apple.com</a>> wrote:<o:p class=""></o:p></p><p class="MsoNormal">I agree this is a great idea. I think it needs to be fleshed out a little though.<br class="">
<br class="">
It would still be wise to run the regression detection algorithm, because the test suite changes and the machines change, and the algorithm is not perfect yet. It would be a valuable source of information though.<o:p class=""></o:p></p>
<div class=""><p class="MsoNormal"> <o:p class=""></o:p></p>
</div>
<div class=""><p class="MsoNormal">How would running it as part of regular testing change anything? Presumably the only purpose it would serve is retrospectively going back and seeing false-positives in the aggregate. But if we are already doing offline analysis, we can
run the regression detection algorithm (or any prospective ones) offline on the raw data; it doesn't take that long.<o:p class=""></o:p></p>
</div>
<div class=""><p class="MsoNormal"> <o:p class=""></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt" class=""><p class="MsoNormal"><br class="">
This is not a small change to how LNT works, so I think some due diligence is necessary. Is clang *really* that deterministic, especially over successive revs? <o:p class=""></o:p></p>
</blockquote>
<div class=""><p class="MsoNormal"> <o:p class=""></o:p></p>
</div>
<div class=""><p class="MsoNormal">Yes. Actually, google's build system depends on this for its caching strategy to work and so the google guys are usually on top of any issues in this respect (thanks google guys!).<o:p class=""></o:p></p>
</div>
<div class=""><p class="MsoNormal"> <o:p class=""></o:p></p>
</div>
<div class=""><p class="MsoNormal"> <o:p class=""></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt" class=""><p class="MsoNormal">I know it is supposed to be. Does anyone have any data to show this is going to be an effective approach? It seems like there are benchmarks in the test-suite which use __DATE__ and __TIME__ in them. I assume that will be a problem?<o:p class=""></o:p></p>
</blockquote>
<div class=""><p class="MsoNormal"> <o:p class=""></o:p></p>
</div>
<div class=""><p class="MsoNormal">__DATE__ and __TIME__ should be easy to solve by modifying the benchmark, or teaching clang to always return a fixed value for them (maybe we already have this? IIRC google's build system does something like this; or maybe the do it at
the OS level).<o:p class=""></o:p></p>
</div>
<div class=""><p class="MsoNormal"> <o:p class=""></o:p></p>
</div>
<div class=""><p class="MsoNormal">-- Sean Silva<o:p class=""></o:p></p>
</div>
<div class=""><p class="MsoNormal"> <o:p class=""></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt" class="">
<div class="">
<div class=""><p class="MsoNormal"><br class="">
> On May 21, 2015, at 1:43 AM, Renato Golin <<a href="mailto:renato.golin@linaro.org" class="">renato.golin@linaro.org</a>> wrote:<br class="">
><br class="">
> On 20 May 2015 at 23:31, Sean Silva <<a href="mailto:chisophugis@gmail.com" class="">chisophugis@gmail.com</a>> wrote:<br class="">
>> In the last 10,000 revisions of LLVM+Clang, only 10 revisions actually<br class="">
>> caused the binary of MultiSource/Benchmarks/BitBench/five11 to change. So if<br class="">
>> just store a hash of the binary in the database, we should be able to pool<br class="">
>> all samples we have collected while the binary is the the same as it<br class="">
>> currently is, which will let us use significantly more datapoints for the<br class="">
>> reference.<br class="">
><br class="">
> +1<br class="">
><br class="">
><br class="">
>> Also, we can trivially eliminate running the regression detection algorithm<br class="">
>> if the binary hasn't changed.<br class="">
><br class="">
> +2!<br class="">
><br class="">
> --renato<o:p class=""></o:p></p>
</div>
</div>
<div class="">
<div class=""><p class="MsoNormal" style="margin-bottom:12.0pt">> _______________________________________________<br class="">
> LLVM Developers mailing list<br class="">
> <a href="mailto:LLVMdev@cs.uiuc.edu" class="">LLVMdev@cs.uiuc.edu</a> <a href="http://llvm.cs.uiuc.edu/" target="_blank" class="">http://llvm.cs.uiuc.edu</a><br class="">
> <a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev" target="_blank" class="">http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev</a><o:p class=""></o:p></p>
</div>
</div>
</blockquote>
</div><p class="MsoNormal"> <o:p class=""></o:p></p>
</div>
</div><p class="MsoNormal"><br class="">
<br class="">
<br class="">
<br class="">
<o:p class=""></o:p></p>
<pre class="">_______________________________________________<o:p class=""></o:p></pre>
<pre class="">LLVM Developers mailing list<o:p class=""></o:p></pre>
<pre class=""><a href="mailto:LLVMdev@cs.uiuc.edu" class="">LLVMdev@cs.uiuc.edu</a> <a href="http://llvm.cs.uiuc.edu/" class="">http://llvm.cs.uiuc.edu</a><o:p class=""></o:p></pre>
<pre class=""><a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev" class="">http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev</a><o:p class=""></o:p></pre>
</blockquote><p class="MsoNormal"> <o:p class=""></o:p></p>
</div>
</div>
</blockquote>
</div><p class="MsoNormal"> <o:p class=""></o:p></p>
</div>
</div>
</div>
</blockquote>
</div><p class="MsoNormal"><o:p class=""> </o:p></p>
</div>
</div>
</div>
</div>
</div></blockquote></div><br class=""></div></div></body></html>