<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">I agree. I think there are a lot of exciting uses for this tool. A stage 3 build bot would be another one.<div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On May 28, 2015, at 10:14 AM, Philip Reames <<a href="mailto:listmail@philipreames.com" class="">listmail@philipreames.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta content="text/html; charset=windows-1252" http-equiv="Content-Type" class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
I'd love to see this tool contributed, even it isn't used for
regression detection work. I've got a couple of hacked up scripts
which do similar things and having a robust tool available for this
would be very useful. <br class="">
<br class="">
Philip<br class="">
<br class="">
<div class="moz-cite-prefix">On 05/26/2015 09:53 AM, Smith, Kevin B
wrote:<br class="">
</div>
<blockquote cite="mid:4F440F6BE90FC64193D0A9A486C6274C9F6E3975@FMSMSX103.amr.corp.intel.com" type="cite" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)" class="">
<style class=""><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Arial","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1"><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">Intel
has a binary comparator tool that we have been using for
several years for comparing output binaries<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">to
see if the code within them is considered identical. We use
it to eliminate runs (and therefore some performance noise)<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">from
our own performance tracking tools.<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""><o:p class=""> </o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">We
are willing to contribute the source code for this to the
LLVM community if there is interest.<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""><o:p class=""> </o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">There
are two programs involved: getdep, which displays the list
of DLL/.so dependencies of the image in question, and
cmpimage itself, which does the comparison ignoring the
parts not contributed by the compiler. The cmpimage program
is also almost completely derived from the published object
format descriptions.<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""><o:p class=""> </o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">Let
me know if there is interest in these pieces of tooling, and
if so, what you think next steps should be.<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""><o:p class=""> </o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class="">Kevin
B. Smith<o:p class=""></o:p></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D" class=""><o:p class=""> </o:p></span></p><p class="MsoNormal"><b class=""><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class="">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class="">
<a class="moz-txt-link-abbreviated" href="mailto:llvmdev-bounces@cs.uiuc.edu">llvmdev-bounces@cs.uiuc.edu</a>
[<a class="moz-txt-link-freetext" href="mailto:llvmdev-bounces@cs.uiuc.edu">mailto:llvmdev-bounces@cs.uiuc.edu</a>]
<b class="">On Behalf Of </b>Sean Silva<br class="">
<b class="">Sent:</b> Thursday, May 21, 2015 2:14 PM<br class="">
<b class="">To:</b> Chris Matthews<br class="">
<b class="">Cc:</b> LLVM Developers Mailing List<br class="">
<b class="">Subject:</b> Re: [LLVMdev] Proposal: change LNT’s
regression detection algorithm and how it is used to reduce
false positives<o:p class=""></o:p></span></p><p class="MsoNormal"><o:p class=""> </o:p></p>
<div class=""><p class="MsoNormal"><o:p class=""> </o:p></p>
<div class=""><p class="MsoNormal"><o:p class=""> </o:p></p>
<div class=""><p class="MsoNormal">On Thu, May 21, 2015 at 11:24 AM,
Chris Matthews <<a moz-do-not-send="true" href="mailto:chris.matthews@apple.com" target="_blank" class="">chris.matthews@apple.com</a>>
wrote:<o:p class=""></o:p></p><p class="MsoNormal">I agree this is a great idea. I
think it needs to be fleshed out a little though.<br class="">
<br class="">
It would still be wise to run the regression detection
algorithm, because the test suite changes and the
machines change, and the algorithm is not perfect yet.
It would be a valuable source of information though.<o:p class=""></o:p></p>
<div class=""><p class="MsoNormal"><o:p class=""> </o:p></p>
</div>
<div class=""><p class="MsoNormal">How would running it as part of
regular testing change anything? Presumably the only
purpose it would serve is retrospectively going back
and seeing false-positives in the aggregate. But if we
are already doing offline analysis, we can run the
regression detection algorithm (or any prospective
ones) offline on the raw data; it doesn't take that
long.<o:p class=""></o:p></p>
</div>
<div class=""><p class="MsoNormal"> <o:p class=""></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC
1.0pt;padding:0in 0in 0in
6.0pt;margin-left:4.8pt;margin-right:0in" class=""><p class="MsoNormal"><br class="">
This is not a small change to how LNT works, so I
think some due diligence is necessary. Is clang
*really* that deterministic, especially over
successive revs? <o:p class=""></o:p></p>
</blockquote>
<div class=""><p class="MsoNormal"><o:p class=""> </o:p></p>
</div>
<div class=""><p class="MsoNormal">Yes. Actually, google's build
system depends on this for its caching strategy to
work and so the google guys are usually on top of any
issues in this respect (thanks google guys!).<o:p class=""></o:p></p>
</div>
<div class=""><p class="MsoNormal"><o:p class=""> </o:p></p>
</div>
<div class=""><p class="MsoNormal"> <o:p class=""></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC
1.0pt;padding:0in 0in 0in
6.0pt;margin-left:4.8pt;margin-right:0in" class=""><p class="MsoNormal">I know it is supposed to be. Does
anyone have any data to show this is going to be an
effective approach? It seems like there are
benchmarks in the test-suite which use __DATE__ and
__TIME__ in them. I assume that will be a problem?<o:p class=""></o:p></p>
</blockquote>
<div class=""><p class="MsoNormal"><o:p class=""> </o:p></p>
</div>
<div class=""><p class="MsoNormal">__DATE__ and __TIME__ should be
easy to solve by modifying the benchmark, or teaching
clang to always return a fixed value for them (maybe
we already have this? IIRC google's build system does
something like this; or maybe the do it at the OS
level).<o:p class=""></o:p></p>
</div>
<div class=""><p class="MsoNormal"><o:p class=""> </o:p></p>
</div>
<div class=""><p class="MsoNormal">-- Sean Silva<o:p class=""></o:p></p>
</div>
<div class=""><p class="MsoNormal"> <o:p class=""></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC
1.0pt;padding:0in 0in 0in
6.0pt;margin-left:4.8pt;margin-right:0in" class="">
<div class="">
<div class=""><p class="MsoNormal"><br class="">
> On May 21, 2015, at 1:43 AM, Renato Golin
<<a moz-do-not-send="true" href="mailto:renato.golin@linaro.org" class="">renato.golin@linaro.org</a>>
wrote:<br class="">
><br class="">
> On 20 May 2015 at 23:31, Sean Silva <<a moz-do-not-send="true" href="mailto:chisophugis@gmail.com" class="">chisophugis@gmail.com</a>>
wrote:<br class="">
>> In the last 10,000 revisions of
LLVM+Clang, only 10 revisions actually<br class="">
>> caused the binary of
MultiSource/Benchmarks/BitBench/five11 to change.
So if<br class="">
>> just store a hash of the binary in the
database, we should be able to pool<br class="">
>> all samples we have collected while the
binary is the the same as it<br class="">
>> currently is, which will let us use
significantly more datapoints for the<br class="">
>> reference.<br class="">
><br class="">
> +1<br class="">
><br class="">
><br class="">
>> Also, we can trivially eliminate running
the regression detection algorithm<br class="">
>> if the binary hasn't changed.<br class="">
><br class="">
> +2!<br class="">
><br class="">
> --renato<o:p class=""></o:p></p>
</div>
</div>
<div class="">
<div class=""><p class="MsoNormal" style="margin-bottom:12.0pt">>
_______________________________________________<br class="">
> LLVM Developers mailing list<br class="">
> <a moz-do-not-send="true" href="mailto:LLVMdev@cs.uiuc.edu" class="">LLVMdev@cs.uiuc.edu</a>
<a moz-do-not-send="true" href="http://llvm.cs.uiuc.edu/" target="_blank" class="">http://llvm.cs.uiuc.edu</a><br class="">
> <a moz-do-not-send="true" href="http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev" target="_blank" class="">http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev</a><o:p class=""></o:p></p>
</div>
</div>
</blockquote>
</div><p class="MsoNormal"><o:p class=""> </o:p></p>
</div>
</div>
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre wrap="" class="">_______________________________________________
LLVM Developers mailing list
<a class="moz-txt-link-abbreviated" href="mailto:LLVMdev@cs.uiuc.edu">LLVMdev@cs.uiuc.edu</a> <a class="moz-txt-link-freetext" href="http://llvm.cs.uiuc.edu/">http://llvm.cs.uiuc.edu</a>
<a class="moz-txt-link-freetext" href="http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev">http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev</a>
</pre>
</blockquote>
<br class="">
</div>
</div></blockquote></div><br class=""></div></body></html>