<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">On the original thread topic, in r238965 I committed a much better detection algorithm, which uses a min of diffs approach. And in r238968 I updated the daily report to pass more data to make use of this. The change brings the false positive rate down to about ~15% on our internal reports. For the last two weeks I have actually been able to detect real regression in the 1% range!<div class=""><br class=""></div><div class="">This approach only works when the previous samples set is large enough to have some meaningful previous information in it. Call sites of the regression detection have to be changed to pass more data. For the daily report I changed it from comparing last run on current and previous days, to comparing last run on first day to all runs on previous day.</div><div class=""><br class=""></div><div class="">If this works out well, lets consider changing Run comparisons and Field comparisons to work in a similar way. <br class=""><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jun 2, 2015, at 6:04 PM, Sean Silva <<a href="mailto:chisophugis@gmail.com" class="">chisophugis@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Tue, Jun 2, 2015 at 2:24 PM, Philip Reames <span dir="ltr" class=""><<a href="mailto:listmail@philipreames.com" target="_blank" class="">listmail@philipreames.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000" class="">
Personally, I would prefer this either live in it's own repository,
or llvm/tools/. None of my use cases will likely involve the
test-suite.<br class="">
<br class="">
p.s. If this is going to end up an llvm tool, it will need to follow
LLVM style. <br class="">
<br class="">
p.p.s. We should probably start a new thread with the proposed
addition since I imagine many folks are ignoring this one by now
given how deep it's gotten.</div></blockquote><div class=""><br class=""></div><div class="">Maybe just putting it on github for now is easiest to at least make it generally available for review. If we later want to officially pull it in or integrate it with our build system we can do that.</div><div class=""><br class=""></div><div class="">-- Sean Silva</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000" class=""><span class="HOEnZb"><font color="#888888" class=""><br class="">
<br class="">
Philip</font></span><div class=""><div class="h5"><br class="">
<br class="">
<div class="">On 06/02/2015 12:04 PM, Chris Matthews
wrote:<br class="">
</div>
<blockquote type="cite" class="">
I like that idea!
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Jun 2, 2015, at 12:00 PM, Smith, Kevin B
<<a href="mailto:kevin.b.smith@intel.com" target="_blank" class="">kevin.b.smith@intel.com</a>>
wrote:</div>
<br class="">
<div class="">
<div link="blue" vlink="purple" lang="EN-US" class="">
<div class=""><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">The code for cmpimage and getdep
consists of five source files, with the
following sizes<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""><u class=""></u> <u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">$ wc *<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""> 5912 20353 191869 cmpimage.cpp<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""> 290 1328 10668 elf.h<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""> 1496 5006 41691 getdep.cpp<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""> 233 959 7692 macho.h<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""> 403 1831 18394 pecoff.h<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""> 8334 29477 270314 total<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""><u class=""></u> <u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">to build each of them is just a simple
compilation for whatever C++ compiler you happen
to be using (clang, icc, cl, g++)<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""><u class=""></u> <u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">$(CXX) –o cmpimage –O2 cmpimage.cpp<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">$(CXX) –o getdep –O2 getdep.cpp<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""><u class=""></u> <u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">This seems like it would fit rather
easily into test-suite/tools, which already
exists and has a Makefile that the commands to
build<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">these could be integrated into.<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""><u class=""></u> <u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">This is my best guess/opinion based on
a cursory look over the test-suite directory
structure.<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""><u class=""></u> <u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">Kevin<u class=""></u><u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""><u class=""></u> <u class=""></u></span></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""><u class=""></u> <u class=""></u></span></p>
<div class="">
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in" class=""><p class="MsoNormal"><b class=""><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class="">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class=""> Chris Matthews [<a href="mailto:chris.matthews@apple.com" target="_blank" class="">mailto:chris.matthews@apple.com</a>]
<br class="">
<b class="">Sent:</b> Thursday, May 28, 2015
1:02 PM<br class="">
<b class="">To:</b> Smith, Kevin B<br class="">
<b class="">Cc:</b> Philip Reames; Sean
Silva; LLVM Developers Mailing List<br class="">
<b class="">Subject:</b> Re: [LLVMdev]
Proposal: change LNT’s regression detection
algorithm and how it is used to reduce false
positives<u class=""></u><u class=""></u></span></p>
</div>
</div><p class="MsoNormal"><u class=""></u> <u class=""></u></p><p class="MsoNormal">Where is the best place to keep
this? <u class=""></u><u class=""></u></p>
<div class=""><p class="MsoNormal"><u class=""></u> <u class=""></u></p>
</div>
<div class=""><p class="MsoNormal">- As third party tool we all
use?<u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal">- Contribute as new project?<u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal">- Lives in test-suite/utils?<u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal">- Lives in llvm/utils?<u class=""></u><u class=""></u></p>
<div class=""><p class="MsoNormal"><u class=""></u> <u class=""></u></p>
<div class="">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt" class="">
<div class=""><p class="MsoNormal">On May 28, 2015, at
11:22 AM, Smith, Kevin B <<a href="mailto:kevin.b.smith@intel.com" target="_blank" class="">kevin.b.smith@intel.com</a>>
wrote:<u class=""></u><u class=""></u></p>
</div><p class="MsoNormal"><u class=""></u> <u class=""></u></p>
<div class="">
<div class=""><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">OK, there is interest from
at least a couple of people. What
should next steps be?</span><u class=""></u><u class=""></u></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""> </span><u class=""></u><u class=""></u></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">Kevin</span><u class=""></u><u class=""></u></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""> </span><u class=""></u><u class=""></u></p>
<div class="">
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in" class=""><p class="MsoNormal"><b class=""><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class="">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class=""> Chris Matthews [<a href="mailto:chris.matthews@apple.com" target="_blank" class="">mailto:chris.matthews@apple.com</a>]
<br class="">
<b class="">Sent:</b> Thursday,
May 28, 2015 10:57 AM<br class="">
<b class="">To:</b> Philip
Reames<br class="">
<b class="">Cc:</b> Smith, Kevin
B; Sean Silva; LLVM Developers
Mailing List<br class="">
<b class="">Subject:</b> Re:
[LLVMdev] Proposal: change LNT’s
regression detection algorithm
and how it is used to reduce
false positives</span><u class=""></u><u class=""></u></p>
</div>
</div><p class="MsoNormal"> <u class=""></u><u class=""></u></p><p class="MsoNormal">I agree. I think
there are a lot of exciting uses for
this tool. A stage 3 build bot would
be another one.<u class=""></u><u class=""></u></p>
<div class=""><p class="MsoNormal"> <u class=""></u><u class=""></u></p>
</div>
<div class="">
<div class="">
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt" class="">
<div class=""><p class="MsoNormal">On May 28,
2015, at 10:14 AM, Philip
Reames <<a href="mailto:listmail@philipreames.com" target="_blank" class="">listmail@philipreames.com</a>>
wrote:<u class=""></u><u class=""></u></p>
</div><p class="MsoNormal"> <u class=""></u><u class=""></u></p>
<div class="">
<div class=""><p class="MsoNormal" style="margin-bottom:12.0pt">I'd
love to see this tool
contributed, even it isn't
used for regression
detection work. I've got a
couple of hacked up scripts
which do similar things and
having a robust tool
available for this would be
very useful. <br class="">
<br class="">
Philip<u class=""></u><u class=""></u></p>
<div class=""><p class="MsoNormal">On
05/26/2015 09:53 AM,
Smith, Kevin B wrote:<u class=""></u><u class=""></u></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt" class=""><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">Intel has a
binary comparator tool
that we have been using
for several years for
comparing output
binaries</span><u class=""></u><u class=""></u></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">to see if the
code within them is
considered identical.
We use it to eliminate
runs (and therefore some
performance noise)</span><u class=""></u><u class=""></u></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">from our own
performance tracking
tools.</span><u class=""></u><u class=""></u></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""> </span><u class=""></u><u class=""></u></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">We are willing
to contribute the source
code for this to the
LLVM community if there
is interest.</span><u class=""></u><u class=""></u></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""> </span><u class=""></u><u class=""></u></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">There are two
programs involved:
getdep, which displays
the list of DLL/.so
dependencies of the
image in question, and
cmpimage itself, which
does the comparison
ignoring the parts not
contributed by the
compiler. The cmpimage
program is also almost
completely derived from
the published object
format descriptions.</span><u class=""></u><u class=""></u></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""> </span><u class=""></u><u class=""></u></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">Let me know if
there is interest in
these pieces of tooling,
and if so, what you
think next steps should
be.</span><u class=""></u><u class=""></u></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""> </span><u class=""></u><u class=""></u></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class="">Kevin B. Smith</span><u class=""></u><u class=""></u></p><p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d" class=""> </span><u class=""></u><u class=""></u></p><p class="MsoNormal"><b class=""><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class="">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" class="">
<a href="mailto:llvmdev-bounces@cs.uiuc.edu" target="_blank" class="">llvmdev-bounces@cs.uiuc.edu</a>
[<a href="mailto:llvmdev-bounces@cs.uiuc.edu" target="_blank" class="">mailto:llvmdev-bounces@cs.uiuc.edu</a>]
<b class="">On Behalf Of
</b>Sean Silva<br class="">
<b class="">Sent:</b>
Thursday, May 21, 2015
2:14 PM<br class="">
<b class="">To:</b>
Chris Matthews<br class="">
<b class="">Cc:</b> LLVM
Developers Mailing List<br class="">
<b class="">Subject:</b>
Re: [LLVMdev] Proposal:
change LNT’s regression
detection algorithm and
how it is used to reduce
false positives</span><u class=""></u><u class=""></u></p><p class="MsoNormal"> <u class=""></u><u class=""></u></p>
<div class=""><p class="MsoNormal"> <u class=""></u><u class=""></u></p>
<div class=""><p class="MsoNormal"> <u class=""></u><u class=""></u></p>
<div class=""><p class="MsoNormal">On
Thu, May 21, 2015 at
11:24 AM, Chris
Matthews <<a href="mailto:chris.matthews@apple.com" target="_blank" class="">chris.matthews@apple.com</a>>
wrote:<u class=""></u><u class=""></u></p><p class="MsoNormal">I
agree this is a
great idea. I think
it needs to be
fleshed out a little
though.<br class="">
<br class="">
It would still be
wise to run the
regression detection
algorithm, because
the test suite
changes and the
machines change, and
the algorithm is not
perfect yet. It
would be a valuable
source of
information though.<u class=""></u><u class=""></u></p>
<div class=""><p class="MsoNormal"> <u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal">How
would running it
as part of regular
testing change
anything?
Presumably the
only purpose it
would serve is
retrospectively
going back and
seeing
false-positives in
the aggregate. But
if we are already
doing offline
analysis, we can
run the regression
detection
algorithm (or any
prospective ones)
offline on the raw
data; it doesn't
take that long.<u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal"> <u class=""></u><u class=""></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt" class=""><p class="MsoNormal"><br class="">
This is not a
small change to
how LNT works, so
I think some due
diligence is
necessary. Is
clang *really*
that
deterministic,
especially over
successive revs? <u class=""></u><u class=""></u></p>
</blockquote>
<div class=""><p class="MsoNormal"> <u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal">Yes.
Actually, google's
build system
depends on this
for its caching
strategy to work
and so the google
guys are usually
on top of any
issues in this
respect (thanks
google guys!).<u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal"> <u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal"> <u class=""></u><u class=""></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt" class=""><p class="MsoNormal">I
know it is
supposed to be.
Does anyone have
any data to show
this is going to
be an effective
approach? It
seems like there
are benchmarks in
the test-suite
which use __DATE__
and __TIME__ in
them. I assume
that will be a
problem?<u class=""></u><u class=""></u></p>
</blockquote>
<div class=""><p class="MsoNormal"> <u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal">__DATE__
and __TIME__
should be easy to
solve by modifying
the benchmark, or
teaching clang to
always return a
fixed value for
them (maybe we
already have this?
IIRC google's
build system does
something like
this; or maybe the
do it at the OS
level).<u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal"> <u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal">--
Sean Silva<u class=""></u><u class=""></u></p>
</div>
<div class=""><p class="MsoNormal"> <u class=""></u><u class=""></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt" class="">
<div class="">
<div class=""><p class="MsoNormal"><br class="">
> On May
21, 2015, at
1:43 AM,
Renato Golin
<<a href="mailto:renato.golin@linaro.org" target="_blank" class="">renato.golin@linaro.org</a>>
wrote:<br class="">
><br class="">
> On 20 May
2015 at 23:31,
Sean Silva
<<a href="mailto:chisophugis@gmail.com" target="_blank" class="">chisophugis@gmail.com</a>>
wrote:<br class="">
>> In
the last
10,000
revisions of
LLVM+Clang,
only 10
revisions
actually<br class="">
>>
caused the
binary of
MultiSource/Benchmarks/BitBench/five11
to change. So
if<br class="">
>> just
store a hash
of the binary
in the
database, we
should be able
to pool<br class="">
>> all
samples we
have collected
while the
binary is the
the same as it<br class="">
>>
currently is,
which will let
us use
significantly
more
datapoints for
the<br class="">
>>
reference.<br class="">
><br class="">
> +1<br class="">
><br class="">
><br class="">
>> Also,
we can
trivially
eliminate
running the
regression
detection
algorithm<br class="">
>> if
the binary
hasn't
changed.<br class="">
><br class="">
> +2!<br class="">
><br class="">
> --renato<u class=""></u><u class=""></u></p>
</div>
</div>
<div class="">
<div class=""><p class="MsoNormal" style="margin-bottom:12.0pt">>
_______________________________________________<br class="">
> LLVM
Developers
mailing list<br class="">
> <a href="mailto:LLVMdev@cs.uiuc.edu" target="_blank" class="">LLVMdev@cs.uiuc.edu</a>
<a href="http://llvm.cs.uiuc.edu/" target="_blank" class="">http://llvm.cs.uiuc.edu</a><br class="">
> <a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev" target="_blank" class="">http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev</a><u class=""></u><u class=""></u></p>
</div>
</div>
</blockquote>
</div><p class="MsoNormal"> <u class=""></u><u class=""></u></p>
</div>
</div><p class="MsoNormal"><br class="">
<br class="">
<br class="">
<br class="">
<u class=""></u><u class=""></u></p>
<pre class="">_______________________________________________<u class=""></u><u class=""></u></pre>
<pre class="">LLVM Developers mailing list<u class=""></u><u class=""></u></pre>
<pre class=""><a href="mailto:LLVMdev@cs.uiuc.edu" target="_blank" class="">LLVMdev@cs.uiuc.edu</a> <a href="http://llvm.cs.uiuc.edu/" target="_blank" class="">http://llvm.cs.uiuc.edu</a><u class=""></u><u class=""></u></pre>
<pre class=""><a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev" target="_blank" class="">http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev</a><u class=""></u><u class=""></u></pre>
</blockquote><p class="MsoNormal"> <u class=""></u><u class=""></u></p>
</div>
</div>
</blockquote>
</div><p class="MsoNormal"> <u class=""></u><u class=""></u></p>
</div>
</div>
</div>
</blockquote>
</div><p class="MsoNormal"><u class=""></u> <u class=""></u></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
<br class="">
</div></div></div>
</blockquote></div><br class=""></div></div>
</div></blockquote></div><br class=""></div></div></body></html>