<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 2, 2015 at 2:24 PM, Philip Reames <span dir="ltr"><<a href="mailto:listmail@philipreames.com" target="_blank">listmail@philipreames.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Personally, I would prefer this either live in it's own repository,
or llvm/tools/. None of my use cases will likely involve the
test-suite.<br>
<br>
p.s. If this is going to end up an llvm tool, it will need to follow
LLVM style. <br>
<br>
p.p.s. We should probably start a new thread with the proposed
addition since I imagine many folks are ignoring this one by now
given how deep it's gotten.</div></blockquote><div><br></div><div>Maybe just putting it on github for now is easiest to at least make it generally available for review. If we later want to officially pull it in or integrate it with our build system we can do that.</div><div><br></div><div>-- Sean Silva</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><span class="HOEnZb"><font color="#888888"><br>
<br>
Philip</font></span><div><div class="h5"><br>
<br>
<div>On 06/02/2015 12:04 PM, Chris Matthews
wrote:<br>
</div>
<blockquote type="cite">
I like that idea!
<div><br>
</div>
<div><br>
<div>
<div>
<blockquote type="cite">
<div>On Jun 2, 2015, at 12:00 PM, Smith, Kevin B
<<a href="mailto:kevin.b.smith@intel.com" target="_blank">kevin.b.smith@intel.com</a>>
wrote:</div>
<br>
<div>
<div link="blue" vlink="purple" lang="EN-US">
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">The code for cmpimage and getdep
consists of five source files, with the
following sizes<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">$ wc *<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"> 5912 20353 191869 cmpimage.cpp<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"> 290 1328 10668 elf.h<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"> 1496 5006 41691 getdep.cpp<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"> 233 959 7692 macho.h<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"> 403 1831 18394 pecoff.h<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"> 8334 29477 270314 total<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">to build each of them is just a simple
compilation for whatever C++ compiler you happen
to be using (clang, icc, cl, g++)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">$(CXX) –o cmpimage –O2 cmpimage.cpp<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">$(CXX) –o getdep –O2 getdep.cpp<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">This seems like it would fit rather
easily into test-suite/tools, which already
exists and has a Makefile that the commands to
build<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">these could be integrated into.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">This is my best guess/opinion based on
a cursory look over the test-suite directory
structure.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">Kevin<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Chris Matthews [<a href="mailto:chris.matthews@apple.com" target="_blank">mailto:chris.matthews@apple.com</a>]
<br>
<b>Sent:</b> Thursday, May 28, 2015
1:02 PM<br>
<b>To:</b> Smith, Kevin B<br>
<b>Cc:</b> Philip Reames; Sean
Silva; LLVM Developers Mailing List<br>
<b>Subject:</b> Re: [LLVMdev]
Proposal: change LNT’s regression detection
algorithm and how it is used to reduce false
positives<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Where is the best place to keep
this? <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">- As third party tool we all
use?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- Contribute as new project?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- Lives in test-suite/utils?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- Lives in llvm/utils?<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On May 28, 2015, at
11:22 AM, Smith, Kevin B <<a href="mailto:kevin.b.smith@intel.com" target="_blank">kevin.b.smith@intel.com</a>>
wrote:<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">OK, there is interest from
at least a couple of people. What
should next steps be?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">Kevin</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Chris Matthews [<a href="mailto:chris.matthews@apple.com" target="_blank">mailto:chris.matthews@apple.com</a>]
<br>
<b>Sent:</b> Thursday,
May 28, 2015 10:57 AM<br>
<b>To:</b> Philip
Reames<br>
<b>Cc:</b> Smith, Kevin
B; Sean Silva; LLVM Developers
Mailing List<br>
<b>Subject:</b> Re:
[LLVMdev] Proposal: change LNT’s
regression detection algorithm
and how it is used to reduce
false positives</span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">I agree. I think
there are a lot of exciting uses for
this tool. A stage 3 build bot would
be another one.<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On May 28,
2015, at 10:14 AM, Philip
Reames <<a href="mailto:listmail@philipreames.com" target="_blank">listmail@philipreames.com</a>>
wrote:<u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I'd
love to see this tool
contributed, even it isn't
used for regression
detection work. I've got a
couple of hacked up scripts
which do similar things and
having a robust tool
available for this would be
very useful. <br>
<br>
Philip<u></u><u></u></p>
<div>
<p class="MsoNormal">On
05/26/2015 09:53 AM,
Smith, Kevin B wrote:<u></u><u></u></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">Intel has a
binary comparator tool
that we have been using
for several years for
comparing output
binaries</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">to see if the
code within them is
considered identical.
We use it to eliminate
runs (and therefore some
performance noise)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">from our own
performance tracking
tools.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">We are willing
to contribute the source
code for this to the
LLVM community if there
is interest.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">There are two
programs involved:
getdep, which displays
the list of DLL/.so
dependencies of the
image in question, and
cmpimage itself, which
does the comparison
ignoring the parts not
contributed by the
compiler. The cmpimage
program is also almost
completely derived from
the published object
format descriptions.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">Let me know if
there is interest in
these pieces of tooling,
and if so, what you
think next steps should
be.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d">Kevin B. Smith</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a href="mailto:llvmdev-bounces@cs.uiuc.edu" target="_blank">llvmdev-bounces@cs.uiuc.edu</a>
[<a href="mailto:llvmdev-bounces@cs.uiuc.edu" target="_blank">mailto:llvmdev-bounces@cs.uiuc.edu</a>]
<b>On Behalf Of
</b>Sean Silva<br>
<b>Sent:</b>
Thursday, May 21, 2015
2:14 PM<br>
<b>To:</b>
Chris Matthews<br>
<b>Cc:</b> LLVM
Developers Mailing List<br>
<b>Subject:</b>
Re: [LLVMdev] Proposal:
change LNT’s regression
detection algorithm and
how it is used to reduce
false positives</span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On
Thu, May 21, 2015 at
11:24 AM, Chris
Matthews <<a href="mailto:chris.matthews@apple.com" target="_blank">chris.matthews@apple.com</a>>
wrote:<u></u><u></u></p>
<p class="MsoNormal">I
agree this is a
great idea. I think
it needs to be
fleshed out a little
though.<br>
<br>
It would still be
wise to run the
regression detection
algorithm, because
the test suite
changes and the
machines change, and
the algorithm is not
perfect yet. It
would be a valuable
source of
information though.<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">How
would running it
as part of regular
testing change
anything?
Presumably the
only purpose it
would serve is
retrospectively
going back and
seeing
false-positives in
the aggregate. But
if we are already
doing offline
analysis, we can
run the regression
detection
algorithm (or any
prospective ones)
offline on the raw
data; it doesn't
take that long.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal"><br>
This is not a
small change to
how LNT works, so
I think some due
diligence is
necessary. Is
clang *really*
that
deterministic,
especially over
successive revs? <u></u><u></u></p>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Yes.
Actually, google's
build system
depends on this
for its caching
strategy to work
and so the google
guys are usually
on top of any
issues in this
respect (thanks
google guys!).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal">I
know it is
supposed to be.
Does anyone have
any data to show
this is going to
be an effective
approach? It
seems like there
are benchmarks in
the test-suite
which use __DATE__
and __TIME__ in
them. I assume
that will be a
problem?<u></u><u></u></p>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">__DATE__
and __TIME__
should be easy to
solve by modifying
the benchmark, or
teaching clang to
always return a
fixed value for
them (maybe we
already have this?
IIRC google's
build system does
something like
this; or maybe the
do it at the OS
level).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">--
Sean Silva<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><br>
> On May
21, 2015, at
1:43 AM,
Renato Golin
<<a href="mailto:renato.golin@linaro.org" target="_blank">renato.golin@linaro.org</a>>
wrote:<br>
><br>
> On 20 May
2015 at 23:31,
Sean Silva
<<a href="mailto:chisophugis@gmail.com" target="_blank">chisophugis@gmail.com</a>>
wrote:<br>
>> In
the last
10,000
revisions of
LLVM+Clang,
only 10
revisions
actually<br>
>>
caused the
binary of
MultiSource/Benchmarks/BitBench/five11
to change. So
if<br>
>> just
store a hash
of the binary
in the
database, we
should be able
to pool<br>
>> all
samples we
have collected
while the
binary is the
the same as it<br>
>>
currently is,
which will let
us use
significantly
more
datapoints for
the<br>
>>
reference.<br>
><br>
> +1<br>
><br>
><br>
>> Also,
we can
trivially
eliminate
running the
regression
detection
algorithm<br>
>> if
the binary
hasn't
changed.<br>
><br>
> +2!<br>
><br>
> --renato<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">>
_______________________________________________<br>
> LLVM
Developers
mailing list<br>
> <a href="mailto:LLVMdev@cs.uiuc.edu" target="_blank">LLVMdev@cs.uiuc.edu</a>
<a href="http://llvm.cs.uiuc.edu/" target="_blank">http://llvm.cs.uiuc.edu</a><br>
> <a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev</a><u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><br>
<br>
<br>
<br>
<u></u><u></u></p>
<pre>_______________________________________________<u></u><u></u></pre>
<pre>LLVM Developers mailing list<u></u><u></u></pre>
<pre><a href="mailto:LLVMdev@cs.uiuc.edu" target="_blank">LLVMdev@cs.uiuc.edu</a> <a href="http://llvm.cs.uiuc.edu/" target="_blank">http://llvm.cs.uiuc.edu</a><u></u><u></u></pre>
<pre><a href="http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev</a><u></u><u></u></pre>
</blockquote>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div></div>