<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Have you seen the prototype for googlebenchmark integration I did in the past:</div><div class=""><br class=""></div><div class=""><a href="https://reviews.llvm.org/D18428" class="">https://reviews.llvm.org/D18428</a> (though probably out of date for todays test-suite)</div><div class=""><br class=""></div><div class="">+1 for copying the googlebechmark into the test-suite.</div><div class=""><br class=""></div><div class="">However I do not think this should simply go into MultiSource: We currently have a number of additional plugins in the lit test runner such as measuring the runtime of the benchmark executable, determining code size, we still plan to add a mode to run benchmarks multiple times, we run the bechmark under perf (or iOS specific tools) to collect performance counters… Many of those are questionable measurements for a googlebenchmark executable which has varying runtime because it runs the test more/less often.</div><div class="">We really should introduce a new benchmarking mode for this.</div><div class=""><br class=""></div><div class="">- Matthias</div><div class=""><br class=""></div><div><blockquote type="cite" class=""><div class="">On Sep 14, 2016, at 8:29 AM, Eric Christopher via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""><br class=""><div class="gmail_quote"><div dir="ltr" class="">On Wed, Sep 14, 2016 at 8:23 AM Mehdi Amini via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div class="gmail_msg" style="word-wrap: break-word;"><br class="gmail_msg"><div class="gmail_msg"><blockquote type="cite" class="gmail_msg"><div class="gmail_msg">On Sep 14, 2016, at 12:50 AM, Dean Michael Berris via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:</div><br class="gmail_msg m_-6377964279478798610Apple-interchange-newline"><div class="gmail_msg"><span class="gmail_msg" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; float: none; display: inline !important;">I'm working on this now, and I had a few more questions below for Renato and the list in general. Please see inline below.</span><br class="gmail_msg" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br class="gmail_msg" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><blockquote type="cite" class="gmail_msg" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">On 2 Sep 2016, at 11:27, Dean Michael Berris <<a href="mailto:dean.berris@gmail.com" class="gmail_msg" target="_blank">dean.berris@gmail.com</a>> wrote:<br class="gmail_msg"><br class="gmail_msg"><blockquote type="cite" class="gmail_msg">On 2 Sep 2016, at 11:19, Renato Golin <<a href="mailto:renato.golin@linaro.org" class="gmail_msg" target="_blank">renato.golin@linaro.org</a>> wrote:<br class="gmail_msg"><br class="gmail_msg">On 2 September 2016 at 02:13, Dean Michael Berris <<a href="mailto:dean.berris@gmail.com" class="gmail_msg" target="_blank">dean.berris@gmail.com</a>> wrote:<br class="gmail_msg"><blockquote type="cite" class="gmail_msg">I think it should be possible to have a snapshot of it included. I don't know what the licensing implications are (I'm not a lawyer, but I know someone who is -- paging Danny Berlin).<br class="gmail_msg"></blockquote><br class="gmail_msg">The test-suite has a very large number of licenses (compared to LLVM),<br class="gmail_msg">so licensing should be less of a problem there. Though Dan can help<br class="gmail_msg">more than I can. :)<br class="gmail_msg"><br class="gmail_msg"></blockquote><br class="gmail_msg">Cool, let's wait for what Danny thinks on the patch I'll be preparing. :)<br class="gmail_msg"><br class="gmail_msg"><blockquote type="cite" class="gmail_msg"><br class="gmail_msg"><blockquote type="cite" class="gmail_msg">I'm not as concerned about falling behind on versions there though mostly because it should be trivial to update it if we need it. Though like you, I agree this isn't the best way of doing it. :)<br class="gmail_msg"></blockquote><br class="gmail_msg">If we start using it more (maybe we should, at least for the<br class="gmail_msg">benchmarks, I've been long wanting to do something decent there), then<br class="gmail_msg">we'd need to add a proper update procedure.<br class="gmail_msg"><br class="gmail_msg">I'm fine with some checkout if it's a stable release, not trunk, as it<br class="gmail_msg">would make things a lot easier to update later (patch releases, new<br class="gmail_msg">releases, etc).<br class="gmail_msg"><br class="gmail_msg"></blockquote><br class="gmail_msg">SGTM.<br class="gmail_msg"><br class="gmail_msg"></blockquote><br class="gmail_msg" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><span class="gmail_msg" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; float: none; display: inline !important;">Is there a preference on where to place the library? I had a look at {SingleSource/MultiSource}/Benchmarks/ and I didn't find a common location for libraries used. I'm tempted to create a top-level "libs" directory that will host common libraries but I'm also fine with just having the benchmark library living alongside the XRay benchmarks.</span><br class="gmail_msg" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br class="gmail_msg" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><span class="gmail_msg" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; float: none; display: inline !important;">So two options here:</span><br class="gmail_msg" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><br class="gmail_msg" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><span class="gmail_msg" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; float: none; display: inline !important;">1) libs/googlebenchmark/</span><br class="gmail_msg" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><span class="gmail_msg" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; float: none; display: inline !important;">2) MultiSource/Benchmarks/XRay/googlebench/</span><br class="gmail_msg" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div><div class="gmail_msg" style="word-wrap: break-word;"><div class="gmail_msg">This is something that may be used (or is intended) to be used by others in the future, the first option makes it easier (or encouraging at least).</div><div class="gmail_msg"><br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">+1 to this.</div><div class=""><br class=""></div><div class="">Looks like there is reasonably active development going on right now (primarily by EricWF who is also contributing to llvm), so we'll probably want to coordinate how and how often we sync with top of tree. (Probably more often than google unittests :)</div><div class=""><br class=""></div><div class="">-eric </div></div></div><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">LLVM Developers mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="mailto:llvm-dev@lists.llvm.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">llvm-dev@lists.llvm.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a></div></blockquote></div><br class=""></body></html>