<div dir="ltr"><br><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Currently the script uses perf and cset for running the benchmarks on<br>
linux. That part will have to be abstracted in the future.<br>
<br>
What statistics can one collect on windows? I ask to see if the lnt<br>
schema should be extended.<br></blockquote><div><br></div><div>Reading between the lines of the MSDN documentation at <a href="https://msdn.microsoft.com/en-us/library/bb385772.aspx" target="_blank">https://msdn.microsoft.com/en-<wbr>us/library/bb385772.aspx</a>, it is possible to collect a whole range of stats if you have the Visual Studio Performance tools installed, but the complete set may vary from machine to machine. The list of so-called "portable" counters available on my machines, which are supposedly independent of processor, are:</div><div><br></div><div> InstructionsRetired<br> NonHaltedCycles<br> L2Misses<br> L2References<br> ITLBMisses<br> BranchesRetired<br> MispredictedBranches</div><div><br></div><div>It's also possible to gather memory information, via the Windows API using GetProcessMemoryInfo, which returns a struct containing the following members:</div><div><br></div> PageFaultCount;<br> PeakWorkingSetSize;<br> WorkingSetSize;<br> QuotaPeakPagedPoolUsage;<br> QuotaPagedPoolUsage;<br> QuotaPeakNonPagedPoolUsage;<br> QuotaNonPagedPoolUsage;<br> PagefileUsage;<br> PeakPagefileUsage;<br> PrivateUsage;<div> </div><div>This is for the 64-bit variant - it may be necessary to create a simple program to wrap the function to enable it to be run by a 32-bit program, such as a typical python installation.</div><div><br></div><div>I don't know if there are other values available, but I couldn't immediately find any more information.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="m_675379160789730823gmail-"><br>
<br>
> 2) We would find it useful if there were hooks we could easily<br>
> configure to do various environment setup/tear-down steps. Perhaps this<br>
> could be achieved by adding the ability to specify a wrapper process that<br>
> in turn runs some specified link-line after/before doing some work itself<br>
> such as configuring process priorities etc.<br>
<br>
</span>That is one part I am undecided. Currently the script just runs the<br>
benchmark and sends the result to a lnt server. It is up to something<br>
else to build the various lld versions.<br>
<br>
I did the split like this for two reasons. First, it is useful to just<br>
compare two versions manually to test a patch. Second, I assume that<br>
most of the logic for repeatedly building lld is already coded in the<br>
existing bot infrastructure.<br></blockquote><div><br></div><div>I think maybe I wasn't as clear as I could be - in fact the script up for review already does what I meant, but explicitly for cset and then perf. Providing a way of extending this list of programs to call before running the test would be helpful. One Windows example would be to run a program that disables/re-enables Superfetch during the course of the run.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="m_675379160789730823gmail-"><br>
> 3) Finally, it would be really useful to support implied or<br>
> externally-imposed variants for tests, where each variant runs the test<br>
> from a distinct location. This would allow us to run each case from an HDD,<br>
> SSD, and from RAM. RAM's good for reducing noise, but we have also found it<br>
> useful to have some real-world numbers available.<br>
<br>
</span>In the current setup the script is oblivious to that. It is just given a<br>
directory, that directory can be in tmpfs, ssd or hd.<br>
<span class="m_675379160789730823gmail-"><br>
> I also noticed that you didn’t mention measuring memory usage anywhere. I<br>
> assume that this is something that you’d be measuring as well?<br>
<br>
</span>Currently I am not recording that, but it would be a good addition.<br></blockquote><div><br></div><div>From my point of view, memory usage is nearly as important as runtime performance, because some links can use up many GB of memory, which already pushes the limits of what some users have available, so it definitely should be added at some point. I think it's also useful to measure the output size. In general, this should not change, but we might, for example, improve the effectiveness of icf or gc-sections somehow, and it would be useful to point to the improvements quickly. I'm less concerned about this, however.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Cheers,<br>
Rafael<br>
<br>
<br>
</blockquote></div><br></div></div>