[llvm-dev] Benchmarks for LLVM-generated Binaries

Mon Sep 12 15:31:54 PDT 2016

I added support for extensions to the test-suite cmake/lit a while ago: In a default cmake run it will look for */CMakeLists.txt and compiles/runs the benchmarks there. Usually that will be External/SingleSource/MultiSource, but if you drop additional directories in your test-suite checkout it should pick those up. You can also manually specify the TEST_SUITE_SUBDIRS variable instead if you do not want to run everything or want it to pick up directories outside the llvm test-suite.

That should be enough to move specialized tests into independent repositories while re-using the running/cmake infrastructure. It would probably be a good idea to this with the External and Bitcode directories today, we just have to follow through and figure out which buildbots need to run what and which additional repositories they need to check out.

- Matthias

> On Sep 7, 2016, at 10:40 AM, Eric Christopher via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> That's correct. It seems like a fairly well contained local patch for your local testsuite, otherwise you could probably change test-suite to try to understand arbitrary directories. 
> 
> On Wed, Sep 7, 2016 at 10:31 AM Michael Zolotukhin <mzolotukhin at apple.com <mailto:mzolotukhin at apple.com>> wrote:
> Hi Eric,
> 
> Yeah, I know about Externals and SPEC specifically. But as far as I understand, you have to have kind of description of the tests in test-suite even if you don’t provide the source codes - that’s what I would like to avoid. I.e. you have to have CMakeLists.txt and other files in place all the time, open to everyone.
> 
> Now, imagine I have a small testsuite, which probably is not very interesting to anyone else, but extremely interesting for me. AFAIU, to make it a part of LNT now, I have to modify ‘test-suite’ repo so that it’s aware of this new test-suite. I will probably not be able to upstream these changes (as they are only interesting to me), and that’s the source of some inconvenience. I’d be happy to be mistaken here, but that how I understand the current infrastructure, and that where my question came from.
> 
> Thanks,
> Michael
> 
>> On Sep 7, 2016, at 10:13 AM, Eric Christopher <echristo at gmail.com <mailto:echristo at gmail.com>> wrote:
>> 
>> Hi Michael,
>> 
>> You'll want to look into the externals part of the test suite :) It's how things like SPEC etc are run.
>> 
>> -eric
>> 
>> On Tue, Sep 6, 2016 at 4:08 PM Michael Zolotukhin via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> 
>> > On Sep 1, 2016, at 8:14 AM, Renato Golin via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> >
>> > On 1 September 2016 at 07:45, Dean Michael Berris via llvm-dev
>> > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> >> I've lately been wondering where benchmarks for LLVM-generated binaries are hosted, and whether they're tracked over time.
>> >
>> > Hi Dean,
>> >
>> > Do you mean Perf?
>> >
>> > http://llvm.org/perf/ <http://llvm.org/perf/>
>> >
>> > Example, ARM and AArch64 tracking performance at:
>> >
>> > http://llvm.org/perf/db_default/v4/nts/machine/41 <http://llvm.org/perf/db_default/v4/nts/machine/41>
>> > http://llvm.org/perf/db_default/v4/nts/machine/46 <http://llvm.org/perf/db_default/v4/nts/machine/46>
>> >
>> >
>> >> - Is the test-suite repository the right place to put these generated-code benchmarks?
>> >
>> > I believe there would be the best place, yes.
>> >
>> >
>> >> - Are there any objections to using a later version of the Google benchmarking library [0] in the test-suite?
>> >
>> > While this looks like a very nice tool set, I wonder how we're going
>> > to integrate it.
>> >
>> > Checking it out in the test-suite wouldn't be the best option (version
>> > rot), but neither would be requiring people to install it before
>> > running the test-suite, especially if the installation process isn't
>> > as easy as "apt-get install", like all the other dependencies.
>> >
>> >
>> >> - Are the docs on the Testing Infrastructure Guide still relevant and up-to-date, and is that a good starting point for exploration here?
>> >
>> > Unfortunately, that's mostly for the "make check" tests, not for the
>> > test-suite. The test-suite execution is covered by LNT's doc
>> > (http://llvm.org/docs/lnt <http://llvm.org/docs/lnt>), but it's mostly about LNT internals and
>> > not the test-suite itself.
>> >
>> > However, it's not that hard to understand the test-suite structure. To
>> > add new tests, you just need to find a suitable place { (SingleSource
>> > / MultiSource) / Benchmarks / YourBench } and copy ( CMakeFiles.txt,
>> > Makefile, lit.local.cfg ), change to your needs, and it should be
>> > done.
>> Hi Renato and others,
>> 
>> Is it possible/how hard will it be to make the testsuite kind of extendable? Like, I copy a folder with some tests that comply to some rules (e.g. have CMakeLists.txt, lit.local.cfg, etc.) and run them via the standard infrastructure without changing anything in test-suite files?
>> 
>> The motivation of this question is the following: we (and probably many other companies too) have internal tests that we can’t share, but still want to track. Currently, the process of adding them to the existing test-suite is not clear to me (or at least not very well documented), and while I can figure it out, it would be cool if we can streamline this process.
>> 
>> Ideally, I’d like to see this process like this:
>> 1) Add following files to your benchmark suite:
>>   1.a) CMakeLists.txt having this and that target doing this and that.
>>   1.b) lit.local.cfg script having this and that.
>>   …
>> 2) Make sure the test report results in the following format/provide a wrapper script to convert results to the specified form. /* TODO: Results format is specified here */
>> 3) Run your tests using the standard LNT command, like  "lnt runtest … --only-test=External/MyTestSuite/TestA”
>> 
>> If that’s already implemented, then I’ll be glad to help with documentation, and if not, I can try implementing it. What do you think?
>> 
>> Thanks,
>> Michael
>> 
>> >
>> > cheers,
>> > --renato
>> > _______________________________________________
>> > LLVM Developers mailing list
>> > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160912/2fdea9c8/attachment.html>