<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">I added support for extensions to the test-suite cmake/lit a while ago: In a default cmake run it will look for */CMakeLists.txt and compiles/runs the benchmarks there. Usually that will be External/SingleSource/MultiSource, but if you drop additional directories in your test-suite checkout it should pick those up. You can also manually specify the TEST_SUITE_SUBDIRS variable instead if you do not want to run everything or want it to pick up directories outside the llvm test-suite.</div><div class=""><br class=""></div><div class="">That should be enough to move specialized tests into independent repositories while re-using the running/cmake infrastructure. It would probably be a good idea to this with the External and Bitcode directories today, we just have to follow through and figure out which buildbots need to run what and which additional repositories they need to check out.</div><div class=""><br class=""></div><div class="">- Matthias</div><div class=""><br class=""></div><div><blockquote type="cite" class=""><div class="">On Sep 7, 2016, at 10:40 AM, Eric Christopher via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">That's correct. It seems like a fairly well contained local patch for your local testsuite, otherwise you could probably change test-suite to try to understand arbitrary directories. </div><br class=""><div class="gmail_quote"><div dir="ltr" class="">On Wed, Sep 7, 2016 at 10:31 AM Michael Zolotukhin <<a href="mailto:mzolotukhin@apple.com" class="">mzolotukhin@apple.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class="gmail_msg">Hi Eric,<div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Yeah, I know about Externals and SPEC specifically. But as far as I understand, you have to have kind of description of the tests in test-suite even if you don’t provide the source codes - that’s what I would like to avoid. I.e. you have to have CMakeLists.txt and other files in place all the time, open to everyone.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Now, imagine I have a small testsuite, which probably is not very interesting to anyone else, but extremely interesting for me. AFAIU, to make it a part of LNT now, I have to modify ‘test-suite’ repo so that it’s aware of this new test-suite. I will probably not be able to upstream these changes (as they are only interesting to me), and that’s the source of some inconvenience. I’d be happy to be mistaken here, but that how I understand the current infrastructure, and that where my question came from.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Thanks,</div><div class="gmail_msg">Michael</div></div><div style="word-wrap:break-word" class="gmail_msg"><div class="gmail_msg"><br class="gmail_msg"><div class="gmail_msg"><blockquote type="cite" class="gmail_msg"><div class="gmail_msg">On Sep 7, 2016, at 10:13 AM, Eric Christopher <<a href="mailto:echristo@gmail.com" class="gmail_msg" target="_blank">echristo@gmail.com</a>> wrote:</div><br class="gmail_msg m_3370304052884748815Apple-interchange-newline"><div class="gmail_msg"><div dir="ltr" class="gmail_msg">Hi Michael,<div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">You'll want to look into the externals part of the test suite :) It's how things like SPEC etc are run.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">-eric</div></div><br class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Tue, Sep 6, 2016 at 4:08 PM Michael Zolotukhin via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br class="gmail_msg">
> On Sep 1, 2016, at 8:14 AM, Renato Golin via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br class="gmail_msg">
><br class="gmail_msg">
> On 1 September 2016 at 07:45, Dean Michael Berris via llvm-dev<br class="gmail_msg">
> <<a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br class="gmail_msg">
>> I've lately been wondering where benchmarks for LLVM-generated binaries are hosted, and whether they're tracked over time.<br class="gmail_msg">
><br class="gmail_msg">
> Hi Dean,<br class="gmail_msg">
><br class="gmail_msg">
> Do you mean Perf?<br class="gmail_msg">
><br class="gmail_msg">
> <a href="http://llvm.org/perf/" rel="noreferrer" class="gmail_msg" target="_blank">http://llvm.org/perf/</a><br class="gmail_msg">
><br class="gmail_msg">
> Example, ARM and AArch64 tracking performance at:<br class="gmail_msg">
><br class="gmail_msg">
> <a href="http://llvm.org/perf/db_default/v4/nts/machine/41" rel="noreferrer" class="gmail_msg" target="_blank">http://llvm.org/perf/db_default/v4/nts/machine/41</a><br class="gmail_msg">
> <a href="http://llvm.org/perf/db_default/v4/nts/machine/46" rel="noreferrer" class="gmail_msg" target="_blank">http://llvm.org/perf/db_default/v4/nts/machine/46</a><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
>> - Is the test-suite repository the right place to put these generated-code benchmarks?<br class="gmail_msg">
><br class="gmail_msg">
> I believe there would be the best place, yes.<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
>> - Are there any objections to using a later version of the Google benchmarking library [0] in the test-suite?<br class="gmail_msg">
><br class="gmail_msg">
> While this looks like a very nice tool set, I wonder how we're going<br class="gmail_msg">
> to integrate it.<br class="gmail_msg">
><br class="gmail_msg">
> Checking it out in the test-suite wouldn't be the best option (version<br class="gmail_msg">
> rot), but neither would be requiring people to install it before<br class="gmail_msg">
> running the test-suite, especially if the installation process isn't<br class="gmail_msg">
> as easy as "apt-get install", like all the other dependencies.<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
>> - Are the docs on the Testing Infrastructure Guide still relevant and up-to-date, and is that a good starting point for exploration here?<br class="gmail_msg">
><br class="gmail_msg">
> Unfortunately, that's mostly for the "make check" tests, not for the<br class="gmail_msg">
> test-suite. The test-suite execution is covered by LNT's doc<br class="gmail_msg">
> (<a href="http://llvm.org/docs/lnt" rel="noreferrer" class="gmail_msg" target="_blank">http://llvm.org/docs/lnt</a>), but it's mostly about LNT internals and<br class="gmail_msg">
> not the test-suite itself.<br class="gmail_msg">
><br class="gmail_msg">
> However, it's not that hard to understand the test-suite structure. To<br class="gmail_msg">
> add new tests, you just need to find a suitable place { (SingleSource<br class="gmail_msg">
> / MultiSource) / Benchmarks / YourBench } and copy ( CMakeFiles.txt,<br class="gmail_msg">
> Makefile, lit.local.cfg ), change to your needs, and it should be<br class="gmail_msg">
> done.<br class="gmail_msg">
Hi Renato and others,<br class="gmail_msg">
<br class="gmail_msg">
Is it possible/how hard will it be to make the testsuite kind of extendable? Like, I copy a folder with some tests that comply to some rules (e.g. have CMakeLists.txt, lit.local.cfg, etc.) and run them via the standard infrastructure without changing anything in test-suite files?<br class="gmail_msg">
<br class="gmail_msg">
The motivation of this question is the following: we (and probably many other companies too) have internal tests that we can’t share, but still want to track. Currently, the process of adding them to the existing test-suite is not clear to me (or at least not very well documented), and while I can figure it out, it would be cool if we can streamline this process.<br class="gmail_msg">
<br class="gmail_msg">
Ideally, I’d like to see this process like this:<br class="gmail_msg">
1) Add following files to your benchmark suite:<br class="gmail_msg">
1.a) CMakeLists.txt having this and that target doing this and that.<br class="gmail_msg">
1.b) lit.local.cfg script having this and that.<br class="gmail_msg">
…<br class="gmail_msg">
2) Make sure the test report results in the following format/provide a wrapper script to convert results to the specified form. /* TODO: Results format is specified here */<br class="gmail_msg">
3) Run your tests using the standard LNT command, like "lnt runtest … --only-test=External/MyTestSuite/TestA”<br class="gmail_msg">
<br class="gmail_msg">
If that’s already implemented, then I’ll be glad to help with documentation, and if not, I can try implementing it. What do you think?<br class="gmail_msg">
<br class="gmail_msg">
Thanks,<br class="gmail_msg">
Michael<br class="gmail_msg">
<br class="gmail_msg">
><br class="gmail_msg">
> cheers,<br class="gmail_msg">
> --renato<br class="gmail_msg">
> _______________________________________________<br class="gmail_msg">
> LLVM Developers mailing list<br class="gmail_msg">
> <a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" target="_blank">llvm-dev@lists.llvm.org</a><br class="gmail_msg">
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="gmail_msg">
<br class="gmail_msg">
_______________________________________________<br class="gmail_msg">
LLVM Developers mailing list<br class="gmail_msg">
<a href="mailto:llvm-dev@lists.llvm.org" class="gmail_msg" target="_blank">llvm-dev@lists.llvm.org</a><br class="gmail_msg">
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br class="gmail_msg">
</blockquote></div>
</div></blockquote></div><br class="gmail_msg"></div></div></blockquote></div>
_______________________________________________<br class="">LLVM Developers mailing list<br class=""><a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a><br class="">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev<br class=""></div></blockquote></div><br class=""></body></html>