[llvm-dev] Benchmarks for LLVM-generated Binaries

Wed Sep 7 10:40:34 PDT 2016

That's correct. It seems like a fairly well contained local patch for your
local testsuite, otherwise you could probably change test-suite to try to
understand arbitrary directories.

On Wed, Sep 7, 2016 at 10:31 AM Michael Zolotukhin <mzolotukhin at apple.com>
wrote:

> Hi Eric,
>
> Yeah, I know about Externals and SPEC specifically. But as far as I
> understand, you have to have kind of description of the tests in test-suite
> even if you don’t provide the source codes - that’s what I would like to
> avoid. I.e. you have to have CMakeLists.txt and other files in place all
> the time, open to everyone.
>
> Now, imagine I have a small testsuite, which probably is not very
> interesting to anyone else, but extremely interesting for me. AFAIU, to
> make it a part of LNT now, I have to modify ‘test-suite’ repo so that it’s
> aware of this new test-suite. I will probably not be able to upstream these
> changes (as they are only interesting to me), and that’s the source of some
> inconvenience. I’d be happy to be mistaken here, but that how I understand
> the current infrastructure, and that where my question came from.
>
> Thanks,
> Michael
>
> On Sep 7, 2016, at 10:13 AM, Eric Christopher <echristo at gmail.com> wrote:
>
> Hi Michael,
>
> You'll want to look into the externals part of the test suite :) It's how
> things like SPEC etc are run.
>
> -eric
>
> On Tue, Sep 6, 2016 at 4:08 PM Michael Zolotukhin via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>
> > On Sep 1, 2016, at 8:14 AM, Renato Golin via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >
> > On 1 September 2016 at 07:45, Dean Michael Berris via llvm-dev
> > <llvm-dev at lists.llvm.org> wrote:
> >> I've lately been wondering where benchmarks for LLVM-generated binaries
> are hosted, and whether they're tracked over time.
> >
> > Hi Dean,
> >
> > Do you mean Perf?
> >
> > http://llvm.org/perf/
> >
> > Example, ARM and AArch64 tracking performance at:
> >
> > http://llvm.org/perf/db_default/v4/nts/machine/41
> > http://llvm.org/perf/db_default/v4/nts/machine/46
> >
> >
> >> - Is the test-suite repository the right place to put these
> generated-code benchmarks?
> >
> > I believe there would be the best place, yes.
> >
> >
> >> - Are there any objections to using a later version of the Google
> benchmarking library [0] in the test-suite?
> >
> > While this looks like a very nice tool set, I wonder how we're going
> > to integrate it.
> >
> > Checking it out in the test-suite wouldn't be the best option (version
> > rot), but neither would be requiring people to install it before
> > running the test-suite, especially if the installation process isn't
> > as easy as "apt-get install", like all the other dependencies.
> >
> >
> >> - Are the docs on the Testing Infrastructure Guide still relevant and
> up-to-date, and is that a good starting point for exploration here?
> >
> > Unfortunately, that's mostly for the "make check" tests, not for the
> > test-suite. The test-suite execution is covered by LNT's doc
> > (http://llvm.org/docs/lnt), but it's mostly about LNT internals and
> > not the test-suite itself.
> >
> > However, it's not that hard to understand the test-suite structure. To
> > add new tests, you just need to find a suitable place { (SingleSource
> > / MultiSource) / Benchmarks / YourBench } and copy ( CMakeFiles.txt,
> > Makefile, lit.local.cfg ), change to your needs, and it should be
> > done.
> Hi Renato and others,
>
> Is it possible/how hard will it be to make the testsuite kind of
> extendable? Like, I copy a folder with some tests that comply to some rules
> (e.g. have CMakeLists.txt, lit.local.cfg, etc.) and run them via the
> standard infrastructure without changing anything in test-suite files?
>
> The motivation of this question is the following: we (and probably many
> other companies too) have internal tests that we can’t share, but still
> want to track. Currently, the process of adding them to the existing
> test-suite is not clear to me (or at least not very well documented), and
> while I can figure it out, it would be cool if we can streamline this
> process.
>
> Ideally, I’d like to see this process like this:
> 1) Add following files to your benchmark suite:
>   1.a) CMakeLists.txt having this and that target doing this and that.
>   1.b) lit.local.cfg script having this and that.
>   …
> 2) Make sure the test report results in the following format/provide a
> wrapper script to convert results to the specified form. /* TODO: Results
> format is specified here */
> 3) Run your tests using the standard LNT command, like  "lnt runtest …
> --only-test=External/MyTestSuite/TestA”
>
> If that’s already implemented, then I’ll be glad to help with
> documentation, and if not, I can try implementing it. What do you think?
>
> Thanks,
> Michael
>
> >
> > cheers,
> > --renato
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160907/d5ebadac/attachment-0001.html>