[LLVMdev] YA Vectorization Benchmark

[Daniel Dunbar]

On Wed, Nov 7, 2012 at 12:39 AM, Renato Golin <rengolin at systemcall.org>wrote:

> On 6 November 2012 22:28, Daniel Dunbar <daniel at zuster.org> wrote:
> > Is it possible instead to refactor the tests so that each binary
> corresponds
> > to one test? For example, look at how Hal went about integrating TSVC:
>
> It should be possible. I'll have to understand better what the
> preamble does to make sure I'm not stripping out important stuff, but
> also what to copy to each kernel's initialization.
>
> Also, I don't know how the timing functions perform across platforms.
> I'd have to implement a decent enough timing system, platform
> independent, to factor out the initialization step.
>

The way we handle timing of all the other tests is just by timing the
executable. This isn't perfect, but its what we use everywhere else so we
should stick with keeping it outside the tests.


>
>
> > Other things that I would *like* before integrating it:
> >  - Rip out the CPU ID stuff, this isn't useful and adds messiness.
>
> Absolutely, that is meaningless.
>
>
> >  - Have the test just produce output that can be compared, instead of
> > including its own check routines
>
> I can make it print numbers in order, is that good enough for the
> comparison routines?
>

Yup.


> If I got it right, the tests self-validates the results, so at least
> we know it executed correctly in the end. I can make it produce "OK"
> and "FAIL" either way with some numbers stating the timing.
>

Yeah, we can get rid of that, the way we check all the other tests is by
comparing to reference outputs or output from a reference binary (compiled
by the system compiler).


> >  - Have the tests run for fixed iterations, instead of doing their own
> > adaptive run
>
> Yes, that's rubbish. That was needed to compare the results based on
> CPU specific features, but we don't need that.
>
>
> >  - Produce reference output files, so it works with
> USE_REFERENCE_OUTPUT=1
>
> Is this a simple diff or do you compare the numerical results by
> value+-stdev?
>

We have support for value + tolerance. You can set FP_TOLERANCE=.0001 or so
in the Makefile to tune the limit. For example
see SingleSource/Benchmarks/Misc/Makefile.

 - Daniel

cheers,
> --renato
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20121107/7ee1705a/attachment.html>