<div class="gmail_quote"><div><br></div><div>How do other benchmarks deal with unstable algorithms or differences in floating point results?</div><div><br></div><div>>> haven't been following this thread, but this sounds like a typical</div>
<div>>> unstable algorithm problem. Are you always operating that close to<br>>> the tolerance level of the algorithm or are there some sets of inputs<br>>> that will behave reasonably?</div><div><br></div>
<div>What do you mean by "reasonably" or "affect codes so horribly"?</div><div><br></div><div>The accumulation of algorithms in a physics pipeline is unstable and unless the compiler/platform</div><div>
guarantees 100% identical floating point results, the outcome will diverge.</div><div><br></div><div><div>Do you think LLVM can be forced to produce identical floating point results?</div><div>Even when using different optimization levels or even different CPUs?</div>
<div><br></div></div><div>Some CPUs use 80bit FPU precision for intermediate results (on-chip in registers),</div><div>while variables in-memory only use 32-bit or 64bit precision. </div><div>In combination with cancellation and other re-ordering this can give slightly different results.</div>
<div><br></div><div>>> If not, the code doesn't seem very useful to me. How could anyone rely</div><div>>> on the results, ever?</div><div><br></div><div>The code has proven to be useful for games and special effects in film,</div>
<div>but this particular benchmark might not suite LLVM testing indeed.</div><div><br></div><div>I suggest working on a better benchmark that tests independent parts of the pipeline,</div><div>so we don't accumulate results (several frames) but we test a single algorithm at a time,</div>
<div>with known input and expected output. This avoid unstability and we can measure the error of the output.</div><div><br></div><div>Anton, are you interested in working together on such improved benchmark?</div><div>Thanks,</div>
<div>Erwin</div><div><br></div><div><font class="Apple-style-span" color="#790619" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse: collapse; white-space: nowrap; -webkit-border-horizontal-spacing: 2px; -webkit-border-vertical-spacing: 2px;"><b><br>
</b></span></font></div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Date: Mon, 4 Jan 2010 20:24:23 -0600<br>
From: David Greene <<a href="mailto:dag@cray.com">dag@cray.com</a>><br>
Subject: Re: [LLVMdev] Help adding the Bullet physics sdk benchmark to<br>
the LLVM test suite?<br>
To: <a href="mailto:llvmdev@cs.uiuc.edu">llvmdev@cs.uiuc.edu</a><br>
Message-ID: <<a href="mailto:201001042024.23451.dag@cray.com">201001042024.23451.dag@cray.com</a>><br>
Content-Type: text/plain; charset="iso-8859-15"<br>
<br>
On Monday 04 January 2010 20:11, Erwin Coumans wrote:<br>
> Hi Anton, and happy new year all,<br>
><br>
> >>One questions though: is it possible to "verify" the results of all<br>
> >>the computations somehow?<br>
><br>
> Good point, and there is no automated way currently, but we can work on<br>
> that.<br>
> Note that simulation suffers from the 'butterfly effect', so the smallest<br>
> change anywhere,<br>
> (cpu, compiler etc) diverges into totally different results after a while.<br>
<br>
I haven't been following this thread, but this sounds like a typical<br>
unstable algorithm problem. Are you always operating that close to<br>
the tolerance level of the algorithm or are there some sets of inputs<br>
that will behave reasonably?<br>
<br>
If not, the code doesn't seem very useful to me. How could anyone rely<br>
on the results, ever?<br>
<br>
In the worst case, you could experiment with different optimization levels<br>
and/or Pass combinations to find something that is reasonably stable.<br>
<br>
Perhaps LLVM needs a flag to disable sometimes undesireable transformations.<br>
Like anything involving floating-point calculations. Compiler changes should<br>
not affect codes so horribly unless the user tells them to. :) The Cray<br>
compiler provides various -Ofp (-Ofp0, -Ofp1, etc.) levels for this very<br>
reason.<br>
<br>
> There are a few ways of verification I can think of:<br>
><br>
> 1) verifying by adding unit tests for all stages in the physics pipeline<br>
> (broadphase acceleration structures, closest point computation, constraint<br>
> solver)<br>
> Given known input and output we can check if the solution is within a<br>
> certain tolerance.<br>
<br>
At each stage? That's reasonable. It could also help identify the parts of<br>
the pipeline that are unstable (if not already known).<br>
<br>
> 2) using the benchmark simulation and verifying the results frame by frame<br>
> and check for unusual behaviour<br>
<br>
Sounds expensive.<br>
<br>
> 3) modify the benchmark so that it is easier to test the end result, even<br>
> through it might be different.<br>
<br>
We really don't want to do this. Either LLVM needs to be fixed to respect<br>
floating-point evaluation in unstable cases or the benchmark and upstream code<br>
needs to be fixed to be more stable.<br>
<br>
-Dave<br></blockquote></div>