[lldb-dev] Benchmark tests

[Enrico Granata via lldb-dev]

I have actually added a few benchmark tests recently. We admittedly are not that good with running those tests ever (because they're not run by default most likely - and I do wonder if some of them would take a long time to run.. I don't think I have ever run the full set, just my own as I increment on performance work).

Maybe we could try flipping the default to be "run the benchmarks", see if test suite run times explode and take it from there in terms of feasibility as well as whether they all still make sense.

The other problem with the tests as they stand is that they mark themselves as PASS or FAIL purely on the basis of whether they encounter command or API errors, and do nothing to track performance regressions. That is admittedly a harder problem to tackle given heterogeneous hardware and workload - but maybe we could have them fail if the timings go wildly crazy over some threshold?

Sent from my iPhone

> On Dec 9, 2015, at 1:22 PM, Todd Fiala via lldb-dev <lldb-dev at lists.llvm.org> wrote:
> 
> Hey Jason,
> 
> Are you the benchmark user?
> 
> -Todd
> 
>> On Wed, Dec 9, 2015 at 12:32 PM, Zachary Turner via lldb-dev <lldb-dev at lists.llvm.org> wrote:
>> Is anyone using the benchmark tests?  None of the command line options related to the benchmark tests were claimed as being used by anyone.  Which makes me wonder if the tests are even being used by anyone.  
>> 
>> What I really want to know is: Is it really ok to delete the -x and -y command line options?  And what is the status of these tests?  Does anyone use them?
>> 
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> 
> 
> 
> -- 
> -Todd
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151209/b3a18915/attachment-0001.html>