[lldb-dev] Benchmark tests

[Zachary Turner via lldb-dev]

When you do run the benchmark tests, what command line options do you use?
At the moment I'm mostly just trying to remove dead options from the test
suite.  I removed one already that allowed you to specify the benchmark
executable, but then when I started looking at the rest and seeing how
tightly integrated they are with the benchamrk tests in general, I started
to wonder.

The three benchmark related command line options are:

1. An option to specify the benchmark executable (defaults to lldb.exe)
2. An option to specify the breakpoint spec (defaults to -n main)
3. An option to specify the breakpoint iteration count (defaults to 30 I
think)
4. An option to specify that you only want to run benchmark tests and no
other tests.

I deleted #4 because you can use the category system for that.  I deleted
#1 because nobody said they needed it on the spreadsheet.  Nobody said they
needed #2 or #3 either, but I just want to double check that deleting them
is fine.

On Wed, Dec 9, 2015 at 2:01 PM Enrico Granata <egranata at apple.com> wrote:

> I have actually added a few benchmark tests recently. We admittedly are
> not that good with running those tests ever (because they're not run by
> default most likely - and I do wonder if some of them would take a long
> time to run.. I don't think I have ever run the full set, just my own as I
> increment on performance work).
>
> Maybe we could try flipping the default to be "run the benchmarks", see if
> test suite run times explode and take it from there in terms of feasibility
> as well as whether they all still make sense.
>
> The other problem with the tests as they stand is that they mark
> themselves as PASS or FAIL purely on the basis of whether they encounter
> command or API errors, and do nothing to track performance regressions.
> That is admittedly a harder problem to tackle given heterogeneous hardware
> and workload - but maybe we could have them fail if the timings go wildly
> crazy over some threshold?
>
> Sent from my iPhone
>
> On Dec 9, 2015, at 1:22 PM, Todd Fiala via lldb-dev <
> lldb-dev at lists.llvm.org> wrote:
>
> Hey Jason,
>
> Are you the benchmark user?
>
> -Todd
>
> On Wed, Dec 9, 2015 at 12:32 PM, Zachary Turner via lldb-dev <
> lldb-dev at lists.llvm.org> wrote:
>
>> Is anyone using the benchmark tests?  None of the command line options
>> <https://docs.google.com/spreadsheets/d/1wkxAY7l0_cJOHhhsSlh3aKKlQShlX1D7X1Dn8kpqxy4/edit?pli=1#gid=0>
>> related to the benchmark tests were claimed as being used by anyone.  Which
>> makes me wonder if the tests are even being used by anyone.
>>
>> What I really want to know is: Is it really ok to delete the -x and -y
>> command line options?  And what is the status of these tests?  Does anyone
>> use them?
>>
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>>
>
>
> --
> -Todd
>
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151209/c1be8a70/attachment.html>