[PATCH] [libcxx] Checking LIT benchmark testing framework.

[Eric Fiselier]

I would love to put this in-tree. I originally thought there would be
some licensing issues but if gtest lives in llvm/utils/unittests then
this probably can too.
That is the direction I will head with this.

It's also not a bad idea to add ways for CMake to build the benchmark
tests. Typically that seems like the preferred method to build the
tests and I'm fully committed to adding this functionality.

However libc++ should probably still use LIT to build the tests for a
couple of reasons:
1. If a test is broken or unsupported it shouldn't break the build.
This frequently occurs when a test targets newer standard versions (ex
c++1z).
2. LIT is the only way we can properly build tests in libc++. It would
take significantly more code to teach CMake how to properly build
tests.
3. I don't want to have to re-configure CMake every time I want to run
the benchmarks against a different standard library, optimization
level, or a new change.

It should be reasonable to add functionality to LLVM LIT to run the
tests and parse the output in such a way that everybody can use it.

I'll start working on a solution to this.

/Eric

On Fri, Mar 6, 2015 at 3:00 PM, Chandler Carruth <chandlerc at gmail.com> wrote:
>
> On Fri, Mar 6, 2015 at 11:47 AM, Dmitri Gribenko <gribozavr at gmail.com>
> wrote:
>>
>> On Fri, Mar 6, 2015 at 11:29 AM, Chandler Carruth <chandlerc at gmail.com>
>> wrote:
>> >
>> > On Fri, Mar 6, 2015 at 10:21 AM, hfinkel at anl.gov <hfinkel at anl.gov>
>> > wrote:
>> >>
>> >> Almost none of my benchmarking machines can initiate outgoing
>> >> connections,
>> >> so I'd need to download/checkout the Google 'benchmark' library
>> >> manually.
>> >> Can you please add some comments about exactly what should be
>> >> retrieved, and
>> >> where it should be put.
>> >
>> >
>> > The entire thing seems odd.
>> >
>> > I would much rather just have the benchmark code live in some utility
>> > place
>> > in one of the repositories. For example, we keep gtest in utils/unittest
>> > in
>> > the LLVM repository. I wonder if we could check the benchmark stuff into
>> > utils/benchmark, and add it to CMake normally.
>> >
>> > Then the benchmarks can be built normally with CMake (or Makefiles) and
>> > LIT
>> > can just be responsible for actually running them.
>> >
>> > This would make the benchmarks unavailable from a stand-alone build of
>> > libc++ but that seems a not-terrible tradeoff.
>>
>> +1 from me.  It would be great to have a benchmarking framework
>> in-tree so that other projects could use it.  For example, we could
>> have latency tests for Clang code completion or re-parsing with
>> implicit PCH.
>
>
> Eric reminded me that I may have led to this design. Sorry for that if so...
>
> I think the concern I had was that we don't currently have any unusually
> licensed stuff in the libc++ tree, and adding it might be problematic.
> Keeping the license of libc++ super clear and explicit is really important
> because it is a runtime library and can be used in even more diverse
> contexts than the rest of LLVM.
>
> I had originally thought that we might not want to re-use an existing
> benchmarking framework, but I can also see lots of good reasons to re-use an
> existing framework.
>
> My hope with the suggestion of keeping the benchmarking framework in the
> core LLVM repository is that it would make it much more clear that none of
> the functionality in libc++ would have anything to do with it, only
> benchmarks. The benchmarks themselves of course could always live in the
> libc++ repository and be built with an explicitly specified copy of the
> benchmark framework if libc++ were being used outside of an LLVM checkout.
>
>
> Hopefully that clarifies some of why it might be useful to have *some*
> levels of separation between the project and the benchmarking stuff...
>
> _______________________________________________
> cfe-commits mailing list
> cfe-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-commits
>