[llvm-dev] Proposal: introduce dependency on abseil when building benchmarks

[Mircea Trofin via llvm-dev]

On Thu, Oct 7, 2021 at 10:00 AM Renato Golin <rengolin at gmail.com> wrote:

> On Thu, 7 Oct 2021 at 17:46, Mircea Trofin <mtrofin at google.com> wrote:
>
>> Making sure we're talking about the same thing: this is strictly about
>> benchmarks that require flipping LLVM_BUILD_BENCHMARKS
>> or LIBCXX_INCLUDE_BENCHMARKS; and it's about exploring how those specific
>> benchmarks relate to the 'benchmark' project which we currently have as a
>> source copy in the llvm tree (i.e. we're sometimes manually cherry-picking
>> changes over, but we're not updating it through any automated means)
>>
>
> Honestly, I didn't understand what this means... :)
>
> But I mostly run the benchmarks via CMake from a recently built LLVM copy,
> and I don't do that "professionally", so I'm not the right person to such
> things.
>
Oh, OK - so what's your cmake  invocation like - i.e. if you don't flip
those 2 flags, then you probably don't build those benchmarks.

Another way to put it: do you build anything under:

llvm-test-suite/MicroBenchmarks/libs/benchmark/
llvm/utils/benchmark
libcxx/utils/google-benchmark



> I looked under llvm-zorg, and can't find any bot configuration that flips
>> those flags in the first place, so perhaps these specific benchmarks are
>> run elsewhere (or my search was naive - I just grep-ed for '_BENCHMARKS' -
>> and only got one entry turning off TEST_SUITE_RUN_BENCHMARKS, which appears
>> to be unrelated)?
>>
>
> AFAIK, most of the benchmarks using our test-suite are not done via
> buildbots. I've set up one in the long past but even then it didn't seem to
> be the norm.
>
> Back then, I remember every single major target was running the test-suite
> in benchmark mode in an internal infrastructure, because they were also
> running private benchmarks and had their own local reporting system that
> allowed them to track regressions.
>
> Looking for bot-owners to add to the thread only gets you as far as
> knowing who they think should be copied, or if you're luck, they're the
> same people. :)
>
> From the OS x CPU matrix above, it seems the main gap is in the CPU column
>> - since abseil supports Linux, MacOS, and Windows, but doesn't appear to
>> specifically support mips, powerpc, systemz, or riscv. So if anyone runs
>> these kinds of benchmarks (i.e. the ones that require flipping those cmake
>> flags) on those CPUs, *and* we wanted, in llvm, to periodically auto-update
>> the source copy of 'benchmark' project, only then we would have a breaking
>> change.
>>
>
> There's also the fact that abseil only supports "Linux with Arm64" and
> "ChromeOS on Armv7", so it's not really a matrix, just a list of
> combinations.
>
> - I think the main one is: was there ever any intent to automatically
>> update the 2 copies of 'benchmarks' in the llvm tree? what about the one in
>> llvm-test-suite?
>>
>
> Right, and only the people who actually run benchmarks can answer that.
>

> - the second is getting a better understanding of where folks build and
>> run those specific benchmarks (it'd help the upstream 'benchmark' project
>> with data points).
>>
>
> Yes, unfortunately, because many of those are internal, people don't
> communicate what they do too well. And if some of the additional benchmarks
> are secret, they may not even be able to communicate.
>
> Renato, would you mind detailing your scenario (i.e. which of those 2
>> types of benchmarks, on which OSxCPU)
>>
>
> I'm afraid I cannot, as I haven't done any benchmarking for a long time.
> Hopefully Tom may be able to help you more than I can.
>
> cheers,
> --renato
>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20211007/7c373b02/attachment.html>