[llvm-dev] Proposal: introduce dependency on abseil when building benchmarks

[Renato Golin via llvm-dev]

On Thu, 7 Oct 2021 at 17:46, Mircea Trofin <mtrofin at google.com> wrote:

> Making sure we're talking about the same thing: this is strictly about
> benchmarks that require flipping LLVM_BUILD_BENCHMARKS
> or LIBCXX_INCLUDE_BENCHMARKS; and it's about exploring how those specific
> benchmarks relate to the 'benchmark' project which we currently have as a
> source copy in the llvm tree (i.e. we're sometimes manually cherry-picking
> changes over, but we're not updating it through any automated means)
>

Honestly, I didn't understand what this means... :)

But I mostly run the benchmarks via CMake from a recently built LLVM copy,
and I don't do that "professionally", so I'm not the right person to such
things.

I looked under llvm-zorg, and can't find any bot configuration that flips
> those flags in the first place, so perhaps these specific benchmarks are
> run elsewhere (or my search was naive - I just grep-ed for '_BENCHMARKS' -
> and only got one entry turning off TEST_SUITE_RUN_BENCHMARKS, which appears
> to be unrelated)?
>

AFAIK, most of the benchmarks using our test-suite are not done via
buildbots. I've set up one in the long past but even then it didn't seem to
be the norm.

Back then, I remember every single major target was running the test-suite
in benchmark mode in an internal infrastructure, because they were also
running private benchmarks and had their own local reporting system that
allowed them to track regressions.

Looking for bot-owners to add to the thread only gets you as far as knowing
who they think should be copied, or if you're luck, they're the same
people. :)

>From the OS x CPU matrix above, it seems the main gap is in the CPU column
> - since abseil supports Linux, MacOS, and Windows, but doesn't appear to
> specifically support mips, powerpc, systemz, or riscv. So if anyone runs
> these kinds of benchmarks (i.e. the ones that require flipping those cmake
> flags) on those CPUs, *and* we wanted, in llvm, to periodically auto-update
> the source copy of 'benchmark' project, only then we would have a breaking
> change.
>

There's also the fact that abseil only supports "Linux with Arm64" and
"ChromeOS on Armv7", so it's not really a matrix, just a list of
combinations.

- I think the main one is: was there ever any intent to automatically
> update the 2 copies of 'benchmarks' in the llvm tree? what about the one in
> llvm-test-suite?
>

Right, and only the people who actually run benchmarks can answer that.

- the second is getting a better understanding of where folks build and run
> those specific benchmarks (it'd help the upstream 'benchmark' project with
> data points).
>

Yes, unfortunately, because many of those are internal, people don't
communicate what they do too well. And if some of the additional benchmarks
are secret, they may not even be able to communicate.

Renato, would you mind detailing your scenario (i.e. which of those 2 types
> of benchmarks, on which OSxCPU)
>

I'm afraid I cannot, as I haven't done any benchmarking for a long time.
Hopefully Tom may be able to help you more than I can.

cheers,
--renato

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20211007/e75a8a37/attachment.html>