[libcxx-commits] [libcxx] [libc++][format] Adds print benchmarks. (PR #129765)

[Mark de Wever via libcxx-commits]

================
@@ -0,0 +1,79 @@
+//===----------------------------------------------------------------------===//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+// REQUIRES: std-at-least-c++23
+
+// This benchmark writes the print and benchmark data to stdout. In order to
+// preserve the benchmark output it needs to be stored in a file (using the
+// console format). Another issue with the benchmark is the time it takes to
+// write output to the real terminal. In order to avoid that overhead write the
+// output to a fast "terminal", like /dev/null. For example, the printf
+//   console    1546   ns
+//   /dev/null    70.9 ns
+// An example of a good test invocation.
+// BENCHMARK_OUT=benchmark.txt BENCHMARK_OUT_FORMAT=console <exe> >/dev/null
----------------
mordante wrote:

I think we could add a new lit keyword. The issue with this test is the same for other potential benchmarks that write to the standard output facilities. So something along the lines of 'BENCHMARK_USES_TERMINAL`. Then we can parse this in `format.py`. Typically we parse this in `parseScript`, here I think it makes sense to parse in `execute` and based on its presence use the current step or do
```
%dbg(EXECUTED AS) %{exec} %t.exe --benchmark_out=%T/benchmark-result.json --benchmark_out_format=json >/dev/null"
",
```
(Obviously `/dev/null` needs to be a portable solution.)

Changing the output to a `.txt` file feels wrong, then not all tests provide a `json` file.

Another solution is to use `/dev/null` unconditionally. Then add a lit parameter benchmark_out="csv|json|txt" to select the output format. By default you get json. If you want human readable you select txt. Since the suite supports csv, I think we should add it. The file will always get the same name + the appropriate extension.

IMO the cleanest solution is to always write the benchmark output to "`/dev/null`" and add the lit parameter to select the output format. WDYT?

https://github.com/llvm/llvm-project/pull/129765