[PATCH] D43314: [lit] - Allow 1 test to report multiple micro-test results to provide support for microbenchmarks.
Renato Golin via Phabricator via llvm-commits
llvm-commits at lists.llvm.org
Mon Feb 19 07:26:45 PST 2018
rengolin added inline comments.
================
Comment at: utils/lit/tests/test-data-micro.py:4
+
+# RUN: %{lit} -j 1 -v %{inputs}/test-data-micro > %t.out
+# RUN: FileCheck < %t.out %s
----------------
Join these two lines with a pipe, otherwise .out files will be kept and can break further tests.
================
Comment at: utils/lit/tests/test-data-micro.py:14
+# CHECK-NEXT: ***
+# CHECK-NEXT: *** MICRO-TEST: test{{[0-2]}}
+# CHECK-NEXT: micro_value0: 4
----------------
Both micro-test and micro-results are being stored in dictionaries, that means their order is not guaranteed.
I assume that's the reason for the `{{[0-2]}}` regex here, but a similar solution would be needed for the values as well.
I recommend you look at CHECK-DAG:
# CHECK-NEXT: *** MICRO-TEST: test{{[0-2]}}
# CHECK-DAG: micro_value0: 4
# CHECK-DAG: micro_value1: 1.3
This means there are two lines after `MICRO-TEST` which have those two values, in any order.
It could match to a list after the next `MICRO-TEST`, however, if you have three `CHECK-NEXT:` of `MICRO-TEST`, then the last one would fail if you match the `CHECK-DAG` on the wrong line, so you're covered.
Just repeat that pattern three times and it should be fine.
================
Comment at: utils/lit/tests/test-output-micro.py:1
+# RUN: %{lit} -j 1 -v %{inputs}/test-data-micro --output %t.results.out > %t.out
+# RUN: FileCheck < %t.results.out %s
----------------
Same thing here, join the lines with a pipe.
================
Comment at: utils/lit/tests/test-output-micro.py:9
+# CHECK-NEXT: {
+# CHECK-NEXT: "code": "PASS",
+# CHECK-NEXT: "elapsed": null,
----------------
Here, the `CHECK-DAG` trick won't work, because you have two levels: one for the result and another for the metrics, and there's no way to identify `CHECK-DAG1` and `CHECK-DAG2`.
One way to solve this is to sort the keys before dumping JSON, as to get a reproducible answer (that is, of course, if Python's default sort is stable).
https://reviews.llvm.org/D43314
More information about the llvm-commits
mailing list