[PATCH] D43314: [lit] - Allow 1 test to report multiple micro-test results to provide support for microbenchmarks.

[Renato Golin via Phabricator via llvm-commits]

rengolin added inline comments.


================
Comment at: utils/lit/tests/test-data-micro.py:4
+
+# RUN: %{lit} -j 1 -v %{inputs}/test-data-micro > %t.out
+# RUN: FileCheck < %t.out %s
----------------
Join these two lines with a pipe, otherwise .out files will be kept and can break further tests.


================
Comment at: utils/lit/tests/test-data-micro.py:14
+# CHECK-NEXT: ***
+# CHECK-NEXT: *** MICRO-TEST: test{{[0-2]}}
+# CHECK-NEXT: micro_value0: 4
----------------
Both micro-test and micro-results are being stored in dictionaries, that means their order is not guaranteed.

I assume that's the reason for the `{{[0-2]}}` regex here, but a similar solution would be needed for the values as well.

I recommend you look at CHECK-DAG:

    # CHECK-NEXT: *** MICRO-TEST: test{{[0-2]}}
    # CHECK-DAG: micro_value0: 4
    ​# CHECK-DAG: micro_value1: 1.3

This means there are two lines after `MICRO-TEST` which have those two values, in any order.

It could match to a list after the next `MICRO-TEST`, however, if you have three `CHECK-NEXT:` of `MICRO-TEST`, then the last one would fail if you match the `CHECK-DAG` on the wrong line, so you're covered.

Just repeat that pattern three times and it should be fine.


================
Comment at: utils/lit/tests/test-output-micro.py:1
+# RUN: %{lit} -j 1 -v %{inputs}/test-data-micro --output %t.results.out > %t.out
+# RUN: FileCheck < %t.results.out %s
----------------
Same thing here, join the lines with a pipe.


================
Comment at: utils/lit/tests/test-output-micro.py:9
+# CHECK-NEXT:   {
+# CHECK-NEXT:     "code": "PASS",
+# CHECK-NEXT:     "elapsed": null,
----------------
Here, the `CHECK-DAG` trick won't work, because you have two levels: one for the result and another for the metrics, and there's no way to identify `CHECK-DAG1` and `CHECK-DAG2`.

One way to solve this is to sort the keys before dumping JSON, as to get a reproducible answer (that is, of course, if Python's default sort is stable).


https://reviews.llvm.org/D43314