[PATCH] D43314: [lit] - Allow 1 test to report multiple micro-test results to provide support for microbenchmarks.

[Renato Golin via Phabricator via llvm-commits]

rengolin added a comment.

Thanks @MatzeB, apart from the missing pipe in the RUN line, I'm happy when you are.

cheers,
--renato



================
Comment at: utils/lit/tests/test-data-micro.py:14
+# CHECK-NEXT: ***
+# CHECK-NEXT: *** MICRO-TEST: test{{[0-2]}}
+# CHECK-NEXT: micro_value0: 4
----------------
MatzeB wrote:
> rengolin wrote:
> > Both micro-test and micro-results are being stored in dictionaries, that means their order is not guaranteed.
> > 
> > I assume that's the reason for the `{{[0-2]}}` regex here, but a similar solution would be needed for the values as well.
> > 
> > I recommend you look at CHECK-DAG:
> > 
> >     # CHECK-NEXT: *** MICRO-TEST: test{{[0-2]}}
> >     # CHECK-DAG: micro_value0: 4
> >     ​# CHECK-DAG: micro_value1: 1.3
> > 
> > This means there are two lines after `MICRO-TEST` which have those two values, in any order.
> > 
> > It could match to a list after the next `MICRO-TEST`, however, if you have three `CHECK-NEXT:` of `MICRO-TEST`, then the last one would fail if you match the `CHECK-DAG` on the wrong line, so you're covered.
> > 
> > Just repeat that pattern three times and it should be fine.
> Actually I would recommend changing the printing code above to something like:
> ```
> sorted_results = sorted(test.result.microResults.items())
> for key, micro_test in sorted_results:
> ```
> to avoid the indeterministic output.
Indeed, better.


================
Comment at: utils/lit/tests/test-output-micro.py:1
+# RUN: %{lit} -j 1 -v %{inputs}/test-data-micro --output %t.results.out > %t.out
+# RUN: FileCheck < %t.results.out %s
----------------
rengolin wrote:
> Same thing here, join the lines with a pipe.
Forgot the pipe here. :)


================
Comment at: utils/lit/tests/test-output-micro.py:9
+# CHECK-NEXT:   {
+# CHECK-NEXT:     "code": "PASS",
+# CHECK-NEXT:     "elapsed": null,
----------------
MatzeB wrote:
> rengolin wrote:
> > Here, the `CHECK-DAG` trick won't work, because you have two levels: one for the result and another for the metrics, and there's no way to identify `CHECK-DAG1` and `CHECK-DAG2`.
> > 
> > One way to solve this is to sort the keys before dumping JSON, as to get a reproducible answer (that is, of course, if Python's default sort is stable).
> lit does the json dumping with `sort_keys=True` so the output should be deterministic so we don't need the `CHECK-DAG`s.
Ah, perfect!


https://reviews.llvm.org/D43314