[PATCH] D43316: [test-suite] Update litsupport/module/microbenchmark.py to report individual timing results from 1 test.

[Brian Homerding via Phabricator via llvm-commits]

homerdin created this revision.
homerdin added reviewers: MatzeB, dberlin, hfinkel, rengolin.
Herald added a subscriber: mgorny.

These changes are dependent on the changes purposed in https://reviews.llvm.org/D43314.

With the purposed changes to lit we can report the individual timing results for each microbenchmark with one test.  This updates the litsupport plugin to make further use of the benchmark library.

I have also changed the current microbenchmarks that use this plugin by removing the individual tests from the XRay microbenchmarks as they should no longer be needed.   Please let me know if this causes any problems with the XRay tests uses @eizan .

With the purposed changes example output is:

Terminal:

  PASS: test-suite :: MicroBenchmarks/XRay/ReturnReference/retref-bench.test (1 of 1)
  ********** TEST 'test-suite :: MicroBenchmarks/XRay/ReturnReference/retref-bench.test' RESULTS **********
  MicroBenchmarks: 6 
  compile_time: 0.6487 
  hash: "daec0d37414da26fdae56f0d9bfa2cc0" 
  link_time: 0.0637 
  **********
  *** MICRO-TEST 'BM_RDTSCP_Cost RESULTS ***
      exec_time:  12.7087 
  *** MICRO-TEST 'BM_ReturnInstrumentedPatched RESULTS ***
      exec_time:  9.0223 
  *** MICRO-TEST 'BM_ReturnInstrumentedPatchedThenUnpatched RESULTS ***
      exec_time:  2.6744 
  *** MICRO-TEST 'BM_ReturnInstrumentedPatchedWithLogHandler RESULTS ***
      exec_time:  36.5216 
  *** MICRO-TEST 'BM_ReturnInstrumentedUnPatched RESULTS ***
      exec_time:  2.6745 
  *** MICRO-TEST 'BM_ReturnNeverInstrumented RESULTS ***
      exec_time:  2.0074 
  **********

JSON output:

  "tests": [
    {
      "code": "PASS",
      "elapsed": null,
      "metrics": {
        "exec_time": 2.67445
      },
      "name": "test-suite :: MicroBenchmarks/XRay/ReturnReference/retref-bench/BM_ReturnInstrumentedUnPatched.test",
      "output": ""
    },
    {
      "code": "PASS",
      "elapsed": null,
      "metrics": {
        "exec_time": 2.67435
      },
      "name": "test-suite :: MicroBenchmarks/XRay/ReturnReference/retref-bench/BM_ReturnInstrumentedPatchedThenUnpatched.test",
      "output": ""
    },
    {
      "code": "PASS",
      "elapsed": null,
      "metrics": {
        "exec_time": 36.5216
      },
      "name": "test-suite :: MicroBenchmarks/XRay/ReturnReference/retref-bench/BM_ReturnInstrumentedPatchedWithLogHandler.test",
      "output": ""
    },
    {
      "code": "PASS",
      "elapsed": null,
      "metrics": {
        "exec_time": 12.7087
      },
      "name": "test-suite :: MicroBenchmarks/XRay/ReturnReference/retref-bench/BM_RDTSCP_Cost.test",
      "output": ""
    },
    {
      "code": "PASS",
      "elapsed": null,
      "metrics": {
        "exec_time": 2.00737
      },
      "name": "test-suite :: MicroBenchmarks/XRay/ReturnReference/retref-bench/BM_ReturnNeverInstrumented.test",
      "output": ""
    },
    {
      "code": "PASS",
      "elapsed": null,
      "metrics": {
        "exec_time": 9.02227
      },
      "name": "test-suite :: MicroBenchmarks/XRay/ReturnReference/retref-bench/BM_ReturnInstrumentedPatched.test",
      "output": ""
    },
    {
      "code": "PASS",
      "elapsed": 5.6960108280181885,
      "metrics": {
        "MicroBenchmarks": 6,
        "compile_time": 0.6487,
        "hash": "daec0d37414da26fdae56f0d9bfa2cc0",
        "link_time": 0.0637
      },
      "name": "test-suite :: MicroBenchmarks/XRay/ReturnReference/retref-bench.test",
      "output": "\nMicroBenchmarks/XRay/ReturnReference/retref-bench --benchmark_format=csv > MicroBenchmarks/XRay/ReturnReference/Output/retref-bench.test.bench.csv"
    }


https://reviews.llvm.org/D43316

Files:
  MicroBenchmarks/XRay/FDRMode/CMakeLists.txt
  MicroBenchmarks/XRay/FDRMode/fdrmode-bench_BM_XRayFDRMultiThreaded_16_thread.test
  MicroBenchmarks/XRay/FDRMode/fdrmode-bench_BM_XRayFDRMultiThreaded_1_thread.test
  MicroBenchmarks/XRay/FDRMode/fdrmode-bench_BM_XRayFDRMultiThreaded_2_thread.test
  MicroBenchmarks/XRay/FDRMode/fdrmode-bench_BM_XRayFDRMultiThreaded_32_thread.test
  MicroBenchmarks/XRay/FDRMode/fdrmode-bench_BM_XRayFDRMultiThreaded_4_thread.test
  MicroBenchmarks/XRay/FDRMode/fdrmode-bench_BM_XRayFDRMultiThreaded_8_thread.test
  MicroBenchmarks/XRay/ReturnReference/CMakeLists.txt
  MicroBenchmarks/XRay/ReturnReference/retref-bench_BM_RDTSCP_Cost.test
  MicroBenchmarks/XRay/ReturnReference/retref-bench_BM_ReturnInstrumentedPatched.test
  MicroBenchmarks/XRay/ReturnReference/retref-bench_BM_ReturnInstrumentedPatchedThenUnpatched.test
  MicroBenchmarks/XRay/ReturnReference/retref-bench_BM_ReturnInstrumentedPatchedWithLogHandler.test
  MicroBenchmarks/XRay/ReturnReference/retref-bench_BM_ReturnInstrumentedUnPatched.test
  MicroBenchmarks/XRay/ReturnReference/retref-bench_BM_ReturnNeverInstrumented.test
  litsupport/modules/microbenchmark.py
  litsupport/testplan.py

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D43316.134309.patch
Type: text/x-patch
Size: 9528 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20180214/0302535c/attachment.bin>