[PATCH] D40077: [RFC][LNT][test-suite] - Allow 1 test to report multiple individual test results

[Brian Homerding via Phabricator via llvm-commits]

homerdin created this revision.
Herald added a subscriber: mgorny.

Hello,

I know that there has been some discussion on this subject.  I'm not sure if this approach is the way to go, but hopefully it will aide the discussion.

I included some minor changes to the litsupport module and the XRay benchmark to provide a use case.  The changes in LNT break down to:

1. Extract the processing of 1 individual test into a function.
2. Add check for `'microbenchmarks'` field and make a recursive call.

With these changes we get the following outputs:

lit json output
---------------

  "tests": [
    {
      "code": "PASS",
      "elapsed": 42.87996697425842,
      "metrics": {
        "compile_time": 0.6165,
        "hash": "5863047f3ce3ad782b7f03edf21e0359",
        "link_time": 0.1186,
        "microbenchmarks": {
          "BM_RDTSCP_Cost": {
            "metrics": {
              "exec_time": 26.9471,
              "std_dev": 0.00923222
            }
          },
          "BM_ReturnInstrumentedPatched": {
            "metrics": {
              "exec_time": 18.6394,
              "std_dev": 0.0191427
            }
          },
          "BM_ReturnInstrumentedPatchedThenUnpatched": {
            "metrics": {
              "exec_time": 9.42276,
              "std_dev": 0.000323646
            }
          },
          "BM_ReturnInstrumentedPatchedWithLogHandler": {
            "metrics": {
              "exec_time": 54.954,
              "std_dev": 0.00254871
            }
          },
          "BM_ReturnInstrumentedUnPatched": {
            "metrics": {
              "exec_time": 9.46574,
              "std_dev": 0.00167303
            }
          },
          "BM_ReturnNeverInstrumented": {
            "metrics": {
              "exec_time": 9.58165,
              "std_dev": 0.0144976
            }
          }
        }
      },
      "name": "test-suite :: MicroBenchmarks/XRay/retref-bench.test",
      "output": "\n/home/bhomerding/testbox/SANDBOX/test-2017-11-15_14-40-39/MicroBenchmarks/XRay/retref-bench --benchmark_repetitions=10 --benchmark_format=csv --benchmark_report_aggregates_only=true > /home/bhomerding/testbox/SANDBOX/test-2017-11-15_14-40-39/MicroBenchmarks/XRay/Output/retref-bench.test.bench.csv"
    }
  ]



LNT report.json
---------------

  "Tests": [
      {
          "Data": [
              0.6165
          ],
          "Info": {},
          "Name": "nts.MicroBenchmarks/XRay/retref-bench.compile"
      },
      {
          "Data": [
              "5863047f3ce3ad782b7f03edf21e0359"
          ],
          "Info": {},
          "Name": "nts.MicroBenchmarks/XRay/retref-bench.hash"
      },
      {
          "Data": [
              0.1186
          ],
          "Info": {},
          "Name": "nts.MicroBenchmarks/XRay/retref-bench-link.compile"
      },
      {
          "Data": [
              9.46574
          ],
          "Info": {},
          "Name": "nts.MicroBenchmarks/XRay/retref-bench/BM_ReturnInstrumentedUnPatched.exec"
      },
      {
          "Data": [
              9.42276
          ],
          "Info": {},
          "Name": "nts.MicroBenchmarks/XRay/retref-bench/BM_ReturnInstrumentedPatchedThenUnpatched.exec"
      },
      {
          "Data": [
              54.954
          ],
          "Info": {},
          "Name": "nts.MicroBenchmarks/XRay/retref-bench/BM_ReturnInstrumentedPatchedWithLogHandler.exec"
      },
      {
          "Data": [
              26.9471
          ],
          "Info": {},
          "Name": "nts.MicroBenchmarks/XRay/retref-bench/BM_RDTSCP_Cost.exec"
      },
      {
          "Data": [
              9.58165
          ],
          "Info": {},
          "Name": "nts.MicroBenchmarks/XRay/retref-bench/BM_ReturnNeverInstrumented.exec"
      },
      {
          "Data": [
              18.6394
          ],
          "Info": {},
          "Name": "nts.MicroBenchmarks/XRay/retref-bench/BM_ReturnInstrumentedPatched.exec"
      }
  ]


https://reviews.llvm.org/D40077

Files:
  MicroBenchmarks/XRay/CMakeLists.txt
  litsupport/modules/microbenchmark.py
  lnt/tests/test_suite.py

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D40077.123022.patch
Type: text/x-patch
Size: 10295 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20171115/b6142db6/attachment.bin>