<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/129443>129443</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            [mlir] Testing infra improvements ideas
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
            mlir
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
            banach-space
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          banach-space
      </td>
    </tr>
</table>

<pre>
    Meta-ticket to collect ideas for improving the testing infrastructure in MLIR. 

At the moment, this is a collection of suggestions extracted from:
* https://discourse.llvm.org/t/rfc-should-we-aim-for-more-consistency-in-tests/, and
* https://github.com/llvm/mlir-www/pull/216.

The order of these items is random. I will try to add more data/notes over time.

_Please, if you decide to work on any of these, leave a comment so that we avoid duplicating efforts._

### IDEA 1

Separate examples from ODS definitions into a new field: `let examples = {}`. Build infra so that `ninja check-mlir` would extra those examples and sanity-check them with `mlir-opt` (to avoid the examples becoming out-dated). 

This could be used to demonstrate both "valid" and invalid usage. 

### IDEA 2

Generate, somehow, examples from the ODS `assemblyFormat` (even if we will likely want to update them manually).

### IDEA 3

Revisit the concept of round-trip tests in "ops.mlir" files. They shouldn't be strictly necessary for ops with `let assemblyFormat` MLIR thoroughly tests the printer/parser generator.

### IDEA 4

Improve "split" locations when encountering errors. Currently these are OK for humans to read - make them easier for tools to parse (and to jump to locations).

### IDEA 5

Add more metadata to “splits” so that one could do something like:

```
---- Test: Verify the folding behavior with single dynamic rank.
```

This would then be printed alongside the error/failure message. This should be integrated with the existing metadata like [here](https://github.com/llvm/llvm-project/blob/1aea0241f1cce9eb4eba3e4add3be9370e30e415/mlir/test/Dialect/Vector/vector-transfer-permutation-lowering.mlir#L8-L13).

### IDEA 6

Make [generate-test-checks.py](https://github.com/llvm/llvm-project/blob/main/mlir/utils/generate-test-checks.py) generate better names. For example:
```mlir
// Automatically generated - possible near-future version (mild improvement)
// CHECK-LABEL:   func.func @bitcast_2d(
// CHECK-SAME:                          %[[ARG_0]]: vector<2x4xi32>) -> vector<2x16xi8> {
// CHECK:           %[[BITCAST_1:.*]] = vector.bitcast %[[ARG_0]] : vector<2x4xi32> to vector<2x2xi64>
// CHECK:           %[[BITCAST_2.:*]] = vector.bitcast %[[BITCAST_1]] : vector<2x2xi64> to vector<2x16xi8>
// CHECK:           return %[[BITCAST_2]] : vector<2x16xi8>
func.func @bitcast_2d(%arg0: vector<2x4xi32>) -> vector<2x16xi8> {
  %0 = vector.bitcast %arg0 : vector<2x4xi32> to vector<2x2xi64>
  %1 = vector.bitcast %0 : vector<2x2xi64> to vector<2x16xi8>
  return %1 : vector<2x16xi8>
}
```

instead of:
```mlir
// Automatically generated - today’s version
// CHECK-LABEL:   func.func @bitcast_2d(
// CHECK-SAME:                          %[[VAL_0:[0-9]+|[a-zA-Z$._-][a-zA-Z0-9$._-]*]]: vector<2x4xi32>) -> vector<2x16xi8> {
// CHECK:           %[[VAL_1:.*]] = vector.bitcast %[[VAL_0]] : vector<2x4xi32> to vector<2x2xi64>
// CHECK:           %[[VAL_2:.*]] = vector.bitcast %[[VAL_1]] : vector<2x2xi64> to vector<2x16xi8>
// CHECK:           return %[[VAL_2]] : vector<2x16xi8>
func.func @bitcast_2d(%arg0: vector<2x4xi32>) -> vector<2x16xi8> {
  %0 = vector.bitcast %arg0 : vector<2x4xi32> to vector<2x2xi64>
  %1 = vector.bitcast %0 : vector<2x2xi64> to vector<2x16xi8>
  return %1 : vector<2x16xi8>
}
```
</pre>
<img width="1" height="1" alt="" src="http://email.email.llvm.org/o/eJzsWEFv2zoS_jX0ZSBBpiwnPvhgx_Hb4qV4izboYS8BJY4sNhQpkJQd769fDGk7aZv0bd6i77SBEVu0OPPNN_PN0BLeq51BXLJqzTivhRFNl_lBNMg4Z9VmIsbQWbd8-c2ktvK4_IhBZEE1jxggWGis1tgEUBKFh9Y6UP3g7F6ZHYQOIaAP9FmZ1gkf3NiE0SEoAx_vPnzKgRUrVqxWId7c2x5NYPwGQqc8KA_i7EBZA7YFP-52ZNEaD_gUnGgCSmid7VlJhhhfQRfC4OmSbxnfSuUbOzqPudb7Prdux_g2ML51bZP5zo5aZgfMhOqz1rqstw6zxhqvfEDTHDNlMorBR2s3IIx81c1OhW6s88b2jG_JE-PbXiuXHQ4HxrfDqDXjWz6d5yni-w7BOomOogodegQVsI9BO2Gk7XP4AAelNQR3JKaFlEDoQIogGN8aG9CD3aODoHo82X34p0bhkaCqFo52BImNkkgWDtY9gjUgzPHilW7UKPYYqe6Jf_AWQicCHBDE3ioJchy0akRMJLatdcHnD8kf42V6wYfN7QqmafUzDsKJgIBPoh80-pgi-GPzGSS2yqiUQWUoLjB4gFahlqxcAZsXGsPzRlZugF2t2dWGzYsc1qPSMlXTBSebF0aZrwKaDpvHjGhn8wIOlNtUJRA661-gEUaCF0aFYxb3EBc9HFToyFjMmx0CGWH8mjBGGqhELyZqbGxPhNgxZFIElIwvzvV8T-XbRP81wuhREv8Se2t8iMTUlnxxvhdaScZ5hKRMvITRix2ebX3HME-rv6FBMkT587bHzh7o47d8E2DinM0L4T32tT5urevFOTLco6EyOWCqNK0eUR_hIEzU9jhQXImbXphRaH2kIF_FVabVT7hXXiU5N9Y0OASqNWdHI7Pg1BA7AqWeoreDz2O6OIdWafQ53Hd4hKRLw_hVIAJ9cKoJ-ggGG_ReuGPsNHbwl5xRzfwYI_UYyr2z467Tx5NvwjY4ZQI6kqZwHh3sEqHWvR7eLK1-iM0NCbsftAoEXFuSBtXzoUMDaBo7ku2oFues8zncjM6hoRCS1oVD-OP3GEU39sJ44tuhkJBBLx5PpKPwCl28K1ir400RLiWPCiZY-Dr2A71fULyZoerUa899pMcgqJfQbnbL2XXBFjcxKn--3Fw0Zg2eClraWHCho_ioYk6dt1ixeXF6FassyzK4Rx9I01_QqTaGDq3VkjbW2Im9si4l0Cuz0wjyaESvGmqAj_l3Bs-qSqoOxHR9TqMEoa3Z-djnSKTEOuPbVig9xkh9UlS0kIqLdtPeHalIJhhJ4CpNrAs9FCOwat2hQ1ZtGL_-89ZPb9ng7FdsaNbU2taMb6cCRcFn03baNLjAeoa1KHEmpCxrXJRXBZYFzqbVaXbQoCIG-XajhE6WvmATYmz7-CELThjfossGdP0YYglk2h5i-Z20Vd5dZ3fT8s3CmKfVjyLFeVICxrmX-qPPh-P_FnovlHkOawxK00R9yxVfnPWIUGMI6MCIntrD1rpzkzvV3blGouniBAxWY7C9CKqhpnUxRvIarPeq1ggGhcvaMZ5H9ug8HTEYv-7jgEk6T8eRxbPZm3_c3vye3a3Wt3dU2ADtaJqc_gGbFbUKjfDhgUvGr7_f9Hn18TbteeOP8YqOY9V69em3h4Lorja0IWWalTf8afakSs7KWyIoY-Xty--m8yd1TWs0Lb_1_a3bi5_1h_ub1ef7hykrVznjq-QyTtxkNz8F9Co0eAMbtZMXy_xJzWcE-V2QeB5r7L-B9BzFq7DO_r-DdWbrp7AchtGZV9C97uqFzZ-UBa-E2xV_PbGRreINSsj0-xMTTU7fMPmDvT9n9CVz05-wRIe6H7q8Mj7QILTtXxR4sFIczxNs4c_a_ptF_GV190BZZtW6yBaxea7Z1Q2r1iL79yr7F-Oz_CGLhXRaodsui-fS_6UNgDC-Q_wppF8nfrLP3wfnbxB9QvV_wf8CwU_kspSLciEmuJxezYpFyctpOemWi7ks53N-hYuqrFs5v14IOV8UM2yxlULyiVrygldFWfDpfHpdljlWzVVdzSopUOJ0ytmswF4offnVP1Hej7ic8sVsVk60qFH70-OP0y8QVm0mbhmPL_W482xWaOWDf7YQVNDxkUncUG3i-fbycOPlmcGn5yGT0enlu09MESidjk5Y90v-nwAAAP__q0JhDg">