[Mlir-commits] [mlir] [mlir][test] Make SME e2e tests require an emulator (PR #86489)

[Andrzej Warzyński]

banach-space wrote:

Thanks for the feedback!

 > > That seems like a valid use-case, @banach-space how do you see this supported? Also: what about someone who has the actual HW in the future?
> 
> You've touched on a further issue, if we had done this for SVE we wouldn't have been able to run the integration tests on hardware without removing it. However, we don't all have access to hardware at the same time, at what point do we decide this can be removed?

There's a couple of points here.

### 1. binfmt_misc on Linux

I think that in terms of these integration tests, "less is more" and we should try to avoid supporting too many scenarios. When the set of scenarios is limited and well defined, reproducing failures and helping folks fix their issues over "e-mail" is much easier. 

I also feel that we are making our job easier for ourselves if we clearly define what interfaces we'd like our users to use (e.g. `ARM_EMULATOR_EXECUTABLE`). Whatever that interface is, it should be documented. If we don't want to require `ARM_EMULATOR_EXECUTABLE`, then alternatives like `binfmt_misc` should be documented.

### 2. Hardware vs no-hardware

It's not clear to me how to best support such cases. SVE is a great example  - some folks have access to hardware, others don't (i.e. some people require an emulator, others don't). But this isn't really SVE specific, right? It's more about the overall infra that we have in MLIR.

Today, there's no SME hardware, so the approach proposed here is valid. Once SME hardware is available, that won't be the case and that's when I'd revert this. Something longer term would be much better, I agree.

### LIT constraints

> Lit has constraints [1] which are the typically used for situations like this. 

These constraints are "compile-time" rather than "run-time" constraints. For example, things like available targets (for stuff `// REQUIRES: target=aarch64 `) are effectively taken from the list of LLVM targets that you decide to build:
https://github.com/llvm/llvm-project/blob/c8d70e94c4b69e809142054e75b9725ed70418af/llvm/test/lit.site.cfg.py.in#L6
and:
https://github.com/llvm/llvm-project/blob/c8d70e94c4b69e809142054e75b9725ed70418af/llvm/utils/lit/lit/llvm/config.py#L112

I don't see how we could use these here **today** (see below for an idea).

### Alternatives

#### Option 1
Ultimately, on unsupported hardware, it's `mlir-cpu-runner` (instructed by the user) that's attempting to run code requiring feature "X" on hardware that has no such feature. So perhaps `mlir-cpu-runner` should query the host? But then I'm not sure how to "mix" that with potential emulation.

#### Option 2
One other option would involve updating CMake. There's [LLVM_TARGET_ARCH](https://llvm.org/docs/CMake.html) and we could add `LLVM_TARGET_ARCH_FEATURES`. That could be wired up with the LIT logic, e.g.:
```python
// REQUIRES: (target=aarch64 AND target-arch-feature=sme) OR arm-emulator-executable
```

I guess that this could scale nicely to other targets?

Thoughts?

https://github.com/llvm/llvm-project/pull/86489