[clang] [Clang][OpenMP] Fix multi arch compilation for -march option (PR #92290)

[Joseph Huber via cfe-commits]

jhuber6 wrote:

> > I don't think we want to support this. `-march` was the wrong option to use in the first place, and upstream LLVM never supported specifying multiple device images with `-march` so there isn't a legacy argument in trunk. However, AOMP did support this and if it's deemed too disruptive to request users move to `--offload-arch=a,b,c` then we can carry that change in the fork.
> > > It will fix tests like: [targetid_multi_image](https://github.com/ROCm/aomp/tree/aomp-dev/test/smoke/targetid_multi_image)
> > 
> > 
> > I think the easier way to fix this is to update the Makefile.
> 
> Irrespective of what AOMP does, I think it makes sense to ensure parity between the two ways of specifying architecture. People have been historically using `-Xopenmp-target -march` style, and using the same for multiple architectures seems to be the most obvious choice. Isn't it quite confusing to tell the users that they can use `offload-arch` style for single as well as multiple archs, but can use `-march` style only for single arch?

`-march` was the wrong option to use for this from the beginning. It's supposed to be an overriding option and it shouldn't be overloaded to mean something different here. In LLVM / trunk we never supported multiple architectures with the `-march` option so I don't see any reason to start now. `--offload-arch=` is a complete replacement for this behavior and I consider the single `-march` option to be legacy. Even within this it's divergent because HIP / OpenCL / AMDGPU use `-mcpu` but the OpenMP toolchain ignored that and uses `-march=`.

Using `--offload-arch=` is a direct replacement for `-march` in all use-cases. It's also easier to use and interoperable with CUDA. I would just change the test, you can replace every use of `-march` with `--offload-arch` and it will work. See the following.
```console
> clang input.c -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa,nvptx64-nvidia-cuda -Xopenmp-target=amdgcn-amd-amdhsa --offload-arch=gfx1030 -Xopenmp-target=amdgcn-amd-amdhsa --offload-arch=gfx90a -Xopenmp-target=nvptx64-nvidia-cuda --offload-arch=sm_89 
> llvm-objdump --offloading a.out
                                                                                    
a.out:	file format elf64-x86-64

OFFLOADING IMAGE [0]:
kind            elf
arch            sm_89
triple          nvptx64-nvidia-cuda
producer        openmp

OFFLOADING IMAGE [1]:
kind            elf
arch            gfx90a
triple          amdgcn-amd-amdhsa
producer        openmp

OFFLOADING IMAGE [2]:
kind            elf
arch            gfx1030
triple          amdgcn-amd-amdhsa
producer        openmp

```

https://github.com/llvm/llvm-project/pull/92290