[clang] [Clang][AMDGPU] Search TheRock-based device libraries (PR #170590)

[Tsukasa OI via cfe-commits]

a4lg wrote:

## Background

This is originally submitted as ROCm/llvm-project#739 to the `amd-staging` branch.

AMD is experimenting with TheRock, new HIP/ROCm build system and distribution.

With TheRock, we can install full ROCm SDK as Python packages (and multiple versions can coexist through `venv`).

To fully utilize TheRock-based toolchain, we normally set the environment variable `${ROCM_PATH}` to `$(rocm-sdk --path root)` (`rocm-sdk` is TheRock-specific command to prepare/test ROCm SDK) to make sure that external programs/libraries can locate TheRock-based ROCm SDK.

One of the problems is: some programs (such as vLLM) don't use `hipcc` but instead, attempt to use `$(rocm-sdk path --root)/lib/llvm/bin/clang++` directly.  Second, TheRock has slightly different toolchain layout than the release versions of ROCm SDK.

| ROCm SDK                 | Device libraries path                  |
| ------------------------ | -------------------------------------- |
| Production Release (7.1) | `${ROCM_PATH}/amdgcn/bitcode`          |
| TheRock (7.9 / Nightly)  | `${ROCM_PATH}/lib/llvm/amdgcn/bitcode` |

Clang does not search the latter by default, causing a compiler error and fails to configure the ROCm toolchain on such Clang-dependent programs.

Note that, this error does not occur if `ROCM_PATH` is *not set* because in the fallback path, a ROCm installation candidate path equivalent to `${ROCM_PATH}/lib/llvm` is generated.  However, it prevents configuring ROCm-dependent programs using standard `ROCM_PATH` environment variable.

The workaround below is working yet inconvenient so I hope this issue is resolved in either:

1.  TheRock distribution (separate PR required): Change the path of device libraries or make links/copies to the same path as the production releases.
2.  LLVM (this PR): Add TheRock search path

## Workaround (without this PR)

With the current state, we can avoid device library issues by:

```sh
export HIP_DEVICE_LIB_PATH=$(rocm-sdk path --root)/lib/llvm/amdgcn/bitcode
```

https://github.com/llvm/llvm-project/pull/170590