<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/55989>55989</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            Clang can't find ROCM device library when using FHS compliant path
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          JacekJagosz
      </td>
    </tr>
</table>

<pre>
    I have packaged rocm for Solus closely following [packages for Fedora](https://copr.fedorainfracloud.org/coprs/mystro256/rocm-opencl/packages/), not using `/opt/rocm` but instead `GNUInstallDirs`. The paths are `/usr/lib64/cmake/AMDDeviceLibs/AMDDeviceLibsConfig.cmake
` and`/usr/lib64/amdgcn/bitcode/`, same as on Fedora. I used [this patch without modifications](https://src.fedoraproject.org/rpms/rocm-device-libs/blob/rawhide/f/0001-Use-FHS-compliant-install.patch). I assume this will affect Fedora as well.
All packages work so far, up to rocm-opencl and hip, but when building rocBLAS I am getting:
```
clang-13: error: cannot find ROCm device library. Provide its path via --rocm-path or --rocm-device-lib-path, or pass -nogpulib to build without ROCm device library.
```
This is happening no matter if using LLVM 13 or 14, when defining `HIP_DEVICE_LIB_PATH`/`--rocm-device-lib-path` or not. 
I [found the function it is failing at](https://github.com/llvm/llvm-project/blob/f1e7ecaa18a7b1c77fc95e4b83d3db02c4e2d211/clang/lib/Driver/ToolChains/AMDGPU.cpp#L362), but have no idea why `RocmInstallationDetector::detectDeviceLibrary()` is failing.

I found a piece of code that will definitely fail to find HIP when it is installed in FHS paths, but I think my problem also lies somewhere else: https://github.com/llvm/llvm-project/blob/50f846d63443781a5c00ec7022c381c4f09f543d/mlir/lib/ExecutionEngine/CMakeLists.txt#L191
```
  if (NOT DEFINED ROCM_PATH)
    if (NOT DEFINED ENV{ROCM_PATH})
      set(ROCM_PATH "/opt/rocm" CACHE PATH "Path to which ROCm has been installed")
    else()
      set(ROCM_PATH $ENV{ROCM_PATH} CACHE PATH "Path to which ROCm has been installed")
    endif()
    set(HIP_PATH "${ROCM_PATH}/hip" CACHE PATH "Path to which HIP has been installed")
  endif()
  set(CMAKE_MODULE_PATH "${HIP_PATH}/cmake" ${CMAKE_MODULE_PATH})
  find_package(HIP)
  if (NOT HIP_FOUND)
    message(SEND_ERROR "Building mlir with ROCm support requires a working ROCm and HIP install")
  else()
    message(STATUS "ROCm HIP version: ${HIP_VERSION}")
  endif()

  # Locate HIP runtime library.
  find_library(ROCM_RUNTIME_LIBRARY amdhip64
               PATHS "${HIP_PATH}/lib")
```
With that logic it is not possible to find both `/usr/include/hip/` and `/usr/lib64/cmake/Modules/` with one `HIP_PATH`.

Could anyone point me tho where the problem could lie? Are there any flags I still need to set? Or is it a problem with LLVM's logic?
Here are my [rocm-device-libs build script](https://dev.getsol.us/D13161#change-dIsnTU8Mrbn6), and here the [rocBLAS one that fails](https://github.com/JacekJagosz/rocm-Solus/blob/main/rocblas/package.yml).
</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJytV1tz6jYQ_jXwosHjG7cHHgiXE1pIMlzS6VNGtmWsRrZcSU4O_fXdlW1IOJx02mnGMTYraXe_3f12iWRymqxIRt8YKWn8So8sIUrGOUmlIjspKk1iITUTJ_hGCPnOiyPp9O-axdquW7JEKtrpzzv-KDOm1J1g2vGXcMWyVE5qxbxIFYWzqsSR6tjINHzmJ22U9PsDeEbVPVmyIhbw1iqxZ407_owU0pBKWxsGLnwtS9PsgncSVYbwQhtGE5R_ezis4I0KMeegaeA6ZJ-hnybThCrWnFFpBXfBo0GIVuX0lcHndDOfszceszWP9PX7TBYpPzr1WnfecaeonhbJrRNpnhzjAh4ibmKZ4OF22YxomjNCNZFFA6FDVuAehAAQNhnXaGuckXduMgm-5TLhKY-p4bLQt-DWKm7QLpX8g8WmgVqVuW7RTawXPVG7FQkZoYS-Z9yalsK_67pe76BZb3m_68UyLwWnhenxGkzHGgUBQWup1hU4YY1950IQmqagt_EHnXtnsKUGaQryc-K8S_VKtCQpVYhFVRIjyYf4I5wk4yUKMbDvGSvggYsEww8L79bTHVqQkyMzBr5EGNpgNJd9jQUtjj0vADlhSkmFDzEtMJlSDlq2j7Oc1LgQwEVRdXLIk5JvAAnhxoYhI2-ckl7PWmjfIfGb1wukVoIWg7AEbEivkMeyAgl6Z60_R_OW1pv27xFcuDJaAjLofSFJTo1hivC0qYf1-nlDvAAVeyFaYPFKGDjYlMv96ullvnhezRYv69Xdy9N0f1-nK9x_4ggkNZwHODmkNmWFmZnKCkAzUEppVcSYjAASGphSLlAZNbeS8wh-V5ED-YTFId7aj16Tq5dsTD02ZDGl3ogOIy8eDtN43GdhNAqSIIlcPw6Zn_ieh-WKsa2LDe5zxd8Y1t5eSjHLgHOa0v32dHDiEnIpWAcDvyETzCrLfAAnRJoCZCcEagtQNMRhS23ODJhn0wauxL6duQDDBq7ikYDWBYQ2lA1sNWaUlJxBuGVKkAkAQ2rqsqkDZSzPwgGYLTY1IWh1JGuEmxIEhuDAGfe7msxaZ1ZYh8UryU8EMI0EywkVUGKCQ71pmTM4CWiPCc2wBP5rePpuOgoHySAIw2A48mg_dl0WD13fj4ORF4epO077YZAgtwuuztFZfGdxhYAuiiMvkGxmGyDQNddGO-a7weh4Y-9mDRDMdMD54XFP5ovl6mExxwLa1GkM4Derbq1bPDx3hneX1cP5pw2EaAa6R-cFsN__3Ft8n8yms_sFacVPSAAQI2BN4GdbyRlwXcQwUm2M7DEfFFnc61T5Unn4o8H_i_oCuse1_lo7UsPF8_AHtJaWh78GATP1n4y4YUJtwGwz_XXxsnmcH9aLK0ta22pDmvbsk1r4w77PwcUSemk6Tu3mB-ElUVDF8vHw8DkvcqZ1vW-3eJi_LLbbxy2addf2IExuy-Z1BHRVllIZotifFVdQcdT2OFxp5bQp5wabK2Ru5MYHA_bT_WGHyu1JeArwnIZSwjq-4PS82O5Wjw8WhC9xbwVQcmQtYaJg9lBVFYbn1-2owVGcyc4mx_bwsF9tbC_ZTre_Qx9OIEtg4vmQ2-c_DM3uZzG17HAx96ryf0OALVEKeeRxQ4TYu0upNQeWO5NlJGHpxxGMwxxR2bHGJvCymdK-HPw2MqlEPXPCahteWbC2gTZd8xO7z2QFbZ0WJ1xXSl7ApIbcjpWBfIudsuXj2K4FQu4ESzKthXCHzSQV9KiBw7XBhlAwIHlwDOsDlj4qS_8GW0hzlDUN237HH-oaHFhZm3RvD4V_aAXQsa8Hv2YS0bHi5c1eDWsdmKq0FE6FUMy9wBtAvw3iDPot6yUrXewPo42KikHTTe241rpb67QTGoJiw4d97ebU-qnz_EJj9voLPUr9Vzux2p8hl_aTQ1uvRZGg-vIzwTnlAqfSbjIJknEwpl3DjWCTGY4IOO8BTpeBb3M1etVdtp6lsLGex17bYruVEpN_3S85DMc2k_r98WjczSbe0KX-IA1GwyD0h_1B5A6TNHSZPxiG0TBKu4JGwAQTgA-A6vKJD03VHXieF_rjfuhElKV92A3dN03HvtuBvQCHcFAxzvpdNbE2RNVRg1DY1noWwkDKj5BY7fm0giRVkw-Qd63JE2vv3wErYoA">