<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/59341>59341</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            commit 9223315 ISSUE: Unpredictable and wrong behavior of openmp offload application on AMD GPU
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
            new issue
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          fel-cab
      </td>
    </tr>
</table>

<pre>
    Application behaves unpredictable after commit 92233159035d1b50face95d886901cf99035bd99.
The application is SPEChpc tealeaf compiled for openmp offload.
CPU: AMD EPYC 7A53
GPU: AMD MI250X
Compilation Flags: -fopenmp -Wall -Ofast -foffload-lto -fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa -march=gfx90a

The problem appears in different iterations of the execution, for different executions.
I have simplified as much as I could the application and still get the problem and added some printf to show the unexpected behavior. As an example, in one of the executions the error appears on iteration 175, where the value of the variable **rrn** shouldn't jump to the value on iteration 175:

`iteration 173, error: 1.563668706240932e+08   pw = 8.489109156989025e+12, rrn = 1.530719718661226e+08`
`iteration 174, error: 1.530719718661226e+08   pw = 8.237148061653313e+12, rrn = 1.521889487492852e+08`
`iteration 175, error: 1.521889487492852e+08   pw = 8.261711869147320e+12, rrn = 3.903098816183680e+299`

In a different execution the **rrn** value can get a wrong value on iteration 38 instead.

`iteration 36, error: 3.821804208275762e+09   pw = 2.165156939490312e+14, rrn = 3.561339554743209e+09`
`iteration 37, error: 3.561339554743209e+09   pw = 1.907022120691036e+14, rrn = 3.446392781811089e+09`
`iteration 38, error: 3.446392781811089e+09   pw = 1.945475027719899e+14, rrn = -1.080323384166538e+85`

If the code is compiled with the previous commit (a8ec170e01dd0d0cbf8c059aa7addaaf3d886876) the application doesn't presents this unpredictable behavior and all the executions are the same (as it should be).

The problem seems to be in one or both of functions `cg_calc_ur` or `cg_calc_w`  from the `cg.c` file.

I'm attaching the application code, with an compilation script, and the IR with the "good" and the "bad" commit obtained by compiling with `-Rpass=openmp-opt -Rpass-missed=openmp-opt -Rpass-analysis=openmp-opt -save-temps` and `opt -S -O3`.

[simplifiedTealeaf.zip](https://github.com/llvm/llvm-project/files/10155376/simplifiedTealeaf.zip)

</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJyUVktv4zgS_jX0hbDAh0iRBx_cSWeQw2CCyQQ7expQYsligxIFkUrS--sXlBw7drwN7CFQzHp99eBXNDG6wwCwQ-IbEvcbM6cuTLsW_LYx9aYO9uduP47eNSa5MOAaOvMKEc_DOIF1TTK1B2zaBBNuQt-7hDVjnFOhCReW1oK0pgEtrFJSE9q0Ogtqq3WByD0i-786wOZTBBfx89P3u25scALjwbTZ8eg8WNyGCYcRhn7EoW19MPbo5O7pBfE93v9-j78__fsOV3vBV8lvZ8nvj0yQv48Gi8s14oM3h5iVtu3R-fZfxnu8_aM1MeXTNdbWp3DS2SYzHSBFxO9Nbw_NsDW9zX9dNHj794XSbZ3eTE2H-P2hfdfErLDOJRmnUHvoc2nATBG7AVvXtjDBkLBLMC3YIw4tTh1geIdmzieI3S1lOiufRPFYrEece4ij60fvWgcWm4j7ueny9xE3YfZ2cfq5LWawOCbnPT5AWqQnhIPFxlqwOIY-H7shtTgFHLvwtmjOA7yP0CSw6_y4MBV4H7EZMLybfvSQUbsBhwG-JBTXn9MUplMx8ph8lADTSmTztw4mWHRfjZ9Pfl7N5JYZRWyP2H6ahvWfjG72dkCsSvjH3I8Z8Sfr6wh8_7lDSJLPYp4BLBDzGNFCSC6lqohkJdGcAWLfiMIYj28Y8XusilJpSjQVUitNmMgKlGUn0zQsKrQQnFRUV1RJSRmTqw8kyU0A5TWAW8afATBe0VIRSaXgnPKbABhVSpeqKjVTgv0agLgGcMv4AoCkFaVKalpWnJGvAHihCSdaKSqp4lItKkzrM4J1mAdsbg370svrnq-9bcywDLHBb1MYDrc6zhV2Q0xwIpgbSXN5kTMvFKOKlIwoVolKrjnrc86soFLklnNdasLpokDLy5yFpJxrIcqq5Izo1cftovPqKv5N23N8WmhSEcYoI1JTwuWt-GUpuWaVoopSon4dX13Fv2l7Eb8UZSUIqyqqldZf429pQRThjHNVUikFV1lHieuer1e7CRbyujithzeXuiM1wasLc_xYSYgpo6ChFQFCrSWWNHWrGiK0MZWx1piW5wWlKomY_sJ9NkBciWKcIMKQMie56yX4QW0rIXp_TWPmSE_R9LAgitilIw3hGhDTxf9aAhGgj5mgajjR5ITrkLpMc-08NGsIJElz-KcxvvlnnpAkWevT2Vs-wu0U-vV2ZEnR5MPWebiI_ohY1WOTkmk6Nxy-lCTXfmHdXHIzHHuwymIzuTFlaa5Etnz889wbxNghBIsYO4kRY7VZTo79CnUybsjr4ufRc8awuECSbP8cTcyL97hkw5jwerbtXYxgb4rMYPzP6K7tonmFbYJ-jLkOGRGSZJE84-0fHElyyQDi23lv_rW-T4r_uBGJe8RUl9KYXxKIPSD2cHCpm-uiCT1iD96_fny24xR-QJMQe8h1j4g9UEKF4Hn6Hm67Z3qNv7E7bjXXZgM7KivCspXcdDtT26alraFtK6jkiqsSGlNVlVISlDYbt2Pr5RdUMFbKwkquNWVCNDXVQhBUEuiN80XGWITpsHExzrATmpd0400NPi7PRMYGeMOLEDGWX43Tbsmrng8RlcS7mOLZS3LJw-7ycYgfn59fvmfaeLl8SA72SMqn2xTaqyffxRyGYXna_fb0spknv_u_G7BkkTuwZPnfAAAA__9ekl-m">