[flang-commits] [flang] [llvm] [OpenMPIRBuilder] Always inline device-outlined target regions (PR #211136)

[Spencer Bryngelson via flang-commits]

sbryngelson wrote:

Root-caused why the inliner refuses it. I think this resolves the conflict with ROCm/llvm-project#3485.

The 495 in the remark is 45 x 11: `-inline-cold-callsite-threshold` (45) times AMDGPU's inlining threshold multiplier (11). Setting it to 46 gives 506, so that is where it comes from. The ordinary threshold would be 225 x 11 = 2475, and the cost is 1280, well under. The region is rejected only because the callsite is classified cold.

Reproducible with `opt` alone on the pre-inline device LTO module:

```
$ opt -passes='cgscc(inline)' -pass-remarks-missed=inline mod.ll
... '..omp_par.2' not inlined into '__omp_offloading_..._kern__l27' \
    because too costly to inline (cost=1280, threshold=495)

$ opt -passes='cgscc(inline)' -inline-cold-callsite-threshold=200 ... mod.ll
(inlined, no remark)

$ sed -E 's/,? !prof ![0-9]+//g' mod.ll | opt -passes='cgscc(inline)' ...
(inlined, no remark)
```

Where the coldness comes from: `__kmpc_parallel_60` is `always_inline`, so it lands in every kernel and brings its `OMP_UNLIKELY` branch weights with it. Two calls to the microtask survive in the kernel afterwards, one on the SPMD path and one on the serialized path. The SPMD one inlines. The serialized one sits behind `enterDataEnvironment` at roughly 1/4000 of the entry frequency, so it does not.

That leftover cold call is what costs the occupancy. The kernel is no longer a leaf and its resource numbers reflect the call. Raising only `-inline-cold-callsite-threshold` from 45 to 200 removes it and reproduces the patched codegen exactly, 94 VGPR / 0 scratch / occupancy 5 on gfx90a, with no `alwaysinline` anywhere.

So the defect is not that the outlined region is too costly to inline. It is that a serialized-path call which never executes in an SPMD kernel stops the kernel from being allocated as a leaf. Fixing it there, by dropping the serialized path once OpenMPOpt has established SPMD mode or by keeping that copy from forcing the kernel's worst case, leaves the cost model in charge, which is what ROCm/llvm-project#3485 needs: there the problem is precisely that `alwaysinline` takes the cost model out of the loop and the enlarged kernel blows the AMDGPU basic-block budget.

Also checked, since it was the obvious way to split the two cases: restricting `alwaysinline` to the workshare-loop outline from `applyWorkshareLoopTarget` and leaving the parallel outline alone gives 212 / 48 / 2, i.e. no benefit at all. Both cases are the same outlined function, so they cannot be separated by construct.

@skatrak @abidh, if that reading looks right I will close this PR and open one against the narrower cause instead.

This affects performance-critical applications on large AMD GPU supercomputers, including [MFC](https://github.com/MFlowCode/MFC).

All numbers above come from the validated reproducers included with this report and are independently reproducible; they stand on their own.

This was found and root-caused with the assistance of AI tools.


https://github.com/llvm/llvm-project/pull/211136