[llvm] [Offload][AMDGPU] Only allow memory pool access to valid agents (PR #93969)

[Johannes Doerfert via llvm-commits]

https://github.com/jdoerfert approved this pull request.

LG, but I think this should be reworked to minimize what we do per allocation.

https://github.com/llvm/llvm-project/pull/93969