[llvm] [Offload][AMDGPU] Add env var to control device-to-device memory access (PR #215385)

[Joseph Huber via llvm-commits]

================
@@ -4514,10 +4522,12 @@ Expected<void *> AMDGPUDeviceTy::allocate(size_t Size, void *,
   if (auto Err = MemoryPool->allocate(Size, &Alloc, Alignment))
     return std::move(Err);
 
-  if (Alloc) {
+  if (Alloc && (Kind == TARGET_ALLOC_HOST || Kind == TARGET_ALLOC_SHARED ||
----------------
jhuber6 wrote:

Okay, so this is a behavior change, but one I generally agree with. This should not be an environment variable, or at least not one at the plugin level. CUDA also has this concept and it's opt-in, so we should probably do something similar and expose it more consistently. E.g. for CUDA:

```c
  int ok = 0;
  cudaDeviceCanAccessPeer(&ok, /*device=*/0, /*peer=*/1);
  cudaSetDevice(0);
  cudaDeviceEnablePeerAccess(/*peerDevice=*/1, 0);  // 0 can now load/store from 1
```

So, I'm in agreement we should not do this by default but disagree in this implementation's choices.

https://github.com/llvm/llvm-project/pull/215385