[llvm] [offload] add support for aligned allocations (PR #203353)
via llvm-commits
llvm-commits at lists.llvm.org
Fri Jun 12 08:22:54 PDT 2026
https://github.com/EuphoricThinking updated https://github.com/llvm/llvm-project/pull/203353
>From 50503d466d71b830246dc8bb8fb8d348f6cd35b9 Mon Sep 17 00:00:00 2001
From: Agata Momot <agata.momot at intel.com>
Date: Fri, 8 May 2026 14:26:43 +0000
Subject: [PATCH] [offload] add support for aligned allocations
This patch is the first step towards introducing alignment support in memory allocations using liboffload, in order to enable SYCL implementation of aligned allocations.
At the level of device allocators, it does not modify the Level Zero code except for forwarding the alignment parameter, since Level Zero already allows for specifying the alignment in its device allocator implementation. For AMD and CUDA, it checks whether the alignment passed by the caller is supported by the given backend; the reasoning behind this verification is described in the following paragraphs. At the API level, it adds a new function olMemAllocAligned, which is expected to work similarly to olMemAlloc, with the difference that the buffers returned by olMemAllocAligned should be aligned to the alignment passed by the user. At the level of the plugin interface internal abstractions, it adds a new argument Alignment to existing functions and delegates memory allocation between olMemAllocAligned and olMemAlloc implementations by using a common helper function.
The goal of the anticipated series of patches is to implement handling of the alignment in the memory manager at the plugin interface level. At the first stage, presented in this patch, the information about the passed alignment is used mainly for checking whether the buffer returned by the device allocators meets the requirements. In the case of the requested memory size exceeding the thresholds of allocations handled by the memory manager, the request is forwarded directly to the device. Otherwise, the memory manager is responsible for allocating memory in full pages and pooling it according to the requested chunks. In the first scenario, the requested size, which is greater than the aforementioned threshold, is usually a multiple of the page size. Therefore, any alignment smaller than the page size would be correct.
Neither CUDA nor HSA provides users with the ability to specify the alignment of the allocated memory. Their APIs include the alignment as one of the possible arguments only in functions that reserve virtual address space. CUDA enables users only to check the allocation granularity, which is usually synonymous with the page size. In the case of HSA, if the memory is allocated using a pool, the user is able to check not only the granularity, but also the alignment of the buffers allocated using the given pool. However, these values - granularity and alignment - are defined only if HSA_AMD_MEMORY_POOL_INFO_RUNTIME_ALLOC_ALLOWED is set to true, which should always be set since the current implementation of the memory pool for the AMD plugin in liboffload uses only hsa_amd_memory_pool_ts for allocations. Only Level Zero accepts the alignment parameter during the memory allocation process, but manages returned pointers internally. Since pooling is also implemented in the memory manager at the plugin level, such a design in the Level Zero device allocator duplicates pooling in the whole application.
In the final version, the memory manager at the plugin interface level will handle memory pooling and the buffer alignment from different devices instead of delegating memory management to device allocator implementations, as this would simplify the Level Zero plugin design and provide AMD and CUDA plugins with support for the alignment in memory allocations, which is not natively included in their APIs.
---
offload/liboffload/API/Memory.td | 27 +++
offload/liboffload/src/OffloadImpl.cpp | 21 +-
offload/plugins-nextgen/amdgpu/src/rtl.cpp | 56 ++++-
.../common/include/MemoryManager.h | 33 +--
.../common/include/PluginInterface.h | 3 +-
.../common/src/PluginInterface.cpp | 24 +-
offload/plugins-nextgen/cuda/src/rtl.cpp | 33 ++-
offload/plugins-nextgen/host/src/rtl.cpp | 3 +-
.../level_zero/include/L0Device.h | 4 +-
.../level_zero/src/L0Device.cpp | 9 +-
offload/unittests/OffloadAPI/CMakeLists.txt | 1 +
.../OffloadAPI/memory/olMemAllocAligned.cpp | 211 ++++++++++++++++++
12 files changed, 379 insertions(+), 46 deletions(-)
create mode 100644 offload/unittests/OffloadAPI/memory/olMemAllocAligned.cpp
diff --git a/offload/liboffload/API/Memory.td b/offload/liboffload/API/Memory.td
index 78cfdd2f8855a..78f136801885a 100644
--- a/offload/liboffload/API/Memory.td
+++ b/offload/liboffload/API/Memory.td
@@ -51,6 +51,33 @@ def olMemAlloc : Function {
];
}
+def olMemAllocAligned : Function {
+ let desc = "Creates a memory allocation on the specified device with specified properties.";
+ let details = [
+ "All allocations through olMemAllocWithProp regardless of source share a single virtual address range. There is no risk of multiple devices returning equal pointers to different memory."
+ ];
+ let params = [
+ Param<"ol_device_handle_t", "Device", "handle of the device to allocate on", PARAM_IN>,
+ Param<"ol_alloc_type_t", "Type", "type of the allocation", PARAM_IN>,
+ Param<"size_t", "Size", "size of the allocation in bytes", PARAM_IN>,
+ Param<"size_t", "Alignment",
+ "alignment of the allocation im bytes. Must be non-zero and a power of two",
+ PARAM_IN>,
+ Param<"void**", "AllocationOut", "output for the allocated pointer", PARAM_OUT>
+ ];
+ let returns = [
+ Return<"OL_ERRC_INVALID_SIZE", [
+ "`Size == 0`"
+ ]>,
+ Return<"OL_ERRC_INVALID_ARGUMENT", [
+ "`Alignment == 0`"
+ ]>,
+ Return<"OL_ERRC_INVALID_ARGUMENT", [
+ "`(Alignment & (Alignment - 1)) != 0`"
+ ]>,
+ ];
+}
+
def olMemFree : Function {
let desc = "Frees a memory allocation previously made by olMemAlloc.";
let params = [
diff --git a/offload/liboffload/src/OffloadImpl.cpp b/offload/liboffload/src/OffloadImpl.cpp
index 63294886e520e..741ef836c3433 100644
--- a/offload/liboffload/src/OffloadImpl.cpp
+++ b/offload/liboffload/src/OffloadImpl.cpp
@@ -597,16 +597,17 @@ TargetAllocTy convertOlToPluginAllocTy(ol_alloc_type_t Type) {
}
constexpr size_t MAX_ALLOC_TRIES = 50;
-Error olMemAlloc_impl(ol_device_handle_t Device, ol_alloc_type_t Type,
- size_t Size, void **AllocationOut) {
+Error olMemAllocImplHelper(ol_device_handle_t Device, ol_alloc_type_t Type,
+ size_t Size, size_t Alignment,
+ void **AllocationOut) {
SmallVector<void *> Rejects;
// Repeat the allocation up to a certain amount of times. If it happens to
// already be allocated (e.g. by a device from another vendor) throw it away
// and try again.
for (size_t Count = 0; Count < MAX_ALLOC_TRIES; Count++) {
- auto NewAlloc = Device->Device->dataAlloc(Size, nullptr,
- convertOlToPluginAllocTy(Type));
+ auto NewAlloc = Device->Device->dataAlloc(
+ Size, nullptr, convertOlToPluginAllocTy(Type), Alignment);
if (!NewAlloc)
return NewAlloc.takeError();
@@ -653,6 +654,18 @@ Error olMemAlloc_impl(ol_device_handle_t Device, ol_alloc_type_t Type,
"failed to allocate non-overlapping memory");
}
+Error olMemAlloc_impl(ol_device_handle_t Device, ol_alloc_type_t Type,
+ size_t Size, void **AllocationOut) {
+ return olMemAllocImplHelper(Device, Type, Size, /*Alignment=*/0,
+ AllocationOut);
+}
+
+Error olMemAllocAligned_impl(ol_device_handle_t Device, ol_alloc_type_t Type,
+ size_t Size, size_t Alignment,
+ void **AllocationOut) {
+ return olMemAllocImplHelper(Device, Type, Size, Alignment, AllocationOut);
+}
+
Error olMemFree_impl(void *Address) {
ol_device_handle_t Device;
ol_alloc_type_t Type;
diff --git a/offload/plugins-nextgen/amdgpu/src/rtl.cpp b/offload/plugins-nextgen/amdgpu/src/rtl.cpp
index d1c6d0de11280..8c7d3d84f7519 100644
--- a/offload/plugins-nextgen/amdgpu/src/rtl.cpp
+++ b/offload/plugins-nextgen/amdgpu/src/rtl.cpp
@@ -291,6 +291,10 @@ struct AMDGPUMemoryPoolTy {
if (auto Err = getAttr(HSA_AMD_MEMORY_POOL_INFO_GLOBAL_FLAGS, GlobalFlags))
return Err;
+ if (auto Err = getAttr(HSA_AMD_MEMORY_POOL_INFO_RUNTIME_ALLOC_ALIGNMENT,
+ PoolAllocationAlignment))
+ return Err;
+
return getAttr(HSA_AMD_MEMORY_POOL_INFO_RUNTIME_ALLOC_GRANULE, Granule);
}
@@ -324,10 +328,32 @@ struct AMDGPUMemoryPoolTy {
/// Get the allocation granularity of the pool.
size_t getGranule() const { return Granule; }
+ /// Get the allocation alignment of the pool.
+ size_t getAlignment() const { return PoolAllocationAlignment; }
+
/// Allocate memory on the memory pool.
- Error allocate(size_t Size, void **PtrStorage) {
+ Error allocate(size_t Size, void **PtrStorage, size_t Alignment) {
+ // A non-zero value passed as the Alignment indicates that the user expects
+ // the allocation to have a specific alignment. However, the HSA API does
+ // not allow users to define alignment. Therefore, the passed alignment is
+ // compared with the alignment of the memory allocated using the given pool.
+ // If the default alignment is greater than or equal to the alignment
+ // requested by the user, it would still meet the user's requirements.
+ if (Alignment > 0 && Alignment >= PoolAllocationAlignment) {
+ return Plugin::error(ErrorCode::UNSUPPORTED,
+ "requested alignment (%lu) larger than maximum "
+ "supported pool alignment (%lu)",
+ Alignment, PoolAllocationAlignment);
+ }
+
hsa_status_t Status =
hsa_amd_memory_pool_allocate(MemoryPool, Size, 0, PtrStorage);
+
+ if (Alignment > 0 && !isAddrAligned(Align(Alignment), *PtrStorage)) {
+ return Plugin::error(ErrorCode::UNSUPPORTED,
+ "unsupported alignment size");
+ }
+
return Plugin::check(Status, "error in hsa_amd_memory_pool_allocate: %s");
}
@@ -407,6 +433,14 @@ struct AMDGPUMemoryPoolTy {
/// The page size in this memory pool.
size_t Granule;
+
+ /// The alignment of the buffers allocated by
+ /// hsa_amd_memory_pool_allocate(...). This attribute is defined only if
+ /// HSA_AMD_MEMORY_POOL_INFO_RUNTIME_ALLOC_ALLOWED is set to true. Since
+ /// hsa_amd_memory_pool_allocate is the only memory-allocating function that
+ /// is used by the memory pool, HSA_AMD_MEMORY_POOL_INFO_RUNTIME_ALLOC_ALLOWED
+ /// should always be set.
+ size_t PoolAllocationAlignment;
};
/// Class that implements a memory manager that gets memory from a specific
@@ -442,7 +476,8 @@ struct AMDGPUMemoryManagerTy : public DeviceAllocatorTy {
assert(MemoryManager && "Invalid memory manager");
assert(PtrStorage && "Invalid pointer storage");
- auto PtrStorageOrErr = MemoryManager->allocate(Size, nullptr);
+ auto PtrStorageOrErr =
+ MemoryManager->allocate(Size, nullptr, /*Alignment=*/0);
if (!PtrStorageOrErr)
return PtrStorageOrErr.takeError();
@@ -466,8 +501,8 @@ struct AMDGPUMemoryManagerTy : public DeviceAllocatorTy {
private:
/// Allocation callback that will be called once the memory manager does not
/// have more previously allocated buffers.
- Expected<void *> allocate(size_t Size, void *HstPtr,
- TargetAllocTy Kind) override;
+ Expected<void *> allocate(size_t Size, void *HstPtr, TargetAllocTy Kind,
+ size_t Alignment) override;
/// Deallocation callback that will be called by the memory manager.
Error free(void *TgtPtr, TargetAllocTy Kind) override {
@@ -2648,7 +2683,8 @@ struct AMDGPUDeviceTy : public GenericDeviceTy, AMDGenericDeviceTy {
}
/// Allocate memory on the device or related to the device.
- Expected<void *> allocate(size_t Size, void *, TargetAllocTy Kind) override;
+ Expected<void *> allocate(size_t Size, void *, TargetAllocTy Kind,
+ size_t Alignment) override;
/// Deallocate memory on the device or related to the device.
Error free(void *TgtPtr, TargetAllocTy Kind) override {
@@ -4352,10 +4388,11 @@ static Error Plugin::check(int32_t Code, const char *ErrFmt, ArgsTy... Args) {
}
Expected<void *> AMDGPUMemoryManagerTy::allocate(size_t Size, void *HstPtr,
- TargetAllocTy Kind) {
+ TargetAllocTy Kind,
+ size_t Alignment) {
// Allocate memory from the pool.
void *Ptr = nullptr;
- if (auto Err = MemoryPool->allocate(Size, &Ptr))
+ if (auto Err = MemoryPool->allocate(Size, &Ptr, Alignment))
return std::move(Err);
assert(Ptr && "Invalid pointer");
@@ -4373,7 +4410,8 @@ Expected<void *> AMDGPUMemoryManagerTy::allocate(size_t Size, void *HstPtr,
}
Expected<void *> AMDGPUDeviceTy::allocate(size_t Size, void *,
- TargetAllocTy Kind) {
+ TargetAllocTy Kind,
+ size_t Alignment) {
if (Size == 0)
return nullptr;
@@ -4398,7 +4436,7 @@ Expected<void *> AMDGPUDeviceTy::allocate(size_t Size, void *,
// Allocate from the corresponding memory pool.
void *Alloc = nullptr;
- if (auto Err = MemoryPool->allocate(Size, &Alloc))
+ if (auto Err = MemoryPool->allocate(Size, &Alloc, Alignment))
return std::move(Err);
if (Alloc) {
diff --git a/offload/plugins-nextgen/common/include/MemoryManager.h b/offload/plugins-nextgen/common/include/MemoryManager.h
index 9dd4ee684f85a..4b57be45e7551 100644
--- a/offload/plugins-nextgen/common/include/MemoryManager.h
+++ b/offload/plugins-nextgen/common/include/MemoryManager.h
@@ -38,9 +38,9 @@ class DeviceAllocatorTy {
/// Allocate a memory of size \p Size . \p HstPtr is used to assist the
/// allocation.
- virtual Expected<void *>
- allocate(size_t Size, void *HstPtr,
- TargetAllocTy Kind = TARGET_ALLOC_DEFAULT) = 0;
+ virtual Expected<void *> allocate(size_t Size, void *HstPtr,
+ TargetAllocTy Kind = TARGET_ALLOC_DEFAULT,
+ size_t Alignment = 0) = 0;
/// Delete the pointer \p TgtPtr on the device
virtual Error free(void *TgtPtr,
@@ -143,8 +143,10 @@ class MemoryManagerTy {
size_t SizeThreshold = 1U << 13;
/// Request memory from target device
- Expected<void *> allocateOnDevice(size_t Size, void *HstPtr) const {
- return DeviceAllocator.allocate(Size, HstPtr, TARGET_ALLOC_DEVICE);
+ Expected<void *> allocateOnDevice(size_t Size, void *HstPtr,
+ size_t Alignment) const {
+ return DeviceAllocator.allocate(Size, HstPtr, TARGET_ALLOC_DEVICE,
+ Alignment);
}
/// Deallocate data on device
@@ -153,7 +155,8 @@ class MemoryManagerTy {
/// This function is called when it tries to allocate memory on device but the
/// device returns out of memory. It will first free all memory in the
/// FreeList and try to allocate again.
- Expected<void *> freeAndAllocate(size_t Size, void *HstPtr) {
+ Expected<void *> freeAndAllocate(size_t Size, void *HstPtr,
+ size_t Alignment) {
std::vector<void *> RemoveList;
// Deallocate all memory in FreeList
@@ -178,16 +181,16 @@ class MemoryManagerTy {
}
// Try allocate memory again
- return allocateOnDevice(Size, HstPtr);
+ return allocateOnDevice(Size, HstPtr, Alignment);
}
/// The goal is to allocate memory on the device. It first tries to
/// allocate directly on the device. If a \p nullptr is returned, it might
/// be because the device is OOM. In that case, it will free all unused
/// memory and then try again.
- Expected<void *> allocateOrFreeAndAllocateOnDevice(size_t Size,
- void *HstPtr) {
- auto TgtPtrOrErr = allocateOnDevice(Size, HstPtr);
+ Expected<void *> allocateOrFreeAndAllocateOnDevice(size_t Size, void *HstPtr,
+ size_t Alignment) {
+ auto TgtPtrOrErr = allocateOnDevice(Size, HstPtr, Alignment);
if (!TgtPtrOrErr)
return TgtPtrOrErr.takeError();
@@ -197,7 +200,7 @@ class MemoryManagerTy {
if (TgtPtr == nullptr) {
ODBG(OLDT_Alloc) << "Failed to get memory on device. Free all memory "
<< "in FreeLists and try again.";
- TgtPtrOrErr = freeAndAllocate(Size, HstPtr);
+ TgtPtrOrErr = freeAndAllocate(Size, HstPtr, Alignment);
if (!TgtPtrOrErr)
return TgtPtrOrErr.takeError();
TgtPtr = *TgtPtrOrErr;
@@ -231,7 +234,7 @@ class MemoryManagerTy {
/// Allocate memory of size \p Size from target device. \p HstPtr is used to
/// assist the allocation.
- Expected<void *> allocate(size_t Size, void *HstPtr) {
+ Expected<void *> allocate(size_t Size, void *HstPtr, size_t Alignment) {
// If the size is zero, we will not bother the target device. Just return
// nullptr directly.
if (Size == 0)
@@ -245,7 +248,8 @@ class MemoryManagerTy {
if (Size > SizeThreshold) {
ODBG(OLDT_Alloc) << Size << " is greater than the threshold "
<< SizeThreshold << ". Allocate it directly from device";
- auto TgtPtrOrErr = allocateOrFreeAndAllocateOnDevice(Size, HstPtr);
+ auto TgtPtrOrErr =
+ allocateOrFreeAndAllocateOnDevice(Size, HstPtr, Alignment);
if (!TgtPtrOrErr)
return TgtPtrOrErr.takeError();
@@ -281,7 +285,8 @@ class MemoryManagerTy {
ODBG(OLDT_Alloc) << "Cannot find a node in the FreeLists. "
<< "Allocate on device.";
// Allocate one on device
- auto TgtPtrOrErr = allocateOrFreeAndAllocateOnDevice(Size, HstPtr);
+ auto TgtPtrOrErr =
+ allocateOrFreeAndAllocateOnDevice(Size, HstPtr, Alignment);
if (!TgtPtrOrErr)
return TgtPtrOrErr.takeError();
diff --git a/offload/plugins-nextgen/common/include/PluginInterface.h b/offload/plugins-nextgen/common/include/PluginInterface.h
index f99a0e817fd58..bca49fb5207a8 100644
--- a/offload/plugins-nextgen/common/include/PluginInterface.h
+++ b/offload/plugins-nextgen/common/include/PluginInterface.h
@@ -949,7 +949,8 @@ struct GenericDeviceTy : public DeviceAllocatorTy {
virtual Error memoryVAUnMap(void *VAddr, size_t Size);
/// Allocate data on the device or involving the device.
- Expected<void *> dataAlloc(int64_t Size, void *HostPtr, TargetAllocTy Kind);
+ Expected<void *> dataAlloc(int64_t Size, void *HostPtr, TargetAllocTy Kind,
+ size_t Alignment);
/// Deallocate data from the device or involving the device.
Error dataDelete(void *TgtPtr, TargetAllocTy Kind);
diff --git a/offload/plugins-nextgen/common/src/PluginInterface.cpp b/offload/plugins-nextgen/common/src/PluginInterface.cpp
index 1e05c6ae66fdf..74ee3bf95f44d 100644
--- a/offload/plugins-nextgen/common/src/PluginInterface.cpp
+++ b/offload/plugins-nextgen/common/src/PluginInterface.cpp
@@ -124,9 +124,9 @@ GenericKernelTy::getKernelLaunchEnvironment(
KernelArgs.DynCGroupMem == 0)
return reinterpret_cast<KernelLaunchEnvironmentTy *>(~0);
- auto AllocOrErr = GenericDevice.dataAlloc(sizeof(KernelLaunchEnvironmentTy),
- /*HostPtr=*/nullptr,
- TargetAllocTy::TARGET_ALLOC_DEVICE);
+ auto AllocOrErr = GenericDevice.dataAlloc(
+ sizeof(KernelLaunchEnvironmentTy),
+ /*HostPtr=*/nullptr, TargetAllocTy::TARGET_ALLOC_DEVICE, /*Alignment=*/0);
if (!AllocOrErr)
return AllocOrErr.takeError();
@@ -148,7 +148,8 @@ GenericKernelTy::getKernelLaunchEnvironment(
auto AllocOrErr = GenericDevice.dataAlloc(
KernelEnvironment.Configuration.ReductionDataSize *
KernelEnvironment.Configuration.ReductionBufferLength,
- /*HostPtr=*/nullptr, TargetAllocTy::TARGET_ALLOC_DEVICE);
+ /*HostPtr=*/nullptr, TargetAllocTy::TARGET_ALLOC_DEVICE,
+ /*Alignment=*/0);
if (!AllocOrErr)
return AllocOrErr.takeError();
LocalKLE.ReductionBuffer = *AllocOrErr;
@@ -226,7 +227,8 @@ GenericKernelTy::prepareBlockMemory(GenericDeviceTy &GenericDevice,
// Get global memory as fallback.
auto AllocOrErr = GenericDevice.dataAlloc(
NumBlocks * DynBlockMemSize,
- /*HostPtr=*/nullptr, TargetAllocTy::TARGET_ALLOC_DEVICE);
+ /*HostPtr=*/nullptr, TargetAllocTy::TARGET_ALLOC_DEVICE,
+ /*Alignment=*/0);
if (!AllocOrErr)
return AllocOrErr.takeError();
DynFallbackPtr = *AllocOrErr;
@@ -977,9 +979,11 @@ Error GenericDeviceTy::getDeviceMemorySize(uint64_t &DSize) {
}
Expected<void *> GenericDeviceTy::dataAlloc(int64_t Size, void *HostPtr,
- TargetAllocTy Kind) {
+ TargetAllocTy Kind,
+ size_t Alignment) {
void *Alloc = nullptr;
+ // TODO Check alignment.
if (RecordReplay && RecordReplay->isRecordingOrReplaying())
return RecordReplay->allocate(Size);
@@ -987,7 +991,7 @@ Expected<void *> GenericDeviceTy::dataAlloc(int64_t Size, void *HostPtr,
case TARGET_ALLOC_DEFAULT:
case TARGET_ALLOC_DEVICE:
if (MemoryManager) {
- auto AllocOrErr = MemoryManager->allocate(Size, HostPtr);
+ auto AllocOrErr = MemoryManager->allocate(Size, HostPtr, Alignment);
if (!AllocOrErr)
return AllocOrErr.takeError();
Alloc = *AllocOrErr;
@@ -999,7 +1003,7 @@ Expected<void *> GenericDeviceTy::dataAlloc(int64_t Size, void *HostPtr,
[[fallthrough]];
case TARGET_ALLOC_HOST:
case TARGET_ALLOC_SHARED: {
- auto AllocOrErr = allocate(Size, HostPtr, Kind);
+ auto AllocOrErr = allocate(Size, HostPtr, Kind, Alignment);
if (!AllocOrErr)
return AllocOrErr.takeError();
Alloc = *AllocOrErr;
@@ -1536,8 +1540,8 @@ int32_t GenericPluginTy::load_binary(int32_t DeviceId,
void *GenericPluginTy::data_alloc(int32_t DeviceId, int64_t Size, void *HostPtr,
int32_t Kind) {
- auto AllocOrErr =
- getDevice(DeviceId).dataAlloc(Size, HostPtr, (TargetAllocTy)Kind);
+ auto AllocOrErr = getDevice(DeviceId).dataAlloc(
+ Size, HostPtr, (TargetAllocTy)Kind, /*Alignment=*/0);
if (!AllocOrErr) {
auto Err = AllocOrErr.takeError();
REPORT() << "Failure to allocate device memory: "
diff --git a/offload/plugins-nextgen/cuda/src/rtl.cpp b/offload/plugins-nextgen/cuda/src/rtl.cpp
index 51e2bdb0c01dc..73ca03622cbbb 100644
--- a/offload/plugins-nextgen/cuda/src/rtl.cpp
+++ b/offload/plugins-nextgen/cuda/src/rtl.cpp
@@ -397,6 +397,20 @@ struct CUDADeviceTy : public GenericDeviceTy {
return Err;
MaxBlockSharedMemSize = MaxSharedMem;
+ CUmemAllocationProp Prop = {};
+ Prop.type = CU_MEM_ALLOCATION_TYPE_PINNED;
+ Prop.location.type = CU_MEM_LOCATION_TYPE_DEVICE;
+ Prop.location.id = DeviceId;
+
+ Res = cuMemGetAllocationGranularity(&Granularity, &Prop,
+ CU_MEM_ALLOC_GRANULARITY_MINIMUM);
+ if (auto Err = Plugin::check(
+ Res, "error in cuMemGetAllocationGranularity for the device: %s"))
+ return Err;
+ if (Granularity == 0)
+ return Plugin::error(ErrorCode::INVALID_ARGUMENT,
+ "wrong device page size");
+
return Plugin::success();
}
@@ -586,7 +600,8 @@ struct CUDADeviceTy : public GenericDeviceTy {
}
/// Allocate memory on the device or related to the device.
- Expected<void *> allocate(size_t Size, void *, TargetAllocTy Kind) override {
+ Expected<void *> allocate(size_t Size, void *, TargetAllocTy Kind,
+ size_t Alignment = 0) override {
if (Size == 0)
return nullptr;
@@ -597,6 +612,13 @@ struct CUDADeviceTy : public GenericDeviceTy {
CUdeviceptr DevicePtr;
CUresult Res;
+ if (Alignment > 0 && Alignment > Granularity) {
+ return Plugin::error(ErrorCode::UNSUPPORTED,
+ "requested alignment (%lu) larger than maximum "
+ "supported alignment (%lu)",
+ Alignment, Granularity);
+ }
+
switch (Kind) {
case TARGET_ALLOC_DEFAULT:
case TARGET_ALLOC_DEVICE:
@@ -614,6 +636,12 @@ struct CUDADeviceTy : public GenericDeviceTy {
if (auto Err = Plugin::check(Res, "error in cuMemAlloc[Host|Managed]: %s"))
return std::move(Err);
+
+ if (Alignment > 0 && !isAddrAligned(Align(Alignment), MemAlloc)) {
+ return Plugin::error(ErrorCode::UNSUPPORTED,
+ "unsupported alignment size");
+ }
+
return MemAlloc;
}
@@ -1460,6 +1488,9 @@ struct CUDADeviceTy : public GenericDeviceTy {
/// simultaneously.
uint32_t HardwareParallelism = 0;
+ /// Device page size.
+ size_t Granularity = 0;
+
/// Tracker for virtual address reservations.
VMemTrackerTy<CUmemGenericAllocationHandle> VMemTracker;
};
diff --git a/offload/plugins-nextgen/host/src/rtl.cpp b/offload/plugins-nextgen/host/src/rtl.cpp
index d977a3e0a9793..316e641b3f444 100644
--- a/offload/plugins-nextgen/host/src/rtl.cpp
+++ b/offload/plugins-nextgen/host/src/rtl.cpp
@@ -233,7 +233,8 @@ struct GenELF64DeviceTy : public GenericDeviceTy {
}
/// Allocate memory. Use std::malloc in all cases.
- Expected<void *> allocate(size_t Size, void *, TargetAllocTy Kind) override {
+ Expected<void *> allocate(size_t Size, void *, TargetAllocTy Kind,
+ size_t /* Alignment */) override {
if (Size == 0)
return nullptr;
diff --git a/offload/plugins-nextgen/level_zero/include/L0Device.h b/offload/plugins-nextgen/level_zero/include/L0Device.h
index 275182faebfd6..b49c788bbd7b5 100644
--- a/offload/plugins-nextgen/level_zero/include/L0Device.h
+++ b/offload/plugins-nextgen/level_zero/include/L0Device.h
@@ -484,8 +484,8 @@ class L0DeviceTy final : public GenericDeviceTy {
loadBinaryImpl(std::unique_ptr<MemoryBuffer> &&TgtImage,
int32_t ImageId) override;
Error unloadBinaryImpl(DeviceImageTy *Image) override;
- Expected<void *> allocate(size_t Size, void *HstPtr,
- TargetAllocTy Kind) override;
+ Expected<void *> allocate(size_t Size, void *HstPtr, TargetAllocTy Kind,
+ size_t Alignment) override;
Error free(void *TgtPtr, TargetAllocTy Kind = TARGET_ALLOC_DEFAULT) override;
/// This plugin does nothing to lock buffers. Do not return an error, just
diff --git a/offload/plugins-nextgen/level_zero/src/L0Device.cpp b/offload/plugins-nextgen/level_zero/src/L0Device.cpp
index fd664f33b81e9..a1a6bfd47ee6e 100644
--- a/offload/plugins-nextgen/level_zero/src/L0Device.cpp
+++ b/offload/plugins-nextgen/level_zero/src/L0Device.cpp
@@ -334,8 +334,8 @@ Error L0DeviceTy::queryAsyncImpl(__tgt_async_info &AsyncInfo, bool ReleaseQueue,
}
Expected<void *> L0DeviceTy::allocate(size_t Size, void *HstPtr,
- TargetAllocTy Kind) {
- return dataAlloc(Size, /*Align=*/0, Kind,
+ TargetAllocTy Kind, size_t Alignment) {
+ return dataAlloc(Size, Alignment, Kind,
/*Offset=*/0, /*UserAlloc=*/HstPtr == nullptr,
/*DevMalloc=*/false);
}
@@ -879,8 +879,9 @@ Error L0DeviceTy::callGlobalCtorDtorCommon(GenericPluginTy &Plugin,
llvm::sort(Funcs,
[](const auto &X, const auto &Y) { return X.second < Y.second; });
- auto BufferOrErr = allocate(Funcs.size() * sizeof(void *),
- /*HostPtr=*/nullptr, TARGET_ALLOC_DEVICE);
+ auto BufferOrErr =
+ allocate(Funcs.size() * sizeof(void *),
+ /*HostPtr=*/nullptr, TARGET_ALLOC_DEVICE, /*Alignment=*/0);
if (!BufferOrErr)
return HandleErr(BufferOrErr.takeError());
diff --git a/offload/unittests/OffloadAPI/CMakeLists.txt b/offload/unittests/OffloadAPI/CMakeLists.txt
index e862d8635e0b7..d960da4b98f82 100644
--- a/offload/unittests/OffloadAPI/CMakeLists.txt
+++ b/offload/unittests/OffloadAPI/CMakeLists.txt
@@ -28,6 +28,7 @@ add_offload_unittest("kernel"
add_offload_unittest("memory"
memory/olMemAlloc.cpp
+ memory/olMemAllocAligned.cpp
memory/olMemFill.cpp
memory/olMemFree.cpp
memory/olMemcpy.cpp
diff --git a/offload/unittests/OffloadAPI/memory/olMemAllocAligned.cpp b/offload/unittests/OffloadAPI/memory/olMemAllocAligned.cpp
new file mode 100644
index 0000000000000..18feafbf2b325
--- /dev/null
+++ b/offload/unittests/OffloadAPI/memory/olMemAllocAligned.cpp
@@ -0,0 +1,211 @@
+//===--------------- Offload API tests - olMemAllocAligned ----------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "../common/Fixtures.hpp"
+#include <OffloadAPI.h>
+#include <gtest/gtest.h>
+
+using olMemAllocAlignedTest = OffloadDeviceTest;
+
+OFFLOAD_TESTS_INSTANTIATE_DEVICE_FIXTURE(olMemAllocAlignedTest);
+
+constexpr size_t DefaultAlignment = 16;
+constexpr size_t TestAllocsNum = 1000;
+
+TEST_P(olMemAllocAlignedTest, SuccessAllocMany) {
+ std::vector<void *> Allocs;
+ Allocs.reserve(1000);
+
+ constexpr ol_alloc_type_t TYPES[3] = {
+ OL_ALLOC_TYPE_DEVICE, OL_ALLOC_TYPE_MANAGED, OL_ALLOC_TYPE_HOST};
+
+ for (size_t I = 1; I < TestAllocsNum; I++) {
+ void *Alloc = nullptr;
+ ASSERT_SUCCESS(olMemAllocAligned(Device, TYPES[I % 3], 1024 * I,
+ DefaultAlignment, &Alloc));
+ ASSERT_NE(Alloc, nullptr);
+
+ Allocs.push_back(Alloc);
+ }
+
+ for (auto *A : Allocs) {
+ olMemFree(A);
+ }
+}
+
+TEST_P(olMemAllocAlignedTest, InvalidNullDevice) {
+ void *Alloc = nullptr;
+ ASSERT_ERROR(OL_ERRC_INVALID_NULL_HANDLE,
+ olMemAllocAligned(nullptr, OL_ALLOC_TYPE_DEVICE, 1024,
+ DefaultAlignment, &Alloc));
+}
+
+TEST_P(olMemAllocAlignedTest, InvalidNullOutPtr) {
+ ASSERT_ERROR(OL_ERRC_INVALID_NULL_POINTER,
+ olMemAllocAligned(Device, OL_ALLOC_TYPE_DEVICE, 1024,
+ DefaultAlignment, nullptr));
+}
+
+TEST_P(olMemAllocAlignedTest, InvalidAlignmentZero) {
+ void *Alloc = nullptr;
+
+ ASSERT_ERROR(
+ OL_ERRC_INVALID_ARGUMENT,
+ olMemAllocAligned(Device, OL_ALLOC_TYPE_DEVICE, 1024, 0, &Alloc));
+}
+
+TEST_P(olMemAllocAlignedTest, InvalidAlignmentNotAPowerOfTwo) {
+ void *Alloc = nullptr;
+
+ ASSERT_ERROR(
+ OL_ERRC_INVALID_ARGUMENT,
+ olMemAllocAligned(Device, OL_ALLOC_TYPE_DEVICE, 1024, 3, &Alloc));
+}
+
+TEST_P(olMemAllocAlignedTest, CudaExceedDefaultAlignment) {
+ if (getPlatformBackend() != OL_PLATFORM_BACKEND_CUDA) {
+ GTEST_SKIP() << "Test inteded for CUDA backend";
+ }
+
+ void *Alloc = nullptr;
+ // The default page size for cuda is 64 KB.
+ ASSERT_ERROR(OL_ERRC_UNSUPPORTED,
+ olMemAllocAligned(Device, OL_ALLOC_TYPE_DEVICE, 1024,
+ 1024 * 64 * 64, &Alloc));
+ ASSERT_EQ(Alloc, nullptr);
+}
+
+TEST_P(olMemAllocAlignedTest, SuccessAllocManagedDifferentAlignments) {
+ void *Alloc = nullptr;
+ size_t NumAlignments = 6;
+ size_t Alignments[] = {8, 16, 32, 64, 128, 256};
+ size_t Alignment;
+
+ for (size_t i = 0; i < NumAlignments; i++) {
+ Alignment = Alignments[i];
+ SCOPED_TRACE("alignment: " + std::to_string(Alignment));
+ ASSERT_SUCCESS(olMemAllocAligned(Device, OL_ALLOC_TYPE_MANAGED, 1024,
+ Alignment, &Alloc));
+ ASSERT_NE(Alloc, nullptr);
+ olMemFree(Alloc);
+ }
+}
+
+TEST_P(olMemAllocAlignedTest, SuccessAllocHostDifferentAlignments) {
+ void *Alloc = nullptr;
+ size_t NumAlignments = 6;
+ size_t Alignments[] = {8, 16, 32, 64, 128, 256};
+ size_t Alignment;
+
+ for (size_t i = 0; i < NumAlignments; i++) {
+ Alignment = Alignments[i];
+ SCOPED_TRACE("alignment: " + std::to_string(Alignment));
+ ASSERT_SUCCESS(
+ olMemAllocAligned(Device, OL_ALLOC_TYPE_HOST, 1024, Alignment, &Alloc));
+ ASSERT_NE(Alloc, nullptr);
+ olMemFree(Alloc);
+ }
+}
+
+TEST_P(olMemAllocAlignedTest, SuccessAllocDeviceDifferentAlignments) {
+ void *Alloc = nullptr;
+ size_t NumAlignments = 6;
+ size_t Alignments[] = {8, 16, 32, 64, 128, 256};
+ size_t Alignment;
+
+ for (size_t i = 0; i < NumAlignments; i++) {
+ Alignment = Alignments[i];
+ SCOPED_TRACE("alignment: " + std::to_string(Alignment));
+ ASSERT_SUCCESS(olMemAllocAligned(Device, OL_ALLOC_TYPE_DEVICE, 1024,
+ Alignment, &Alloc));
+ ASSERT_NE(Alloc, nullptr);
+
+ olMemFree(Alloc);
+ }
+}
+
+TEST_P(olMemAllocAlignedTest, SuccessMemcpyManagedDiferentAlignments) {
+ constexpr size_t Size = 1024;
+ void *Alloc;
+ std::vector<uint8_t> Input(Size, 42);
+ std::vector<uint8_t> Output(Size, 0);
+
+ size_t NumAlignments = 6;
+ size_t Alignments[] = {8, 16, 32, 64, 128, 256};
+ size_t Alignment;
+ for (size_t i = 0; i < NumAlignments; i++) {
+ Alignment = Alignments[i];
+ SCOPED_TRACE("alignment: " + std::to_string(Alignment));
+
+ ASSERT_SUCCESS(olMemAllocAligned(Device, OL_ALLOC_TYPE_MANAGED, Size,
+ Alignment, &Alloc));
+ // memcpy is synchronous when queue is unspecified.
+ ASSERT_SUCCESS(olMemcpy(nullptr, Alloc, Device, Input.data(), Host, Size));
+ ASSERT_SUCCESS(olMemcpy(nullptr, Output.data(), Host, Alloc, Device, Size));
+
+ for (uint8_t Val : Output) {
+ ASSERT_EQ(Val, 42);
+ }
+
+ ASSERT_SUCCESS(olMemFree(Alloc));
+ }
+}
+
+TEST_P(olMemAllocAlignedTest, SuccessMemcpyDeviceDiferentAlignments) {
+ constexpr size_t Size = 1024;
+ void *Alloc;
+ std::vector<uint8_t> Input(Size, 42);
+ std::vector<uint8_t> Output(Size, 0);
+
+ size_t NumAlignments = 6;
+ size_t Alignments[] = {8, 16, 32, 64, 128, 256};
+ size_t Alignment;
+ for (size_t i = 0; i < NumAlignments; i++) {
+ Alignment = Alignments[i];
+ SCOPED_TRACE("alignment: " + std::to_string(Alignment));
+
+ ASSERT_SUCCESS(olMemAllocAligned(Device, OL_ALLOC_TYPE_DEVICE, Size,
+ Alignment, &Alloc));
+ // memcpy is synchronous when queue is unspecified.
+ ASSERT_SUCCESS(olMemcpy(nullptr, Alloc, Device, Input.data(), Host, Size));
+ ASSERT_SUCCESS(olMemcpy(nullptr, Output.data(), Host, Alloc, Device, Size));
+
+ for (uint8_t Val : Output) {
+ ASSERT_EQ(Val, 42);
+ }
+
+ ASSERT_SUCCESS(olMemFree(Alloc));
+ }
+}
+
+TEST_P(olMemAllocAlignedTest, SuccessMemcpyHostDiferentAlignments) {
+ constexpr size_t Size = 1024;
+ void *Alloc;
+ std::vector<uint8_t> Input(Size, 42);
+ std::vector<uint8_t> Output(Size, 0);
+
+ size_t NumAlignments = 6;
+ size_t Alignments[] = {8, 16, 32, 64, 128, 256};
+ size_t Alignment;
+ for (size_t i = 0; i < NumAlignments; i++) {
+ Alignment = Alignments[i];
+ SCOPED_TRACE("alignment: " + std::to_string(Alignment));
+
+ ASSERT_SUCCESS(
+ olMemAllocAligned(Device, OL_ALLOC_TYPE_HOST, Size, Alignment, &Alloc));
+ // memcpy is synchronous when queue is unspecified.
+ ASSERT_SUCCESS(olMemcpy(nullptr, Alloc, Device, Input.data(), Host, Size));
+ ASSERT_SUCCESS(olMemcpy(nullptr, Output.data(), Host, Alloc, Device, Size));
+
+ for (uint8_t Val : Output) {
+ ASSERT_EQ(Val, 42);
+ }
+
+ ASSERT_SUCCESS(olMemFree(Alloc));
+ }
+}
More information about the llvm-commits
mailing list