[all-commits] [llvm/llvm-project] 028969: [OpenMP] Introduce target memory manager

[Shilei Tian via All-commits]

  Branch: refs/heads/master
  Home:   https://github.com/llvm/llvm-project
  Commit: 0289696751e9a959b2413ca26624fc6c91be1eea
      https://github.com/llvm/llvm-project/commit/0289696751e9a959b2413ca26624fc6c91be1eea
  Author: Shilei Tian <tianshilei1992 at gmail.com>
  Date:   2020-08-19 (Wed, 19 Aug 2020)

  Changed paths:
    M openmp/libomptarget/src/CMakeLists.txt
    A openmp/libomptarget/src/MemoryManager.cpp
    A openmp/libomptarget/src/MemoryManager.h
    M openmp/libomptarget/src/device.cpp
    M openmp/libomptarget/src/device.h
    A openmp/libomptarget/test/offloading/memory_manager.cpp

  Log Message:
  -----------
  [OpenMP] Introduce target memory manager

Target memory manager is introduced in this patch which aims to manage target
memory such that they will not be freed immediately when they are not used
because the overhead of memory allocation and free is very large. For CUDA
device, cuMemFree even blocks the context switch on device which affects
concurrent kernel execution.

The memory manager can be taken as a memory pool. It divides the pool into
multiple buckets according to the size such that memory allocation/free
distributed to different buckets will not affect each other.

In this version, we use the exact-equality policy to find a free buffer. This
is an open question: will best-fit work better here? IMO, best-fit is not good
for target memory management because computation on GPU usually requires GBs of
data. Best-fit might lead to a serious waste. For example, there is a free
buffer of size 1960MB, and now we need a buffer of size 1200MB. If best-fit,
the free buffer will be returned, leading to a 760MB waste.

The allocation will happen when there is no free memory left, and the memory
free on device will take place in the following two cases:
1. The program ends. Obviously. However, there is a little problem that plugin
library is destroyed before the memory manager is destroyed, leading to a fact
that the call to target plugin will not succeed.
2. Device is out of memory when we request a new memory. The manager will walk
through all free buffers from the bucket with largest base size, pick up one
buffer, free it, and try to allocate immediately. If it succeeds, it will
return right away rather than freeing all buffers in free list.

Update:
A threshold (8KB by default) is set such that users could control what size of memory
will be managed by the manager. It can also be configured by an environment variable
`LIBOMPTARGET_MEMORY_MANAGER_THRESHOLD`.

Reviewed By: jdoerfert, ye-luo, JonChesterfield

Differential Revision: https://reviews.llvm.org/D81054