[docs][AMDGPU] move DMA operations to a separate file (#206917)
This includes the various `.async` and `.tensor` operations. Various scattered bits of information are now in one place. Also added the missing documentation for tensor intrinsics.
This commit is contained in:
Sameer Sahasrabuddhe authored and 
GitHub committed
1 parent
29fa6a8247
commit
deecf8dce7
4 files changed
+189
-75
No files matched your search
@@ -10,52 +10,14 @@
|
||||
Introduction
|
||||
============
|
||||
|
||||
Asynchronous operations are operations that are completed independently at an
|
||||
unspecified scope. A thread that initiates one or more async operations can use
|
||||
Asynchronous operations are operations whose completion is not tracked
|
||||
internally by the compiler. A thread that initiates one or more async operations can use
|
||||
*asyncmarks* to track their completion.
|
||||
|
||||
Operations
|
||||
==========
|
||||
|
||||
Async Instructions
|
||||
------------------
|
||||
|
||||
The following instructions initiate async operations that transfer data between
|
||||
global memory and LDS memory.
|
||||
|
||||
.. note::
|
||||
|
||||
These listings are *merely representative*. The actual function signatures
|
||||
and supported architectures are documented in the :ref:`amdgpu-usage-guide`.
|
||||
|
||||
**GFX9 Async Instructions (LDS DMA)**
|
||||
|
||||
.. code-block:: llvm
|
||||
|
||||
void @llvm.amdgcn.load.async.to.lds(ptr %src, ptr %dst)
|
||||
void @llvm.amdgcn.global.load.async.lds(ptr %src, ptr %dst)
|
||||
void @llvm.amdgcn.raw.buffer.load.async.lds(ptr %src, ptr %dst)
|
||||
void @llvm.amdgcn.raw.ptr.buffer.load.async.lds(ptr %src, ptr %dst)
|
||||
void @llvm.amdgcn.struct.buffer.load.async.lds(ptr %src, ptr %dst)
|
||||
void @llvm.amdgcn.struct.ptr.buffer.load.async.lds(ptr %src, ptr %dst)
|
||||
|
||||
**GFX12 Async Instructions**
|
||||
|
||||
.. code-block:: llvm
|
||||
|
||||
void @llvm.amdgcn.global.load.async.to.lds.type(ptr %dst, ptr %src)
|
||||
void @llvm.amdgcn.global.store.async.from.lds.type(ptr %dst, ptr %src)
|
||||
void @llvm.amdgcn.cluster.load.async.to.lds.type(ptr %dst, ptr %src)
|
||||
|
||||
**GFX1250 Tensor DMA Instructions**
|
||||
|
||||
.. code-block:: llvm
|
||||
|
||||
void @llvm.amdgcn.tensor.load.to.lds(...)
|
||||
void @llvm.amdgcn.tensor.store.from.lds(...)
|
||||
- Most :ref:`DMA operations<amdgpu-dma-operations>` are asynchronous.
|
||||
|
||||
Asyncmarks
|
||||
----------
|
||||
==========
|
||||
|
||||
An *asyncmark* created by a thread can be used to track async operations
|
||||
initiated by that thread. The abstract machine maintains a sequence of
|
||||
@@ -65,18 +27,20 @@ executing function. The state of this sequence at each program point in the
|
||||
function is called the *current sequence*.
|
||||
|
||||
``@llvm.amdgcn.asyncmark()``
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
----------------------------
|
||||
|
||||
Produces an asyncmark and appends it to the current sequence.
|
||||
|
||||
``@llvm.amdgcn.wait.asyncmark(i16 %N)``
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
---------------------------------------
|
||||
|
||||
Ensures that the length of the current sequence is at most ``N`` by removing
|
||||
asyncmarks from the start of the sequence if it is more than ``N``.
|
||||
|
||||
Memory Consistency Model
|
||||
========================
|
||||
.. _amdgpu-asyncmark-memory-model:
|
||||
|
||||
Memory Model
|
||||
============
|
||||
|
||||
An ``asyncmark()`` operation ``X`` that produces an asyncmark ``M`` is
|
||||
*completed-at* a ``wait.asyncmark()`` operation ``Y`` in the same function body
|
||||
|
||||
@@ -0,0 +1,146 @@
|
||||
(amdgpu-dma-operations)=
|
||||
|
||||
# AMDGPU DMA Operations
|
||||
|
||||
```{contents}
|
||||
:local:
|
||||
```
|
||||
|
||||
## Introduction
|
||||
|
||||
DMA (or "Direct Memory Access") operations transfer data between different kinds
|
||||
of memory directly without occupying registers in the invoking wave. They are
|
||||
usually {ref}`asynchronous<amdgpu-async-operations>` asynchronous, and require
|
||||
the user to explicitly track completion using
|
||||
{ref}`asyncmarks<amdgpu-async-operations>`.
|
||||
|
||||
All DMA operations support the same cache modifiers as ordinary load/store
|
||||
operations from registers. They cannot be performed atomically.
|
||||
|
||||
### GFX9 DMA
|
||||
|
||||
Each GFX9 DMA instruction has a synchronous counterpart (e.g.,
|
||||
``@llvm.amdgcn.load.to.lds`` for ``@llvm.amdgcn.load.async.to.lds``). The
|
||||
synchronous variants perform the same operation, but the compiler automatically
|
||||
ensures completion before their side-effects are used.
|
||||
|
||||
GFX9 DMA instructions implement volatile (via ``aux/cpol`` bit 31) and
|
||||
nontemporal (via metadata) as if they were loads from the global address space.
|
||||
|
||||
**Flat/Global Addressing**
|
||||
|
||||
```llvm
|
||||
void @llvm.amdgcn.load[.async].to.lds.pN(
|
||||
ptr addrspace(N) %src, ; base pointer to load from (per-lane)
|
||||
ptr addrspace(3) %lds_base, ; LDS base pointer (wave-uniform)
|
||||
i32 immarg %size, ; data byte size: 1/2/4 (12/16 for gfx950)
|
||||
i32 immarg %offset, ; offset applied to both src and LDS address
|
||||
i32 immarg %cpol) ; cache policy
|
||||
```
|
||||
|
||||
Loads data from global memory to LDS. The data size can be 1, 2, or 4 bytes
|
||||
(gfx950 also allows 12 or 16 bytes). The LDS address is implicitly offset by
|
||||
``4 * lane_id`` bytes for sizes up to 4 bytes, and by ``16 * lane_id`` bytes
|
||||
for larger sizes.
|
||||
|
||||
The ``%lds_base`` pointer must be wave-uniform.
|
||||
|
||||
The source pointer is overloaded on address space. Supported address spaces are
|
||||
flat (0), global (1), and buffer fat pointer (7).
|
||||
|
||||
``@llvm.amdgcn.load[.async].to.lds.p7`` (buffer pointer) is lowered to
|
||||
``@llvm.amdgcn.raw.ptr.buffer.load[.async].lds`` before instruction selection.
|
||||
|
||||
```llvm
|
||||
void @llvm.amdgcn.global.load[.async].lds(
|
||||
ptr addrspace(1) %src, ; global base pointer to load from (per-lane)
|
||||
ptr addrspace(3) %lds_base, ; LDS base pointer (wave-uniform)
|
||||
i32 immarg %size, ; data byte size: 1/2/4 (12/16 for gfx950)
|
||||
i32 immarg %offset, ; offset applied to both global and LDS address
|
||||
i32 immarg %cpol) ; cache policy
|
||||
```
|
||||
|
||||
This is identical to ``@llvm.amdgcn.load[.async].to.lds.p1``.
|
||||
|
||||
**Buffer Addressing**
|
||||
|
||||
```llvm
|
||||
void @llvm.amdgcn.{raw|struct}[.ptr].buffer.load[.async].lds(
|
||||
%rsrc, ; buffer resource descriptor (SGPR):
|
||||
; <4 x i32> or ptr addrspace(8)
|
||||
ptr addrspace(3) %lds_base, ; LDS base pointer (wave-uniform)
|
||||
i32 immarg %size, ; data byte size: 1/2/4 (12/16 for gfx950)
|
||||
[i32 %vindex,] ; VGPR buffer index (struct variants only)
|
||||
i32 %voffset, ; VGPR offset (included in bounds checking)
|
||||
i32 %soffset, ; SGPR/imm offset (excluded from bounds checking)
|
||||
i32 immarg %offset, ; imm offset (included in bounds checking)
|
||||
i32 immarg %cpol) ; cache policy
|
||||
```
|
||||
|
||||
Loads data from a buffer resource to LDS.
|
||||
|
||||
The ``%lds_base`` pointer must be wave-uniform.
|
||||
|
||||
The intrinsics differ in two orthogonal ways:
|
||||
|
||||
- **raw** vs **struct**: The ``struct`` variants add a ``%vindex`` argument for
|
||||
indexed buffer addressing.
|
||||
- **ptr** vs non-ptr: The ``ptr`` variants use ``ptr addrspace(8)`` for the
|
||||
buffer resource descriptor; the non-ptr variants use ``<4 x i32>``.
|
||||
|
||||
### GFX1250
|
||||
|
||||
GFX1250 LDS DMA instructions implement nontemporal (via metadata) as if they
|
||||
were loads from the global address space. Tensor DMA instructions do not support
|
||||
volatile or nontemporal.
|
||||
|
||||
**Global Addressing**
|
||||
|
||||
```llvm
|
||||
void @llvm.amdgcn.{global|cluster}.load.async.to.lds.b<N>(
|
||||
ptr addrspace(1) %src, ; global base pointer to load from (per-lane)
|
||||
ptr addrspace(3) %lds_base, ; LDS base pointer (per-lane)
|
||||
i32 immarg %offset, ; offset applied to both global and LDS address
|
||||
i32 immarg %cpol, ; cache policy
|
||||
[i32 %m0]) ; workgroup broadcast mask, cluster variants only (in M0)
|
||||
```
|
||||
|
||||
The bit-size encoded in the name can be 8, 32, 64 or 128.
|
||||
|
||||
Loads data from global memory to LDS. The ``%offset`` is applied to both the
|
||||
global and LDS addresses.
|
||||
|
||||
The ``cluster`` variants add a ``%m0`` argument for workgroup broadcast. The
|
||||
broadcast mask selects which workgroups within a cluster participate in the load.
|
||||
|
||||
```llvm
|
||||
void @llvm.amdgcn.global.store.async.from.lds.b<N>(
|
||||
ptr addrspace(1) %dst, ; global base pointer to store to (per-lane)
|
||||
ptr addrspace(3) %lds_base, ; LDS base pointer to load from (per-lane)
|
||||
i32 immarg %offset, ; offset applied to both global and LDS address
|
||||
i32 immarg %cpol) ; cache policy
|
||||
```
|
||||
|
||||
Stores data from LDS to global memory.
|
||||
|
||||
**Tensor Addressing**
|
||||
|
||||
```llvm
|
||||
void @llvm.amdgcn.tensor.{load.to|store.from}.lds(
|
||||
<4 x i32> %desc0, ; D# group 0
|
||||
<8 x i32> %desc1, ; D# group 1
|
||||
<4 x i32> %desc2, ; D# group 2 (zero-init for D# up to 2D)
|
||||
<4 x i32> %desc3, ; D# group 3 (zero-init for D# up to 2D)
|
||||
<8 x i32> %desc4, ; D# group 4 (reserved, use zeroinitializer)
|
||||
i32 immarg %cpol) ; cache policy
|
||||
```
|
||||
|
||||
Loads or stores data between global memory and LDS using a tensor descriptor
|
||||
(D#). The descriptor is split across multiple groups. GFX1250 supports up to 4
|
||||
descriptor groups; ``%desc4`` is reserved for future targets and must be
|
||||
zero-initialized.
|
||||
|
||||
Despite the absence of ``.async`` in their names, these intrinsics are
|
||||
asynchronous.
|
||||
|
||||
All arguments must be wave-uniform.
|
||||
+26
-28
@@ -1520,21 +1520,6 @@ The AMDGPU backend implements the following LLVM IR intrinsics.
|
||||
The format is a 64-bit concatenation of the MODE and TRAPSTS registers.
|
||||
|
||||
:ref:`llvm.set.fpenv<int_set_fpenv>` Sets the floating point environment to the specified state.
|
||||
llvm.amdgcn.load.to.lds.p<1/7> Loads values from global memory (either in the form of a global
|
||||
a raw fat buffer pointer) to LDS. The size of the data copied can be 1, 2,
|
||||
or 4 bytes (and gfx950 also allows 12 or 16 bytes). The LDS pointer
|
||||
argument should be wavefront-uniform; the global pointer need not be.
|
||||
The LDS pointer is implicitly offset by 4 * lane_id bytes for size <= 4 bytes
|
||||
and 16 * lane_id bytes for larger sizes. This lowers to `global_load_lds`,
|
||||
`buffer_load_* ... lds`, or `global_load__* ... lds` depending on address
|
||||
space and architecture. `amdgcn.global.load.lds` has the same semantics as
|
||||
`amdgcn.load.to.lds.p1`.
|
||||
|
||||
llvm.amdgcn.load.async.to.lds.p<1/7> Same as `llvm.amdgcn.load.to.lds.p<1/7>`, but the completion of this
|
||||
:ref:`asynchronous version<amdgpu-async-operations>` is not automatically tracked
|
||||
by the compiler. The user must explicitly track the completion with `asyncmark`
|
||||
operations before using their side-effects.
|
||||
|
||||
llvm.amdgcn.readfirstlane Provides direct access to v_readfirstlane_b32. Returns the value in
|
||||
the lowest active lane of the input operand. Currently implemented
|
||||
for i16, i32, float, half, bfloat, <2 x i16>, <2 x half>, <2 x bfloat>,
|
||||
@@ -2097,6 +2082,31 @@ The 3rd operand for ``.store`` or 2nd for ``.load`` intrinsics is the
|
||||
The last operand of the intrinsic is the
|
||||
:ref:`synchronization scope<amdgpu-intrinsics-syncscope-metadata-operand>` of the operation.
|
||||
|
||||
DMA Intrinsics
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
DMA intrinsics transfer data between global memory and LDS without occupying
|
||||
registers. See :ref:`amdgpu-dma-operations` for full documentation.
|
||||
|
||||
::
|
||||
|
||||
llvm.amdgcn.load[.async].to.lds
|
||||
llvm.amdgcn.global.load[.async].lds
|
||||
llvm.amdgcn.{raw|struct}[.ptr].buffer.load[.async].lds
|
||||
llvm.amdgcn.{global|cluster}.load.async.to.lds.b{8,32,64,128}
|
||||
llvm.amdgcn.global.store.async.from.lds.b{8,32,64,128}
|
||||
|
||||
Tensor Intrinsics
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
Tensor intrinsics transfer data between global memory and LDS using a tensor
|
||||
descriptor. Despite the absence of ``.async`` in their names, these intrinsics
|
||||
are asynchronous. See :ref:`amdgpu-dma-operations` for full documentation.
|
||||
|
||||
::
|
||||
|
||||
llvm.amdgcn.tensor.{load.to|store.from}.lds
|
||||
|
||||
Intrinsic Operands
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
@@ -7204,19 +7214,7 @@ operations.
|
||||
termed vector memory operations.
|
||||
|
||||
``global_load_lds`` or ``buffer/global_load`` instructions with the `lds` flag
|
||||
are LDS DMA loads. They interact with caches as if the loaded data were
|
||||
being loaded to registers and not to LDS, and so therefore support the same
|
||||
cache modifiers. They cannot be performed atomically. They implement volatile
|
||||
(via aux/cpol bit 31) and nontemporal (via metadata) as if they were loads
|
||||
from the global address space.
|
||||
|
||||
The LDS DMA instructions are synchronous by default, which means that the
|
||||
compiler will automatically ensure that the corresponding operation has
|
||||
completed before its side-effects are used. The :ref:`asynchronous
|
||||
versions<amdgpu-async-operations>` of these same instructions perform the same
|
||||
operations, but without automatic tracking in the compiler; the user must
|
||||
explicitly track the completion of these instructions before using their
|
||||
side-effects.
|
||||
are :ref:`LDS DMA operations<amdgpu-dma-operations>`.
|
||||
|
||||
Private address space uses ``buffer_load/store`` using the scratch V#
|
||||
(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX11). Since only a single thread
|
||||
|
||||
@@ -63,6 +63,7 @@ CompileCudaWithLLVM
|
||||
NVPTXUsage
|
||||
AMDGPUUsage
|
||||
AMDGPUAsyncOperations
|
||||
AMDGPUDMAOperations
|
||||
AMDGPUExecutionSynchronization
|
||||
AMDGPUMemoryModel
|
||||
SPIRVUsage
|
||||
@@ -338,7 +339,12 @@ yaml2obj
|
||||
|
||||
- {doc}`AMDGPUAsyncOperations`
|
||||
|
||||
Builtins for invoking asynchronous data transfer operations in AMD GPUs.
|
||||
Asynchronous operations and the asyncmark mechanism for tracking their
|
||||
completion on AMD GPUs.
|
||||
|
||||
- {doc}`AMDGPUDMAOperations`
|
||||
|
||||
DMA operations that transfer data between global memory and LDS on AMD GPUs.
|
||||
|
||||
- {doc}`AMDGPUMemoryModel`
|
||||
|
||||
|
||||
Reference in new issue
Block a user