[llvm] cddf206 - [docs] Rewrite 19 LLVM docs from reST to markdown (#208798)
via llvm-commits
llvm-commits at lists.llvm.org
Fri Jul 17 21:23:27 PDT 2026
Author: Reid Kleckner
Date: 2026-07-18T04:23:20Z
New Revision: cddf2069ded94cb47048ad2a5f340d12c35f1651
URL: https://github.com/llvm/llvm-project/commit/cddf2069ded94cb47048ad2a5f340d12c35f1651
DIFF: https://github.com/llvm/llvm-project/commit/cddf2069ded94cb47048ad2a5f340d12c35f1651.diff
LOG: [docs] Rewrite 19 LLVM docs from reST to markdown (#208798)
Tracking issue: #201242
[Migration
guide](https://llvm.org/docs/SphinxQuickstartTemplate.html#markdown-migration-guidelines)
This is a stacked PR based on #208800, which does the file rename to
preserve history.
This was prepared with rst2myst plus LLM-assisted cleanup. I paged
through all the generated HTML looking for migration artifacts, and all
of the differences I could find appear to be formatting error
corrections.
Added:
Modified:
llvm/docs/AArch64SME.md
llvm/docs/AMDGPUAsyncOperations.md
llvm/docs/AMDGPUExecutionSynchronization.md
llvm/docs/AMDGPUMemoryModel.md
llvm/docs/AddingConstrainedIntrinsics.md
llvm/docs/BigEndianNEON.md
llvm/docs/CompileCudaWithLLVM.md
llvm/docs/DebuggingJITedCode.md
llvm/docs/ExtendingLLVM.md
llvm/docs/HowToBuildWindowsItaniumPrograms.md
llvm/docs/HowToCrossCompileBuiltinsOnArm.md
llvm/docs/HowToUpdateDebugInfo.md
llvm/docs/Instrumentor.md
llvm/docs/JITLink.md
llvm/docs/MCJITDesignAndImplementation.md
llvm/docs/NVPTXUsage.md
llvm/docs/ORCv2.md
llvm/docs/Remarks.md
llvm/docs/SPIRVUsage.md
Removed:
################################################################################
diff --git a/llvm/docs/AArch64SME.md b/llvm/docs/AArch64SME.md
index d633dc2fbce2d..24292559f2f1c 100644
--- a/llvm/docs/AArch64SME.md
+++ b/llvm/docs/AArch64SME.md
@@ -1,16 +1,14 @@
-*****************************************************
-Support for AArch64 Scalable Matrix Extension in LLVM
-*****************************************************
+# Support for AArch64 Scalable Matrix Extension in LLVM
-.. contents::
- :local:
+```{contents}
+:local: true
+```
-1. Introduction
-===============
+## 1. Introduction
-The :ref:`AArch64 SME ACLE <aarch64_sme_acle>` provides a number of
+The {ref}`AArch64 SME ACLE <aarch64_sme_acle>` provides a number of
attributes for users to control PSTATE.SM and PSTATE.ZA.
-The :ref:`AArch64 SME ABI<aarch64_sme_abi>` describes the requirements for
+The {ref}`AArch64 SME ABI <aarch64_sme_abi>` describes the requirements for
calls between functions when at least one of those functions uses PSTATE.SM or
PSTATE.ZA.
@@ -21,36 +19,46 @@ requirements of the ABI.
Below, we describe the LLVM IR attributes and their relation to the
C/C++-level ACLE attributes:
-``aarch64_pstate_sm_enabled``
- is used for functions with ``__arm_streaming``
+`aarch64_pstate_sm_enabled`
-``aarch64_pstate_sm_compatible``
- is used for functions with ``__arm_streaming_compatible``
+: is used for functions with `__arm_streaming`
-``aarch64_pstate_sm_body``
- is used for functions with ``__arm_locally_streaming`` and is
+`aarch64_pstate_sm_compatible`
+
+: is used for functions with `__arm_streaming_compatible`
+
+`aarch64_pstate_sm_body`
+
+: is used for functions with `__arm_locally_streaming` and is
only valid on function definitions (not declarations)
-``aarch64_new_za``
- is used for functions with ``__arm_new("za")``
+`aarch64_new_za`
+
+: is used for functions with `__arm_new("za")`
+
+`aarch64_in_za`
+
+: is used for functions with `__arm_in("za")`
-``aarch64_in_za``
- is used for functions with ``__arm_in("za")``
+`aarch64_out_za`
-``aarch64_out_za``
- is used for functions with ``__arm_out("za")``
+: is used for functions with `__arm_out("za")`
-``aarch64_inout_za``
- is used for functions with ``__arm_inout("za")``
+`aarch64_inout_za`
-``aarch64_preserves_za``
- is used for functions with ``__arm_preserves("za")``
+: is used for functions with `__arm_inout("za")`
-``aarch64_expanded_pstate_za``
- is used for functions with ``__arm_new_za``
+`aarch64_preserves_za`
-``aarch64_zt0_undef``
- Deprecated. Previously used internally to prevent spills/reloads of ZT0 in
+: is used for functions with `__arm_preserves("za")`
+
+`aarch64_expanded_pstate_za`
+
+: is used for functions with `__arm_new_za`
+
+`aarch64_zt0_undef`
+
+: Deprecated. Previously used internally to prevent spills/reloads of ZT0 in
some cases.
Clang must ensure that the above attributes are added both to the
@@ -58,18 +66,14 @@ function's declaration/definition as well as to their call-sites. This is
important for calls to attributed function pointers, where no
definition or declaration is available.
-
-2. Handling PSTATE.SM
-=====================
+## 2. Handling PSTATE.SM
When changing PSTATE.SM the execution of FP/vector operations may be transferred
to another processing element. This has three important implications:
-* The runtime SVE vector length may change.
-
-* The contents of FP/AdvSIMD/SVE registers are zeroed.
-
-* The set of allowable instructions changes.
+- The runtime SVE vector length may change.
+- The contents of FP/AdvSIMD/SVE registers are zeroed.
+- The set of allowable instructions changes.
This leads to certain restrictions on IR and optimizations. For example, it
is undefined behaviour to share vector-length dependent state between functions
@@ -78,8 +82,8 @@ these restrictions when generating LLVM IR.
Even though the runtime SVE vector length may change, for the purpose of LLVM IR
and almost all parts of CodeGen we can assume that the runtime value for
-``vscale`` does not. If we let the compiler insert the appropriate ``smstart``
-and ``smstop`` instructions around call boundaries, then the effects on SVE
+`vscale` does not. If we let the compiler insert the appropriate `smstart`
+and `smstop` instructions around call boundaries, then the effects on SVE
state can be mitigated. By limiting the state changes to a very brief window
around the call, we can control how the operations are scheduled and how live
values remain preserved between state transitions.
@@ -87,73 +91,64 @@ values remain preserved between state transitions.
In order to control PSTATE.SM at this level of granularity, we use function and
callsite attributes rather than intrinsics.
+### Restrictions on attributes
-Restrictions on attributes
---------------------------
-
-* It is undefined behaviour to pass or return (pointers to) scalable vector
+- It is undefined behaviour to pass or return (pointers to) scalable vector
objects to/from functions which may use a
diff erent SVE vector length.
This includes functions with a non-streaming interface but marked with
- ``aarch64_pstate_sm_body``.
-
-* It is not allowed for a function to be decorated with both
- ``aarch64_pstate_sm_compatible`` and ``aarch64_pstate_sm_enabled``.
-
-* It is not allowed for a function to be decorated with more than one of the
+ `aarch64_pstate_sm_body`.
+- It is not allowed for a function to be decorated with both
+ `aarch64_pstate_sm_compatible` and `aarch64_pstate_sm_enabled`.
+- It is not allowed for a function to be decorated with more than one of the
following attributes:
- ``aarch64_new_za``, ``aarch64_in_za``, ``aarch64_out_za``, ``aarch64_inout_za``,
- ``aarch64_preserves_za``.
+ `aarch64_new_za`, `aarch64_in_za`, `aarch64_out_za`, `aarch64_inout_za`,
+ `aarch64_preserves_za`.
These restrictions also apply in the higher-level SME ACLE, which means we can
emit diagnostics in Clang to signal users about incorrect behaviour.
-
-Compiler inserted streaming-mode changes
-----------------------------------------
+### Compiler inserted streaming-mode changes
The table below describes the transitions in PSTATE.SM the compiler has to
account for when doing calls between functions with
diff erent attributes.
In this table, we use the following abbreviations:
-``N``
- functions with a normal interface (PSTATE.SM=0 on entry, PSTATE.SM=0 on
+`N`
+
+: functions with a normal interface (PSTATE.SM=0 on entry, PSTATE.SM=0 on
return)
-``S``
- functions with a Streaming interface (PSTATE.SM=1 on entry, PSTATE.SM=1
+`S`
+
+: functions with a Streaming interface (PSTATE.SM=1 on entry, PSTATE.SM=1
on return)
-``SC``
- functions with a Streaming-Compatible interface (PSTATE.SM can be
+`SC`
+
+: functions with a Streaming-Compatible interface (PSTATE.SM can be
either 0 or 1 on entry, and is unchanged on return).
-Functions with ``__attribute__((arm_locally_streaming))`` are excluded from this
+Functions with `__attribute__((arm_locally_streaming))` are excluded from this
table because for the caller the attribute is synonymous with 'streaming', and
for the callee it is merely an implementation detail that is explicitly not
exposed to the caller.
-.. table:: Combinations of calls for functions with
diff erent attributes
-
- ==== ==== =============================== ============================== ==============================
- From To Before call After call After exception
- ==== ==== =============================== ============================== ==============================
- N N
- N S SMSTART SMSTOP
- N SC
- S N SMSTOP SMSTART SMSTART
- S S SMSTART
- S SC SMSTART
- SC N If PSTATE.SM before call is 1, If PSTATE.SM before call is 1, If PSTATE.SM before call is 1,
- then SMSTOP then SMSTART then SMSTART
- SC S If PSTATE.SM before call is 0, If PSTATE.SM before call is 0, If PSTATE.SM before call is 1,
- then SMSTART then SMSTOP then SMSTART
- SC SC If PSTATE.SM before call is 1,
- then SMSTART
- ==== ==== =============================== ============================== ==============================
+**Combinations of calls for functions with
diff erent attributes**
+| From | To | Before call | After call | After exception |
+| --- | --- | --- | --- | --- |
+| N | N | | | |
+| N | S | SMSTART | SMSTOP | |
+| N | SC | | | |
+| S | N | SMSTOP | SMSTART | SMSTART |
+| S | S | | | SMSTART |
+| S | SC | | | SMSTART |
+| SC | N | If PSTATE.SM before call is 1, then SMSTOP | If PSTATE.SM before call is 1, then SMSTART | If PSTATE.SM before call is 1, then SMSTART |
+| SC | S | If PSTATE.SM before call is 0, then SMSTART | If PSTATE.SM before call is 0, then SMSTOP | If PSTATE.SM before call is 1, then SMSTART |
+| SC | SC | | | If PSTATE.SM before call is 1, then SMSTART |
Because changing PSTATE.SM zeroes the FP/vector registers, it is best to emit
-the ``smstart`` and ``smstop`` instructions before register allocation, so that
+the `smstart` and `smstop` instructions before register allocation, so that
the register allocator can spill/reload registers around the mode change.
The compiler should also have sufficient information on which operations are
@@ -164,107 +159,98 @@ the call's arguments/return values to implement the specified calling convention
SelectionDAG provides Chains and Glue to specify the order of operations and give
preliminary control over instruction scheduling.
+### Example of preserving state
-Example of preserving state
----------------------------
-
-When passing and returning a ``float`` value to/from a function
+When passing and returning a `float` value to/from a function
that has a streaming interface from a function that has a normal interface, the
call-site will need to ensure that the argument/result registers are preserved
-and that no other code is scheduled in between the ``smstart/smstop`` and the call.
+and that no other code is scheduled in between the `smstart/smstop` and the call.
-.. code-block:: llvm
+```llvm
+define float @foo(float %f) nounwind {
+ %res = call float @bar(float %f) "aarch64_pstate_sm_enabled"
+ ret float %res
+}
- define float @foo(float %f) nounwind {
- %res = call float @bar(float %f) "aarch64_pstate_sm_enabled"
- ret float %res
- }
-
- declare float @bar(float) "aarch64_pstate_sm_enabled"
+declare float @bar(float) "aarch64_pstate_sm_enabled"
+```
The program needs to preserve the value of the floating point argument and
-return value in register ``s0``:
-
-.. code-block:: none
-
- foo: // @foo
- // %bb.0:
- stp d15, d14, [sp, #-80]! // 16-byte Folded Spill
- stp d13, d12, [sp, #16] // 16-byte Folded Spill
- stp d11, d10, [sp, #32] // 16-byte Folded Spill
- stp d9, d8, [sp, #48] // 16-byte Folded Spill
- str x30, [sp, #64] // 8-byte Folded Spill
- str s0, [sp, #76] // 4-byte Folded Spill
- smstart sm
- ldr s0, [sp, #76] // 4-byte Folded Reload
- bl bar
- str s0, [sp, #76] // 4-byte Folded Spill
- smstop sm
- ldp d9, d8, [sp, #48] // 16-byte Folded Reload
- ldp d11, d10, [sp, #32] // 16-byte Folded Reload
- ldp d13, d12, [sp, #16] // 16-byte Folded Reload
- ldr s0, [sp, #76] // 4-byte Folded Reload
- ldr x30, [sp, #64] // 8-byte Folded Reload
- ldp d15, d14, [sp], #80 // 16-byte Folded Reload
- ret
+return value in register `s0`:
+
+```none
+foo: // @foo
+// %bb.0:
+ stp d15, d14, [sp, #-80]! // 16-byte Folded Spill
+ stp d13, d12, [sp, #16] // 16-byte Folded Spill
+ stp d11, d10, [sp, #32] // 16-byte Folded Spill
+ stp d9, d8, [sp, #48] // 16-byte Folded Spill
+ str x30, [sp, #64] // 8-byte Folded Spill
+ str s0, [sp, #76] // 4-byte Folded Spill
+ smstart sm
+ ldr s0, [sp, #76] // 4-byte Folded Reload
+ bl bar
+ str s0, [sp, #76] // 4-byte Folded Spill
+ smstop sm
+ ldp d9, d8, [sp, #48] // 16-byte Folded Reload
+ ldp d11, d10, [sp, #32] // 16-byte Folded Reload
+ ldp d13, d12, [sp, #16] // 16-byte Folded Reload
+ ldr s0, [sp, #76] // 4-byte Folded Reload
+ ldr x30, [sp, #64] // 8-byte Folded Reload
+ ldp d15, d14, [sp], #80 // 16-byte Folded Reload
+ ret
+```
Setting the correct register masks on the ISD nodes and inserting the
-``smstart/smstop`` in the right places should ensure this is done correctly.
-
-
-Instruction Selection Nodes
----------------------------
+`smstart/smstop` in the right places should ensure this is done correctly.
-.. code-block:: none
+### Instruction Selection Nodes
- AArch64ISD::SMSTART Chain, [SM|ZA|Both][, RegMask]
- AArch64ISD::SMSTOP Chain, [SM|ZA|Both][, RegMask]
- AArch64ISD::COND_SMSTART Chain, [SM|ZA|Both], CurrentState, ExpectedState[, RegMask]
- AArch64ISD::COND_SMSTOP Chain, [SM|ZA|Both], CurrentState, ExpectedState[, RegMask]
+```none
+AArch64ISD::SMSTART Chain, [SM|ZA|Both][, RegMask]
+AArch64ISD::SMSTOP Chain, [SM|ZA|Both][, RegMask]
+AArch64ISD::COND_SMSTART Chain, [SM|ZA|Both], CurrentState, ExpectedState[, RegMask]
+AArch64ISD::COND_SMSTOP Chain, [SM|ZA|Both], CurrentState, ExpectedState[, RegMask]
+```
-The ``COND_SMSTART/COND_SMSTOP`` nodes additionally take ``CurrentState`` and
-``ExpectedState``, in this case the instruction will only be executed if
-``CurrentState != ExpectedState``.
+The `COND_SMSTART/COND_SMSTOP` nodes additionally take `CurrentState` and
+`ExpectedState`, in this case the instruction will only be executed if
+`CurrentState != ExpectedState`.
-When ``CurrentState`` and ``ExpectedState`` can be evaluated at compile-time
-(i.e. they are both constants) then an unconditional ``smstart/smstop``
+When `CurrentState` and `ExpectedState` can be evaluated at compile-time
+(i.e. they are both constants) then an unconditional `smstart/smstop`
instruction is emitted. Otherwise, the node is matched to a Pseudo instruction
-which expands to a compare/branch and a ``smstart/smstop``. This is necessary to
-implement transitions from ``SC -> N`` and ``SC -> S``.
+which expands to a compare/branch and a `smstart/smstop`. This is necessary to
+implement transitions from `SC -> N` and `SC -> S`.
+### Unchained Function calls
-Unchained Function calls
-------------------------
-When a function with "``aarch64_pstate_sm_enabled``" calls a function that is not
+When a function with "`aarch64_pstate_sm_enabled`" calls a function that is not
streaming compatible, the compiler has to insert an SMSTOP before the call and
insert an SMSTOP after the call.
If the function that is called is an intrinsic with no side-effects which in
-turn is lowered to a function call (e.g., ``@llvm.cos()``), then the call to
-``@llvm.cos()`` is not part of any Chain; it can be scheduled freely.
+turn is lowered to a function call (e.g., `@llvm.cos()`), then the call to
+`@llvm.cos()` is not part of any Chain; it can be scheduled freely.
Lowering of a Callsite creates a small chain of nodes which:
- starts a call sequence
-
- copies input values from virtual registers to physical registers specified by
the ABI
-
- executes a branch-and-link
-
- stops the call sequence
-
- copies the output values from their physical registers to virtual registers
When the callsite's Chain is not used, only the result value from the chained
sequence is used, but the Chain itself is discarded.
-The ``SMSTART`` and ``SMSTOP`` ISD nodes return a Chain, but no real
-values, so when the ``SMSTART/SMSTOP`` nodes are part of a Chain that isn't
+The `SMSTART` and `SMSTOP` ISD nodes return a Chain, but no real
+values, so when the `SMSTART/SMSTOP` nodes are part of a Chain that isn't
used, these nodes are not considered for scheduling and are
-removed from the DAG. In order to prevent these nodes
+removed from the DAG. In order to prevent these nodes
from being removed, we need a way to ensure the results from the
-``CopyFromReg`` can only be **used after** the ``SMSTART/SMSTOP`` has been
+`CopyFromReg` can only be **used after** the `SMSTART/SMSTOP` has been
executed.
We can use a CopyToReg -> CopyFromReg sequence for this, which moves the
@@ -276,120 +262,114 @@ allocator.
The example below shows how this is used in a DAG that does not link
together the result by a Chain, but rather by a value:
-.. code-block:: none
-
- t0: ch,glue = AArch64ISD::SMSTOP ...
- t1: ch,glue = ISD::CALL ....
- t2: res,ch,glue = CopyFromReg t1, ...
- t3: ch,glue = AArch64ISD::SMSTART t2:1, .... <- this is now part of the expression that returns the result value.
- t4: ch = CopyToReg t3, Register:f64 %vreg, t2
- t5: res,ch = CopyFromReg t4, Register:f64 %vreg
- t6: res = FADD t5, t9
-
-We also need this for locally streaming functions, where an ``SMSTART`` needs to
+```none
+ t0: ch,glue = AArch64ISD::SMSTOP ...
+ t1: ch,glue = ISD::CALL ....
+ t2: res,ch,glue = CopyFromReg t1, ...
+ t3: ch,glue = AArch64ISD::SMSTART t2:1, .... <- this is now part of the expression that returns the result value.
+ t4: ch = CopyToReg t3, Register:f64 %vreg, t2
+ t5: res,ch = CopyFromReg t4, Register:f64 %vreg
+t6: res = FADD t5, t9
+```
+
+We also need this for locally streaming functions, where an `SMSTART` needs to
be inserted into the DAG at the start of the function.
-Functions with __attribute__((arm_locally_streaming))
------------------------------------------------------
+### Functions with \_\_attribute\_\_((arm_locally_streaming))
-If a function is marked as ``arm_locally_streaming``, then the runtime SVE
+If a function is marked as `arm_locally_streaming`, then the runtime SVE
vector length in the prologue/epilogue may be
diff erent from the vector length
in the function's body. This happens because we invoke smstart after setting up
the stack-frame and similarly invoke smstop before deallocating the stack-frame.
To ensure we use the correct SVE vector length to allocate the locals with, we
can use the streaming vector-length to allocate the stack-slots through the
-``ADDSVL`` instruction, even when the CPU is not yet in streaming mode.
+`ADDSVL` instruction, even when the CPU is not yet in streaming mode.
This works only for locals and not callee-save slots, since LLVM doesn't support
mixing two
diff erent scalable vector lengths in one stack frame. That means that the
-case where a function is marked ``arm_locally_streaming`` and needs to spill SVE
-callee-saves in the prologue is currently unsupported. However, it is unlikely
-for this to happen without user intervention because ``arm_locally_streaming``
+case where a function is marked `arm_locally_streaming` and needs to spill SVE
+callee-saves in the prologue is currently unsupported. However, it is unlikely
+for this to happen without user intervention because `arm_locally_streaming`
functions cannot take or return vector-length-dependent values. This would otherwise
-require forcing both the SVE PCS using '``aarch64_sve_pcs``' combined with using
-``arm_locally_streaming`` in order to encounter this problem. This combination
+require forcing both the SVE PCS using '`aarch64_sve_pcs`' combined with using
+`arm_locally_streaming` in order to encounter this problem. This combination
can be prevented in Clang through emitting a diagnostic.
-
An example of how the prologue/epilogue would look for a function that is
-attributed with ``arm_locally_streaming``:
+attributed with `arm_locally_streaming`:
-.. code-block:: c++
+```c++
+#define N 64
- #define N 64
+void __attribute__((arm_streaming_compatible)) some_use(svfloat32_t *);
- void __attribute__((arm_streaming_compatible)) some_use(svfloat32_t *);
+// Use a float argument type, to check the value isn't clobbered by smstart.
+// Use a float return type to check the value isn't clobbered by smstop.
+float __attribute__((noinline, arm_locally_streaming)) foo(float arg) {
+ // Create local for SVE vector to check local is created with correct
+ // size when not yet in streaming mode (ADDSVL).
+ float array[N];
+ svfloat32_t vector;
- // Use a float argument type, to check the value isn't clobbered by smstart.
- // Use a float return type to check the value isn't clobbered by smstop.
- float __attribute__((noinline, arm_locally_streaming)) foo(float arg) {
- // Create local for SVE vector to check local is created with correct
- // size when not yet in streaming mode (ADDSVL).
- float array[N];
- svfloat32_t vector;
+ some_use(&vector);
+ svst1_f32(svptrue_b32(), &array[0], vector);
+ return array[N - 1] + arg;
+}
+```
- some_use(&vector);
- svst1_f32(svptrue_b32(), &array[0], vector);
- return array[N - 1] + arg;
- }
-
-should use ``ADDSVL`` for allocating the stack space and should avoid clobbering
+should use `ADDSVL` for allocating the stack space and should avoid clobbering
the return/argument values.
-.. code-block:: none
-
- _Z3foof: // @_Z3foof
- // %bb.0: // %entry
- stp d15, d14, [sp, #-96]! // 16-byte Folded Spill
- stp d13, d12, [sp, #16] // 16-byte Folded Spill
- stp d11, d10, [sp, #32] // 16-byte Folded Spill
- stp d9, d8, [sp, #48] // 16-byte Folded Spill
- stp x29, x30, [sp, #64] // 16-byte Folded Spill
- add x29, sp, #64
- str x28, [sp, #80] // 8-byte Folded Spill
- addsvl sp, sp, #-1
- sub sp, sp, #256
- str s0, [x29, #28] // 4-byte Folded Spill
- smstart sm
- sub x0, x29, #64
- addsvl x0, x0, #-1
- bl _Z10some_usePu13__SVFloat32_t
- sub x8, x29, #64
- ptrue p0.s
- ld1w { z0.s }, p0/z, [x8, #-1, mul vl]
- ldr s1, [x29, #28] // 4-byte Folded Reload
- st1w { z0.s }, p0, [sp]
- ldr s0, [sp, #252]
- fadd s0, s0, s1
- str s0, [x29, #28] // 4-byte Folded Spill
- smstop sm
- ldr s0, [x29, #28] // 4-byte Folded Reload
- addsvl sp, sp, #1
- add sp, sp, #256
- ldp x29, x30, [sp, #64] // 16-byte Folded Reload
- ldp d9, d8, [sp, #48] // 16-byte Folded Reload
- ldp d11, d10, [sp, #32] // 16-byte Folded Reload
- ldp d13, d12, [sp, #16] // 16-byte Folded Reload
- ldr x28, [sp, #80] // 8-byte Folded Reload
- ldp d15, d14, [sp], #96 // 16-byte Folded Reload
- ret
-
-
-Preventing the use of illegal instructions in Streaming Mode
-------------------------------------------------------------
-
-* When executing a program in streaming-mode (PSTATE.SM=1) a subset of SVE/SVE2
+```none
+_Z3foof: // @_Z3foof
+// %bb.0: // %entry
+ stp d15, d14, [sp, #-96]! // 16-byte Folded Spill
+ stp d13, d12, [sp, #16] // 16-byte Folded Spill
+ stp d11, d10, [sp, #32] // 16-byte Folded Spill
+ stp d9, d8, [sp, #48] // 16-byte Folded Spill
+ stp x29, x30, [sp, #64] // 16-byte Folded Spill
+ add x29, sp, #64
+ str x28, [sp, #80] // 8-byte Folded Spill
+ addsvl sp, sp, #-1
+ sub sp, sp, #256
+ str s0, [x29, #28] // 4-byte Folded Spill
+ smstart sm
+ sub x0, x29, #64
+ addsvl x0, x0, #-1
+ bl _Z10some_usePu13__SVFloat32_t
+ sub x8, x29, #64
+ ptrue p0.s
+ ld1w { z0.s }, p0/z, [x8, #-1, mul vl]
+ ldr s1, [x29, #28] // 4-byte Folded Reload
+ st1w { z0.s }, p0, [sp]
+ ldr s0, [sp, #252]
+ fadd s0, s0, s1
+ str s0, [x29, #28] // 4-byte Folded Spill
+ smstop sm
+ ldr s0, [x29, #28] // 4-byte Folded Reload
+ addsvl sp, sp, #1
+ add sp, sp, #256
+ ldp x29, x30, [sp, #64] // 16-byte Folded Reload
+ ldp d9, d8, [sp, #48] // 16-byte Folded Reload
+ ldp d11, d10, [sp, #32] // 16-byte Folded Reload
+ ldp d13, d12, [sp, #16] // 16-byte Folded Reload
+ ldr x28, [sp, #80] // 8-byte Folded Reload
+ ldp d15, d14, [sp], #96 // 16-byte Folded Reload
+ ret
+```
+
+### Preventing the use of illegal instructions in Streaming Mode
+
+- When executing a program in streaming-mode (PSTATE.SM=1) a subset of SVE/SVE2
instructions and most AdvSIMD/NEON instructions are invalid.
-
-* When executing a program in normal mode (PSTATE.SM=0), a subset of SME
+- When executing a program in normal mode (PSTATE.SM=0), a subset of SME
instructions are invalid.
-
-* Streaming-compatible functions must use only instructions that are valid when
+- Streaming-compatible functions must use only instructions that are valid when
either PSTATE.SM=0 or PSTATE.SM=1.
The value of PSTATE.SM is not controlled by the feature flags, but rather by the
-function attributes. This means that we can compile for '``+sme``', and the compiler
+function attributes. This means that we can compile for '`+sme`', and the compiler
will code-generate any instructions, even if they are not legal under the requested
streaming mode. The compiler needs to use the function attributes to ensure the
compiler doesn't perform transformations under the assumption that certain operations
@@ -398,13 +378,13 @@ are available at runtime.
We made a conscious choice not to model this with feature flags because we
still want to support inline-asm in either mode (with the user placing
smstart/smstop manually), and this became rather complicated to implement at the
-individual instruction level (see `D120261 <https://reviews.llvm.org/D120261>`_
-and `D121208 <https://reviews.llvm.org/D121208>`_) because of limitations in
+individual instruction level (see [D120261](https://reviews.llvm.org/D120261)
+and [D121208](https://reviews.llvm.org/D121208)) because of limitations in
TableGen.
As a first step, this means we'll disable vectorization (LoopVectorize/SLP)
-entirely when a function has either of the ``aarch64_pstate_sm_enabled``,
-``aarch64_pstate_sm_body`` or ``aarch64_pstate_sm_compatible`` attributes,
+entirely when a function has either of the `aarch64_pstate_sm_enabled`,
+`aarch64_pstate_sm_body` or `aarch64_pstate_sm_compatible` attributes,
in order to avoid the use of vector instructions.
Later on, we'll aim to relax these restrictions to enable scalable
@@ -415,21 +395,16 @@ We will also emit diagnostics in Clang to prevent the use of
non-streaming(-compatible) operations, e.g., through ACLE intrinsics, when a
function is decorated with the streaming mode attributes.
+### Other things to consider
-Other things to consider
-------------------------
-
-* Inlining must be disabled when the call-site needs to toggle PSTATE.SM or
+- Inlining must be disabled when the call-site needs to toggle PSTATE.SM or
when the callee's function body is executed in a
diff erent streaming mode from
its caller. This is needed because function calls are the boundaries for
streaming mode changes.
-
-* Tail call optimization must be disabled when the call-site needs to toggle
+- Tail call optimization must be disabled when the call-site needs to toggle
PSTATE.SM, such that the caller can restore the original value of PSTATE.SM.
-
-3. Handling PSTATE.ZA
-=====================
+## 3. Handling PSTATE.ZA
In contrast to PSTATE.SM, enabling PSTATE.ZA does not affect the SVE vector
length and also doesn't clobber FP/AdvSIMD/SVE registers. This means it is safe
@@ -437,27 +412,22 @@ to toggle PSTATE.ZA using intrinsics. This also makes it simpler to setup a
lazy-save mechanism for calls to private-ZA functions (i.e. functions that may
either directly or indirectly clobber ZA state).
-For the purpose of handling functions marked with ``aarch64_new_za``,
+For the purpose of handling functions marked with `aarch64_new_za`,
we have introduced a new LLVM IR pass (SMEABIPass) that runs just before
SelectionDAG. Any such functions handled by this pass are marked with
-``aarch64_expanded_pstate_za``.
+`aarch64_expanded_pstate_za`.
-Setting up a lazy-save
-----------------------
+### Setting up a lazy-save
-Committing a lazy-save
-----------------------
+### Committing a lazy-save
-Exception handling and ZA
--------------------------
+### Exception handling and ZA
-4. Types
-========
+## 4. Types
-AArch64 Predicate-as-Counter Type
----------------------------------
+### AArch64 Predicate-as-Counter Type
-:Overview:
+**Overview:**
The predicate-as-counter type represents the type of a predicate-as-counter
value held in an AArch64 SVE predicate register. Such a value contains
@@ -467,28 +437,24 @@ used to move the predicate-as-counter value to/from a predicate vector.
There are certain limitations on the type:
-* The type can be used for function parameters and return values.
-
-* The supported LLVM operations on this type are limited to ``load``, ``store``,
- ``phi``, ``select``, and ``alloca`` instructions.
+- The type can be used for function parameters and return values.
+- The supported LLVM operations on this type are limited to `load`, `store`,
+ `phi`, `select`, and `alloca` instructions.
The predicate-as-counter type is a scalable type.
-:Syntax:
-
-::
-
- target("aarch64.svcount")
-
+**Syntax:**
+```
+target("aarch64.svcount")
+```
-5. References
-=============
+## 5. References
- .. _aarch64_sme_acle:
+ (aarch64_sme_acle)=
-1. `SME ACLE Pull-request <https://github.com/ARM-software/acle/pull/188>`__
+1. [SME ACLE Pull-request](https://github.com/ARM-software/acle/pull/188)
- .. _aarch64_sme_abi:
+ (aarch64_sme_abi)=
-2. `SME ABI Pull-request <https://github.com/ARM-software/abi-aa/pull/123>`__
+2. [SME ABI Pull-request](https://github.com/ARM-software/abi-aa/pull/123)
diff --git a/llvm/docs/AMDGPUAsyncOperations.md b/llvm/docs/AMDGPUAsyncOperations.md
index 93f5cf10ff448..1863a516fdd19 100644
--- a/llvm/docs/AMDGPUAsyncOperations.md
+++ b/llvm/docs/AMDGPUAsyncOperations.md
@@ -1,23 +1,20 @@
-.. _amdgpu-async-operations:
+(amdgpu-async-operations)=
-===============================
- AMDGPU Asynchronous Operations
-===============================
+# AMDGPU Asynchronous Operations
-.. contents::
- :local:
+```{contents}
+:local: true
+```
-Introduction
-============
+## Introduction
Asynchronous operations are operations whose completion is not tracked
internally by the compiler. A thread that initiates one or more async operations can use
*asyncmarks* to track their completion.
-- Most :ref:`DMA operations<amdgpu-dma-operations>` are asynchronous.
+- Most {ref}`DMA operations <amdgpu-dma-operations>` are asynchronous.
-Asyncmarks
-==========
+## Asyncmarks
An *asyncmark* created by a thread can be used to track async operations
initiated by that thread. The abstract machine maintains a sequence of
@@ -26,239 +23,229 @@ asyncmarks produced by calls to other functions encountered in the currently
executing function. The state of this sequence at each program point in the
function is called the *current sequence*.
-``@llvm.amdgcn.asyncmark()``
-----------------------------
+### `@llvm.amdgcn.asyncmark()`
Produces an asyncmark and appends it to the current sequence.
-``@llvm.amdgcn.wait.asyncmark(i16 %N)``
----------------------------------------
+### `@llvm.amdgcn.wait.asyncmark(i16 %N)`
-Ensures that the length of the current sequence is at most ``N`` by removing
-asyncmarks from the start of the sequence if it is more than ``N``.
+Ensures that the length of the current sequence is at most `N` by removing
+asyncmarks from the start of the sequence if it is more than `N`.
-.. _amdgpu-asyncmark-memory-model:
+(amdgpu-asyncmark-memory-model)=
-Memory Model
-============
+## Memory Model
-An ``asyncmark()`` operation ``X`` that produces an asyncmark ``M`` is
-*completed-at* a ``wait.asyncmark()`` operation ``Y`` in the same function body
+An `asyncmark()` operation `X` that produces an asyncmark `M` is
+*completed-at* a `wait.asyncmark()` operation `Y` in the same function body
if:
-- ``X`` is *program-ordered* before ``Y``, and
-- ``M`` is not in the current sequence at any operation ``Z`` that immediately
- follows ``Y`` in *program-order*.
+- `X` is *program-ordered* before `Y`, and
+- `M` is not in the current sequence at any operation `Z` that immediately
+ follows `Y` in *program-order*.
-Each dynamic instance ``I`` of an async *instruction* initiates a corresponding
-async *operation* ``A`` such that ``I`` *happens-before* ``A``. Then ``A``
-*happens-before* a ``wait.asyncmark()`` operation ``Y`` if there exists an
-``asyncmark()`` operation ``X`` such that:
+Each dynamic instance `I` of an async *instruction* initiates a corresponding
+async *operation* `A` such that `I` *happens-before* `A`. Then `A`
+*happens-before* a `wait.asyncmark()` operation `Y` if there exists an
+`asyncmark()` operation `X` such that:
-- ``I`` is *program-ordered* before ``X``, and
-- ``X`` is *completed-at* ``Y``.
+- `I` is *program-ordered* before `X`, and
+- `X` is *completed-at* `Y`.
-Examples
-========
+## Examples
-Uneven blocks of async operations
----------------------------------
+### Uneven blocks of async operations
-.. code-block:: c++
+```c++
+void foo(global int *g, local int *l) {
+ // first block
+ async_load_to_lds(l, g);
+ async_load_to_lds(l, g);
+ async_load_to_lds(l, g);
+ asyncmark();
- void foo(global int *g, local int *l) {
- // first block
- async_load_to_lds(l, g);
- async_load_to_lds(l, g);
- async_load_to_lds(l, g);
- asyncmark();
+ // second block; longer
+ async_load_to_lds(l, g);
+ async_load_to_lds(l, g);
+ async_load_to_lds(l, g);
+ async_load_to_lds(l, g);
+ async_load_to_lds(l, g);
+ asyncmark();
- // second block; longer
- async_load_to_lds(l, g);
- async_load_to_lds(l, g);
- async_load_to_lds(l, g);
- async_load_to_lds(l, g);
- async_load_to_lds(l, g);
- asyncmark();
+ // third block; shorter
+ async_load_to_lds(l, g);
+ async_load_to_lds(l, g);
+ asyncmark();
- // third block; shorter
- async_load_to_lds(l, g);
- async_load_to_lds(l, g);
- asyncmark();
+ // Wait for first block
+ wait.asyncmark(2);
+}
+```
- // Wait for first block
- wait.asyncmark(2);
- }
-
-Software pipeline
------------------
-
-.. code-block:: c++
+### Software pipeline
- void foo(global int *g, local int *l) {
- // first block
- asyncmark();
+```c++
+void foo(global int *g, local int *l) {
+ // first block
+ asyncmark();
- // second block
- asyncmark();
+ // second block
+ asyncmark();
- // third block
- asyncmark();
+ // third block
+ asyncmark();
- for (;;) {
- wait.asyncmark(2);
- // use data
+ for (;;) {
+ wait.asyncmark(2);
+ // use data
- // next block
- asyncmark();
- }
+ // next block
+ asyncmark();
+ }
- // flush one block
- wait.asyncmark(2);
+ // flush one block
+ wait.asyncmark(2);
- // flush one more block
- wait.asyncmark(1);
+ // flush one more block
+ wait.asyncmark(1);
- // flush last block
- wait.asyncmark(0);
- }
-
-Ordinary function call
-----------------------
+ // flush last block
+ wait.asyncmark(0);
+}
+```
-.. code-block:: c++
+### Ordinary function call
- extern void bar(); // may or may not initiate async operations
+```c++
+extern void bar(); // may or may not initiate async operations
- void foo(global int *g, local int *l) {
- // first block
- asyncmark();
+void foo(global int *g, local int *l) {
+ // first block
+ asyncmark();
- // second block
- asyncmark();
+ // second block
+ asyncmark();
- // function call
- bar();
+ // function call
+ bar();
- // third block
- asyncmark();
+ // third block
+ asyncmark();
- // wait for the second block
- wait.asyncmark(1);
+ // wait for the second block
+ wait.asyncmark(1);
- // wait for the third block, including bar()
- wait.asyncmark(0);
- }
+ // wait for the third block, including bar()
+ wait.asyncmark(0);
+}
+```
-Implementation notes
-====================
+## Implementation notes
[This section is informational.]
-Function Calls
---------------
+### Function Calls
In general, at a function call, if the caller uses sufficient waits to track
its own async operations, the actions performed by the callee cannot affect
correctness. But inlining such a call may result in redundant waits.
-.. code-block:: c++
-
- void foo() {
- ...
- asyncmark(); // X
- ... // no wait.asyncmark()
- }
+```c++
+void foo() {
+ ...
+ asyncmark(); // X
+ ... // no wait.asyncmark()
+}
+
+void bar() {
+ asyncmark(); // B
+ asyncmark(); // C
+ foo();
+ wait.asyncmark(1); // D
+}
+```
+
+Before inlining, it is unspecified whether `X` is *completed-at* `D`, while
+`C` is **not** *completed-at* `D`. The programmer can only rely on `B`
+being *completed-at* `D`.
+
+```c++
+void bar() {
+ asyncmark(); // B
+ asyncmark(); // C
+ ...
+ asyncmark(); // X
+ ... // no wait.asyncmark()
+ wait.asyncmark(1); // D
+}
+```
+
+After inlining, `C` is also *completed-at* `D` and `X` is **not**
+*completed-at* `D`.
+
+Conversely, a `wait.asyncmark` call inside a callee cannot be used to track
+asyncmarks from the caller, since this `wait.asyncmark` can only
+observe the current sequence of the callee.
- void bar() {
- asyncmark(); // B
- asyncmark(); // C
- foo();
- wait.asyncmark(1); // D
- }
+```c++
+void foo() {
+ ... // no asyncmark()
+ wait.asyncmark(0); // Y
+ ...
+}
+
+void bar() {
+ asyncmark(); // B
+ asyncmark(); // C
+ foo();
+ wait.asyncmark(1); // D
+}
+```
+
+In the above example, it is unspecified whether `B` and `C` in `bar()` are
+*completed-at* `Y`, because they are not included in the sequence that can be
+examined at `Y`.
+
+```c++
+void bar() {
+ asyncmark(); // B
+ asyncmark(); // C
+ ... // no asyncmark()
+ wait.asyncmark(0); // Y
+ ...
+ wait.asyncmark(1); // D
+}
+```
+
+After inlining, both `B` and `C` are *completed-at* `Y`.
+
+### Optimization
-Before inlining, it is unspecified whether ``X`` is *completed-at* ``D``, while
-``C`` is **not** *completed-at* ``D``. The programmer can only rely on ``B``
-being *completed-at* ``D``.
+The implementation may eliminate asyncmark/wait intrinsics in the following
+cases. These are just examples and not meant to be an exhaustive list.
-.. code-block:: c++
+1. An `asyncmark` operation which remains in the current sequence along every
+ path that reaches the function exit.
- void bar() {
- asyncmark(); // B
- asyncmark(); // C
+ ```c++
+ void foo() {
...
asyncmark(); // X
... // no wait.asyncmark()
- wait.asyncmark(1); // D
}
+ ```
-After inlining, ``C`` is also *completed-at* ``D`` and ``X`` is **not**
-*completed-at* ``D``.
-
-Conversely, a ``wait.asyncmark`` call inside a callee cannot be used to track
-asyncmarks from the caller, since this ``wait.asyncmark`` can only
-observe the current sequence of the callee.
+ Here, `X` can be eliminated.
-.. code-block:: c++
+2. A `wait.asyncmark` which sees an empty sequence of asyncmarks along every
+ path that reaches it.
+ ```c++
void foo() {
... // no asyncmark()
wait.asyncmark(0); // Y
...
}
+ ```
- void bar() {
- asyncmark(); // B
- asyncmark(); // C
- foo();
- wait.asyncmark(1); // D
- }
-
-In the above example, it is unspecified whether ``B`` and ``C`` in ``bar()`` are
-*completed-at* ``Y``, because they are not included in the sequence that can be
-examined at ``Y``.
-
-.. code-block:: c++
-
- void bar() {
- asyncmark(); // B
- asyncmark(); // C
- ... // no asyncmark()
- wait.asyncmark(0); // Y
- ...
- wait.asyncmark(1); // D
- }
-
-After inlining, both ``B`` and ``C`` are *completed-at* ``Y``.
-
-Optimization
-------------
-
-The implementation may eliminate asyncmark/wait intrinsics in the following
-cases. These are just examples and not meant to be an exhaustive list.
-
-1. An ``asyncmark`` operation which remains in the current sequence along every
- path that reaches the function exit.
-
- .. code-block:: c++
-
- void foo() {
- ...
- asyncmark(); // X
- ... // no wait.asyncmark()
- }
-
- Here, ``X`` can be eliminated.
-
-2. A ``wait.asyncmark`` which sees an empty sequence of asyncmarks along every
- path that reaches it.
-
- .. code-block:: c++
-
- void foo() {
- ... // no asyncmark()
- wait.asyncmark(0); // Y
- ...
- }
-
- Here, ``Y`` can be eliminated.
+ Here, `Y` can be eliminated.
diff --git a/llvm/docs/AMDGPUExecutionSynchronization.md b/llvm/docs/AMDGPUExecutionSynchronization.md
index f02d13b35e075..09c8bc602f8b9 100644
--- a/llvm/docs/AMDGPUExecutionSynchronization.md
+++ b/llvm/docs/AMDGPUExecutionSynchronization.md
@@ -1,298 +1,318 @@
-.. _amdgpu-execution-synchronization:
+(amdgpu-execution-synchronization)=
-================================
-AMDGPU Execution Synchronization
-================================
+# AMDGPU Execution Synchronization
-.. contents::
- :local:
+```{contents}
+:local: true
+```
-.. _amdgpu-execution-synchronization-barriers:
+(amdgpu-execution-synchronization-barriers)=
This document covers
diff erent ways of synchronizing execution of threads on AMD GPUs.
-.. note::
+:::{note}
+This document is not exhaustive. There may be more ways of synchronizing execution
+that are not covered by this document.
+:::
- This document is not exhaustive. There may be more ways of synchronizing execution
- that are not covered by this document.
-
-********
-Barriers
-********
+## Barriers
This section covers execution synchronization using barrier-style primitives.
-.. _amdgpu-execution-synchronization-barriers-execution-model:
+(amdgpu-execution-synchronization-barriers-execution-model)=
-Execution Model
-===============
+### Execution Model
This section contains a formal execution model that can be used to model the behavior of
barriers on AMDGPU targets.
Barriers only synchronize execution and do not affect the visibility of memory operations between threads.
-Refer to the :ref:`execution barriers memory model<amdgpu-amdhsa-execution-barriers-memory-model>`
+Refer to the {ref}`execution barriers memory model<amdgpu-amdhsa-execution-barriers-memory-model>`
to determine how to synchronize memory operations through *barrier-executes-before*.
-.. note::
-
- The barrier execution model is experimental and subject to change.
+:::{note}
+The barrier execution model is experimental and subject to change.
+:::
-.. rubric:: Barrier *Objects*
+:::{rubric} Barrier *Objects*
+:::
Threads can synchronize execution by performing barrier operations on barrier *objects* as described below:
Each barrier *object* has the following state:
-* An unsigned positive integer *expected count*: counts the number of *arrive* operations
+- An unsigned positive integer *expected count*: counts the number of *arrive* operations
expected for this barrier *object*.
-* An unsigned non-negative integer *arrive count*: counts the number of *arrive* operations
+
+- An unsigned non-negative integer *arrive count*: counts the number of *arrive* operations
already performed on this barrier *object*.
- * The initial value of *arrive count* is zero.
- * When an operation causes *arrive count* to be equal to *expected count*, the barrier is completed,
+ - The initial value of *arrive count* is zero.
+ - When an operation causes *arrive count* to be equal to *expected count*, the barrier is completed,
and the *arrive count* is reset to zero.
-Barrier *objects* exist within a *scope* instance (see :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`),
+Barrier *objects* exist within a *scope* instance (see {ref}`amdgpu-amdhsa-llvm-sync-scopes-table`),
and thus can only be accessed by threads in the same *scope* instance.
-.. _amdgpu-execution-synchronization-barriers-execution-model-barrier-operations:
-.. rubric:: Barrier Operations
+(amdgpu-execution-synchronization-barriers-execution-model-barrier-operations)=
+
+:::{rubric} Barrier Operations
+:::
Barrier operations are performed on barrier *objects*. A barrier operation is a dynamic instance
of one of the following:
-* Barrier *init*
+- Barrier *init*
- * Barrier *init* takes an additional unsigned positive integer argument *k*.
- * Sets the *expected count* of the *barrier object* to *k*.
- * Resets the *arrive count* of the *barrier object* to zero.
+ - Barrier *init* takes an additional unsigned positive integer argument *k*.
+ - Sets the *expected count* of the *barrier object* to *k*.
+ - Resets the *arrive count* of the *barrier object* to zero.
-* Barrier *drop*.
+- Barrier *drop*.
- * Decrements *expected count* of the barrier *object* by one.
- * A barrier *drop* cannot cause the *expected count* of the barrier *object*
+ - Decrements *expected count* of the barrier *object* by one.
+ - A barrier *drop* cannot cause the *expected count* of the barrier *object*
to become negative; otherwise, the behavior is undefined.
-* Barrier *arrive*.
+- Barrier *arrive*.
- * Increments the *arrive count* of the barrier *object* by one.
- * If supported, an additional argument to *arrive* can also update the *expected count* of the
+ - Increments the *arrive count* of the barrier *object* by one.
+ - If supported, an additional argument to *arrive* can also update the *expected count* of the
barrier *object* before the *arrive count* is incremented;
the new *expected count* cannot be less than or equal to the *arrive count*,
otherwise the behavior is undefined.
-* Barrier *wait*.
+- Barrier *wait*.
- * Introduces execution dependencies between threads; this operation depends on
+ - Introduces execution dependencies between threads; this operation depends on
other barrier operations to complete.
Barrier modification operations are barrier operations that modify the barrier *object* state:
-* Barrier *init*.
-* Barrier *drop*.
-* Barrier *arrive*.
+- Barrier *init*.
+- Barrier *drop*.
+- Barrier *arrive*.
-.. rubric:: *Thread-barrier-order<BO>*
+:::{rubric} `Thread-barrier-order<BO>`
+:::
-*Thread-barrier-order<BO>* is the subset of *program-order* that only relates barrier operations
-performed on a barrier *object* ``BO``.
+`Thread-barrier-order<BO>` is the subset of *program-order* that only relates barrier operations
+performed on a barrier *object* `BO`.
-.. rubric:: *Barrier-modification-order<BO>*
+:::{rubric} `Barrier-modification-order<BO>`
+:::
-All barrier modification operations on a barrier *object* ``BO`` occur in a strict total order called
-*barrier-modification-order<BO>*; it is the order in which ``BO`` observes barrier
-operations that change its state. For any valid *barrier-modification-order<BO>*, the
+All barrier modification operations on a barrier *object* `BO` occur in a strict total order called
+`barrier-modification-order<BO>`; it is the order in which `BO` observes barrier
+operations that change its state. For any valid `barrier-modification-order<BO>`, the
following must be true:
-* Let ``A`` and ``B`` be two barrier modification operations where ``A -> B`` in
- *thread-barrier-order<BO>*, then ``A -> B`` is also in *barrier-modification-order<BO>*.
-* The first element in *barrier-modification-order<BO>* is always a barrier *init*, otherwise
+- Let `A` and `B` be two barrier modification operations where `A -> B` in
+ `thread-barrier-order<BO>`, then `A -> B` is also in `barrier-modification-order<BO>`.
+- The first element in `barrier-modification-order<BO>` is always a barrier *init*, otherwise
the behavior is undefined.
-.. rubric:: *Barrier-participates-in*
+:::{rubric} *Barrier-participates-in*
+:::
*Barrier-participates-in* relates barrier operations to the barrier *waits* that depend on them
-to complete. A barrier operation ``X`` *barrier-participates-in* a barrier *wait* ``W``
+to complete. A barrier operation `X` *barrier-participates-in* a barrier *wait* `W`
if and only if all of the following is true:
-* ``X`` and ``W`` are both performed on the same barrier *object* ``BO``.
-* ``X`` is a barrier *arrive* or *drop* operation.
-* ``X`` does not *barrier-participate-in* another distinct barrier *wait* ``W'`` in the same thread as ``W``.
-* ``W -> X`` not in *thread-barrier-order<BO>*.
-* All dependent constraint and relations are satisfied as well. [0]_
+- `X` and `W` are both performed on the same barrier *object* `BO`.
+- `X` is a barrier *arrive* or *drop* operation.
+- `X` does not *barrier-participate-in* another distinct barrier *wait* `W'` in the same thread as `W`.
+- `W -> X` not in `thread-barrier-order<BO>`.
+- All dependent constraint and relations are satisfied as well. [^0]
-For the set ``S`` consisting of all barrier operations that *barrier-participate-in* a barrier *wait* ``W`` for some
-barrier *object* ``BO``:
+For the set `S` consisting of all barrier operations that *barrier-participate-in* a barrier *wait* `W` for some
+barrier *object* `BO`:
-* The elements of ``S`` all exist in a continuous, uninterrupted interval of *barrier-modification-order<BO>*.
-* The *arrive count* of ``BO`` is zero before the first operation of ``S`` in *barrier-modification-order<BO>*.
-* The *arrive count* and *expected count* of ``BO`` are equal after the last operation of ``S`` in
- *barrier-modification-order<BO>*. The *arrive count* and *expected count* of ``BO`` cannot
- equal at any other point in ``S``.
+- The elements of `S` all exist in a continuous, uninterrupted interval of `barrier-modification-order<BO>`.
+- The *arrive count* of `BO` is zero before the first operation of `S` in `barrier-modification-order<BO>`.
+- The *arrive count* and *expected count* of `BO` are equal after the last operation of `S` in
+ `barrier-modification-order<BO>`. The *arrive count* and *expected count* of `BO` cannot
+ equal at any other point in `S`.
-.. [0] The definition of *barrier-participates-in* (in its current state) is non-deterministic and
- will be improved in the future: Within a valid execution, there may be multiple ways
- to build *barrier-participates-in*, however there is only one way to build it that also satisfies all
- other relations and constraints that depend on *barrier-participates-in* and relations derived from it.
+[^0]: The definition of *barrier-participates-in* (in its current state) is non-deterministic and
+ will be improved in the future: Within a valid execution, there may be multiple ways
+ to build *barrier-participates-in*, however there is only one way to build it that also satisfies all
+ other relations and constraints that depend on *barrier-participates-in* and relations derived from it.
-.. rubric:: *Barrier-executes-before*
+:::{rubric} *Barrier-executes-before*
+:::
-A barrier operation ``A`` *barrier-executes-before* another barrier operation ``B`` if any of the
+A barrier operation `A` *barrier-executes-before* another barrier operation `B` if any of the
following is true:
-* ``A -> B`` in *program-order*.
-* ``A -> B`` in *barrier-participates-in*.
-* ``A`` *barrier-executes-before* some barrier operation ``X``, and ``X``
- *barrier-executes-before* ``B``.
+- `A -> B` in *program-order*.
+- `A -> B` in *barrier-participates-in*.
+- `A` *barrier-executes-before* some barrier operation `X`, and `X`
+ *barrier-executes-before* `B`.
-*Barrier-executes-before* is consistent with *barrier-modification-order<BO>* for every barrier object ``BO``.
+*Barrier-executes-before* is consistent with `barrier-modification-order<BO>` for every barrier object `BO`.
-.. rubric:: Barrier *drop* races
+:::{rubric} Barrier *drop* races
+:::
-For every pair of barrier *arrive* ``A`` and barrier *drop* ``D`` performed on a barrier *object*
-``BO``, such that ``A -> D`` in *thread-barrier-order<BO>*, one of the following must be true:
+For every pair of barrier *arrive* `A` and barrier *drop* `D` performed on a barrier *object*
+`BO`, such that `A -> D` in `thread-barrier-order<BO>`, one of the following must be true:
-* ``A`` does not *barrier-participates-in* any barrier *wait*.
-* ``A`` *barrier-participates-in* at least one barrier *wait* ``W``
- such that ``W -> D`` in *barrier-executes-before*.
+- `A` does not *barrier-participates-in* any barrier *wait*.
+- `A` *barrier-participates-in* at least one barrier *wait* `W`
+ such that `W -> D` in *barrier-executes-before*.
-.. rubric:: *barrier-phase-with*
+:::{rubric} *barrier-phase-with*
+:::
*barrier-phase-with* is a symmetric relation over barrier operations defined as the
transitive closure of: *barrier-participates-in* and its inverse relation.
-.. rubric:: Barrier phase separation
+:::{rubric} Barrier phase separation
+:::
-For every barrier operation ``A`` that *barrier-participates-in* a barrier *wait* ``W`` on a barrier *object* ``BO``:
+For every barrier operation `A` that *barrier-participates-in* a barrier *wait* `W` on a barrier *object* `BO`:
-* There is no barrier operation ``X`` on ``BO`` such that ``A -> X -> W`` in
- *barrier-executes-before*, and ``X`` *barrier-phase-with* a non-empty set of operations
- that does not include ``W``.
+- There is no barrier operation `X` on `BO` such that `A -> X -> W` in
+ *barrier-executes-before*, and `X` *barrier-phase-with* a non-empty set of operations
+ that does not include `W`.
-Informational Notes
-~~~~~~~~~~~~~~~~~~~
+#### Informational Notes
Informally, we can deduce from the above formal model that execution barriers behave as follows:
-* *Barrier-executes-before* relates the dynamic instances of operations from
diff erent threads together.
- For example, if ``A -> B`` in *barrier-executes-before*, then the execution of ``A`` must complete
- before the execution of ``B`` can complete.
+- *Barrier-executes-before* relates the dynamic instances of operations from
diff erent threads together.
+ For example, if `A -> B` in *barrier-executes-before*, then the execution of `A` must complete
+ before the execution of `B` can complete.
+
+ - This property can also be combined with *program-order*. For example, let two (non-barrier) operations
+ `X` and `Y` where `X -> A` and `B -> Y` in *program-order*, then we know that the execution
+ of `X` completes before the execution of `Y` does.
- * This property can also be combined with *program-order*. For example, let two (non-barrier) operations
- ``X`` and ``Y`` where ``X -> A`` and ``B -> Y`` in *program-order*, then we know that the execution
- of ``X`` completes before the execution of ``Y`` does.
+- Barriers do not complete "out-of-thin-air"; a barrier *wait* `W` cannot depend on a barrier operation
+ `X` to complete if `W -> X` in *barrier-executes-before*.
-* Barriers do not complete "out-of-thin-air"; a barrier *wait* ``W`` cannot depend on a barrier operation
- ``X`` to complete if ``W -> X`` in *barrier-executes-before*.
-* It is undefined behavior to operate on an uninitialized barrier object.
-* It is undefined behavior for a barrier *wait* to never complete.
-* It is not mandatory to *drop* a barrier after *joining* it.
-* A thread may not *arrive* and then *drop* a barrier *object* unless the barrier completes before the
+- It is undefined behavior to operate on an uninitialized barrier object.
+
+- It is undefined behavior for a barrier *wait* to never complete.
+
+- It is not mandatory to *drop* a barrier after *joining* it.
+
+- A thread may not *arrive* and then *drop* a barrier *object* unless the barrier completes before the
barrier *drop*. Incrementing the *arrive count* and decrementing the *expected count* directly
after may cause undefined behavior.
-* *Joining* a barrier is only useful if the thread will *wait* on that same barrier *object* later.
-Barrier Implementations on AMDGPU Targets
-=========================================
+- *Joining* a barrier is only useful if the thread will *wait* on that same barrier *object* later.
-``s_barrier``
-~~~~~~~~~~~~~
+### Barrier Implementations on AMDGPU Targets
-``s_barrier`` are the primary barrier implementation of AMD GPUs.
+#### `s_barrier`
-``s_barrier`` instructions can only be used to synchronize threads at a wavefront granularity.
-``s_barrier`` instructions are convergent within a wave, and thus can only be performed
+`s_barrier` are the primary barrier implementation of AMD GPUs.
+
+`s_barrier` instructions can only be used to synchronize threads at a wavefront granularity.
+`s_barrier` instructions are convergent within a wave, and thus can only be performed
in wave-uniform control flow.
-The ``s_barrier`` family of instructions is available in some form on all GFX targets,
+The `s_barrier` family of instructions is available in some form on all GFX targets,
and has evolved over time. The sub-sections below cover the capabilities offered by every major
iteration of this feature separately.
-GFX6-11
--------
+##### GFX6-11
Targets from GFX6 through GFX11 included do not have the "split barrier" feature.
The barrier *arrive* and barrier *wait* operations **cannot** be performed independently
-using ``s_barrier``.
+using `s_barrier`.
-There is only one *workgroup barrier* object of ``workgroup`` scope that is implicitly used
-by all ``s_barrier`` instructions.
+There is only one *workgroup barrier* object of `workgroup` scope that is implicitly used
+by all `s_barrier` instructions.
The following code sequences can be used to implement the barrier operations defined by the
-:ref:`execution synchronization model<amdgpu-execution-synchronization-barriers-execution-model>` using
-``s_barrier`` on GFX6 through GFX11:
-
-.. table:: s_barrier GFX6-11
- :name: amdgpu-execution-synchronization-barriers-sbarrier-gfx6-11
- :widths: 15 15 70
-
- ===================== ====================== ===========================================================
- Barrier Operation(s) Barrier *Object* AMDGPU Machine Code
- ===================== ====================== ===========================================================
- **Init and Drop**
- --------------------------------------------------------------------------------------------------------
- *init* - *Workgroup barrier* Automatically initialized by the hardware when a workgroup
- is launched. The *expected count* of this barrier is set
- to the number of waves in the workgroup.
-
- *drop* - *Workgroup barrier* When a thread ends, it automatically *drops* this barrier
- *object* if it had previously *joined* it.
-
- **Arrive and Wait**
- --------------------------------------------------------------------------------------------------------
- *arrive* then *wait* - *Workgroup barrier* | **BackOffBarrier**
- | ``s_barrier``
- | **No BackOffBarrier**
- | ``s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)``
- | ``s_waitcnt_vscnt null, 0x0``
- | ``s_barrier``
-
- - If the target does not have the BackOffBarrier feature,
- then there cannot be any outstanding memory operations
- before issuing the ``s_barrier`` instruction.
- - The waitcnts can independently be moved earlier, or
- removed entirely as long as the associated
- counter remains at zero before issuing the
- ``s_barrier`` instruction.
- - The ``s_barrier`` instruction cannot complete
- before all waves of the workgroup have launched.
-
- *arrive* - *Workgroup barrier* Not available separately, see *arrive* then *wait*
-
- *wait* - *Workgroup barrier* Not available separately, see *arrive* then *wait*
- ===================== ====================== ===========================================================
-
-GFX12
------
-
-GFX12 targets have the split-barrier feature, and also allow ``s_barrier`` instructions to use
-one of multiple barrier *objects* available per workgroup. ``s_barrier`` instruction use the
+{ref}`execution synchronization model<amdgpu-execution-synchronization-barriers-execution-model>` using
+`s_barrier` on GFX6 through GFX11:
+
+```{list-table} s_barrier GFX6-11
+:name: amdgpu-execution-synchronization-barriers-sbarrier-gfx6-11
+:widths: 15 15 70
+:header-rows: 1
+
+ * - Barrier Operation(s)
+ - Barrier *Object*
+ - AMDGPU Machine Code
+ * - **Init and Drop**
+ -
+ -
+ * - *init*
+ - *Workgroup barrier*
+ - Automatically initialized by the hardware when a workgroup is launched.
+ The *expected count* of this barrier is set to the number of waves in the
+ workgroup.
+ * - *drop*
+ - *Workgroup barrier*
+ - When a thread ends, it automatically *drops* this barrier *object* if it
+ had previously *joined* it.
+ * - **Arrive and Wait**
+ -
+ -
+ * - *arrive* then *wait*
+ - *Workgroup barrier*
+ - **BackOffBarrier**
+
+ `s_barrier`
+
+ **No BackOffBarrier**
+
+ `s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)`
+
+ `s_waitcnt_vscnt null, 0x0`
+
+ `s_barrier`
+
+ - If the target does not have the BackOffBarrier feature, then there
+ cannot be any outstanding memory operations before issuing the
+ `s_barrier` instruction.
+ - The waitcnts can independently be moved earlier, or removed entirely
+ as long as the associated counter remains at zero before issuing the
+ `s_barrier` instruction.
+ - The `s_barrier` instruction cannot complete before all waves of the
+ workgroup have launched.
+ * - *arrive*
+ - *Workgroup barrier*
+ - Not available separately, see *arrive* then *wait*
+ * - *wait*
+ - *Workgroup barrier*
+ - Not available separately, see *arrive* then *wait*
+```
+
+##### GFX12
+
+GFX12 targets have the split-barrier feature, and also allow `s_barrier` instructions to use
+one of multiple barrier *objects* available per workgroup. `s_barrier` instruction use the
barrier ID operand to determine the barrier *object* they operate on.
GFX12.5 additionally introduces new barrier *objects* that offer more flexibility for synchronizing the execution
of a subset of waves of a workgroup, or synchronizing execution across workgroups within a workgroup cluster, via
-``s_barrier``. These are called "named barriers".
-
-.. note::
+`s_barrier`. These are called "named barriers".
- Check the :ref:`the table below<amdgpu-execution-synchronization-barriers-sbarrier-ids-gfx12>` to determine
- which barrier IDs are available to ``s_barrier`` instructions on a given target.
+:::{note}
+Check the {ref}`the table below<amdgpu-execution-synchronization-barriers-sbarrier-ids-gfx12>` to determine
+which barrier IDs are available to `s_barrier` instructions on a given target.
+:::
-.. _amdgpu-execution-synchronization-barriers-execution-model-gfx12-sbarrier:
+(amdgpu-execution-synchronization-barriers-execution-model-gfx12-sbarrier)=
-"Named Barriers" Model Extensions
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+###### "Named Barriers" Model Extensions
-In order to reason about the execution of ``s_barrier`` "named barriers" on GFX12.5 and up,
+In order to reason about the execution of `s_barrier` "named barriers" on GFX12.5 and up,
we define the following extensions to the
-:ref:`barrier execution model<amdgpu-execution-synchronization-barriers-execution-model>`:
+{ref}`barrier execution model<amdgpu-execution-synchronization-barriers-execution-model>`:
-.. note::
+:::{note}
+The aforementioned execution model always applies unless stated otherwise by one of the extensions below.
+:::
- The aforementioned execution model always applies unless stated otherwise by one of the extensions below.
-
-.. rubric:: Barrier *Objects*
+:::{rubric} Barrier *Objects*
+:::
There is a sub-type of barrier *objects* called *named barrier objects*.
*Named barrier objects* inherit all the properties of barrier *objects* as defined by the barrier execution
@@ -300,177 +320,225 @@ model. They are also subject to additional constraints.
Not all barrier *objects* are *named barrier objects*, and both types can coexist in an implementation.
-.. rubric:: Barrier Operations
+:::{rubric} Barrier Operations
+:::
The entirety of the
-:ref:`barrier operations section<amdgpu-execution-synchronization-barriers-execution-model-barrier-operations>`
+{ref}`barrier operations section<amdgpu-execution-synchronization-barriers-execution-model-barrier-operations>`
applies, with the following barrier operation being added:
-* Barrier *join*.
+- Barrier *join*.
- * Allow the thread that executes the operation to *wait* on a barrier *object*.
- * Can only be used on *named barrier objects*.
+ - Allow the thread that executes the operation to *wait* on a barrier *object*.
+ - Can only be used on *named barrier objects*.
-.. rubric:: *Barrier-joined-before*
+:::{rubric} *Barrier-joined-before*
+:::
-A barrier *join* ``J`` is *barrier-joined-before* a barrier operation ``X`` if and only if all
+A barrier *join* `J` is *barrier-joined-before* a barrier operation `X` if and only if all
of the following is true:
-* ``J -> X`` in *thread-barrier-order<BO>*.
-* ``X`` is not a barrier *join*.
-* There is no barrier *join* or *drop* ``JD`` where ``J -> JD -> X`` in *thread-barrier-order<BO>*.
-* There is no barrier *join* ``J'`` on a distinct barrier *object* ``BO'`` such that ``J -> J' -> X`` in
+- `J -> X` in `thread-barrier-order<BO>`.
+- `X` is not a barrier *join*.
+- There is no barrier *join* or *drop* `JD` where `J -> JD -> X` in `thread-barrier-order<BO>`.
+- There is no barrier *join* `J'` on a distinct barrier *object* `BO'` such that `J -> J' -> X` in
*program-order*.
-.. rubric:: Join and Drop Ordering
+:::{rubric} Join and Drop Ordering
+:::
-For every barrier *drop* ``D`` performed on a *named barrier object* ``BO``:
+For every barrier *drop* `D` performed on a *named barrier object* `BO`:
-* There is a barrier *join* ``J`` such that ``J -> D`` in *barrier-joined-before*;
+- There is a barrier *join* `J` such that `J -> D` in *barrier-joined-before*;
otherwise, the behavior is undefined.
-.. rubric:: Join and Wait Ordering
+:::{rubric} Join and Wait Ordering
+:::
-For every barrier *wait* ``W`` performed on a *named barrier object* ``BO``:
+For every barrier *wait* `W` performed on a *named barrier object* `BO`:
-* There is a barrier *join* ``J`` such that ``J -> W`` in *barrier-joined-before*, and
- ``J`` must *barrier-executes-before* at least one operation ``X`` that
- *barrier-participates-in* ``W``; otherwise, the behavior is undefined.
+- There is a barrier *join* `J` such that `J -> W` in *barrier-joined-before*, and
+ `J` must *barrier-executes-before* at least one operation `X` that
+ *barrier-participates-in* `W`; otherwise, the behavior is undefined.
-Code Sequences
-^^^^^^^^^^^^^^
+###### Code Sequences
The following code sequences can be used to implement the barrier operations defined by the
-GFX12 ``s_barrier``
-:ref:`execution synchronization model<amdgpu-execution-synchronization-barriers-execution-model-gfx12-sbarrier>`:
-
-.. table:: s_barrier GFX12
- :name: amdgpu-execution-synchronization-barriers-sbarrier-gfx2
- :widths: 15 15 70
-
- ===================== =========================== ===========================================================
- Barrier Operation(s) Barrier ID AMDGPU Machine Code
- ===================== =========================== ===========================================================
- **Init, Join and Drop**
- -------------------------------------------------------------------------------------------------------------
- *init* - ``-2``, ``-1`` Automatically initialized by the hardware when a workgroup
- is launched. The *expected count* of this barrier is set
- to the number of waves in the workgroup.
-
- *init* - ``-4``, ``-3`` Automatically initialized by the hardware when a workgroup
- is launched as part of a workgroup cluster.
- The *expected count* of this barrier is set to the number
- of workgroups in the workgroup cluster.
-
- *init* - ``0`` Automatically initialized by the hardware and always
- available. This barrier *object* is opaque and immutable
- as all operations other than barrier *join* are no-ops.
-
- *init* - ``[1, 16]`` | ``s_barrier_init <N>``
-
- - ``<N>`` is an immediate constant, or stored in the lower
- half of ``m0``.
- - The value to set as the *expected count* of the barrier
- is stored in the upper half of ``m0``.
-
- *join* - ``-2``, ``-1`` N/A
- - ``-4``, ``-3``
-
- *join* - ``0`` | ``s_barrier_join <N>``
- - ``[1, 16]``
- - ``<N>`` is an immediate constant, or stored in the lower
- half of ``m0``.
-
- *drop* - ``0`` | ``s_barrier_leave``
- - ``[1, 16]``
- - ``s_barrier_leave`` takes no operand. It can only be used
- to *drop* a *named* barrier *object* ``BO`` if ``BO`` was
- previously *joined* using ``s_barrier_join``.
- - *Drops* the *named* barrier *object* ``BO`` if and only if
- there is a barrier *join* ``J`` such that ``J`` is
- *barrier-joined-before* this barrier
- *drop* operation.
-
- *drop* - ``-2``, ``-1`` When a thread ends, it automatically *drops* this barrier
- - ``-4``, ``-3`` *object* if it had previously *joined* it.
-
- **Arrive and Wait**
- -------------------------------------------------------------------------------------------------------------
-
- *arrive* - ``-4``, ``-3`` | ``s_barrier_signal <N>``
- - ``-2``, ``-1`` | Or
- - ``0`` | ``s_barrier_signal_isfirst <N>``
- - ``[1, 16]``
- - ``<N>`` is an immediate constant, or stored in bits ``[4:0]`` of ``m0``.
- - The ``_isfirst`` variant sets ``SCC=1`` if this wave is the first
- to signal the barrier, otherwise ``SCC=0``.
- - For barrier *objects* ``[1, 16]``: When using ``m0`` as an operand,
- if there is a non-zero value contained in the bits ``[22:16]`` of ``m0``,
- the *expected count* of the barrier *object* is set to that value before
- the *arrive count* of the barrier *object* is incremented.
- The new *expected count* value must be greater than or equal to the
- *arrive count*, otherwise the behavior is undefined.
- - For barrier *objects* ``-4`` and ``-3``
- (``cluster`` barriers): only one wave
- per workgroup may arrive at the barrier on behalf of
- its entire workgroup. However, any wave within the workgroup
- cluster can then *wait* on this barrier *object*.
- - This is a no-op on the *NULL barrier*
- (barrier *object* ``0``).
-
- *wait* - ``-4``, ``-3`` ``s_barrier_wait <N>``.
- - ``-2``, ``-1``
- - ``0`` - ``<N>`` is an immediate constant.
- - ``[1, 16]`` - For barrier *objects* ``-2`` and ``-1``: This instruction
- cannot complete before all waves of the
- workgroup have launched.
- - For barrier *objects* ``-4`` and ``-3`` (``cluster`` barriers):
- This instruction cannot complete before all waves of the
- workgroup cluster have launched.
- - This is a no-op on the *NULL barrier* (barrier *object* ``0``).
- - For *named barrier objects*, this instruction always waits on the
- last *named barrier object* that the thread has *joined*, even
- if it is
diff erent from the *barrier object* passed to the
- instruction.
- ===================== =========================== ===========================================================
-
+GFX12 `s_barrier`
+{ref}`execution synchronization model<amdgpu-execution-synchronization-barriers-execution-model-gfx12-sbarrier>`:
+
+```{list-table} s_barrier GFX12
+:name: amdgpu-execution-synchronization-barriers-sbarrier-gfx2
+:widths: 15 15 70
+:header-rows: 1
+
+ * - Barrier Operation(s)
+ - Barrier ID
+ - AMDGPU Machine Code
+ * - **Init, Join and Drop**
+ -
+ -
+ * - *init*
+ - `-2`, `-1`
+ - Automatically initialized by the hardware when a workgroup is launched.
+ The *expected count* of this barrier is set to the number of waves in the
+ workgroup.
+ * - *init*
+ - `-4`, `-3`
+ - Automatically initialized by the hardware when a workgroup is launched
+ as part of a workgroup cluster. The *expected count* of this barrier is
+ set to the number of workgroups in the workgroup cluster.
+ * - *init*
+ - `0`
+ - Automatically initialized by the hardware and always available. This
+ barrier *object* is opaque and immutable as all operations other than
+ barrier *join* are no-ops.
+ * - *init*
+ - `[1, 16]`
+ - `s_barrier_init <N>`
+
+ - `<N>` is an immediate constant, or stored in the lower half of `m0`.
+ - The value to set as the *expected count* of the barrier is stored in
+ the upper half of `m0`.
+ * - *join*
+ - `-2`, `-1`
+
+ `-4`, `-3`
+ - N/A
+ * - *join*
+ - `0`
+
+ `[1, 16]`
+ - `s_barrier_join <N>`
+
+ - `<N>` is an immediate constant, or stored in the lower half of `m0`.
+ * - *drop*
+ - `0`
+
+ `[1, 16]`
+ - `s_barrier_leave`
+
+ - `s_barrier_leave` takes no operand. It can only be used to *drop* a
+ *named* barrier *object* `BO` if `BO` was previously *joined* using
+ `s_barrier_join`.
+ - *Drops* the *named* barrier *object* `BO` if and only if there is a
+ barrier *join* `J` such that `J` is *barrier-joined-before* this
+ barrier *drop* operation.
+ * - *drop*
+ - `-2`, `-1`
+
+ `-4`, `-3`
+ - When a thread ends, it automatically *drops* this barrier *object* if it
+ had previously *joined* it.
+ * - **Arrive and Wait**
+ -
+ -
+ * - *arrive*
+ - `-4`, `-3`
+
+ `-2`, `-1`
+
+ `0`
+
+ `[1, 16]`
+ - `s_barrier_signal <N>`
+
+ Or
+
+ `s_barrier_signal_isfirst <N>`
+
+ - `<N>` is an immediate constant, or stored in bits `[4:0]` of `m0`.
+ - The `_isfirst` variant sets `SCC=1` if this wave is the first to signal
+ the barrier, otherwise `SCC=0`.
+ - For barrier *objects* `[1, 16]`: When using `m0` as an operand, if
+ there is a non-zero value contained in the bits `[22:16]` of `m0`, the
+ *expected count* of the barrier *object* is set to that value before
+ the *arrive count* of the barrier *object* is incremented. The new
+ *expected count* value must be greater than or equal to the
+ *arrive count*, otherwise the behavior is undefined.
+ - For barrier *objects* `-4` and `-3` (`cluster` barriers): only one wave
+ per workgroup may arrive at the barrier on behalf of its entire
+ workgroup. However, any wave within the workgroup cluster can then
+ *wait* on this barrier *object*.
+ - This is a no-op on the *NULL barrier* (barrier *object* `0`).
+ * - *wait*
+ - `-4`, `-3`
+
+ `-2`, `-1`
+
+ `0`
+
+ `[1, 16]`
+ - `s_barrier_wait <N>`.
+
+ - `<N>` is an immediate constant.
+ - For barrier *objects* `-2` and `-1`: This instruction cannot complete
+ before all waves of the workgroup have launched.
+ - For barrier *objects* `-4` and `-3` (`cluster` barriers): This
+ instruction cannot complete before all waves of the workgroup cluster
+ have launched.
+ - This is a no-op on the *NULL barrier* (barrier *object* `0`).
+ - For *named barrier objects*, this instruction always waits on the last
+ *named barrier object* that the thread has *joined*, even if it is
+
diff erent from the *barrier object* passed to the instruction.
+```
The following barrier IDs are available:
-.. table:: s_barrier IDs GFX12
- :name: amdgpu-execution-synchronization-barriers-sbarrier-ids-gfx12
- :widths: 15 15 15 10 45
-
- =============== ============== ============ ======================= ==============================================================
- Barrier ID Scope Availability *Named barrier object*? Description
- =============== ============== ============ ======================= ==============================================================
- ``-4`` ``cluster`` GFX12.5 NO *Cluster trap barrier*; *cluster barrier object* for use by
- all workgroups of a workgroup cluster. Dedicated for the trap
- handler and only available in privileged execution mode
- (not accessible by the shader).
-
- ``-3`` ``cluster`` GFX12.5 NO *Cluster user barrier*; *cluster barrier object* for use by
- all workgroups of a workgroup cluster.
-
- ``-2`` ``workgroup`` GFX12 (all) NO *Workgroup trap barrier*, dedicated for the trap handler and
- only available in privileged execution mode
- (not accessible by the shader).
-
- ``-1`` ``workgroup`` GFX12 (all) NO *Workgroup barrier*.
-
- ``0`` ``workgroup`` GFX12.5 YES *NULL barrier*.
-
- ``[1, 16]`` ``workgroup`` GFX12.5 YES *Named barrier objects* for the shader to assign and use.
- =============== ============== ============ ======================= ==============================================================
-
+```{list-table} s_barrier IDs GFX12
+:name: amdgpu-execution-synchronization-barriers-sbarrier-ids-gfx12
+:widths: 15 15 15 10 45
+:header-rows: 1
+
+ * - Barrier ID
+ - Scope
+ - Availability
+ - *Named barrier object*?
+ - Description
+ * - `-4`
+ - `cluster`
+ - GFX12.5
+ - NO
+ - *Cluster trap barrier*; *cluster barrier object* for use by all
+ workgroups of a workgroup cluster. Dedicated for the trap handler and
+ only available in privileged execution mode (not accessible by the
+ shader).
+ * - `-3`
+ - `cluster`
+ - GFX12.5
+ - NO
+ - *Cluster user barrier*; *cluster barrier object* for use by all
+ workgroups of a workgroup cluster.
+ * - `-2`
+ - `workgroup`
+ - GFX12 (all)
+ - NO
+ - *Workgroup trap barrier*, dedicated for the trap handler and only
+ available in privileged execution mode (not accessible by the shader).
+ * - `-1`
+ - `workgroup`
+ - GFX12 (all)
+ - NO
+ - *Workgroup barrier*.
+ * - `0`
+ - `workgroup`
+ - GFX12.5
+ - YES
+ - *NULL barrier*.
+ * - `[1, 16]`
+ - `workgroup`
+ - GFX12.5
+ - YES
+ - *Named barrier objects* for the shader to assign and use.
+```
Informally, we can note that:
-* All operations on the *NULL named barrier object* other than *join* are no-ops.
+- All operations on the *NULL named barrier object* other than *join* are no-ops.
- * As the *NULL barrier* (barrier ID ``0``) is also a *named* barrier *object*, a thread can
+ - As the *NULL barrier* (barrier ID `0`) is also a *named* barrier *object*, a thread can
use a *join* on the *NULL* barrier as a way to "unjoin" a *named barrier*
(break *barrier-joined-before*) without having to use a *drop* operation.
-* When a thread ends, it does **not** implicitly *drop* any *named barrier objects*
- (barrier IDs ``[0, 16]``) it has *joined*.
+- When a thread ends, it does **not** implicitly *drop* any *named barrier objects*
+ (barrier IDs `[0, 16]`) it has *joined*.
diff --git a/llvm/docs/AMDGPUMemoryModel.md b/llvm/docs/AMDGPUMemoryModel.md
index ac153af5197c7..2bba968de6fb1 100644
--- a/llvm/docs/AMDGPUMemoryModel.md
+++ b/llvm/docs/AMDGPUMemoryModel.md
@@ -1,16 +1,14 @@
-.. _amdgpu-memmodel:
+(amdgpu-memmodel)=
-=====================
- AMDGPU Memory Model
-=====================
+# AMDGPU Memory Model
-.. contents::
- :local:
+```{contents}
+:local: true
+```
-Introduction
-============
+## Introduction
-The :ref:`LLVM memory model<memmodel>` provides broad guarantees that are
+The {ref}`LLVM memory model<memmodel>` provides broad guarantees that are
sufficient to implement inter-thread communication via memory. But in most
communication patterns, not all memory accesses performed by a thread need to be
exposed to other threads. Even when they do need to be exposed, not all threads
@@ -27,23 +25,23 @@ new intrinsics and metadata using operations defined in the default LLVM memory
model. Thus, **there exists a safe-by-default implementation** that produces
executions that are valid in both models.
-Terminology
-===========
+## Terminology
Memory Accesses
- Operations that read or write locations in memory are termed as *memory
- accesses*. Typical examples are ``load``, ``store`` and atomic instructions,
+
+: Operations that read or write locations in memory are termed as *memory
+ accesses*. Typical examples are `load`, `store` and atomic instructions,
as well as many intrinsics.
Synchronizing Operations
- Synchronizing operations control how the side-effects of memory accesses are
+
+: Synchronizing operations control how the side-effects of memory accesses are
propagated in the system. Typical examples are atomic operations (including
- fences) with at least ``release`` or ``acquire`` ordering.
+ fences) with at least `release` or `acquire` ordering.
-.. _amdgpu-scopes:
+(amdgpu-scopes)=
-Scopes
-======
+## Scopes
A *scope* is an abstract description of sets of memory accesses and
synchronizing operations in a multi-threaded execution environment. Each such
@@ -51,28 +49,28 @@ set is called an *instance* of that scope, or a *scope instance* for short.
- Each memory access or synchronizing operation belongs to at most one
instance of every scope defined by the target.
-- When an operation ``X`` specifies a scope ``S``, it indicates the instance of
- ``S`` that contains ``X``. This scope instance is also termed as *X's instance
- of scope S*, or just *X's scope instance* when ``S`` is implied by the
+- When an operation `X` specifies a scope `S`, it indicates the instance of
+ `S` that contains `X`. This scope instance is also termed as *X's instance
+ of scope S*, or just *X's scope instance* when `S` is implied by the
context.
- When an operation does not specify a scope, it indicates the *system*
scope defined below.
-LLVM scopes
------------
+### LLVM scopes
-The LLVM Language Reference defines the following :ref:`scopes<syncscope>`:
+The LLVM Language Reference defines the following {ref}`scopes<syncscope>`:
*system scope* (empty string "")
- There exists a single instance of this scope that contains the memory accesses
+
+: There exists a single instance of this scope that contains the memory accesses
and synchronizing operations performed by all threads.
"singlethread" scope
- Each thread corresponds to a "singlethread" scope instance that contains the
+
+: Each thread corresponds to a "singlethread" scope instance that contains the
memory accesses and synchronizing operations performed by that thread.
-AMDGPU scopes
--------------
+### AMDGPU scopes
The AMDGPU backend further refines the LLVM scopes with the following
target-defined scopes and constraints:
@@ -87,24 +85,23 @@ target-defined scopes and constraints:
These are arranged from largest scope (*system scope*) to smallest scope
("singlethread").
-- Every instance ``X`` of some scope ``S1`` other than "singlethread" scope is
- partitioned by the scope ``S2`` one level below it. Each subset defined by this
- partition is an instance of ``S2`` and is called a *subscope instance* of ``X``.
-- It follows that if two scope instances ``X`` and ``Y`` intersect, then their
- intersection is the smaller of ``X`` and ``Y``.
-- A scope ``S1`` is a *subscope* of a scope ``S2`` if every instance of ``S1``
- is a subscope instance of some instance of ``S2``.
+- Every instance `X` of some scope `S1` other than "singlethread" scope is
+ partitioned by the scope `S2` one level below it. Each subset defined by this
+ partition is an instance of `S2` and is called a *subscope instance* of `X`.
+- It follows that if two scope instances `X` and `Y` intersect, then their
+ intersection is the smaller of `X` and `Y`.
+- A scope `S1` is a *subscope* of a scope `S2` if every instance of `S1`
+ is a subscope instance of some instance of `S2`.
-**Inclusive Scopes**: Two operations ``X`` and ``Y`` are said to have *inclusive
+**Inclusive Scopes**: Two operations `X` and `Y` are said to have *inclusive
scopes* if the scope instance of each operation contains the other operation. In
-that case, the *common scope instance* ``S'`` of ``X`` and ``Y`` is the
-intersection of their scope instances. The scope corresponding to ``S'`` is also
-termed as the *common scope* of ``X`` and ``Y``.
+that case, the *common scope instance* `S'` of `X` and `Y` is the
+intersection of their scope instances. The scope corresponding to `S'` is also
+termed as the *common scope* of `X` and `Y`.
-Availability and Visibility
-===========================
+## Availability and Visibility
-The AMDGPU memory model is built on top of the :ref:`happens-before<memmodel>`
+The AMDGPU memory model is built on top of the {ref}`happens-before<memmodel>`
order defined by the LLVM memory model. But when one of the new intrinsics or
metadata is used, **happens-before by itself is not sufficient** to describe its
observable effects. Instead, the AMDGPU model uses *availability* and
@@ -120,303 +117,315 @@ The AMDGPU memory model *does not change the structure of happens-before*, but
changes the rules that determine how operations may observe the side-effects of
other operations that *happen-before* them.
-Consider a write ``W`` that ``happens-before`` a read ``R`` to the same address:
+Consider a write `W` that `happens-before` a read `R` to the same address:
-- ``R`` can potentially observe the side-effects of ``W`` **only if W is
- visible** to ``R``.
-- ``W`` can potentially be visible to ``R`` **only if W is first made
- available** to ``R``.
+- `R` can potentially observe the side-effects of `W` **only if W is
+ visible** to `R`.
+- `W` can potentially be visible to `R` **only if W is first made
+ available** to `R`.
The instructions used in the default LLVM memory model automatically satisfy
these necessary conditions, and hence they can be explained using the rules from
either memory model. But the new intrinsics and metadata *opt out* of the LLVM
memory model, and can only be explained using the AMDGPU memory model.
-.. _amdgpu-store-available:
+(amdgpu-store-available)=
-store-available
----------------
+### store-available
-.. code-block:: llvm
+```llvm
+ at llvm.amdgcn.av.global.store.b128(ptr, value, scope)
+store atomic [syncscope("<target-scope>")]
+atomicrmw [syncscope("<target-scope>")]
+cmpxchg [syncscope("<target-scope>")]
+```
- @llvm.amdgcn.av.global.store.b128(ptr, value, scope)
- store atomic [syncscope("<target-scope>")]
- atomicrmw [syncscope("<target-scope>")]
- cmpxchg [syncscope("<target-scope>")]
-
-The ``@llvm.amdgcn.av.global.store.b128`` intrinsic performs a non-atomic
-*store-available* operation on ``ptr`` with scope ``scope``.
+The `@llvm.amdgcn.av.global.store.b128` intrinsic performs a non-atomic
+*store-available* operation on `ptr` with scope `scope`.
An atomic operation that results in a store operation is a *store-available*
-operation with scope ``syncscope``.
-
-.. _amdgpu-load-visible:
+operation with scope `syncscope`.
-load-visible
-------------
+(amdgpu-load-visible)=
-.. code-block:: llvm
+### load-visible
- @llvm.amdgcn.av.global.load.b128(ptr, scope)
- load atomic [syncscope("<target-scope>")]
- atomicrmw [syncscope("<target-scope>")]
- cmpxchg [syncscope("<target-scope>")]
+```llvm
+ at llvm.amdgcn.av.global.load.b128(ptr, scope)
+load atomic [syncscope("<target-scope>")]
+atomicrmw [syncscope("<target-scope>")]
+cmpxchg [syncscope("<target-scope>")]
+```
-The ``@llvm.amdgcn.av.global.load.b128`` intrinsic performs a non-atomic
-*load-visible* operation on ``ptr`` with scope ``scope``.
+The `@llvm.amdgcn.av.global.load.b128` intrinsic performs a non-atomic
+*load-visible* operation on `ptr` with scope `scope`.
An atomic operation that results in a read operation is a *load-visible*
-operation with scope ``syncscope``.
-
-.. note::
-
- Metadata cannot be used to model this using ordinary load/store operations,
- because the scope is necessary for correctness. In a hypothetical operation
- like this:
+operation with scope `syncscope`.
- .. code-block:: llvm
+:::{note}
+Metadata cannot be used to model this using ordinary load/store operations,
+because the scope is necessary for correctness. In a hypothetical operation
+like this:
- store ptr, data, !mmra !{!"amdgcn-av", !"workgroup"}
+```llvm
+store ptr, data, !mmra !{!"amdgcn-av", !"workgroup"}
+```
- If the metadata is dropped or ignored, there is no guarantee that the store
- will become available at the intended scope. In implementation terms, the
- store may be completed at a nearer cache than the one required for that
- scope. A corresponding *load-visible* that does not access the same near
- cache will fail to observe this store.
+If the metadata is dropped or ignored, there is no guarantee that the store
+will become available at the intended scope. In implementation terms, the
+store may be completed at a nearer cache than the one required for that
+scope. A corresponding *load-visible* that does not access the same near
+cache will fail to observe this store.
+:::
-.. _amdgpu-av-metadata:
+(amdgpu-av-metadata)=
-AV Metadata
------------
+### AV Metadata
-.. code-block:: llvm
-
- !mmra !{!"amdgcn-av", !"none"}
+```llvm
+!mmra !{!"amdgcn-av", !"none"}
+```
The presence of this metadata removes the ability of synchronizing operations to
establish availability and visibility, and essentially creates *non-av* synchronizing
operations.
-For a synchronizing operation which itself accesses memory (e.g., ``store atomic
-release`` or ``load atomic acquire``), the metadata does not affect the
+For a synchronizing operation which itself accesses memory (e.g., `store atomic
+release` or `load atomic acquire`), the metadata does not affect the
availability or the visibility of the access performed by the operation itself.
It only affects the synchronization of other memory accesses.
-MakeAvailable and MakeVisible
------------------------------
-
-.. code-block:: llvm
-
- store atomic [syncscope("<target-scope>")] <ordering> [, !mmra !{!"amdgcn-av", !"none"}]
- load atomic [syncscope("<target-scope>")] <ordering> [, !mmra !{!"amdgcn-av", !"none"}]
- atomicrmw [syncscope("<target-scope>")] <ordering> [, !mmra !{!"amdgcn-av", !"none"}]
- cmpxchg [syncscope("<target-scope>")] <ordering> [, !mmra !{!"amdgcn-av", !"none"}]
- fence [syncscope("<target-scope>")] <ordering> [, !mmra !{!"amdgcn-av", !"none"}]
-
-A synchronizing operation with at least ``release`` ordering is a
-``MakeAvailable`` operation with scope ``syncscope``, if it is not marked as
-``!{!"amdgcn-av", !"none"}``.
-
-A synchronizing operation with at least ``acquire`` ordering is a
-``MakeVisible`` operation with scope ``syncscope``, if it is not marked as
-``!{!"amdgcn-av", !"none"}``.
-
-.. code-block:: llvm
-
- ; This includes the following operations:
- ; - The atomic store at "agent" scope,
- ; - A store-available operation at "agent" scope on `ptr`,
- ; - A `MakeAvailable` operation at "agent" scope that affects previous memory accesses.
- store atomic syncscope("agent") release ptr
-
- ; This includes the following operations:
- ; - The atomic store at "agent" scope,
- ; - A store-available operation at "agent" scope on `ptr`.
- ; Notably, it does not include a `MakeAvailable` operation on other memory accesses.
- store atomic syncscope("agent") release ptr, !mmra !{!"amdgcn-av", !"none"}
-
-Ordering
-========
-
-.. note::
-
- **TODO:** These ordering operations affect all address spaces. We need to
- eventually make that a parameter similar to the storage class parameter on
- operations and orders in Vulkan.
-
-Availability Operation
-----------------------
-
-An operation ``X`` is an *availability operation* on a write ``W`` if one of the
+### MakeAvailable and MakeVisible
+
+```llvm
+store atomic [syncscope("<target-scope>")] <ordering> [, !mmra !{!"amdgcn-av", !"none"}]
+load atomic [syncscope("<target-scope>")] <ordering> [, !mmra !{!"amdgcn-av", !"none"}]
+atomicrmw [syncscope("<target-scope>")] <ordering> [, !mmra !{!"amdgcn-av", !"none"}]
+cmpxchg [syncscope("<target-scope>")] <ordering> [, !mmra !{!"amdgcn-av", !"none"}]
+fence [syncscope("<target-scope>")] <ordering> [, !mmra !{!"amdgcn-av", !"none"}]
+```
+
+A synchronizing operation with at least `release` ordering is a
+`MakeAvailable` operation with scope `syncscope`, if it is not marked as
+`!{!"amdgcn-av", !"none"}`.
+
+A synchronizing operation with at least `acquire` ordering is a
+`MakeVisible` operation with scope `syncscope`, if it is not marked as
+`!{!"amdgcn-av", !"none"}`.
+
+```llvm
+; This includes the following operations:
+; - The atomic store at "agent" scope,
+; - A store-available operation at "agent" scope on `ptr`,
+; - A `MakeAvailable` operation at "agent" scope that affects previous memory accesses.
+store atomic syncscope("agent") release ptr
+
+; This includes the following operations:
+; - The atomic store at "agent" scope,
+; - A store-available operation at "agent" scope on `ptr`.
+; Notably, it does not include a `MakeAvailable` operation on other memory accesses.
+store atomic syncscope("agent") release ptr, !mmra !{!"amdgcn-av", !"none"}
+```
+
+## Ordering
+
+:::{note}
+**TODO:** These ordering operations affect all address spaces. We need to
+eventually make that a parameter similar to the storage class parameter on
+operations and orders in Vulkan.
+:::
+
+### Availability Operation
+
+An operation `X` is an *availability operation* on a write `W` if one of the
following holds:
-- ``X`` is ``W`` itself, and ``W`` is a *store-available* operation, or,
-- ``X`` is a ``MakeAvailable`` operation that follows ``W`` in program order,
+- `X` is `W` itself, and `W` is a *store-available* operation, or,
+
+- `X` is a `MakeAvailable` operation that follows `W` in program order,
or,
-- ``X`` is a ``MakeAvailable`` operation whose scope instance includes ``W``,
- and there is an availability operation ``Z`` on ``W`` such that:
- - ``Z`` happens-before ``X``, and,
- - ``Z``'s scope instance includes ``X``.
+- `X` is a `MakeAvailable` operation whose scope instance includes `W`,
+ and there is an availability operation `Z` on `W` such that:
-Then ``X`` makes ``W`` available in its own scope instance ``S`` and every
-subscope instance of ``S`` that also includes ``W``.
+ - `Z` happens-before `X`, and,
+ - `Z`'s scope instance includes `X`.
-Visibility Operation
---------------------
+Then `X` makes `W` available in its own scope instance `S` and every
+subscope instance of `S` that also includes `W`.
-An operation ``Y`` is a *visibility operation* on a write ``W`` if ``Y`` is a
-*load-visible* operation to the same address, or a ``MakeVisible`` operation,
+### Visibility Operation
+
+An operation `Y` is a *visibility operation* on a write `W` if `Y` is a
+*load-visible* operation to the same address, or a `MakeVisible` operation,
and one of the following holds:
-- There exists an *availability* operation ``X`` on write ``W`` such that:
+- There exists an *availability* operation `X` on write `W` such that:
- - ``X`` happens-before ``Y``, and,
- - ``X`` and ``Y`` specify inclusive scopes.
+ - `X` happens-before `Y`, and,
+ - `X` and `Y` specify inclusive scopes.
- Then ``Y`` makes ``W`` visible in the common scope instance ``S`` of ``X`` and
- ``Y``, and every subscope instance of ``S`` that includes ``Y``.
+ Then `Y` makes `W` visible in the common scope instance `S` of `X` and
+ `Y`, and every subscope instance of `S` that includes `Y`.
-- There exists a *visibility* operation ``X`` on write ``W`` such that:
+- There exists a *visibility* operation `X` on write `W` such that:
- - ``X`` happens-before ``Y``, and,
- - ``X`` makes ``W`` visible in a scope instance ``S1`` that includes ``Y``, and,
- - ``X`` is included in the scope instance ``S2`` of ``Y``.
+ - `X` happens-before `Y`, and,
+ - `X` makes `W` visible in a scope instance `S1` that includes `Y`, and,
+ - `X` is included in the scope instance `S2` of `Y`.
- Then ``Y`` makes ``W`` visible in the intersection ``S`` of ``S1`` and ``S2``,
- and every subscope instance of ``S`` that includes ``Y``.
+ Then `Y` makes `W` visible in the intersection `S` of `S1` and `S2`,
+ and every subscope instance of `S` that includes `Y`.
-Location Order
---------------
+### Location Order
-A write ``W`` is *location-ordered* before an access ``Y`` to the same address
-if ``W`` is program-ordered before ``Y``.
+A write `W` is *location-ordered* before an access `Y` to the same address
+if `W` is program-ordered before `Y`.
-A write ``W`` is *location-ordered* before a write ``W1`` to the same address if
-there exists an availability operation ``Z`` on ``W`` such that:
+A write `W` is *location-ordered* before a write `W1` to the same address if
+there exists an availability operation `Z` on `W` such that:
-- ``Z`` happens-before ``W1``, and,
-- ``W1`` is included in ``Z``'s scope instance.
+- `Z` happens-before `W1`, and,
+- `W1` is included in `Z`'s scope instance.
-A write ``W`` is *location-ordered* before a read ``R`` to the same address if
-there exists a visibility operation ``Z`` on write ``W`` such that:
+A write `W` is *location-ordered* before a read `R` to the same address if
+there exists a visibility operation `Z` on write `W` such that:
-- ``Z`` is ``R`` itself, or,
-- ``Z`` precedes ``R`` in program order.
+- `Z` is `R` itself, or,
+- `Z` precedes `R` in program order.
The AMDGPU memory model overrides the definition of each byte in the
-:ref:`LLVM memory model<memmodel>` as follows.
+{ref}`LLVM memory model<memmodel>` as follows.
-Every (defined) read operation ``R`` reads a series of bytes written by
+Every (defined) read operation `R` reads a series of bytes written by
(defined) write operations. Each initialized global is assumed to have an
initial *system scoped* atomic write operation that is *location-ordered* before
any other read or write to that same location.
-For each byte of a read ``R``, ``R`` may see any write to the same byte, except:
+For each byte of a read `R`, `R` may see any write to the same byte, except:
-- If a write ``W1`` is *location-ordered* before a write ``W2``, and ``W2`` is
- *location-ordered* before a read ``R``, then ``R`` may not see ``W1``.
-- If a read ``R`` happens-before a write ``W3``, then ``R`` may not see ``W3``.
+- If a write `W1` is *location-ordered* before a write `W2`, and `W2` is
+ *location-ordered* before a read `R`, then `R` may not see `W1`.
+- If a read `R` happens-before a write `W3`, then `R` may not see `W3`.
-The value returned by ``R`` is then defined as follows:
+The value returned by `R` is then defined as follows:
-- If no write is *location-ordered* before a read ``R``, then ``R`` returns
- ``undef``.
-- Otherwise if the set consisting of ``R`` and all writes that ``R`` may see
- contains only atomic operations with inclusive scopes, then ``R`` returns the
+- If no write is *location-ordered* before a read `R`, then `R` returns
+ `undef`.
+- Otherwise if the set consisting of `R` and all writes that `R` may see
+ contains only atomic operations with inclusive scopes, then `R` returns the
value written by one of those writes.
-- Otherwise, if ``R`` may see some write that is not *location-ordered* before
- ``R``, then ``R`` returns ``undef``.
-- Otherwise, if ``R`` may see exactly one write ``W``, then ``R`` returns the
- value written by ``W``.
-- Otherwise, ``R`` returns ``undef``.
-
-Properties
-==========
+- Otherwise, if `R` may see some write that is not *location-ordered* before
+ `R`, then `R` returns `undef`.
+- Otherwise, if `R` may see exactly one write `W`, then `R` returns the
+ value written by `W`.
+- Otherwise, `R` returns `undef`.
-.. tip::
+## Properties
- This section is informational.
+:::{tip}
+This section is informational.
+:::
The following properties follow from the definitions above:
-1. **Happens-before is necessary for location-order.** A write ``W`` is
- *location-ordered* before a read ``R`` only if ``W`` happens-before ``R``.
+1. **Happens-before is necessary for location-order.** A write `W` is
+ *location-ordered* before a read `R` only if `W` happens-before `R`.
This follows from the definition of availability and visibility operations,
which always require a happens-before link with the preceding operation in
the chain.
-
2. **A write cannot be made available in a scope that does not contain it.** The
- definition of an availability operation ``X`` requires that ``X``'s scope
- instance includes ``W`` as a precondition. Since every scope instance that
- includes ``X`` also includes ``W``, availability cannot reach a scope
- instance that excludes ``W``. In other words, availability can only "expand
+ definition of an availability operation `X` requires that `X`'s scope
+ instance includes `W` as a precondition. Since every scope instance that
+ includes `X` also includes `W`, availability cannot reach a scope
+ instance that excludes `W`. In other words, availability can only "expand
outwards" into progressively larger scopes.
-
3. **Visibility is bounded by availability.** When a write is available in a
scope instance, it can be made visible in that scope instance by a visibility
- operation with the corresponding scope. Subsequent ``MakeVisible`` operations
+ operation with the corresponding scope. Subsequent `MakeVisible` operations
make that write visible into narrower scope instances towards the observer.
-
4. **A write can be made visible in a scope instance that does not contain it.**
The definition of a *visibility operation* anchors scope instances to the
- observer (``Y``), not to the original write. The only precondition is that the
+ observer (`Y`), not to the original write. The only precondition is that the
write must already be visible or available in the scope instance of the
visibility operation.
-
-5. **Availability and visibility chains.** For a write ``W`` to be visible to a
- read ``R`` anywhere in the system, the sufficient condition is a chain of
+5. **Availability and visibility chains.** For a write `W` to be visible to a
+ read `R` anywhere in the system, the sufficient condition is a chain of
happens-before edges that include availability and visibility operations with
- inclusive scopes. It is not necessary that ``W`` and ``R`` themselves have
+ inclusive scopes. It is not necessary that `W` and `R` themselves have
inclusive scopes. Each link in the availability and visibility definitions
only checks the immediate predecessor, so intermediate operations can bridge
scope gaps that the endpoints cannot satisfy directly. Such a chain passes
through at least one availability operation and at least one visibility
operation with inclusive scopes, such that their common scope includes both
- ``W`` and ``R``.
+ `W` and `R`.
-.. _amdgcn-av-vulkan:
+(amdgcn-av-vulkan)=
-The Vulkan Memory Model
-=======================
+## The Vulkan Memory Model
The AMDGPU memory model draws heavily on the Vulkan memory model. In
particular, the following instructions are equivalent.
-.. csv-table::
- :header: "LLVM", "SPIRV", "Available/Visible Semantics"
- :widths: 20, 20, 60
-
- "``load``", "``OpLoad NonPrivatePointer``", "\-"
- "``load-visible``", "``OpLoad NonPrivatePointer``", "``MakePointerVisible``"
- "``store``", "``OpStore NonPrivatePointer``", "\-"
- "``store-available``", "``OpStore NonPrivatePointer``", "``MakePointerAvailable``"
- "``load atomic``", "``OpAtomicLoad``", "``MakePointerVisible``. Also ``MakeVisible`` when order is at least ``acquire``."
- "``load atomic !{!""amdgcn-av"", !""none""}``", "``OpAtomicLoad``", "``MakePointerVisible``"
- "``store atomic``", "``OpAtomicStore``", "``MakePointerAvailable``. Also ``MakeAvailable`` when order is at least ``release``."
- "``store atomic !{!""amdgcn-av"", !""none""}``", "``OpAtomicStore``", "``MakePointerAvailable``"
- "``fence``", "``OpMemoryBarrier``", "``MakeAvailable`` when order is at least ``release``, and ``MakeVisible`` when order is at least ``acquire``."
- "``fence !{!""amdgcn-av"", !""none""}``", "``OpMemoryBarrier``", "\-"
-
-.. note::
-
- The above table is representative only, and does not aim to be exhaustive. In
- particular, it does not list composite atomic operations like ``rmw`` and
- ``cmpxchg``. The ordering and semantics of these operations can be determined
- by combining suitable rules such as:
-
- - "``MakeAvailable`` if the order is at least ``release``, and the operation
- results in a store",
- - "Only if it is not marked as ``!{!"amdgcn-av", !"none"}``", etc.
+```{list-table}
+:header-rows: 1
+:widths: 20 20 60
+
+ * - LLVM
+ - SPIRV
+ - Available/Visible Semantics
+ * - `load`
+ - `OpLoad NonPrivatePointer`
+ - \-
+ * - `load-visible`
+ - `OpLoad NonPrivatePointer`
+ - `MakePointerVisible`
+ * - `store`
+ - `OpStore NonPrivatePointer`
+ - \-
+ * - `store-available`
+ - `OpStore NonPrivatePointer`
+ - `MakePointerAvailable`
+ * - `load atomic`
+ - `OpAtomicLoad`
+ - `MakePointerVisible`. Also `MakeVisible` when order is at least `acquire`.
+ * - `load atomic !{!"amdgcn-av", !"none"}`
+ - `OpAtomicLoad`
+ - `MakePointerVisible`
+ * - `store atomic`
+ - `OpAtomicStore`
+ - `MakePointerAvailable`. Also `MakeAvailable` when order is at least `release`.
+ * - `store atomic !{!"amdgcn-av", !"none"}`
+ - `OpAtomicStore`
+ - `MakePointerAvailable`
+ * - `fence`
+ - `OpMemoryBarrier`
+ - `MakeAvailable` when order is at least `release`, and `MakeVisible` when order is at least `acquire`.
+ * - `fence !{!"amdgcn-av", !"none"}`
+ - `OpMemoryBarrier`
+ - \-
+```
+
+:::{note}
+The above table is representative only, and does not aim to be exhaustive. In
+particular, it does not list composite atomic operations like `rmw` and
+`cmpxchg`. The ordering and semantics of these operations can be determined
+by combining suitable rules such as:
+
+- "`MakeAvailable` if the order is at least `release`, and the operation
+ results in a store",
+- "Only if it is not marked as `!{!"amdgcn-av", !"none"}`", etc.
+:::
The AMDGPU memory model is a special case of the Vulkan memory model:
-a. LLVM fence/atomic ordering operations have ``MakeAvailable`` /
- ``MakeVisible`` semantics by default, thus satisfying the availability and
- visibility chains required in Vulkan. Hence the LLVM memory model is a
- "strong" subset of the Vulkan memory model.
-b. The AMDGPU memory model described here makes it possible to opt-out of the
- default ``MakeAvailable`` and ``MakeVisible`` semantics, and instead specify
- it on select places including the new *load-visible* and *store-available*
+1. LLVM fence/atomic ordering operations have `MakeAvailable` / `MakeVisible`
+ semantics by default, thus satisfying the availability and visibility chains
+ required in Vulkan. Hence the LLVM memory model is a "strong" subset of the
+ Vulkan memory model.
+2. The AMDGPU memory model described here makes it possible to opt-out of the
+ default `MakeAvailable` and `MakeVisible` semantics, and instead specify it
+ on select places including the new *load-visible* and *store-available*
operations. This expands the subset of the Vulkan memory model that can now
be expressed in LLVM IR.
diff --git a/llvm/docs/AddingConstrainedIntrinsics.md b/llvm/docs/AddingConstrainedIntrinsics.md
index 4d1739ff11748..2f243dd7ae514 100644
--- a/llvm/docs/AddingConstrainedIntrinsics.md
+++ b/llvm/docs/AddingConstrainedIntrinsics.md
@@ -1,101 +1,112 @@
-==================================================
-How To Add A Constrained Floating-Point Intrinsic
-==================================================
+# How To Add A Constrained Floating-Point Intrinsic
-.. contents::
- :local:
+```{contents}
+:local: true
+```
-.. warning::
- This is a work in progress.
+:::{warning}
+This is a work in progress.
+:::
-Add the intrinsic
-=================
+## Add the intrinsic
Multiple files need to be updated when adding a new constrained intrinsic.
-Add the new intrinsic to the table of intrinsics::
+Add the new intrinsic to the table of intrinsics:
- include/llvm/IR/Intrinsics.td
+```
+include/llvm/IR/Intrinsics.td
+```
-Add SelectionDAG node types
-===========================
+## Add SelectionDAG node types
-Add the new ``STRICT`` version of the node type to the ``ISD::NodeType`` enum::
+Add the new `STRICT` version of the node type to the `ISD::NodeType` enum:
- include/llvm/CodeGen/ISDOpcodes.h
+```
+include/llvm/CodeGen/ISDOpcodes.h
+```
-Strict version name must be a concatenation of prefix ``STRICT_`` and the name
+Strict version name must be a concatenation of prefix `STRICT_` and the name
of the corresponding non-strict node name. For instance, strict version of the
-node ``FADD`` must be ``STRICT_FADD``.
+node `FADD` must be `STRICT_FADD`.
-Update mappings
-===============
+## Update mappings
Add new record to the mapping of instructions to constrained intrinsics and
-DAG nodes::
+DAG nodes:
- include/llvm/IR/ConstrainedOps.def
+```
+include/llvm/IR/ConstrainedOps.def
+```
Follow instructions provided in this file.
-Update IR components
-====================
+## Update IR components
-Update the IR verifier::
+Update the IR verifier:
- lib/IR/Verifier.cpp
+```
+lib/IR/Verifier.cpp
+```
-Update Selector components
-==========================
+## Update Selector components
-Building the SelectionDAG
--------------------------
+### Building the SelectionDAG
-The ``SelectionDAGBuilder::visitConstrainedFPIntrinsic`` function builds DAG nodes
-using mappings specified in ``ConstrainedOps.def``. If however this default build is
+The `SelectionDAGBuilder::visitConstrainedFPIntrinsic` function builds DAG nodes
+using mappings specified in `ConstrainedOps.def`. If however this default build is
not sufficient, the build can be modified, see how it is implemented for
-``STRICT_FP_ROUND``. The new ``STRICT`` node will eventually be converted
-to the matching non-``STRICT`` node. For this reason it should have the same
-operands and values as the non-``STRICT`` version but should also use the chain.
-This makes subsequent sharing of code for ``STRICT`` and non-``STRICT`` code paths
-easier::
+`STRICT_FP_ROUND`. The new `STRICT` node will eventually be converted
+to the matching non-`STRICT` node. For this reason it should have the same
+operands and values as the non-`STRICT` version but should also use the chain.
+This makes subsequent sharing of code for `STRICT` and non-`STRICT` code paths
+easier:
- lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+```
+lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp
+```
-Most of the ``STRICT`` nodes get legalized the same as their matching non-``STRICT``
-counterparts. A new ``STRICT`` node with this property must get added to the
-switch in ``SelectionDAGLegalize::LegalizeOp()``::
+Most of the `STRICT` nodes get legalized the same as their matching non-`STRICT`
+counterparts. A new `STRICT` node with this property must get added to the
+switch in `SelectionDAGLegalize::LegalizeOp()`:
- lib/CodeGen/SelectionDAG/LegalizeDAG.cpp
+```
+lib/CodeGen/SelectionDAG/LegalizeDAG.cpp
+```
Other parts of the legalizer may need to be updated as well. Look for
-places where the non-``STRICT`` counterpart is legalized and update as needed.
-Be careful of the chain since ``STRICT`` nodes use it but their counterparts
+places where the non-`STRICT` counterpart is legalized and update as needed.
+Be careful of the chain since `STRICT` nodes use it but their counterparts
often don't.
-The conversion or mutation of the ``STRICT`` node to a non-``STRICT``
-version of the node happens in ``SelectionDAG::mutateStrictFPToFP()``. In most cases
+The conversion or mutation of the `STRICT` node to a non-`STRICT`
+version of the node happens in `SelectionDAG::mutateStrictFPToFP()`. In most cases
the function can do the conversion using information from ConstrainedOps.def. Be
careful updating this function since some nodes have the same return type
as their input operand, but some are
diff erent. Both of these cases must
-be properly handled::
+be properly handled:
- lib/CodeGen/SelectionDAG/SelectionDAG.cpp
+```
+lib/CodeGen/SelectionDAG/SelectionDAG.cpp
+```
Whether the mutation happens or not depends on how the new node has been
-registered in ``TargetLoweringBase::initActions()``. By default, all strict nodes are
-registered with Expand action::
+registered in `TargetLoweringBase::initActions()`. By default, all strict nodes are
+registered with Expand action:
- lib/CodeGen/TargetLoweringBase.cpp
+```
+lib/CodeGen/TargetLoweringBase.cpp
+```
To make debug logs readable, it is helpful to update the SelectionDAG's
-debug logger:::
+debug logger:
- lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp
+```
+lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp
+```
-Add documentation and tests
-===========================
+## Add documentation and tests
-::
-
- docs/LangRef.md
+```
+docs/LangRef.md
+```
diff --git a/llvm/docs/BigEndianNEON.md b/llvm/docs/BigEndianNEON.md
index a11f292a5a818..233d0518d5344 100644
--- a/llvm/docs/BigEndianNEON.md
+++ b/llvm/docs/BigEndianNEON.md
@@ -1,12 +1,15 @@
-==============================================
-Using ARM NEON instructions in big-endian mode
-==============================================
+---
+myst:
+ footnote_transition: false
+---
-.. contents::
- :local:
+# Using ARM NEON instructions in big-endian mode
-Introduction
-============
+```{contents}
+:local: true
+```
+
+## Introduction
Generating code for big-endian ARM processors is straightforward for the most part. NEON loads and stores, however, have some interesting properties that make code generation decisions less obvious in big-endian mode.
@@ -14,34 +17,38 @@ The aim of this document is to explain the problem with NEON loads and stores, a
In this document, the term "vector" refers to what the ARM ABI calls a "short vector", which is a sequence of items that can fit in a NEON register. This sequence can be 64 or 128 bits in length, and can constitute 8, 16, 32 or 64 bit items. This document refers to A64 instructions throughout, but is almost applicable to the A32/ARMv7 instruction sets also. The ABI format for passing vectors in A32 is slightly
diff erent to A64. Apart from that, the same concepts apply.
-Example: C-level intrinsics -> assembly
----------------------------------------
+### Example: C-level intrinsics -> assembly
It may be helpful to first illustrate how C-level ARM NEON intrinsics are lowered to instructions.
-This trivial C function takes a vector of four ints and sets the zero'th lane to the value "42"::
+This trivial C function takes a vector of four ints and sets the zero'th lane to the value "42":
- #include <arm_neon.h>
- int32x4_t f(int32x4_t p) {
- return vsetq_lane_s32(42, p, 0);
- }
+```
+#include <arm_neon.h>
+int32x4_t f(int32x4_t p) {
+ return vsetq_lane_s32(42, p, 0);
+}
+```
-``arm_neon.h`` intrinsics generate "generic" IR where possible (that is, normal IR instructions, not ``llvm.arm.neon.*`` intrinsic calls). The above generates::
+`arm_neon.h` intrinsics generate "generic" IR where possible (that is, normal IR instructions, not `llvm.arm.neon.*` intrinsic calls). The above generates:
- define <4 x i32> @f(<4 x i32> %p) {
- %vset_lane = insertelement <4 x i32> %p, i32 42, i32 0
- ret <4 x i32> %vset_lane
- }
+```
+define <4 x i32> @f(<4 x i32> %p) {
+ %vset_lane = insertelement <4 x i32> %p, i32 42, i32 0
+ ret <4 x i32> %vset_lane
+}
+```
-Which then becomes the following trivial assembly::
+Which then becomes the following trivial assembly:
- f: // @f
- movz w8, #0x2a
- ins v0.s[0], w8
- ret
+```
+f: // @f
+ movz w8, #0x2a
+ ins v0.s[0], w8
+ ret
+```
-Problem
-=======
+## Problem
The main problem is how vectors are represented in memory and in registers.
@@ -49,156 +56,155 @@ First, a recap. The "endianness" of an item affects its representation in memory
A "little endian" layout has the least significant byte first (lowest in memory address). A "big endian" layout has the *most* significant byte first. This means that when loading an item from big endian memory, the lowest 8-bits in memory must go in the most significant 8-bits, and so forth.
-``LDR`` and ``LD1``
-===================
-
-.. figure:: ARM-BE-ldr.png
- :align: right
+## `LDR` and `LD1`
- Big endian vector load using ``LDR``.
+:::{figure} ARM-BE-ldr.png
+:align: right
+Big endian vector load using `LDR`.
+:::
-A vector is a consecutive sequence of items that are operated on simultaneously. To load a 64-bit vector, 64 bits need to be read from memory. In little-endian mode, we can do this by just performing a 64-bit load - ``LDR q0, [foo]``. However, if we try this in big-endian mode, because of the byte swapping the lane indices end up being swapped! The zero'th item as laid out in memory becomes the n'th lane in the vector.
+A vector is a consecutive sequence of items that are operated on simultaneously. To load a 64-bit vector, 64 bits need to be read from memory. In little-endian mode, we can do this by just performing a 64-bit load - `LDR q0, [foo]`. However, if we try this in big-endian mode, because of the byte swapping the lane indices end up being swapped! The zero'th item as laid out in memory becomes the n'th lane in the vector.
-.. figure:: ARM-BE-ld1.png
- :align: right
+:::{figure} ARM-BE-ld1.png
+:align: right
- Big endian vector load using ``LD1``. Note that the lanes retain the correct ordering.
+Big endian vector load using `LD1`. Note that the lanes retain the correct ordering.
+:::
+Because of this, the `LD1` instruction performs a vector load but performs byte swapping not on the entire 64 bits, but on the individual items within the vector. This means that the register content is the same as it would have been on a little-endian system.
-Because of this, the ``LD1`` instruction performs a vector load but performs byte swapping not on the entire 64 bits, but on the individual items within the vector. This means that the register content is the same as it would have been on a little-endian system.
-
-It may seem that ``LD1`` should suffice to perform vector loads on a big-endian machine. However, there are pros and cons to the two approaches that make it less than simple which register format to pick.
+It may seem that `LD1` should suffice to perform vector loads on a big-endian machine. However, there are pros and cons to the two approaches that make it less than simple which register format to pick.
There are two options:
- 1. The content of a vector register is the same *as if* it had been loaded with an ``LDR`` instruction.
- 2. The content of a vector register is the same *as if* it had been loaded with an ``LD1`` instruction.
-
-Because ``LD1 == LDR + REV`` and similarly ``LDR == LD1 + REV`` (on a big-endian system), we can simulate either type of load with the other type of load plus a ``REV`` instruction. So we're not deciding which instructions to use, but which format to use (which will then influence which instruction is best to use).
+> 1. The content of a vector register is the same *as if* it had been loaded with an `LDR` instruction.
+> 2. The content of a vector register is the same *as if* it had been loaded with an `LD1` instruction.
-.. The 'clearer' container is required to make the following section header come after the floated
- images above.
-.. container:: clearer
+Because `LD1 == LDR + REV` and similarly `LDR == LD1 + REV` (on a big-endian system), we can simulate either type of load with the other type of load plus a `REV` instruction. So we're not deciding which instructions to use, but which format to use (which will then influence which instruction is best to use).
- Note that throughout this section, we only mention loads. Stores have exactly the same problems as their associated loads, so have been skipped for brevity.
+% The 'clearer' container is required to make the following section header come after the floated
+% images above.
+:::{container} clearer
+Note that throughout this section, we only mention loads. Stores have exactly the same problems as their associated loads, so have been skipped for brevity.
+:::
-Considerations
-==============
+## Considerations
-LLVM IR Lane ordering
----------------------
+### LLVM IR Lane ordering
-LLVM IR has first class vector types. In LLVM IR, the zero'th element of a vector resides at the lowest memory address. The optimizer relies on this property in certain areas, for example, when concatenating vectors together. The intention is for arrays and vectors to have identical memory layouts - ``[4 x i8]`` and ``<4 x i8>`` should be represented the same in memory. Without this property, there would be many special cases that the optimizer would have to cleverly handle.
+LLVM IR has first class vector types. In LLVM IR, the zero'th element of a vector resides at the lowest memory address. The optimizer relies on this property in certain areas, for example, when concatenating vectors together. The intention is for arrays and vectors to have identical memory layouts - `[4 x i8]` and `<4 x i8>` should be represented the same in memory. Without this property, there would be many special cases that the optimizer would have to cleverly handle.
-Use of ``LDR`` would break this lane ordering property. This doesn't preclude the use of ``LDR``, but we would have to do one of two things:
+Use of `LDR` would break this lane ordering property. This doesn't preclude the use of `LDR`, but we would have to do one of two things:
- 1. Insert a ``REV`` instruction to reverse the lane order after every ``LDR``.
- 2. Disable all optimizations that rely on lane layout, and for every access to an individual lane (``insertelement``/``extractelement``/``shufflevector``) reverse the lane index.
+> 1. Insert a `REV` instruction to reverse the lane order after every `LDR`.
+> 2. Disable all optimizations that rely on lane layout, and for every access to an individual lane (`insertelement`/`extractelement`/`shufflevector`) reverse the lane index.
-AAPCS
------
+### AAPCS
The ARM procedure call standard (AAPCS) defines the ABI for passing vectors between functions in registers. It states:
- When a short vector is transferred between registers and memory, it is treated as an opaque object. That is a short vector is stored in memory as if it were stored with a single ``STR`` of the entire register; a short vector is loaded from memory using the corresponding ``LDR`` instruction. On a little-endian system, this means that element 0 will always contain the lowest addressed element of a short vector; on a big-endian system element 0 will contain the highest-addressed element of a short vector.
-
- -- Procedure Call Standard for the ARM 64-bit Architecture (AArch64), 4.1.2 Short Vectors
-
-The use of ``LDR`` and ``STR`` as the ABI defines has at least one advantage over ``LD1`` and ``ST1``. ``LDR`` and ``STR`` are oblivious to the size of the individual lanes of a vector. ``LD1`` and ``ST1`` are not - the lane size is encoded within them. This is important across an ABI boundary because it would become necessary to know the lane width the callee expects. Consider the following code:
-
-.. code-block:: c
-
- <callee.c>
- void callee(uint32x2_t v) {
- ...
- }
-
- <caller.c>
- extern void callee(uint32x2_t);
- void caller() {
- callee(...);
- }
-
-If ``callee`` changed its signature to ``uint16x4_t``, which is equivalent in register content, if we passed as ``LD1`` we'd break this code until ``caller`` was updated and recompiled.
+> When a short vector is transferred between registers and memory, it is treated
+> as an opaque object. That is a short vector is stored in memory as if it were
+> stored with a single `STR` of the entire register; a short vector is loaded
+> from memory using the corresponding `LDR` instruction. On a little-endian
+> system, this means that element 0 will always contain the lowest addressed
+> element of a short vector; on a big-endian system element 0 will contain the
+> highest-addressed element of a short vector.
+>
+> --- Procedure Call Standard for the ARM 64-bit Architecture (AArch64),
+> 4.1.2 Short Vectors
+
+The use of `LDR` and `STR` as the ABI defines has at least one advantage over `LD1` and `ST1`. `LDR` and `STR` are oblivious to the size of the individual lanes of a vector. `LD1` and `ST1` are not - the lane size is encoded within them. This is important across an ABI boundary because it would become necessary to know the lane width the callee expects. Consider the following code:
+
+```c
+<callee.c>
+void callee(uint32x2_t v) {
+ ...
+}
+
+<caller.c>
+extern void callee(uint32x2_t);
+void caller() {
+ callee(...);
+}
+```
+
+If `callee` changed its signature to `uint16x4_t`, which is equivalent in register content, if we passed as `LD1` we'd break this code until `caller` was updated and recompiled.
There is an argument that if the signatures of the two functions are
diff erent then the behaviour should be undefined. But there may be functions that are agnostic to the lane layout of the vector, and treating the vector as an opaque value (just loading it and storing it) would be impossible without a common format across ABI boundaries.
-So to preserve ABI compatibility, we need to use the ``LDR`` lane layout across function calls.
+So to preserve ABI compatibility, we need to use the `LDR` lane layout across function calls.
-Alignment
----------
+### Alignment
-In strict alignment mode, ``LDR qX`` requires its address to be 128-bit aligned, whereas ``LD1`` only requires it to be as aligned as the lane size. If we canonicalised on using ``LDR``, we'd still need to use ``LD1`` in some places to avoid alignment faults (the result of the ``LD1`` would then need to be reversed with ``REV``).
+In strict alignment mode, `LDR qX` requires its address to be 128-bit aligned, whereas `LD1` only requires it to be as aligned as the lane size. If we canonicalised on using `LDR`, we'd still need to use `LD1` in some places to avoid alignment faults (the result of the `LD1` would then need to be reversed with `REV`).
Most operating systems, however, do not run with alignment faults enabled, so this is often not an issue.
-Summary
--------
+### Summary
The following table summarises the instructions that are required to be emitted for each property mentioned above for each of the two solutions.
-+-------------------------------+-------------------------------+---------------------+
-| | ``LDR`` layout | ``LD1`` layout |
-+===============================+===============================+=====================+
-| Lane ordering | ``LDR + REV`` | ``LD1`` |
-+-------------------------------+-------------------------------+---------------------+
-| AAPCS | ``LDR`` | ``LD1 + REV`` |
-+-------------------------------+-------------------------------+---------------------+
-| Alignment for strict mode | ``LDR`` / ``LD1 + REV`` | ``LD1`` |
-+-------------------------------+-------------------------------+---------------------+
+| | `LDR` layout | `LD1` layout |
+| ------------------------- | ------------------- | ------------ |
+| Lane ordering | `LDR + REV` | `LD1` |
+| AAPCS | `LDR` | `LD1 + REV` |
+| Alignment for strict mode | `LDR` / `LD1 + REV` | `LD1` |
-Neither approach is perfect, and choosing one boils down to choosing the lesser of two evils. The issue with lane ordering, it was decided, would have to change target-agnostic compiler passes and would result in a strange IR in which lane indices were reversed. It was decided that this was worse than the changes that would have to be made to support ``LD1``, so ``LD1`` was chosen as the canonical vector load instruction (and by inference, ``ST1`` for vector stores).
+Neither approach is perfect, and choosing one boils down to choosing the lesser of two evils. The issue with lane ordering, it was decided, would have to change target-agnostic compiler passes and would result in a strange IR in which lane indices were reversed. It was decided that this was worse than the changes that would have to be made to support `LD1`, so `LD1` was chosen as the canonical vector load instruction (and by inference, `ST1` for vector stores).
-Implementation
-==============
+## Implementation
There are 3 parts to the implementation:
- 1. Predicate ``LDR`` and ``STR`` instructions so that they are never allowed to be selected to generate vector loads and stores. The exception is one-lane vectors [1]_; by definition, these cannot have lane ordering problems so are fine to use ``LDR``/``STR``.
-
- 2. Create code generation patterns for bitconverts that create ``REV`` instructions.
-
- 3. Make sure appropriate bitconverts are created so that vector values get passed over call boundaries as 1-element vectors (which is the same as if they were loaded with ``LDR``).
-
-Bitconverts
------------
+1. Predicate `LDR` and `STR` instructions so that they are never allowed to be selected to generate vector loads and stores. The exception is one-lane vectors [^1]; by definition, these cannot have lane ordering problems so are fine to use `LDR`/`STR`.
+2. Create code generation patterns for bitconverts that create `REV` instructions.
+3. Make sure appropriate bitconverts are created so that vector values get passed over call boundaries as 1-element vectors (which is the same as if they were loaded with `LDR`).
-.. image:: ARM-BE-bitcastfail.png
- :align: right
+### Bitconverts
-The main problem with the ``LD1`` solution is dealing with bitconverts (or bitcasts, or reinterpret casts). These are pseudo instructions that only change the compiler's interpretation of data, not the underlying data itself. A requirement is that if data is loaded and then saved again (called a "round trip"), the memory contents should be the same after the store as before the load. If a vector is loaded and then bitconverted to a
diff erent vector type before being stored, the round trip will currently be broken.
+```{image} ARM-BE-bitcastfail.png
+:align: right
+```
-Take this code sequence, for example::
+The main problem with the `LD1` solution is dealing with bitconverts (or bitcasts, or reinterpret casts). These are pseudo instructions that only change the compiler's interpretation of data, not the underlying data itself. A requirement is that if data is loaded and then saved again (called a "round trip"), the memory contents should be the same after the store as before the load. If a vector is loaded and then bitconverted to a
diff erent vector type before being stored, the round trip will currently be broken.
- %0 = load <4 x i32> %x
- %1 = bitcast <4 x i32> %0 to <2 x i64>
- store <2 x i64> %1, <2 x i64>* %y
+Take this code sequence, for example:
-This would produce a code sequence such as that in the figure on the right. The mismatched ``LD1`` and ``ST1`` cause the stored data to
diff er from the loaded data.
+```
+%0 = load <4 x i32> %x
+%1 = bitcast <4 x i32> %0 to <2 x i64>
+ store <2 x i64> %1, <2 x i64>* %y
+```
-.. container:: clearer
+This would produce a code sequence such as that in the figure on the right. The mismatched `LD1` and `ST1` cause the stored data to
diff er from the loaded data.
- When we see a bitcast from type ``X`` to type ``Y``, what we need to do is to change the in-register representation of the data to be *as if* it had just been loaded by a ``LD1`` of type ``Y``.
+:::{container} clearer
+When we see a bitcast from type `X` to type `Y`, what we need to do is to change the in-register representation of the data to be *as if* it had just been loaded by a `LD1` of type `Y`.
+:::
-.. image:: ARM-BE-bitcastsuccess.png
- :align: right
+```{image} ARM-BE-bitcastsuccess.png
+:align: right
+```
-Conceptually, this is simple - we can insert a ``REV`` undoing the ``LD1`` of type ``X`` (converting the in-register representation to the same as if it had been loaded by ``LDR``) and then insert another ``REV`` to change the representation to be as if it had been loaded by an ``LD1`` of type ``Y``.
+Conceptually, this is simple - we can insert a `REV` undoing the `LD1` of type `X` (converting the in-register representation to the same as if it had been loaded by `LDR`) and then insert another `REV` to change the representation to be as if it had been loaded by an `LD1` of type `Y`.
-For the previous example, this would be::
+For the previous example, this would be:
- LD1 v0.4s, [x]
+```
+LD1 v0.4s, [x]
- REV64 v0.4s, v0.4s // There is no REV128 instruction, so it must be synthesizedcd
- EXT v0.16b, v0.16b, v0.16b, #8 // with a REV64 then an EXT to swap the two 64-bit elements.
+REV64 v0.4s, v0.4s // There is no REV128 instruction, so it must be synthesizedcd
+EXT v0.16b, v0.16b, v0.16b, #8 // with a REV64 then an EXT to swap the two 64-bit elements.
- REV64 v0.2d, v0.2d
- EXT v0.16b, v0.16b, v0.16b, #8
+REV64 v0.2d, v0.2d
+EXT v0.16b, v0.16b, v0.16b, #8
- ST1 v0.2d, [y]
+ST1 v0.2d, [y]
+```
-It turns out that these ``REV`` pairs can, in almost all cases, be squashed together into a single ``REV``. For the example above, a ``REV128 4s`` + ``REV128 2d`` is actually a ``REV64 4s``, as shown in the figure on the right.
+It turns out that these `REV` pairs can, in almost all cases, be squashed together into a single `REV`. For the example above, a `REV128 4s` + `REV128 2d` is actually a `REV64 4s`, as shown in the figure on the right.
-.. [1] One-lane vectors may seem useless as a concept, but they serve to distinguish between values held in general-purpose registers and values held in NEON/VFP registers. For example, an ``i64`` would live in an ``x`` register, but ``<1 x i64>`` would live in a ``d`` register.
+[^1]: One-lane vectors may seem useless as a concept, but they serve to distinguish between values held in general-purpose registers and values held in NEON/VFP registers. For example, an `i64` would live in an `x` register, but `<1 x i64>` would live in a `d` register.
diff --git a/llvm/docs/CompileCudaWithLLVM.md b/llvm/docs/CompileCudaWithLLVM.md
index 7e0c99917795e..8811d1617e319 100644
--- a/llvm/docs/CompileCudaWithLLVM.md
+++ b/llvm/docs/CompileCudaWithLLVM.md
@@ -1,26 +1,21 @@
-=========================
-Compiling CUDA with clang
-=========================
+# Compiling CUDA with clang
-.. contents::
- :local:
+```{contents}
+:local: true
+```
-Introduction
-============
+## Introduction
This document describes how to compile CUDA code with clang, and gives some
details about LLVM and clang's CUDA implementations.
This document assumes a basic familiarity with CUDA. Information about CUDA
programming can be found in the
-`CUDA programming guide
-<http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html>`_.
+[CUDA programming guide](http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html).
-Compiling CUDA Code
-===================
+## Compiling CUDA Code
-Prerequisites
--------------
+### Prerequisites
CUDA has been supported since LLVM 3.9. Clang typically supports the recent
major CUDA releases, though the support for the most recent versions may need
@@ -28,11 +23,9 @@ Clang compiled from recent sources. If clang detects a newer CUDA version,
it will issue a warning and will make a best-effort attempt to use detected
CUDA SDK as if it were the most recent version supported by Clang.
-Before building CUDA code, you'll need to have installed the CUDA SDK. See
-`NVIDIA's CUDA installation guide
-<https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html>`_ for
-details. Note that clang `may not support
-<https://bugs.llvm.org/show_bug.cgi?id=26966>`_ the CUDA toolkit as installed by
+Before building CUDA code, you'll need to have installed the CUDA SDK. See
+[NVIDIA's CUDA installation guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html) for
+details. Note that clang [may not support](https://bugs.llvm.org/show_bug.cgi?id=26966) the CUDA toolkit as installed by
some Linux package managers. Clang does attempt to deal with specific details of
CUDA installation on a handful of common Linux distributions, but in general the
most reliable way to make it work is to install CUDA in a single directory from
@@ -41,67 +34,65 @@ NVIDIA's `.run` package and specify its location via `--cuda-path=...` argument.
CUDA compilation is fully supported on Linux. Compilation on Windows should work, but your mileage may vary.
Compilation on macOS is no longer supported as CUDA support has been dropped by NVIDIA.
-Invoking clang
---------------
+### Invoking clang
Invoking clang for CUDA compilation works similarly to compiling regular C++.
You just need to be aware of a few additional flags.
-You can use `this <https://gist.github.com/855e277884eb6b388cd2f00d956c2fd4>`_
-program as a toy example. Save it as ``axpy.cu``. (Clang detects that you're
-compiling CUDA code by noticing that your filename ends with ``.cu``.
-Alternatively, you can pass ``-x cuda``.)
+You can use [this](https://gist.github.com/855e277884eb6b388cd2f00d956c2fd4)
+program as a toy example. Save it as `axpy.cu`. (Clang detects that you're
+compiling CUDA code by noticing that your filename ends with `.cu`.
+Alternatively, you can pass `-x cuda`.)
To build and run, run the following commands, filling in the parts in angle
brackets as described below:
-.. code-block:: console
-
- $ clang++ axpy.cu -o axpy --offload-arch=<GPU arch> \
- -L<CUDA install path>/<lib64 or lib> \
- -lcudart_static -ldl -lrt -pthread
- $ ./axpy
- y[0] = 2
- y[1] = 4
- y[2] = 6
- y[3] = 8
+```console
+$ clang++ axpy.cu -o axpy --offload-arch=<GPU arch> \
+ -L<CUDA install path>/<lib64 or lib> \
+ -lcudart_static -ldl -lrt -pthread
+$ ./axpy
+y[0] = 2
+y[1] = 4
+y[2] = 6
+y[3] = 8
+```
Note that it has to be `clang++` as CUDA headers rely on C++ features.
-.. note::
- macOS is no longer supported for CUDA compilation.
+:::{note}
+macOS is no longer supported for CUDA compilation.
+:::
-* ``<CUDA install path>`` -- the directory where you installed CUDA SDK.
- Typically, ``/usr/local/cuda``.
+- `<CUDA install path>` -- the directory where you installed CUDA SDK.
+ Typically, `/usr/local/cuda`.
- Pass e.g. ``-L/usr/local/cuda/lib64`` if compiling in 64-bit mode; otherwise,
- pass e.g. ``-L/usr/local/cuda/lib``. (In CUDA, the device code and host code
+ Pass e.g. `-L/usr/local/cuda/lib64` if compiling in 64-bit mode; otherwise,
+ pass e.g. `-L/usr/local/cuda/lib`. (In CUDA, the device code and host code
always have the same pointer widths, so if you're compiling 64-bit code for
the host, you're also compiling 64-bit code for the device.) Note that as of
- v10.0 CUDA SDK `no longer supports compilation of 32-bit
- applications <https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#deprecated-features>`_.
+ v10.0 CUDA SDK [no longer supports compilation of 32-bit
+ applications](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#deprecated-features).
-* ``<GPU arch>`` -- the `compute capability
- <https://developer.nvidia.com/cuda-gpus>`_ of your GPU. For example, if you
+- `<GPU arch>` -- the [compute capability](https://developer.nvidia.com/cuda-gpus) of your GPU. For example, if you
want to run your program on a GPU with compute capability of 8.0, specify
- ``--offload-arch=sm_80``.
+ `--offload-arch=sm_80`.
- Note: You cannot pass ``compute_XX`` as an argument to ``--offload-arch``;
- only ``sm_XX`` is currently supported.
+ Note: You cannot pass `compute_XX` as an argument to `--offload-arch`;
+ only `sm_XX` is currently supported.
CUDA compilation no longer includes PTX by default. If you want to enable it,
- use ``--cuda-include-ptx=all|sm_*``. For example, a binary compiled with
- ``--offload-arch=sm_80`` would need ``--cuda-include-ptx=sm_80`` (or ``all``)
- to be forwards-compatible with e.g. ``sm_86`` GPUs.
+ use `--cuda-include-ptx=all|sm_*`. For example, a binary compiled with
+ `--offload-arch=sm_80` would need `--cuda-include-ptx=sm_80` (or `all`)
+ to be forwards-compatible with e.g. `sm_86` GPUs.
- You can pass ``--offload-arch`` multiple times to compile for multiple archs.
+ You can pass `--offload-arch` multiple times to compile for multiple archs.
-The `-L` and `-l` flags only need to be passed when linking. When compiling,
-you may also need to pass ``--cuda-path=/path/to/cuda`` if you didn't install
-the CUDA SDK into ``/usr/local/cuda`` or ``/usr/local/cuda-X.Y``.
+The `-L` and `-l` flags only need to be passed when linking. When compiling,
+you may also need to pass `--cuda-path=/path/to/cuda` if you didn't install
+the CUDA SDK into `/usr/local/cuda` or `/usr/local/cuda-X.Y`.
-Flags that control numerical code
----------------------------------
+### Flags that control numerical code
If you're using GPUs, you probably care about making numerical code run fast.
GPU hardware allows for more control over numerical operations than most CPUs,
@@ -109,184 +100,173 @@ but this results in more compiler options for you to juggle.
Flags you may wish to tweak include:
-* ``-ffp-contract={on,off,fast}`` (defaults to ``fast`` on host and device when
+- `-ffp-contract={on,off,fast}` (defaults to `fast` on host and device when
compiling CUDA) Controls whether the compiler emits fused multiply-add
operations.
- * ``off``: never emit fma operations, and prevent ptxas from fusing multiply
+ - `off`: never emit fma operations, and prevent ptxas from fusing multiply
and add instructions.
- * ``on``: fuse multiplies and adds within a single statement, but never
- across statements (C11 semantics). Prevent ptxas from fusing other
+ - `on`: fuse multiplies and adds within a single statement, but never
+ across statements (C11 semantics). Prevent ptxas from fusing other
multiplies and adds.
- * ``fast``: fuse multiplies and adds wherever profitable, even across
- statements. Doesn't prevent ptxas from fusing additional multiplies and
+ - `fast`: fuse multiplies and adds wherever profitable, even across
+ statements. Doesn't prevent ptxas from fusing additional multiplies and
adds.
Fused multiply-add instructions can be much faster than the unfused
equivalents, but because the intermediate result in an fma is not rounded,
this flag can affect numerical code.
-* ``-fcuda-flush-denormals-to-zero`` (default: off) When this is enabled,
- floating point operations may flush `denormal
- <https://en.wikipedia.org/wiki/Denormal_number>`_ inputs and/or outputs to 0.
+- `-fcuda-flush-denormals-to-zero` (default: off) When this is enabled,
+ floating point operations may flush [denormal](https://en.wikipedia.org/wiki/Denormal_number) inputs and/or outputs to 0.
Operations on denormal numbers are often much slower than the same operations
on normal numbers.
-* ``-fcuda-approx-transcendentals`` (default: off) When this is enabled, the
+- `-fcuda-approx-transcendentals` (default: off) When this is enabled, the
compiler may emit calls to faster, approximate versions of transcendental
- functions, instead of using the slower, fully IEEE-compliant versions. For
- example, this flag allows clang to emit the ptx ``sin.approx.f32``
+ functions, instead of using the slower, fully IEEE-compliant versions. For
+ example, this flag allows clang to emit the ptx `sin.approx.f32`
instruction.
- This is implied by ``-ffast-math``.
+ This is implied by `-ffast-math`.
-Standard library support
-========================
+## Standard library support
In clang and nvcc, most of the C++ standard library is not supported on the
device side.
-``<math.h>`` and ``<cmath>``
-----------------------------
+### `<math.h>` and `<cmath>`
-In clang, ``math.h`` and ``cmath`` are available and `pass
-<https://github.com/llvm/llvm-test-suite/blob/main/External/CUDA/math_h.cu>`_
-`tests
-<https://github.com/llvm/llvm-test-suite/blob/main/External/CUDA/cmath.cu>`_
+In clang, `math.h` and `cmath` are available and [pass](https://github.com/llvm/llvm-test-suite/blob/main/External/CUDA/math_h.cu)
+[tests](https://github.com/llvm/llvm-test-suite/blob/main/External/CUDA/cmath.cu)
adapted from libc++'s test suite.
-In nvcc ``math.h`` and ``cmath`` are mostly available. Versions of ``::foof``
-in namespace std (e.g. ``std::sinf``) are not available, and where the standard
+In nvcc `math.h` and `cmath` are mostly available. Versions of `::foof`
+in namespace std (e.g. `std::sinf`) are not available, and where the standard
calls for overloads that take integral arguments, these are usually not
available.
-.. code-block:: c++
-
- #include <math.h>
- #include <cmath.h>
-
- // clang is OK with everything in this function.
- __device__ void test() {
- std::sin(0.); // nvcc - ok
- std::sin(0); // nvcc - error, because no std::sin(int) override is available.
- sin(0); // nvcc - same as above.
-
- sinf(0.); // nvcc - ok
- std::sinf(0.); // nvcc - no such function
- }
-
-``<std::complex>``
-------------------
-
-nvcc does not officially support ``std::complex``. It's an error to use
-``std::complex`` in ``__device__`` code, but it often works in ``__host__
-__device__`` code due to nvcc's interpretation of the "wrong-side rule" (see
-below). However, we have heard from implementers that it's possible to get
-into situations where nvcc will omit a call to an ``std::complex`` function,
-especially when compiling without optimizations. Using ``--expt-relaxed-constexpr``
+```c++
+#include <math.h>
+#include <cmath.h>
+
+// clang is OK with everything in this function.
+__device__ void test() {
+ std::sin(0.); // nvcc - ok
+ std::sin(0); // nvcc - error, because no std::sin(int) override is available.
+ sin(0); // nvcc - same as above.
+
+ sinf(0.); // nvcc - ok
+ std::sinf(0.); // nvcc - no such function
+}
+```
+
+### `<std::complex>`
+
+nvcc does not officially support `std::complex`. It's an error to use
+`std::complex` in `__device__` code, but it often works in `__host__
+__device__` code due to nvcc's interpretation of the "wrong-side rule" (see
+below). However, we have heard from implementers that it's possible to get
+into situations where nvcc will omit a call to an `std::complex` function,
+especially when compiling without optimizations. Using `--expt-relaxed-constexpr`
may help.
-Clang supports ``std::complex`` without these caveats.
+Clang supports `std::complex` without these caveats.
-``<algorithm>``
----------------
+### `<algorithm>`
-In C++14, many useful functions from ``<algorithm>`` (notably, ``std::min`` and
-``std::max``) become constexpr. You can therefore use these in device code,
+In C++14, many useful functions from `<algorithm>` (notably, `std::min` and
+`std::max`) become constexpr. You can therefore use these in device code,
when compiling with clang.
-Detecting clang vs NVCC from code
-=================================
+## Detecting clang vs NVCC from code
Although clang's CUDA implementation is largely compatible with NVCC's, you may
still want to detect when you're compiling CUDA code specifically with clang.
This is tricky, because NVCC may invoke clang as part of its own compilation
-process! For example, NVCC uses the host compiler's preprocessor when
+process! For example, NVCC uses the host compiler's preprocessor when
compiling for device code, and that host compiler may in fact be clang.
When clang is actually compiling CUDA code -- rather than being used as a
-subtool of NVCC's -- it defines the ``__CUDA__`` macro. ``__CUDA_ARCH__`` is
+subtool of NVCC's -- it defines the `__CUDA__` macro. `__CUDA_ARCH__` is
defined only in device mode (but will be defined if NVCC is using clang as a
-preprocessor). So you can use the following incantations to detect clang CUDA
+preprocessor). So you can use the following incantations to detect clang CUDA
compilation, in host and device modes:
-.. code-block:: c++
-
- #if defined(__clang__) && defined(__CUDA__) && !defined(__CUDA_ARCH__)
- // clang compiling CUDA code, host mode.
- #endif
+```c++
+#if defined(__clang__) && defined(__CUDA__) && !defined(__CUDA_ARCH__)
+// clang compiling CUDA code, host mode.
+#endif
- #if defined(__clang__) && defined(__CUDA__) && defined(__CUDA_ARCH__)
- // clang compiling CUDA code, device mode.
- #endif
+#if defined(__clang__) && defined(__CUDA__) && defined(__CUDA_ARCH__)
+// clang compiling CUDA code, device mode.
+#endif
+```
-Both clang and nvcc define ``__CUDACC__`` during CUDA compilation. You can
-detect NVCC specifically by looking for ``__NVCC__``.
+Both clang and nvcc define `__CUDACC__` during CUDA compilation. You can
+detect NVCC specifically by looking for `__NVCC__`.
-Dialect Differences Between clang and nvcc
-==========================================
+## Dialect Differences Between clang and nvcc
There is no formal CUDA spec, and clang and nvcc speak slightly
diff erent
-dialects of the language. Below, we describe some of the
diff erences.
+dialects of the language. Below, we describe some of the
diff erences.
This section is painful; hopefully you can skip this section and live your life
blissfully unaware.
-Compilation Models
-------------------
+### Compilation Models
Most of the
diff erences between clang and nvcc stem from the
diff erent
-compilation models used by clang and nvcc. nvcc uses *split compilation*,
+compilation models used by clang and nvcc. nvcc uses *split compilation*,
which works roughly as follows:
- * Run a preprocessor over the input ``.cu`` file to split it into two source
- files: ``H``, containing source code for the host, and ``D``, containing
- source code for the device.
-
- * For each GPU architecture ``arch`` that we're compiling for, do:
+- Run a preprocessor over the input `.cu` file to split it into two source
+ files: `H`, containing source code for the host, and `D`, containing
+ source code for the device.
- * Compile ``D`` using nvcc proper. The result of this is a ``ptx`` file for
- ``P_arch``.
+- For each GPU architecture `arch` that we're compiling for, do:
- * Optionally, invoke ``ptxas``, the PTX assembler, to generate a file,
- ``S_arch``, containing GPU machine code (SASS) for ``arch``.
+ - Compile `D` using nvcc proper. The result of this is a `ptx` file for
+ `P_arch`.
+ - Optionally, invoke `ptxas`, the PTX assembler, to generate a file,
+ `S_arch`, containing GPU machine code (SASS) for `arch`.
- * Invoke ``fatbin`` to combine all ``P_arch`` and ``S_arch`` files into a
- single "fat binary" file, ``F``.
+- Invoke `fatbin` to combine all `P_arch` and `S_arch` files into a
+ single "fat binary" file, `F`.
- * Compile ``H`` using an external host compiler (gcc, clang, or whatever you
- like). ``F`` is packaged up into a header file which is force-included into
- ``H``; nvcc generates code that calls into this header to e.g. launch
- kernels.
+- Compile `H` using an external host compiler (gcc, clang, or whatever you
+ like). `F` is packaged up into a header file which is force-included into
+ `H`; nvcc generates code that calls into this header to e.g. launch
+ kernels.
-clang uses *merged parsing*. This is similar to split compilation, except all
+clang uses *merged parsing*. This is similar to split compilation, except all
of the host and device code is present and must be semantically-correct in both
compilation steps.
- * For each GPU architecture ``arch`` that we're compiling for, do:
+- For each GPU architecture `arch` that we're compiling for, do:
- * Compile the input ``.cu`` file for device, using clang. ``__host__`` code
- is parsed and must be semantically correct, even though we're not
- generating code for the host at this time.
+ - Compile the input `.cu` file for device, using clang. `__host__` code
+ is parsed and must be semantically correct, even though we're not
+ generating code for the host at this time.
- The output of this step is a ``ptx`` file ``P_arch``.
+ The output of this step is a `ptx` file `P_arch`.
- * Invoke ``ptxas`` to generate a SASS file, ``S_arch``. Note that, unlike
- nvcc, clang always generates SASS code.
+ - Invoke `ptxas` to generate a SASS file, `S_arch`. Note that, unlike
+ nvcc, clang always generates SASS code.
- * Invoke ``fatbin`` to combine all ``S_arch`` files (and ``P_arch`` files if
- PTX inclusion was requested) into a single fat binary file, ``F``.
+- Invoke `fatbin` to combine all `S_arch` files (and `P_arch` files if
+ PTX inclusion was requested) into a single fat binary file, `F`.
- * Compile ``H`` using clang. ``__device__`` code is parsed and must be
- semantically correct, even though we're not generating code for the device
- at this time.
+- Compile `H` using clang. `__device__` code is parsed and must be
+ semantically correct, even though we're not generating code for the device
+ at this time.
- ``F`` is passed to this compilation, and clang includes it in a special ELF
- section, where it can be found by tools like ``cuobjdump``.
+ `F` is passed to this compilation, and clang includes it in a special ELF
+ section, where it can be found by tools like `cuobjdump`.
(You may ask at this point, why does clang need to parse the input file
-multiple times? Why not parse it just once, and then use the AST to generate
+multiple times? Why not parse it just once, and then use the AST to generate
code for the host and each device architecture?
Unfortunately this can't work because we have to define
diff erent macros during
@@ -294,163 +274,158 @@ host compilation and during device compilation for each GPU architecture.)
clang's approach allows it to be highly robust to C++ edge cases, as it doesn't
need to decide at an early stage which declarations to keep and which to throw
-away. But it has some consequences you should be aware of.
+away. But it has some consequences you should be aware of.
-Overloading Based on ``__host__`` and ``__device__`` Attributes
----------------------------------------------------------------
+### Overloading Based on `__host__` and `__device__` Attributes
-Let "H", "D", and "HD" stand for "``__host__`` functions", "``__device__``
-functions", and "``__host__ __device__`` functions", respectively. Functions
+Let "H", "D", and "HD" stand for "`__host__` functions", "`__device__`
+functions", and "`__host__ __device__` functions", respectively. Functions
with no attributes behave the same as H.
nvcc does not allow you to create H and D functions with the same signature:
-.. code-block:: c++
-
- // nvcc: error - function "foo" has already been defined
- __host__ void foo() {}
- __device__ void foo() {}
+```c++
+// nvcc: error - function "foo" has already been defined
+__host__ void foo() {}
+__device__ void foo() {}
+```
However, nvcc allows you to "overload" H and D functions with
diff erent
signatures:
-.. code-block:: c++
-
- // nvcc: no error
- __host__ void foo(int) {}
- __device__ void foo() {}
+```c++
+// nvcc: no error
+__host__ void foo(int) {}
+__device__ void foo() {}
+```
-In clang, the ``__host__`` and ``__device__`` attributes are part of a
+In clang, the `__host__` and `__device__` attributes are part of a
function's signature, and so it's legal to have H and D functions with
(otherwise) the same signature:
-.. code-block:: c++
-
- // clang: no error
- __host__ void foo() {}
- __device__ void foo() {}
+```c++
+// clang: no error
+__host__ void foo() {}
+__device__ void foo() {}
+```
HD functions cannot be overloaded by H or D functions with the same signature:
-.. code-block:: c++
-
- // nvcc: error - function "foo" has already been defined
- // clang: error - redefinition of 'foo'
- __host__ __device__ void foo() {}
- __device__ void foo() {}
+```c++
+// nvcc: error - function "foo" has already been defined
+// clang: error - redefinition of 'foo'
+__host__ __device__ void foo() {}
+__device__ void foo() {}
- // nvcc: no error
- // clang: no error
- __host__ __device__ void bar(int) {}
- __device__ void bar() {}
+// nvcc: no error
+// clang: no error
+__host__ __device__ void bar(int) {}
+__device__ void bar() {}
+```
When resolving an overloaded function, clang considers the host/device
-attributes of the caller and callee. These are used as a tiebreaker during
-overload resolution. See `IdentifyCUDAPreference
-<https://clang.llvm.org/doxygen/SemaCUDA_8cpp.html>`_ for the full set of rules,
+attributes of the caller and callee. These are used as a tiebreaker during
+overload resolution. See [IdentifyCUDAPreference](https://clang.llvm.org/doxygen/SemaCUDA_8cpp.html) for the full set of rules,
but at a high level they are:
- * D functions prefer to call other Ds. HDs are given lower priority.
+- D functions prefer to call other Ds. HDs are given lower priority.
- * Similarly, H functions prefer to call other Hs, or ``__global__`` functions
- (with equal priority). HDs are given lower priority.
+- Similarly, H functions prefer to call other Hs, or `__global__` functions
+ (with equal priority). HDs are given lower priority.
- * HD functions prefer to call other HDs.
+- HD functions prefer to call other HDs.
- When compiling for device, HDs will call Ds with lower priority than HD, and
- will call Hs with still lower priority. If it's forced to call an H, the
- program is malformed if we emit code for this HD function. We call this the
- "wrong-side rule", see example below.
+ When compiling for device, HDs will call Ds with lower priority than HD, and
+ will call Hs with still lower priority. If it's forced to call an H, the
+ program is malformed if we emit code for this HD function. We call this the
+ "wrong-side rule", see example below.
- The rules are symmetrical when compiling for host.
+ The rules are symmetrical when compiling for host.
Some examples:
-.. code-block:: c++
+```c++
+__host__ void foo();
+__device__ void foo();
- __host__ void foo();
- __device__ void foo();
+__host__ void bar();
+__host__ __device__ void bar();
- __host__ void bar();
- __host__ __device__ void bar();
+__host__ void test_host() {
+ foo(); // calls H overload
+ bar(); // calls H overload
+}
- __host__ void test_host() {
- foo(); // calls H overload
- bar(); // calls H overload
- }
+__device__ void test_device() {
+ foo(); // calls D overload
+ bar(); // calls HD overload
+}
- __device__ void test_device() {
- foo(); // calls D overload
- bar(); // calls HD overload
- }
-
- __host__ __device__ void test_hd() {
- foo(); // calls H overload when compiling for host, otherwise D overload
- bar(); // always calls HD overload
- }
+__host__ __device__ void test_hd() {
+ foo(); // calls H overload when compiling for host, otherwise D overload
+ bar(); // always calls HD overload
+}
+```
Wrong-side rule example:
-.. code-block:: c++
-
- __host__ void host_only();
+```c++
+__host__ void host_only();
- // We don't codegen inline functions unless they're referenced by a
- // non-inline function. inline_hd1() is called only from the host side, so
- // does not generate an error. inline_hd2() is called from the device side,
- // so it generates an error.
- inline __host__ __device__ void inline_hd1() { host_only(); } // no error
- inline __host__ __device__ void inline_hd2() { host_only(); } // error
+// We don't codegen inline functions unless they're referenced by a
+// non-inline function. inline_hd1() is called only from the host side, so
+// does not generate an error. inline_hd2() is called from the device side,
+// so it generates an error.
+inline __host__ __device__ void inline_hd1() { host_only(); } // no error
+inline __host__ __device__ void inline_hd2() { host_only(); } // error
- __host__ void host_fn() { inline_hd1(); }
- __device__ void device_fn() { inline_hd2(); }
+__host__ void host_fn() { inline_hd1(); }
+__device__ void device_fn() { inline_hd2(); }
- // This function is not inline, so it's always codegen'ed on both the host
- // and the device. Therefore, it generates an error.
- __host__ __device__ void not_inline_hd() { host_only(); }
+// This function is not inline, so it's always codegen'ed on both the host
+// and the device. Therefore, it generates an error.
+__host__ __device__ void not_inline_hd() { host_only(); }
+```
For the purposes of the wrong-side rule, templated functions also behave like
-``inline`` functions: They aren't codegen'ed unless they're instantiated
+`inline` functions: They aren't codegen'ed unless they're instantiated
(usually as part of the process of invoking them).
clang's behavior with respect to the wrong-side rule matches nvcc's, except
-nvcc only emits a warning for ``not_inline_hd``; device code is allowed to call
-``not_inline_hd``. In its generated code, nvcc may omit ``not_inline_hd``'s
-call to ``host_only`` entirely, or it may try to generate code for
-``host_only`` on the device. What you get seems to depend on whether or not
-the compiler chooses to inline ``host_only``.
+nvcc only emits a warning for `not_inline_hd`; device code is allowed to call
+`not_inline_hd`. In its generated code, nvcc may omit `not_inline_hd`'s
+call to `host_only` entirely, or it may try to generate code for
+`host_only` on the device. What you get seems to depend on whether or not
+the compiler chooses to inline `host_only`.
Member functions, including constructors, may be overloaded using H and D
-attributes. However, destructors cannot be overloaded.
+attributes. However, destructors cannot be overloaded.
-Clang Warnings for Host and Device Function Declarations
---------------------------------------------------------
+### Clang Warnings for Host and Device Function Declarations
Clang can emit warnings when it detects that host (H) and device (D) functions are declared or defined with the same signature. These warnings are not enabled by default.
To enable these warnings, use the following compiler flag:
-.. code-block:: console
+```console
+-Wnvcc-compat
+```
- -Wnvcc-compat
+### Deferred Diagnostics
-Deferred Diagnostics
---------------------
-
-In CUDA, a ``__host__ __device__`` function can be called from both host and
+In CUDA, a `__host__ __device__` function can be called from both host and
device code. When such a function contains operations not valid on one side
(e.g., calling a host-only function from device code), clang defers the
diagnostics and only emits them if the function is actually reachable from a
caller where the operation cannot be emitted. This avoids false errors in
-``__host__ __device__`` functions that are only used on the other side.
+`__host__ __device__` functions that are only used on the other side.
For a detailed description of deferred diagnostics, HD-promoted functions,
and call chain notes, see the
-`HIP Support <https://clang.llvm.org/docs/HIPSupport.html#deferred-diagnostics>`_
+[HIP Support](https://clang.llvm.org/docs/HIPSupport.html#deferred-diagnostics)
documentation. The same mechanism applies to both CUDA and HIP.
-Using a Different Class on Host/Device
---------------------------------------
+### Using a Different Class on Host/Device
Occasionally you may want to have a class with
diff erent host/device versions.
@@ -461,98 +436,94 @@ However, if you want your class to have
diff erent members on host/device, you
won't be able to provide working H and D overloads in both classes. In this
case, clang is likely to be unhappy with you.
-.. code-block:: c++
-
- #ifdef __CUDA_ARCH__
- struct S {
- __device__ void foo() { /* use device_only */ }
- int device_only;
- };
- #else
- struct S {
- __host__ void foo() { /* use host_only */ }
- double host_only;
- };
-
- __device__ void test() {
- S s;
- // clang generates an error here, because during host compilation, we
- // have ifdef'ed away the __device__ overload of S::foo(). The __device__
- // overload must be present *even during host compilation*.
- S.foo();
- }
- #endif
+```c++
+#ifdef __CUDA_ARCH__
+struct S {
+ __device__ void foo() { /* use device_only */ }
+ int device_only;
+};
+#else
+struct S {
+ __host__ void foo() { /* use host_only */ }
+ double host_only;
+};
+
+__device__ void test() {
+ S s;
+ // clang generates an error here, because during host compilation, we
+ // have ifdef'ed away the __device__ overload of S::foo(). The __device__
+ // overload must be present *even during host compilation*.
+ S.foo();
+}
+#endif
+```
We posit that you don't really want to have classes with
diff erent members on H
-and D. For example, if you were to pass one of these as a parameter to a
+and D. For example, if you were to pass one of these as a parameter to a
kernel, it would have a
diff erent layout on H and D, so would not work
properly.
To make code like this compatible with clang, we recommend you separate it out
-into two classes. If you need to write code that works on both host and
+into two classes. If you need to write code that works on both host and
device, consider writing an overloaded wrapper function that returns
diff erent
types on host and device.
-.. code-block:: c++
+```c++
+struct HostS { ... };
+struct DeviceS { ... };
- struct HostS { ... };
- struct DeviceS { ... };
+__host__ HostS MakeStruct() { return HostS(); }
+__device__ DeviceS MakeStruct() { return DeviceS(); }
- __host__ HostS MakeStruct() { return HostS(); }
- __device__ DeviceS MakeStruct() { return DeviceS(); }
-
- // Now host and device code can call MakeStruct().
+// Now host and device code can call MakeStruct().
+```
Unfortunately, this idiom isn't compatible with nvcc, because it doesn't allow
-you to overload based on the H/D attributes. Here's an idiom that works with
+you to overload based on the H/D attributes. Here's an idiom that works with
both clang and nvcc:
-.. code-block:: c++
-
- struct HostS { ... };
- struct DeviceS { ... };
+```c++
+struct HostS { ... };
+struct DeviceS { ... };
- #ifdef __NVCC__
- #ifndef __CUDA_ARCH__
- __host__ HostS MakeStruct() { return HostS(); }
- #else
- __device__ DeviceS MakeStruct() { return DeviceS(); }
- #endif
- #else
+#ifdef __NVCC__
+ #ifndef __CUDA_ARCH__
__host__ HostS MakeStruct() { return HostS(); }
+ #else
__device__ DeviceS MakeStruct() { return DeviceS(); }
#endif
+#else
+ __host__ HostS MakeStruct() { return HostS(); }
+ __device__ DeviceS MakeStruct() { return DeviceS(); }
+#endif
- // Now host and device code can call MakeStruct().
+// Now host and device code can call MakeStruct().
+```
Hopefully you don't have to do this sort of thing often.
-Optimizations
-=============
+## Optimizations
Modern CPUs and GPUs are architecturally quite
diff erent, so code that's fast
-on a CPU isn't necessarily fast on a GPU. We've made a number of changes to
-LLVM to make it generate good GPU code. Among these changes are:
+on a CPU isn't necessarily fast on a GPU. We've made a number of changes to
+LLVM to make it generate good GPU code. Among these changes are:
-* `Straight-line scalar optimizations <https://docs.google.com/document/d/1momWzKFf4D6h8H3YlfgKQ3qeZy5ayvMRh6yR-Xn2hUE>`_ -- These
+- [Straight-line scalar optimizations](https://docs.google.com/document/d/1momWzKFf4D6h8H3YlfgKQ3qeZy5ayvMRh6yR-Xn2hUE) -- These
reduce redundancy within straight-line code.
-* `Aggressive speculative execution
- <https://llvm.org/docs/doxygen/html/SpeculativeExecution_8cpp_source.html>`_
+- [Aggressive speculative execution](https://llvm.org/docs/doxygen/html/SpeculativeExecution_8cpp_source.html)
-- This is mainly for promoting straight-line scalar optimizations, which are
most effective on code along dominator paths.
-* `Memory space inference
- <https://llvm.org/doxygen/InferAddressSpaces_8cpp_source.html>`_ --
+- [Memory space inference](https://llvm.org/doxygen/InferAddressSpaces_8cpp_source.html) --
In PTX, we can operate on pointers that are in a particular "address space"
(global, shared, constant, or local), or we can operate on pointers in the
- "generic" address space, which can point to anything. Operations in a
+ "generic" address space, which can point to anything. Operations in a
non-generic address space are faster, but pointers in CUDA are not explicitly
annotated with their address space, so it's up to LLVM to infer it where
possible.
-* `Bypassing 64-bit divides
- <https://llvm.org/docs/doxygen/html/BypassSlowDivision_8cpp_source.html>`_ --
+- [Bypassing 64-bit divides](https://llvm.org/docs/doxygen/html/BypassSlowDivision_8cpp_source.html) --
This was an existing optimization that we enabled for the PTX backend.
64-bit integer divides are much slower than 32-bit ones on NVIDIA GPUs.
@@ -560,33 +531,34 @@ LLVM to make it generate good GPU code. Among these changes are:
which fit in 32-bits at runtime. This optimization provides a fast path for
this common case.
-* Aggressive loop unrolling and function inlining -- Loop unrolling and
+- Aggressive loop unrolling and function inlining -- Loop unrolling and
function inlining need to be more aggressive for GPUs than for CPUs because
control flow transfer in GPU is more expensive. More aggressive unrolling and
inlining also promote other optimizations, such as constant propagation and
SROA, which sometimes speed up code by over 10x.
- (Programmers can force unrolling and inline using clang's `loop unrolling pragmas
- <https://clang.llvm.org/docs/AttributeReference.html#pragma-unroll-pragma-nounroll>`_
- and ``__attribute__((always_inline))``.)
+ (Programmers can force unrolling and inline using clang's [loop unrolling pragmas](https://clang.llvm.org/docs/AttributeReference.html#pragma-unroll-pragma-nounroll)
+ and `__attribute__((always_inline))`.)
-Publication
-===========
+## Publication
The team at Google published a paper in CGO 2016 detailing the optimizations
-they'd made to clang/LLVM. Note that "gpucc" is no longer a meaningful name:
+they'd made to clang/LLVM. Note that "gpucc" is no longer a meaningful name:
The relevant tools are now just vanilla clang/LLVM.
-| `gpucc: An Open-Source GPGPU Compiler <http://dl.acm.org/citation.cfm?id=2854041>`_
-| Jingyue Wu, Artem Belevich, Eli Bendersky, Mark Heffernan, Chris Leary, Jacques Pienaar, Bjarke Roune, Rob Springer, Xuetian Weng, Robert Hundt
-| *Proceedings of the 2016 International Symposium on Code Generation and Optimization (CGO 2016)*
-|
-| `Slides from the CGO talk <http://wujingyue.github.io/docs/gpucc-talk.pdf>`_
-|
-| `Tutorial given at CGO <http://wujingyue.github.io/docs/gpucc-tutorial.pdf>`_
+[gpucc: An Open-Source GPGPU Compiler](http://dl.acm.org/citation.cfm?id=2854041)
+
+Jingyue Wu, Artem Belevich, Eli Bendersky, Mark Heffernan, Chris Leary,
+Jacques Pienaar, Bjarke Roune, Rob Springer, Xuetian Weng, Robert Hundt
+
+*Proceedings of the 2016 International Symposium on Code Generation and
+Optimization (CGO 2016)*
+
+[Slides from the CGO talk](http://wujingyue.github.io/docs/gpucc-talk.pdf)
+
+[Tutorial given at CGO](http://wujingyue.github.io/docs/gpucc-tutorial.pdf)
-Obtaining Help
-==============
+## Obtaining Help
-To obtain help on LLVM in general and its CUDA support, see `the LLVM
-community <https://llvm.org/docs/#mailing-lists>`_.
+To obtain help on LLVM in general and its CUDA support, see [the LLVM
+community](https://llvm.org/docs/#mailing-lists).
diff --git a/llvm/docs/DebuggingJITedCode.md b/llvm/docs/DebuggingJITedCode.md
index 5719c71df4ddc..9c00a42d31095 100644
--- a/llvm/docs/DebuggingJITedCode.md
+++ b/llvm/docs/DebuggingJITedCode.md
@@ -1,182 +1,171 @@
-=====================
-Debugging JIT-ed Code
-=====================
+# Debugging JIT-ed Code
-Background
-==========
+## Background
Without special runtime support, debugging dynamically generated code can be
-quite painful. Debuggers generally read debug information from object files on
+quite painful. Debuggers generally read debug information from object files on
disk, but for JITed code there is no such file to look for.
-In order to hand over the necessary debug info, `GDB established an
-interface <https://sourceware.org/gdb/onlinedocs/gdb/JIT-Interface.html>`_
+In order to hand over the necessary debug info, [GDB established an
+interface](https://sourceware.org/gdb/onlinedocs/gdb/JIT-Interface.html)
for registering JITed code with debuggers. LLDB implements it in the
-JITLoaderGDB plugin. On the JIT side, LLVM MCJIT does implement the interface
+JITLoaderGDB plugin. On the JIT side, LLVM MCJIT does implement the interface
for ELF object files.
At a high level, whenever MCJIT generates new machine code, it does so in an
in-memory object file that contains the debug information in DWARF format.
MCJIT then adds this in-memory object file to a global list of dynamically
generated object files and calls a special function
-``__jit_debug_register_code`` that the debugger knows about. When the debugger
+`__jit_debug_register_code` that the debugger knows about. When the debugger
attaches to a process, it puts a breakpoint in this function and associates a
-special handler with it. Once MCJIT calls the registration function, the
+special handler with it. Once MCJIT calls the registration function, the
debugger catches the breakpoint signal, loads the new object file from the
-inferior's memory and resumes execution. This way it can obtain debug
+inferior's memory and resumes execution. This way it can obtain debug
information for pure in-memory object files.
-
-GDB Version
-===========
+## GDB Version
In order to debug code JIT-ed by LLVM, you need GDB 7.0 or newer, which is
-available on most modern distributions of Linux. The version of GDB that
+available on most modern distributions of Linux. The version of GDB that
Apple ships with Xcode has been frozen at 6.3 for a while.
-
-LLDB Version
-============
+## LLDB Version
Due to a regression in release 6.0, LLDB didn't support JITed code debugging for
-a while. The bug was fixed in mainline recently, so that debugging JITed ELF
+a while. The bug was fixed in mainline recently, so that debugging JITed ELF
objects should be possible again from the upcoming release 12.0 on. On macOS the
-feature must be enabled explicitly using the ``plugin.jit-loader.gdb.enable``
+feature must be enabled explicitly using the `plugin.jit-loader.gdb.enable`
setting.
-
-Debugging MCJIT-ed code
-=======================
+## Debugging MCJIT-ed code
The emerging MCJIT component of LLVM allows full debugging of JIT-ed code with
-GDB. This is due to MCJIT's ability to use the MC emitter to provide full
+GDB. This is due to MCJIT's ability to use the MC emitter to provide full
DWARF debugging information to GDB.
-Note that lli has to be passed the ``--jit-kind=mcjit`` flag to JIT the code
+Note that lli has to be passed the `--jit-kind=mcjit` flag to JIT the code
with MCJIT instead of the newer ORC JIT.
-Example
--------
+### Example
Consider the following C code (with line numbers added to make the example
easier to follow):
-..
- FIXME:
- Sphinx has the ability to automatically number these lines by adding
- :linenos: on the line immediately following the `.. code-block:: c`, but
- it looks like garbage; the line numbers don't even line up with the
- lines. Is this a Sphinx bug, or is it a CSS problem?
-
-.. code-block:: c
-
- 1 int compute_factorial(int n)
- 2 {
- 3 if (n <= 1)
- 4 return 1;
- 5
- 6 int f = n;
- 7 while (--n > 1)
- 8 f *= n;
- 9 return f;
- 10 }
- 11
- 12
- 13 int main(int argc, char** argv)
- 14 {
- 15 if (argc < 2)
- 16 return -1;
- 17 char firstletter = argv[1][0];
- 18 int result = compute_factorial(firstletter - '0');
- 19
- 20 // Returned result is clipped at 255...
- 21 return result;
- 22 }
+% FIXME:
+% Sphinx has the ability to automatically number these lines by adding
+% :linenos: on the line immediately following the `.. code-block:: c`, but
+% it looks like garbage; the line numbers don't even line up with the
+% lines. Is this a Sphinx bug, or is it a CSS problem?
+
+```c
+1 int compute_factorial(int n)
+2 {
+3 if (n <= 1)
+4 return 1;
+5
+6 int f = n;
+7 while (--n > 1)
+8 f *= n;
+9 return f;
+10 }
+11
+12
+13 int main(int argc, char** argv)
+14 {
+15 if (argc < 2)
+16 return -1;
+17 char firstletter = argv[1][0];
+18 int result = compute_factorial(firstletter - '0');
+19
+20 // Returned result is clipped at 255...
+21 return result;
+22 }
+```
Here is a sample command line session that shows how to build and run this
-code via ``lli`` inside LLDB:
-
-.. code-block:: bash
-
- > export BINPATH=/workspaces/llvm-project/build/bin
- > $BINPATH/clang -g -S -emit-llvm --target=x86_64-unknown-unknown-elf showdebug.c
- > lldb $BINPATH/lli
- (lldb) target create "/workspaces/llvm-project/build/bin/lli"
- Current executable set to '/workspaces/llvm-project/build/bin/lli' (x86_64).
- (lldb) settings set plugin.jit-loader.gdb.enable on
- (lldb) b compute_factorial
- Breakpoint 1: no locations (pending).
- WARNING: Unable to resolve breakpoint to any actual locations.
- (lldb) run --jit-kind=mcjit showdebug.ll 5
- 1 location added to breakpoint 1
- Process 21340 stopped
- * thread #1, name = 'lli', stop reason = breakpoint 1.1
- frame #0: 0x00007ffff7fd0007 JIT(0x45c2cb0)`compute_factorial(n=5) at showdebug.c:3:11
- 1 int compute_factorial(int n)
- 2 {
- -> 3 if (n <= 1)
- 4 return 1;
- 5 int f = n;
- 6 while (--n > 1)
- 7 f *= n;
- (lldb) p n
- (int) $0 = 5
- (lldb) b showdebug.c:9
- Breakpoint 2: where = JIT(0x45c2cb0)`compute_factorial + 60 at showdebug.c:9:1, address = 0x00007ffff7fd003c
- (lldb) c
- Process 21340 resuming
- Process 21340 stopped
- * thread #1, name = 'lli', stop reason = breakpoint 2.1
- frame #0: 0x00007ffff7fd003c JIT(0x45c2cb0)`compute_factorial(n=1) at showdebug.c:9:1
- 6 while (--n > 1)
- 7 f *= n;
- 8 return f;
- -> 9 }
- 10
- 11 int main(int argc, char** argv)
- 12 {
- (lldb) p f
- (int) $1 = 120
- (lldb) bt
- * thread #1, name = 'lli', stop reason = breakpoint 2.1
- * frame #0: 0x00007ffff7fd003c JIT(0x45c2cb0)`compute_factorial(n=1) at showdebug.c:9:1
- frame #1: 0x00007ffff7fd0095 JIT(0x45c2cb0)`main(argc=2, argv=0x00000000046122f0) at showdebug.c:16:18
- frame #2: 0x0000000002a8306e lli`llvm::MCJIT::runFunction(this=0x000000000458ed10, F=0x0000000004589ff8, ArgValues=ArrayRef<llvm::GenericValue> @ 0x00007fffffffc798) at MCJIT.cpp:554:31
- frame #3: 0x00000000029bdb45 lli`llvm::ExecutionEngine::runFunctionAsMain(this=0x000000000458ed10, Fn=0x0000000004589ff8, argv=size=0, envp=0x00007fffffffe140) at ExecutionEngine.cpp:467:10
- frame #4: 0x0000000001f2fc2f lli`main(argc=4, argv=0x00007fffffffe118, envp=0x00007fffffffe140) at lli.cpp:643:18
- frame #5: 0x00007ffff788c09b libc.so.6`__libc_start_main(main=(lli`main at lli.cpp:387), argc=4, argv=0x00007fffffffe118, init=<unavailable>, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007fffffffe108) at libc-start.c:308:16
- frame #6: 0x0000000001f2dc7a lli`_start + 42
- (lldb) finish
- Process 21340 stopped
- * thread #1, name = 'lli', stop reason = step out
- Return value: (int) $2 = 120
-
- frame #0: 0x00007ffff7fd0095 JIT(0x45c2cb0)`main(argc=2, argv=0x00000000046122f0) at showdebug.c:16:9
- 13 if (argc < 2)
- 14 return -1;
- 15 char firstletter = argv[1][0];
- -> 16 int result = compute_factorial(firstletter - '0');
- 17
- 18 // Returned result is clipped at 255...
- 19 return result;
- (lldb) p result
- (int) $3 = 73670648
- (lldb) n
- Process 21340 stopped
- * thread #1, name = 'lli', stop reason = step over
- frame #0: 0x00007ffff7fd0098 JIT(0x45c2cb0)`main(argc=2, argv=0x00000000046122f0) at showdebug.c:19:12
- 16 int result = compute_factorial(firstletter - '0');
- 17
- 18 // Returned result is clipped at 255...
- -> 19 return result;
- 20 }
- (lldb) p result
- (int) $4 = 120
- (lldb) expr result=42
- (int) $5 = 42
- (lldb) p result
- (int) $6 = 42
- (lldb) c
- Process 21340 resuming
- Process 21340 exited with status = 42 (0x0000002a)
- (lldb) exit
+code via `lli` inside LLDB:
+
+```bash
+> export BINPATH=/workspaces/llvm-project/build/bin
+> $BINPATH/clang -g -S -emit-llvm --target=x86_64-unknown-unknown-elf showdebug.c
+> lldb $BINPATH/lli
+(lldb) target create "/workspaces/llvm-project/build/bin/lli"
+Current executable set to '/workspaces/llvm-project/build/bin/lli' (x86_64).
+(lldb) settings set plugin.jit-loader.gdb.enable on
+(lldb) b compute_factorial
+Breakpoint 1: no locations (pending).
+WARNING: Unable to resolve breakpoint to any actual locations.
+(lldb) run --jit-kind=mcjit showdebug.ll 5
+1 location added to breakpoint 1
+Process 21340 stopped
+* thread #1, name = 'lli', stop reason = breakpoint 1.1
+ frame #0: 0x00007ffff7fd0007 JIT(0x45c2cb0)`compute_factorial(n=5) at showdebug.c:3:11
+ 1 int compute_factorial(int n)
+ 2 {
+-> 3 if (n <= 1)
+ 4 return 1;
+ 5 int f = n;
+ 6 while (--n > 1)
+ 7 f *= n;
+(lldb) p n
+(int) $0 = 5
+(lldb) b showdebug.c:9
+Breakpoint 2: where = JIT(0x45c2cb0)`compute_factorial + 60 at showdebug.c:9:1, address = 0x00007ffff7fd003c
+(lldb) c
+Process 21340 resuming
+Process 21340 stopped
+* thread #1, name = 'lli', stop reason = breakpoint 2.1
+ frame #0: 0x00007ffff7fd003c JIT(0x45c2cb0)`compute_factorial(n=1) at showdebug.c:9:1
+ 6 while (--n > 1)
+ 7 f *= n;
+ 8 return f;
+-> 9 }
+ 10
+ 11 int main(int argc, char** argv)
+ 12 {
+(lldb) p f
+(int) $1 = 120
+(lldb) bt
+* thread #1, name = 'lli', stop reason = breakpoint 2.1
+* frame #0: 0x00007ffff7fd003c JIT(0x45c2cb0)`compute_factorial(n=1) at showdebug.c:9:1
+ frame #1: 0x00007ffff7fd0095 JIT(0x45c2cb0)`main(argc=2, argv=0x00000000046122f0) at showdebug.c:16:18
+ frame #2: 0x0000000002a8306e lli`llvm::MCJIT::runFunction(this=0x000000000458ed10, F=0x0000000004589ff8, ArgValues=ArrayRef<llvm::GenericValue> @ 0x00007fffffffc798) at MCJIT.cpp:554:31
+ frame #3: 0x00000000029bdb45 lli`llvm::ExecutionEngine::runFunctionAsMain(this=0x000000000458ed10, Fn=0x0000000004589ff8, argv=size=0, envp=0x00007fffffffe140) at ExecutionEngine.cpp:467:10
+ frame #4: 0x0000000001f2fc2f lli`main(argc=4, argv=0x00007fffffffe118, envp=0x00007fffffffe140) at lli.cpp:643:18
+ frame #5: 0x00007ffff788c09b libc.so.6`__libc_start_main(main=(lli`main at lli.cpp:387), argc=4, argv=0x00007fffffffe118, init=<unavailable>, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007fffffffe108) at libc-start.c:308:16
+ frame #6: 0x0000000001f2dc7a lli`_start + 42
+(lldb) finish
+Process 21340 stopped
+* thread #1, name = 'lli', stop reason = step out
+Return value: (int) $2 = 120
+
+ frame #0: 0x00007ffff7fd0095 JIT(0x45c2cb0)`main(argc=2, argv=0x00000000046122f0) at showdebug.c:16:9
+ 13 if (argc < 2)
+ 14 return -1;
+ 15 char firstletter = argv[1][0];
+-> 16 int result = compute_factorial(firstletter - '0');
+ 17
+ 18 // Returned result is clipped at 255...
+ 19 return result;
+(lldb) p result
+(int) $3 = 73670648
+(lldb) n
+Process 21340 stopped
+* thread #1, name = 'lli', stop reason = step over
+ frame #0: 0x00007ffff7fd0098 JIT(0x45c2cb0)`main(argc=2, argv=0x00000000046122f0) at showdebug.c:19:12
+ 16 int result = compute_factorial(firstletter - '0');
+ 17
+ 18 // Returned result is clipped at 255...
+-> 19 return result;
+ 20 }
+(lldb) p result
+(int) $4 = 120
+(lldb) expr result=42
+(int) $5 = 42
+(lldb) p result
+(int) $6 = 42
+(lldb) c
+Process 21340 resuming
+Process 21340 exited with status = 42 (0x0000002a)
+(lldb) exit
+```
diff --git a/llvm/docs/ExtendingLLVM.md b/llvm/docs/ExtendingLLVM.md
index 019fdf5fc3278..2419983f22fed 100644
--- a/llvm/docs/ExtendingLLVM.md
+++ b/llvm/docs/ExtendingLLVM.md
@@ -1,10 +1,6 @@
-============================================================
-Extending LLVM: Adding instructions, intrinsics, types, etc.
-============================================================
-
-Introduction and Warning
-========================
+# Extending LLVM: Adding instructions, intrinsics, types, etc.
+## Introduction and Warning
During the course of using LLVM, you may wish to customize it for your research
project or for experimentation. At this point, you may realize that you need to
@@ -14,14 +10,13 @@ function, or a whole new instruction.
When you come to this realization, stop and think. Do you really need to extend
LLVM? Is it a new fundamental capability that LLVM does not support at its
current incarnation or can it be synthesized from existing LLVM
-elements? If you are not sure, ask on the `LLVM forums
-<https://discourse.llvm.org>`_. The reason is that
+elements? If you are not sure, ask on the [LLVM forums](https://discourse.llvm.org). The reason is that
extending LLVM will get involved as you need to update all the
diff erent passes
-that you intend to use with your extension, and there are ``many`` LLVM analyses
+that you intend to use with your extension, and there are `many` LLVM analyses
and transformations, so it may be quite a bit of work.
-Adding an `intrinsic function`_ is far easier than adding an
-instruction, and is transparent to optimization passes. If your added
+Adding an {ref}`intrinsic function <intrinsic-function>` is far easier than
+adding an instruction, and is transparent to optimization passes. If your added
functionality can be expressed as a function call, an intrinsic function is the
method of choice for LLVM extension.
@@ -30,290 +25,284 @@ Before you invest a significant amount of effort into a non-trivial extension,
existing infrastructure, or if maybe someone else is already working on
it. You will save yourself a lot of time and effort by doing so.
-.. _intrinsic function:
+(intrinsic-function)=
-Adding a new intrinsic function
-===============================
+## Adding a new intrinsic function
Adding a new intrinsic function to LLVM is much easier than adding a new
-instruction. Almost all extensions to LLVM should start as an intrinsic
+instruction. Almost all extensions to LLVM should start as an intrinsic
function and then be turned into an instruction if warranted.
-#. ``llvm/docs/LangRef.html``:
+1. `llvm/docs/LangRef.html`:
- Document the intrinsic. Decide whether it is code generator specific and
- what the restrictions are. Talk to other people about it so that you are
+ Document the intrinsic. Decide whether it is code generator specific and
+ what the restrictions are. Talk to other people about it so that you are
sure it's a good idea.
-#. ``llvm/include/llvm/IR/Intrinsics*.td``:
+2. `llvm/include/llvm/IR/Intrinsics*.td`:
- Add an entry for your intrinsic. Describe its memory access
+ Add an entry for your intrinsic. Describe its memory access
characteristics for optimization (this controls whether it will be
DCE'd, CSE'd, etc). If any arguments need to be immediates, these
must be indicated with the ImmArg property. Note that any intrinsic
- using one of the ``llvm_any*_ty`` types for an argument or return
- type will be deemed by ``tblgen`` as overloaded and the
+ using one of the `llvm_any*_ty` types for an argument or return
+ type will be deemed by `tblgen` as overloaded and the
corresponding suffix will be required on the intrinsic's name.
-#. ``llvm/lib/Analysis/ConstantFolding.cpp``:
+3. `llvm/lib/Analysis/ConstantFolding.cpp`:
If it is possible to constant fold your intrinsic, add support for it in the
- ``canConstantFoldCallTo`` and ``ConstantFoldCall`` functions.
+ `canConstantFoldCallTo` and `ConstantFoldCall` functions.
-#. ``llvm/test/*``:
+4. `llvm/test/*`:
Add test cases for your intrinsic to the test suite
Once the intrinsic has been added to the system, you must add code generator
-support for it. Generally you must do the following steps:
+support for it. Generally you must do the following steps:
Add support to the .td file for the target(s) of your choice in
-``lib/Target/*/*.td``.
+`lib/Target/*/*.td`.
- This is usually a matter of adding a pattern to the .td file that matches the
- intrinsic, though it may obviously require adding the instructions you want to
- generate as well. There are lots of examples in the PowerPC and X86 backends
- to follow.
+This is usually a matter of adding a pattern to the .td file that matches the
+intrinsic, though it may obviously require adding the instructions you want to
+generate as well. There are lots of examples in the PowerPC and X86 backends
+to follow.
-Adding a new SelectionDAG node
-==============================
+## Adding a new SelectionDAG node
As with intrinsics, adding a new SelectionDAG node to LLVM is much easier than
-adding a new instruction. New nodes are often added to help represent
-instructions common to many targets. These nodes often map to an LLVM
-instruction (add, sub) or intrinsic (byteswap, population count). In other
+adding a new instruction. New nodes are often added to help represent
+instructions common to many targets. These nodes often map to an LLVM
+instruction (add, sub) or intrinsic (byteswap, population count). In other
cases, new nodes have been added to allow many targets to perform a common task
(converting between floating point and integer representation) or capture more
complicated behavior in a single node (rotate).
-#. ``include/llvm/CodeGen/ISDOpcodes.h``:
-
- Add an enum value for the new SelectionDAG node.
+1. `include/llvm/CodeGen/ISDOpcodes.h`:
-#. ``lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp``:
+ Add an enum value for the new SelectionDAG node.
- Add code to print the node to ``getOperationName``. If your new node can be
- evaluated at compile time when given constant arguments (such as an add of a
- constant with another constant), find the ``getNode`` method that takes the
- appropriate number of arguments, and add a case for your node to the switch
- statement that performs constant folding for nodes that take the same number
- of arguments as your new node.
+2. `lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp`:
-#. ``lib/CodeGen/SelectionDAG/LegalizeDAG.cpp``:
+ Add code to print the node to `getOperationName`. If your new node can be
+ evaluated at compile time when given constant arguments (such as an add of a
+ constant with another constant), find the `getNode` method that takes the
+ appropriate number of arguments, and add a case for your node to the switch
+ statement that performs constant folding for nodes that take the same number
+ of arguments as your new node.
- Add code to `legalize, promote, and expand
- <CodeGenerator.html#selectiondag_legalize>`_ the node as necessary. At a
- minimum, you will need to add a case statement for your node in
- ``LegalizeOp`` which calls LegalizeOp on the node's operands, and returns a
- new node if any of the operands changed as a result of being legalized. It
- is likely that not all targets supported by the SelectionDAG framework will
- natively support the new node. In this case, you must also add code in your
- node's case statement in ``LegalizeOp`` to Expand your node into simpler,
- legal operations. The case for ``ISD::UREM`` for expanding a remainder into
- a divide, multiply, and a subtract is a good example.
+3. `lib/CodeGen/SelectionDAG/LegalizeDAG.cpp`:
-#. ``lib/CodeGen/SelectionDAG/LegalizeDAG.cpp``:
+ Add code to [legalize, promote, and
+ expand](project:CodeGenerator.md#selectiondag-legalize-phase) the node as
+ necessary. At a minimum, you will need to add a case statement for your node
+ in `LegalizeOp` which calls LegalizeOp on the node's operands, and returns a
+ new node if any of the operands changed as a result of being legalized. It
+ is likely that not all targets supported by the SelectionDAG framework will
+ natively support the new node. In this case, you must also add code in your
+ node's case statement in `LegalizeOp` to Expand your node into simpler,
+ legal operations. The case for `ISD::UREM` for expanding a remainder into
+ a divide, multiply, and a subtract is a good example.
- If targets may support the new node being added only at certain sizes, you
- will also need to add code to your node's case statement in ``LegalizeOp``
- to Promote your node's operands to a larger size, and perform the correct
- operation. You will also need to add code to ``PromoteOp`` to do this as
- well. For a good example, see ``ISD::BSWAP``, which promotes its operand to
- a wider size, performs the byteswap, and then shifts the correct bytes right
- to emulate the narrower byteswap in the wider type.
+4. `lib/CodeGen/SelectionDAG/LegalizeDAG.cpp`:
-#. ``lib/CodeGen/SelectionDAG/LegalizeDAG.cpp``:
+ If targets may support the new node being added only at certain sizes, you
+ will also need to add code to your node's case statement in `LegalizeOp`
+ to Promote your node's operands to a larger size, and perform the correct
+ operation. You will also need to add code to `PromoteOp` to do this as
+ well. For a good example, see `ISD::BSWAP`, which promotes its operand to
+ a wider size, performs the byteswap, and then shifts the correct bytes right
+ to emulate the narrower byteswap in the wider type.
- Add a case for your node in ``ExpandOp`` to teach the legalizer how to
- perform the action represented by the new node on a value that has been split
- into high and low halves. This case will be used to support your node with a
- 64-bit operand on a 32-bit target.
+5. `lib/CodeGen/SelectionDAG/LegalizeDAG.cpp`:
-#. ``lib/CodeGen/SelectionDAG/DAGCombiner.cpp``:
+ Add a case for your node in `ExpandOp` to teach the legalizer how to
+ perform the action represented by the new node on a value that has been split
+ into high and low halves. This case will be used to support your node with a
+ 64-bit operand on a 32-bit target.
- If your node can be combined with itself, or other existing nodes in a
- peephole-like fashion, add a visit function for it, and call that function
- from. There are several good examples for simple combines you can do;
- ``visitFABS`` and ``visitSRL`` are good starting places.
+6. `lib/CodeGen/SelectionDAG/DAGCombiner.cpp`:
-#. ``lib/Target/PowerPC/PPCISelLowering.cpp``:
+ If your node can be combined with itself, or other existing nodes in a
+ peephole-like fashion, add a visit function for it, and call that function
+ from. There are several good examples for simple combines you can do;
+ `visitFABS` and `visitSRL` are good starting places.
- Each target has an implementation of the ``TargetLowering`` class, usually in
- its own file (although some targets include it in the same file as the
- DAGToDAGISel). The default behavior for a target is to assume that your new
- node is legal for all types that are legal for that target. If this target
- does not natively support your node, then tell the target to either Promote
- it (if it is supported at a larger type) or Expand it. This will cause the
- code you wrote in ``LegalizeOp`` above to decompose your new node into other
- legal nodes for this target.
+7. `lib/Target/PowerPC/PPCISelLowering.cpp`:
-#. ``include/llvm/Target/TargetSelectionDAG.td``:
+ Each target has an implementation of the `TargetLowering` class, usually in
+ its own file (although some targets include it in the same file as the
+ DAGToDAGISel). The default behavior for a target is to assume that your new
+ node is legal for all types that are legal for that target. If this target
+ does not natively support your node, then tell the target to either Promote
+ it (if it is supported at a larger type) or Expand it. This will cause the
+ code you wrote in `LegalizeOp` above to decompose your new node into other
+ legal nodes for this target.
- Most current targets supported by LLVM generate code using the DAGToDAG
- method, where SelectionDAG nodes are pattern matched to target-specific
- nodes, which represent individual instructions. In order for the targets to
- match an instruction to your new node, you must add a def for that node to
- the list in this file, with the appropriate type constraints. Look at
- ``add``, ``bswap``, and ``fadd`` for examples.
+8. `include/llvm/Target/TargetSelectionDAG.td`:
-#. ``lib/Target/PowerPC/PPCInstrInfo.td``:
+ Most current targets supported by LLVM generate code using the DAGToDAG
+ method, where SelectionDAG nodes are pattern matched to target-specific
+ nodes, which represent individual instructions. In order for the targets to
+ match an instruction to your new node, you must add a def for that node to
+ the list in this file, with the appropriate type constraints. Look at
+ `add`, `bswap`, and `fadd` for examples.
- Each target has a tablegen file that describes the target's instruction set.
- For targets that use the DAGToDAG instruction selection framework, add a
- pattern for your new node that uses one or more target nodes. Documentation
- for this is a bit sparse right now, but there are several decent examples.
- See the patterns for ``rotl`` in ``PPCInstrInfo.td``.
+9. `lib/Target/PowerPC/PPCInstrInfo.td`:
-#. TODO: document complex patterns.
+ Each target has a tablegen file that describes the target's instruction set.
+ For targets that use the DAGToDAG instruction selection framework, add a
+ pattern for your new node that uses one or more target nodes. Documentation
+ for this is a bit sparse right now, but there are several decent examples.
+ See the patterns for `rotl` in `PPCInstrInfo.td`.
-#. ``llvm/test/CodeGen/*``:
+10. TODO: document complex patterns.
- Add test cases for your new node to the test suite.
- ``llvm/test/CodeGen/X86/bswap.ll`` is a good example.
+11. `llvm/test/CodeGen/*`:
-Adding a new instruction
-========================
+ Add test cases for your new node to the test suite.
+ `llvm/test/CodeGen/X86/bswap.ll` is a good example.
-.. warning::
+## Adding a new instruction
- Adding instructions changes the bitcode format, and it will take some effort
- to maintain compatibility with the previous version. Only add an instruction
- if it is absolutely necessary.
+:::{warning}
+Adding instructions changes the bitcode format, and it will take some effort
+to maintain compatibility with the previous version. Only add an instruction
+if it is absolutely necessary.
+:::
-#. ``llvm/include/llvm/IR/Instruction.def``:
+1. `llvm/include/llvm/IR/Instruction.def`:
- add a number for your instruction and an enum name
+ add a number for your instruction and an enum name
-#. ``llvm/include/llvm/IR/Instructions.h``:
+2. `llvm/include/llvm/IR/Instructions.h`:
- add a definition for the class that will represent your instruction
+ add a definition for the class that will represent your instruction
-#. ``llvm/include/llvm/IR/InstVisitor.h``:
+3. `llvm/include/llvm/IR/InstVisitor.h`:
- add a prototype for a visitor to your new instruction type
+ add a prototype for a visitor to your new instruction type
-#. ``llvm/lib/AsmParser/LLLexer.cpp``:
+4. `llvm/lib/AsmParser/LLLexer.cpp`:
- add a new token to parse your instruction from an assembly text file
+ add a new token to parse your instruction from an assembly text file
-#. ``llvm/lib/AsmParser/LLParser.cpp``:
+5. `llvm/lib/AsmParser/LLParser.cpp`:
- add the grammar on how your instruction can be read and what it will
- construct as a result
+ add the grammar on how your instruction can be read and what it will
+ construct as a result
-#. ``llvm/lib/Bitcode/Reader/BitcodeReader.cpp``:
+6. `llvm/lib/Bitcode/Reader/BitcodeReader.cpp`:
- add a case for your instruction and how it will be parsed from bitcode
+ add a case for your instruction and how it will be parsed from bitcode
-#. ``llvm/lib/Bitcode/Writer/BitcodeWriter.cpp``:
+7. `llvm/lib/Bitcode/Writer/BitcodeWriter.cpp`:
- add a case for your instruction and how it will be written to bitcode
+ add a case for your instruction and how it will be written to bitcode
-#. ``llvm/lib/IR/Instruction.cpp``:
+8. `llvm/lib/IR/Instruction.cpp`:
- add a case for how your instruction will be printed out to assembly
+ add a case for how your instruction will be printed out to assembly
-#. ``llvm/lib/IR/Instructions.cpp``:
+9. `llvm/lib/IR/Instructions.cpp`:
- implement the class you defined in ``llvm/include/llvm/Instructions.h``
+ implement the class you defined in `llvm/include/llvm/Instructions.h`
-#. Test your instruction
+10. Test your instruction
-#. ``llvm/lib/Target/*``:
+11. `llvm/lib/Target/*`:
- add support for your instruction to code generators, or add a lowering pass.
+ add support for your instruction to code generators, or add a lowering pass.
-#. ``llvm/test/*``:
+12. `llvm/test/*`:
- add your test cases to the test suite.
+ add your test cases to the test suite.
Also, you need to implement (or modify) any analyses or passes that you want to
understand this new instruction.
-Adding a new type
-=================
-
-.. warning::
+## Adding a new type
- Adding new types changes the bitcode format, and will break compatibility with
- existing LLVM installations. Only add new types if it is absolutely
- necessary.
+:::{warning}
+Adding new types changes the bitcode format, and will break compatibility with
+existing LLVM installations. Only add new types if it is absolutely
+necessary.
+:::
-Adding a fundamental type
--------------------------
+### Adding a fundamental type
-#. ``llvm/include/llvm/IR/Type.h``:
+1. `llvm/include/llvm/IR/Type.h`:
- add enum for the new type; add static ``Type*`` for this type
+ add enum for the new type; add static `Type*` for this type
-#. ``llvm/lib/IR/Type.cpp`` and ``llvm/lib/CodeGen/ValueTypes.cpp``:
+2. `llvm/lib/IR/Type.cpp` and `llvm/lib/CodeGen/ValueTypes.cpp`:
- add mapping from ``TypeID`` => ``Type*``; initialize the static ``Type*``
+ add mapping from `TypeID` => `Type*`; initialize the static `Type*`
-#. ``llvm/include/llvm-c/Core.h`` and ``llvm/lib/IR/Core.cpp``:
+3. `llvm/include/llvm-c/Core.h` and `llvm/lib/IR/Core.cpp`:
- add enum ``LLVMTypeKind`` and modify
- ``LLVMTypeKind LLVMGetTypeKind(LLVMTypeRef Ty)`` for the new type
+ add enum `LLVMTypeKind` and modify
+ `LLVMTypeKind LLVMGetTypeKind(LLVMTypeRef Ty)` for the new type
-#. ``llvm/lib/AsmParser/LLLexer.cpp``:
+4. `llvm/lib/AsmParser/LLLexer.cpp`:
add ability to parse in the type from text assembly
-#. ``llvm/lib/AsmParser/LLParser.cpp``:
+5. `llvm/lib/AsmParser/LLParser.cpp`:
add a token for that type
-#. ``llvm/lib/Bitcode/Writer/BitcodeWriter.cpp``:
+6. `llvm/lib/Bitcode/Writer/BitcodeWriter.cpp`:
- modify ``void ModuleBitcodeWriter::writeTypeTable()`` to serialize your type
+ modify `void ModuleBitcodeWriter::writeTypeTable()` to serialize your type
-#. ``llvm/lib/Bitcode/Reader/BitcodeReader.cpp``:
+7. `llvm/lib/Bitcode/Reader/BitcodeReader.cpp`:
- modify ``Error BitcodeReader::parseTypeTableBody()`` to read your data type
+ modify `Error BitcodeReader::parseTypeTableBody()` to read your data type
-#. ``include/llvm/Bitcode/LLVMBitCodes.h``:
+8. `include/llvm/Bitcode/LLVMBitCodes.h`:
- add enum ``TypeCodes`` for the new type
+ add enum `TypeCodes` for the new type
-Adding a derived type
----------------------
+### Adding a derived type
-#. ``llvm/include/llvm/IR/Type.h``:
+1. `llvm/include/llvm/IR/Type.h`:
add enum for the new type; add a forward declaration of the type also
-#. ``llvm/include/llvm/IR/DerivedTypes.h``:
+2. `llvm/include/llvm/IR/DerivedTypes.h`:
add a new class to represent the new class in the hierarchy; add forward
declaration to the TypeMap value type
-#. ``llvm/lib/IR/Type.cpp`` and ``llvm/lib/CodeGen/ValueTypes.cpp``:
+3. `llvm/lib/IR/Type.cpp` and `llvm/lib/CodeGen/ValueTypes.cpp`:
- add support for derived type, notably ``enum TypeID`` and ``is``, ``get`` methods.
+ add support for derived type, notably `enum TypeID` and `is`, `get` methods.
-#. ``llvm/include/llvm-c/Core.h`` and ``llvm/lib/IR/Core.cpp``:
+4. `llvm/include/llvm-c/Core.h` and `llvm/lib/IR/Core.cpp`:
- add enum ``LLVMTypeKind`` and modify
- ``LLVMTypeKind LLVMGetTypeKind(LLVMTypeRef Ty)`` for the new type
+ add enum `LLVMTypeKind` and modify
+ `LLVMTypeKind LLVMGetTypeKind(LLVMTypeRef Ty)` for the new type
-#. ``llvm/lib/AsmParser/LLLexer.cpp``:
+5. `llvm/lib/AsmParser/LLLexer.cpp`:
- modify ``lltok::Kind LLLexer::LexIdentifier()`` to add ability to
+ modify `lltok::Kind LLLexer::LexIdentifier()` to add ability to
parse in the type from text assembly
-#. ``llvm/lib/Bitcode/Writer/BitcodeWriter.cpp``:
+6. `llvm/lib/Bitcode/Writer/BitcodeWriter.cpp`:
- modify ``void ModuleBitcodeWriter::writeTypeTable()`` to serialize your type
+ modify `void ModuleBitcodeWriter::writeTypeTable()` to serialize your type
-#. ``llvm/lib/Bitcode/Reader/BitcodeReader.cpp``:
+7. `llvm/lib/Bitcode/Reader/BitcodeReader.cpp`:
- modify ``Error BitcodeReader::parseTypeTableBody()`` to read your data type
+ modify `Error BitcodeReader::parseTypeTableBody()` to read your data type
-#. ``include/llvm/Bitcode/LLVMBitCodes.h``:
+8. `include/llvm/Bitcode/LLVMBitCodes.h`:
- add enum ``TypeCodes`` for the new type
+ add enum `TypeCodes` for the new type
-#. ``llvm/lib/IR/AsmWriter.cpp``:
+9. `llvm/lib/IR/AsmWriter.cpp`:
- modify ``void TypePrinting::print(Type *Ty, raw_ostream &OS)``
+ modify `void TypePrinting::print(Type *Ty, raw_ostream &OS)`
to output the new derived type
diff --git a/llvm/docs/HowToBuildWindowsItaniumPrograms.md b/llvm/docs/HowToBuildWindowsItaniumPrograms.md
index a18df992adf10..52bdc77d32ccb 100644
--- a/llvm/docs/HowToBuildWindowsItaniumPrograms.md
+++ b/llvm/docs/HowToBuildWindowsItaniumPrograms.md
@@ -1,9 +1,6 @@
-==========================================
-How to build Windows Itanium applications.
-==========================================
+# How to build Windows Itanium applications.
-Introduction
-============
+## Introduction
This document contains information describing how to create a Windows Itanium toolchain.
@@ -13,26 +10,24 @@ headers or additional runtime machinery (such as is used by mingw).
Windows Itanium Stack:
-* Uses the Itanium C++ ABI.
-* libc++.
-* libc++-abi.
-* libunwind.
-* The MS VS CRT.
-* Is compatible with MS Windows SDK include headers.
-* COFF/PE file format.
-* LLD
+- Uses the Itanium C++ ABI.
+- libc++.
+- libc++-abi.
+- libunwind.
+- The MS VS CRT.
+- Is compatible with MS Windows SDK include headers.
+- COFF/PE file format.
+- LLD
Note: compiler-rt is not used. This functionality is supplied by the MS VCRT.
-Prerequisites
-=============
+## Prerequisites
-* The MS SDK is installed as part of MS Visual Studio.
-* Clang with support for the windows-itanium triple.
-* COFF LLD with support for the -autoimport switch.
+- The MS SDK is installed as part of MS Visual Studio.
+- Clang with support for the windows-itanium triple.
+- COFF LLD with support for the -autoimport switch.
-Known issues:
-=============
+## Known issues:
SJLJ exceptions, "-fsjlj-exceptions", are the only currently supported model.
@@ -53,124 +48,118 @@ store a runtime address from another DLL into this pointer (although runtime
addresses are patched into the IAT). Therefore, the compiler must emit some code,
that runs after IAT patching but before anything that might use the vtable pointers,
and sets the vtable pointer to the address from the IAT. For the special case of
-the references to vtables for __cxxabiv1::__class_type_info from typeinto objects
+the references to vtables for `__cxxabiv1::__class_type_info` from typeinto objects
there is no declaration available to the compiler so this can't be done. To allow
programs to link we currently rely on the -auto-import switch in LLD to auto-import
-references to __cxxabiv1::__class_type_info pointers (see: https://reviews.llvm.org/D43184
+references to `__cxxabiv1::__class_type_info` pointers (see: <https://reviews.llvm.org/D43184>
for a related discussion). This allows for linking; but, code that actually uses
such fields will not work as they these will not be fixed up at runtime. See
-_pei386_runtime_relocator which handles the runtime component of the autoimporting
-scheme used for mingw and comments in https://reviews.llvm.org/D43184 and
-https://reviews.llvm.org/D89518 for more.
+`_pei386_runtime_relocator` which handles the runtime component of the autoimporting
+scheme used for mingw and comments in <https://reviews.llvm.org/D43184> and
+<https://reviews.llvm.org/D89518> for more.
-Assembling a Toolchain:
-=======================
+## Assembling a Toolchain:
The procedure is:
-# Build an LLVM toolchain with support for Windows Itanium.
-# Use the toolchain from step 1. to build libc++, libc++abi, and libunwind.
+1. Build an LLVM toolchain with support for Windows Itanium.
+2. Use the toolchain from step 1. to build libc++, libc++abi, and libunwind.
It is also possible to cross-compile from Linux.
-To build the libraries in step 2, refer to the `libc++ documentation <https://libcxx.llvm.org/VendorDocumentation.html#the-default-build>`_.
+To build the libraries in step 2, refer to the [libc++ documentation](https://libcxx.llvm.org/VendorDocumentation.html#the-default-build).
The next section discusses the salient options and modifications required for building and installing the
libraries. This assumes that we are building libunwind and libc++ as DLLs and statically linking libc++abi
into libc++. Other build configurations are possible, but they are not discussed here.
-Common CMake configuration options:
------------------------------------
+### Common CMake configuration options:
-* ``-D_LIBCPP_ABI_FORCE_ITANIUM'``
+- `-D_LIBCPP_ABI_FORCE_ITANIUM'`
Tell the libc++ headers that the Itanium C++ ABI is being used.
-* ``-DCMAKE_C_FLAGS="-lmsvcrt -llegacy_stdio_definitions -D_NO_CRT_STDIO_INLINE"``
+- `-DCMAKE_C_FLAGS="-lmsvcrt -llegacy_stdio_definitions -D_NO_CRT_STDIO_INLINE"`
Supply CRT definitions including stdio definitions that have been removed from the MS VS CRT.
We don't want the stdio functions declared inline as they will cause multiple definition
errors when the same symbols are pulled in from legacy_stdio_definitions.ib.
-* ``-DCMAKE_INSTALL_PREFIX=<install path>``
+- `-DCMAKE_INSTALL_PREFIX=<install path>`
Where to install the library and headers.
-Building libunwind:
--------------------
+### Building libunwind:
-* ``-DLIBUNWIND_ENABLE_SHARED=ON``
-* ``-DLIBUNWIND_ENABLE_STATIC=OFF``
+- `-DLIBUNWIND_ENABLE_SHARED=ON`
+- `-DLIBUNWIND_ENABLE_STATIC=OFF`
libunwind can be built as a DLL. It is not dependent on other projects.
-* ``-DLIBUNWIND_USE_COMPILER_RT=OFF``
+- `-DLIBUNWIND_USE_COMPILER_RT=OFF`
We use the MS runtime.
The CMake files will need to be edited to prevent them adding GNU specific libraries to the link line.
-Building libc++abi:
--------------------
+### Building libc++abi:
-* ``-DLIBCXXABI_ENABLE_SHARED=OFF``
-* ``-DLIBCXXABI_ENABLE_STATIC=ON``
-* ``-DLIBCXX_ENABLE_SHARED=ON'``
-* ``-DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON``
+- `-DLIBCXXABI_ENABLE_SHARED=OFF`
+- `-DLIBCXXABI_ENABLE_STATIC=ON`
+- `-DLIBCXX_ENABLE_SHARED=ON'`
+- `-DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON`
To break the symbol dependency between libc++abi and libc++ we
build libc++abi as a static library and then statically link it
into the libc++ DLL. This necessitates setting the CMake file
to ensure that the visibility macros (which expand to dllexport/import)
are expanded as they will be needed when creating the final libc++
-DLL later, see: https://reviews.llvm.org/D90021.
+DLL later, see: <https://reviews.llvm.org/D90021>.
-* ``-DLIBCXXABI_LIBCXX_INCLUDES=<path to libcxx>/include``
+- `-DLIBCXXABI_LIBCXX_INCLUDES=<path to libcxx>/include`
Where to find the libc++ headers
-Building libc++:
-----------------
+### Building libc++:
-* ``-DLIBCXX_ENABLE_SHARED=ON``
-* ``-DLIBCXX_ENABLE_STATIC=OFF``
+- `-DLIBCXX_ENABLE_SHARED=ON`
+- `-DLIBCXX_ENABLE_STATIC=OFF`
We build libc++ as a DLL and statically link libc++abi into it.
-* ``-DLIBCXX_INSTALL_HEADERS=ON``
+- `-DLIBCXX_INSTALL_HEADERS=ON`
Install the headers.
-* ``-DLIBCXX_USE_COMPILER_RT=OFF``
+- `-DLIBCXX_USE_COMPILER_RT=OFF`
We use the MS runtime.
-* ``-DLIBCXX_HAS_WIN32_THREAD_API=ON``
+- `-DLIBCXX_HAS_WIN32_THREAD_API=ON`
Windows Itanium does not offer a POSIX-like layer over WIN32.
-* ``-DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON``
-* ``-DLIBCXX_CXX_ABI=libcxxabi``
-* ``-DLIBCXX_CXX_ABI_INCLUDE_PATHS=<libcxxabi src path>/include``
-* ``-DLIBCXX_CXX_ABI_LIBRARY_PATH=<libcxxabi build path>/lib``
+- `-DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON`
+- `-DLIBCXX_CXX_ABI=libcxxabi`
+- `-DLIBCXX_CXX_ABI_INCLUDE_PATHS=<libcxxabi src path>/include`
+- `-DLIBCXX_CXX_ABI_LIBRARY_PATH=<libcxxabi build path>/lib`
Use the static libc++abi library built earlier.
-* ``-DLIBCXX_NO_VCRUNTIME=ON``
+- `-DLIBCXX_NO_VCRUNTIME=ON`
Remove any dependency on the VC runtime - we need libc++abi to supply the C++ runtime.
-* ``-DCMAKE_C_FLAGS=<path to installed unwind.lib>``
+- `-DCMAKE_C_FLAGS=<path to installed unwind.lib>`
As we are statically linking against libcxxabi we need to link
against the unwind import library to resolve unwind references
from the libcxxabi objects.
-* ``-DCMAKE_C_FLAGS+=' -UCLOCK_REALTIME'``
+- `-DCMAKE_C_FLAGS+=' -UCLOCK_REALTIME'`
Prevent the inclusion of sys/time that MS doesn't provide.
-Notes:
-------
+### Notes:
-An example build recipe is available here: https://reviews.llvm.org/D88124
+An example build recipe is available here: <https://reviews.llvm.org/D88124>
diff --git a/llvm/docs/HowToCrossCompileBuiltinsOnArm.md b/llvm/docs/HowToCrossCompileBuiltinsOnArm.md
index 58599404d5cd4..7bcc86ffeb88a 100644
--- a/llvm/docs/HowToCrossCompileBuiltinsOnArm.md
+++ b/llvm/docs/HowToCrossCompileBuiltinsOnArm.md
@@ -1,9 +1,6 @@
-===================================================================
-How to Cross Compile Compiler-rt Builtins For Arm
-===================================================================
+# How to Cross Compile Compiler-rt Builtins For Arm
-Introduction
-============
+## Introduction
This document contains information about building and testing the builtins part
of compiler-rt for an Arm target, from an x86_64 Linux machine.
@@ -16,8 +13,7 @@ The instructions in this document depend on libraries and programs external to
LLVM. There are many ways to install and configure these dependencies, so you
may need to adapt the instructions here to fit your own situation.
-Prerequisites
-=============
+## Prerequisites
In this use case, we will be using cmake on a Debian-based Linux system,
cross-compiling from an x86_64 host to a hard-float Armv7-A target. We will be
@@ -25,238 +21,247 @@ using as many of the LLVM tools as we can, but it is possible to use GNU
equivalents.
You will need:
- * A build of LLVM for the llvm-tools and LLVM CMake files.
- * A clang executable with support for the ``ARM`` target.
- * ``compiler-rt`` sources.
- * The ``qemu-arm`` user mode emulator.
- * An ``arm-linux-gnueabihf`` sysroot.
+- A build of LLVM for the llvm-tools and LLVM CMake files.
+- A clang executable with support for the `ARM` target.
+- `compiler-rt` sources.
+- The `qemu-arm` user mode emulator.
+- An `arm-linux-gnueabihf` sysroot.
-.. note::
- An existing sysroot is required because some of the builtins include C library
- headers and a sysroot is the easiest way to get those.
+:::{note}
+An existing sysroot is required because some of the builtins include C library
+headers and a sysroot is the easiest way to get those.
+:::
-In this example, we will be using ``ninja`` as the build tool.
+In this example, we will be using `ninja` as the build tool.
-See https://compiler-rt.llvm.org/ for information about the dependencies
+See <https://compiler-rt.llvm.org/> for information about the dependencies
on clang and LLVM.
-See https://llvm.org/docs/GettingStarted.html for information about obtaining
+See <https://llvm.org/docs/GettingStarted.html> for information about obtaining
the source for LLVM and compiler-rt.
-``qemu-arm`` should be available as a package for your Linux distribution.
+`qemu-arm` should be available as a package for your Linux distribution.
-The most complicated of the prerequisites to satisfy is the ``arm-linux-gnueabihf``
+The most complicated of the prerequisites to satisfy is the `arm-linux-gnueabihf`
sysroot. In theory, it is possible to use the Linux distributions multiarch
support to fulfill the dependencies for building but unfortunately due to
-``/usr/local/include`` being added some host includes are selected.
+`/usr/local/include` being added some host includes are selected.
-The easiest way to supply a sysroot is to download an ``arm-linux-gnueabihf``
-toolchain from https://developer.arm.com/open-source/gnu-toolchain/gnu-a/downloads.
+The easiest way to supply a sysroot is to download an `arm-linux-gnueabihf`
+toolchain from <https://developer.arm.com/open-source/gnu-toolchain/gnu-a/downloads>.
-Building compiler-rt builtins for Arm
-=====================================
+## Building compiler-rt builtins for Arm
We will be doing a standalone build of compiler-rt. The command is shown below.
-Shell variables are used to simplify some of the options::
-
- LLVM_TOOLCHAIN=<path-to-llvm-install>/
- TARGET_TRIPLE=arm-none-linux-gnueabihf
- GCC_TOOLCHAIN=<path-to-gcc-toolchain>
- SYSROOT=${GCC_TOOLCHAIN}/${TARGET_TRIPLE}/libc
- COMPILE_FLAGS="-march=armv7-a"
-
- cmake ../llvm-project/compiler-rt \
- -G Ninja \
- -DCMAKE_AR=${LLVM_TOOLCHAIN}/bin/llvm-ar \
- -DCMAKE_NM=${LLVM_TOOLCHAIN}/bin/llvm-nm \
- -DCMAKE_RANLIB=${LLVM_TOOLCHAIN}/bin/llvm-ranlib \
- -DLLVM_CMAKE_DIR="${LLVM_TOOLCHAIN}/lib/cmake/llvm" \
- -DCMAKE_SYSROOT="${SYSROOT}" \
- -DCMAKE_ASM_COMPILER_TARGET="${TARGET_TRIPLE}" \
- -DCMAKE_ASM_FLAGS="${COMPILE_FLAGS}" \
- -DCMAKE_C_COMPILER_TARGET="${TARGET_TRIPLE}" \
- -DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \
- -DCMAKE_C_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
- -DCMAKE_C_FLAGS="${COMPILE_FLAGS}" \
- -DCMAKE_CXX_COMPILER_TARGET="${TARGET_TRIPLE}" \
- -DCMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \
- -DCMAKE_CXX_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
- -DCMAKE_CXX_FLAGS="${COMPILE_FLAGS}" \
- -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
- -DCOMPILER_RT_BUILD_BUILTINS=ON \
- -DCOMPILER_RT_BUILD_LIBFUZZER=OFF \
- -DCOMPILER_RT_BUILD_MEMPROF=OFF \
- -DCOMPILER_RT_BUILD_PROFILE=OFF \
- -DCOMPILER_RT_BUILD_CTX_PROFILE=OFF \
- -DCOMPILER_RT_BUILD_SANITIZERS=OFF \
- -DCOMPILER_RT_BUILD_XRAY=OFF \
- -DCOMPILER_RT_BUILD_ORC=OFF \
- -DCOMPILER_RT_BUILD_CRT=OFF \
- -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \
- -DCOMPILER_RT_EMULATOR="qemu-arm -L ${SYSROOT}" \
- -DCOMPILER_RT_INCLUDE_TESTS=ON \
- -DCOMPILER_RT_TEST_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
- -DCOMPILER_RT_TEST_COMPILER_CFLAGS="--target=${TARGET_TRIPLE} ${COMPILE_FLAGS} --gcc-toolchain=${GCC_TOOLCHAIN} --sysroot=${SYSROOT} -fuse-ld=lld"
-
-.. note::
- The command above also enables tests. Enabling tests is not required, more details
- in the testing section.
-
-``CMAKE_<LANGUAGE>_<OPTION>`` options are set so that the correct ``--target``,
-``--sysroot``, ``--gcc-toolchain`` and ``-march`` options will be given to the
+Shell variables are used to simplify some of the options:
+
+```
+LLVM_TOOLCHAIN=<path-to-llvm-install>/
+TARGET_TRIPLE=arm-none-linux-gnueabihf
+GCC_TOOLCHAIN=<path-to-gcc-toolchain>
+SYSROOT=${GCC_TOOLCHAIN}/${TARGET_TRIPLE}/libc
+COMPILE_FLAGS="-march=armv7-a"
+
+cmake ../llvm-project/compiler-rt \
+ -G Ninja \
+ -DCMAKE_AR=${LLVM_TOOLCHAIN}/bin/llvm-ar \
+ -DCMAKE_NM=${LLVM_TOOLCHAIN}/bin/llvm-nm \
+ -DCMAKE_RANLIB=${LLVM_TOOLCHAIN}/bin/llvm-ranlib \
+ -DLLVM_CMAKE_DIR="${LLVM_TOOLCHAIN}/lib/cmake/llvm" \
+ -DCMAKE_SYSROOT="${SYSROOT}" \
+ -DCMAKE_ASM_COMPILER_TARGET="${TARGET_TRIPLE}" \
+ -DCMAKE_ASM_FLAGS="${COMPILE_FLAGS}" \
+ -DCMAKE_C_COMPILER_TARGET="${TARGET_TRIPLE}" \
+ -DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \
+ -DCMAKE_C_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
+ -DCMAKE_C_FLAGS="${COMPILE_FLAGS}" \
+ -DCMAKE_CXX_COMPILER_TARGET="${TARGET_TRIPLE}" \
+ -DCMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \
+ -DCMAKE_CXX_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
+ -DCMAKE_CXX_FLAGS="${COMPILE_FLAGS}" \
+ -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
+ -DCOMPILER_RT_BUILD_BUILTINS=ON \
+ -DCOMPILER_RT_BUILD_LIBFUZZER=OFF \
+ -DCOMPILER_RT_BUILD_MEMPROF=OFF \
+ -DCOMPILER_RT_BUILD_PROFILE=OFF \
+ -DCOMPILER_RT_BUILD_CTX_PROFILE=OFF \
+ -DCOMPILER_RT_BUILD_SANITIZERS=OFF \
+ -DCOMPILER_RT_BUILD_XRAY=OFF \
+ -DCOMPILER_RT_BUILD_ORC=OFF \
+ -DCOMPILER_RT_BUILD_CRT=OFF \
+ -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \
+ -DCOMPILER_RT_EMULATOR="qemu-arm -L ${SYSROOT}" \
+ -DCOMPILER_RT_INCLUDE_TESTS=ON \
+ -DCOMPILER_RT_TEST_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
+ -DCOMPILER_RT_TEST_COMPILER_CFLAGS="--target=${TARGET_TRIPLE} ${COMPILE_FLAGS} --gcc-toolchain=${GCC_TOOLCHAIN} --sysroot=${SYSROOT} -fuse-ld=lld"
+```
+
+:::{note}
+The command above also enables tests. Enabling tests is not required, more details
+in the testing section.
+:::
+
+`CMAKE_<LANGUAGE>_<OPTION>` options are set so that the correct `--target`,
+`--sysroot`, `--gcc-toolchain` and `-march` options will be given to the
compilers.
The combination of these settings needs to be enough to pass CMake's compiler
checks, compile compiler-rt and build the test cases.
The flags need to select:
- * The Arm target (``--target arm-none-linux-gnueabihf``)
- * The Arm architecture level (``-march=armv7-a``)
- * Whether to generate Arm (``-marm``, the default) or Thumb (``-mthumb``) instructions.
+: - The Arm target (`--target arm-none-linux-gnueabihf`)
+ - The Arm architecture level (`-march=armv7-a`)
+ - Whether to generate Arm (`-marm`, the default) or Thumb (`-mthumb`) instructions.
-It is possible to pass all these flags to CMake using ``CMAKE_<LANGUAGE>_FLAGS``,
+It is possible to pass all these flags to CMake using `CMAKE_<LANGUAGE>_FLAGS`,
but the command above uses standard CMake options instead. If you need to
add flags that CMake cannot generate automatically, add them to
-``CMAKE_<LANGUAGE>_FLAGS``.
+`CMAKE_<LANGUAGE>_FLAGS`.
-When CMake has finished, build with Ninja::
+When CMake has finished, build with Ninja:
- ninja builtins
+```
+ninja builtins
+```
-Testing compiler-rt builtins using qemu-arm
-===========================================
+## Testing compiler-rt builtins using qemu-arm
-The following options are required to enable tests::
+The following options are required to enable tests:
- -DCOMPILER_RT_EMULATOR="qemu-arm -L ${SYSROOT}" \
- -DCOMPILER_RT_INCLUDE_TESTS=ON \
- -DCOMPILER_RT_TEST_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
- -DCOMPILER_RT_TEST_COMPILER_CFLAGS="--target=${TARGET_TRIPLE} -march=armv7-a --gcc-toolchain=${GCC_TOOLCHAIN} --sysroot=${SYSROOT} -fuse-ld=lld"
+```
+-DCOMPILER_RT_EMULATOR="qemu-arm -L ${SYSROOT}" \
+-DCOMPILER_RT_INCLUDE_TESTS=ON \
+-DCOMPILER_RT_TEST_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
+-DCOMPILER_RT_TEST_COMPILER_CFLAGS="--target=${TARGET_TRIPLE} -march=armv7-a --gcc-toolchain=${GCC_TOOLCHAIN} --sysroot=${SYSROOT} -fuse-ld=lld"
+```
-This tells compiler-rt that we want to run tests on ``qemu-arm``. If you do not
+This tells compiler-rt that we want to run tests on `qemu-arm`. If you do not
want to run tests, remove these options from the CMake command.
-Note that ``COMPILER_RT_TEST_COMPILER_CFLAGS`` contains the equivalent of the
+Note that `COMPILER_RT_TEST_COMPILER_CFLAGS` contains the equivalent of the
options CMake generated for us with the first command. We must pass them
-manually here because standard options like ``CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN``
+manually here because standard options like `CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN`
do not apply here.
-When CMake has finished, run the tests::
+When CMake has finished, run the tests:
- ninja check-builtins
+```
+ninja check-builtins
+```
-Troubleshooting
-===============
+## Troubleshooting
+
+### The cmake try compile stage fails
-The cmake try compile stage fails
----------------------------------
At an early stage cmake will attempt to compile and link a simple C program to
test if the toolchain is working.
-This stage can often fail at link time if the ``--sysroot=``, ``--target``, or
-``--gcc-toolchain=`` options are not passed to the compiler. Check the
-``CMAKE_<LANGUAGE>_FLAGS`` and ``CMAKE_<LANGAUGE>_COMPILER_TARGET`` flags along
+This stage can often fail at link time if the `--sysroot=`, `--target`, or
+`--gcc-toolchain=` options are not passed to the compiler. Check the
+`CMAKE_<LANGUAGE>_FLAGS` and `CMAKE_<LANGAUGE>_COMPILER_TARGET` flags along
with any of the specific CMake sysroot and toolchain options.
It can be useful to build a simple example outside of cmake with your toolchain
-to make sure it is working. For example::
+to make sure it is working. For example:
+
+```
+clang --target=arm-linux-gnueabi -march=armv7a --gcc-toolchain=/path/to/gcc-toolchain --sysroot=/path/to/gcc-toolchain/arm-linux-gnueabihf/libc helloworld.c
+```
- clang --target=arm-linux-gnueabi -march=armv7a --gcc-toolchain=/path/to/gcc-toolchain --sysroot=/path/to/gcc-toolchain/arm-linux-gnueabihf/libc helloworld.c
+### Clang uses the host header files
-Clang uses the host header files
---------------------------------
On Debian-based systems, it is possible to install multiarch support for
-``arm-linux-gnueabi`` and ``arm-linux-gnueabihf``. In many cases clang can successfully
-use this multiarch support when ``--gcc-toolchain=`` and ``--sysroot=`` are not supplied.
-Unfortunately clang adds ``/usr/local/include`` before
-``/usr/include/arm-linux-gnueabihf`` leading to errors when compiling the hosts
+`arm-linux-gnueabi` and `arm-linux-gnueabihf`. In many cases clang can successfully
+use this multiarch support when `--gcc-toolchain=` and `--sysroot=` are not supplied.
+Unfortunately clang adds `/usr/local/include` before
+`/usr/include/arm-linux-gnueabihf` leading to errors when compiling the hosts
header files.
The multiarch support is not sufficient to build the builtins you will need to
-use a separate ``arm-linux-gnueabihf`` toolchain.
+use a separate `arm-linux-gnueabihf` toolchain.
+
+### No target passed to clang
-No target passed to clang
--------------------------
If clang is not given a target, it will typically use the host target. This will
not understand the Arm assembly language files, resulting in error messages such
-as ``error: unknown directive .syntax unified``.
+as `error: unknown directive .syntax unified`.
You can check the clang invocation in the error message to see if there is no
-``--target`` or if it is set incorrectly. The cause is usually
-``CMAKE_ASM_FLAGS`` not containing ``--target`` or ``CMAKE_ASM_COMPILER_TARGET``
+`--target` or if it is set incorrectly. The cause is usually
+`CMAKE_ASM_FLAGS` not containing `--target` or `CMAKE_ASM_COMPILER_TARGET`
not being present.
-Arm architecture not given
---------------------------
-The ``--target=arm-linux-gnueabihf`` will default to Arm architecture v4t which
-cannot assemble the barrier instructions used in the ``synch_and_fetch`` source
+### Arm architecture not given
+
+The `--target=arm-linux-gnueabihf` will default to Arm architecture v4t which
+cannot assemble the barrier instructions used in the `synch_and_fetch` source
files.
-The cause is usually a missing ``-march=armv7a`` from the ``CMAKE_ASM_FLAGS``.
+The cause is usually a missing `-march=armv7a` from the `CMAKE_ASM_FLAGS`.
+
+### Compiler-rt builds but the tests fail to build
-Compiler-rt builds but the tests fail to build
-----------------------------------------------
The flags used to build the tests are not the same as those used to build the
-builtins. The c flags are provided by ``COMPILER_RT_TEST_COMPILE_CFLAGS`` and
-the ``CMAKE_C_COMPILER_TARGET``, ``CMAKE_ASM_COMPILER_TARGET``,
-``CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN`` and ``CMAKE_SYSROOT`` flags are not
+builtins. The c flags are provided by `COMPILER_RT_TEST_COMPILE_CFLAGS` and
+the `CMAKE_C_COMPILER_TARGET`, `CMAKE_ASM_COMPILER_TARGET`,
+`CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN` and `CMAKE_SYSROOT` flags are not
applied to tests.
-Make sure that ``COMPILER_RT_TEST_COMPILE_CFLAGS`` contains all the necessary
+Make sure that `COMPILER_RT_TEST_COMPILE_CFLAGS` contains all the necessary
flags.
+## Modifications for other Targets
-Modifications for other Targets
-===============================
+### Arm Soft-Float Target
-Arm Soft-Float Target
----------------------
The instructions for the Arm hard-float target can be used for the soft-float
target by substituting soft-float equivalents for the sysroot and target. The
target to use is:
-* ``-DCMAKE_C_COMPILER_TARGET=arm-linux-gnueabi``
+- `-DCMAKE_C_COMPILER_TARGET=arm-linux-gnueabi`
Depending on whether you want to use floating point instructions or not, you
-may need extra c-flags such as ``-mfloat-abi=softfp`` for use of floating-point
-instructions, and ``-mfloat-abi=soft -mfpu=none`` for software floating-point
+may need extra c-flags such as `-mfloat-abi=softfp` for use of floating-point
+instructions, and `-mfloat-abi=soft -mfpu=none` for software floating-point
emulation.
-You will need to use an ``arm-linux-gnueabi`` GNU toolchain for soft-float.
+You will need to use an `arm-linux-gnueabi` GNU toolchain for soft-float.
+
+### AArch64 Target
-AArch64 Target
---------------
The instructions for Arm can be used for AArch64 by substituting AArch64
-equivalents for the sysroot, emulator and target::
+equivalents for the sysroot, emulator and target:
- -DCMAKE_C_COMPILER_TARGET=aarch64-linux-gnu
- -DCOMPILER_RT_EMULATOR="qemu-aarch64 -L /path/to/aarch64/sysroot
+```
+-DCMAKE_C_COMPILER_TARGET=aarch64-linux-gnu
+-DCOMPILER_RT_EMULATOR="qemu-aarch64 -L /path/to/aarch64/sysroot
+```
You will also have to update any use of the target triple in compiler flags.
-For instance in ``CMAKE_C_FLAGS`` and ``COMPILER_RT_TEST_COMPILER_CFLAGS``.
+For instance in `CMAKE_C_FLAGS` and `COMPILER_RT_TEST_COMPILER_CFLAGS`.
+
+### Armv6-m, Armv7-m and Armv7E-M targets
-Armv6-m, Armv7-m and Armv7E-M targets
--------------------------------------
To build and test the libraries using a similar method to Armv7-A is possible
but more
diff icult. The main problems are:
-* There is not a ``qemu-arm`` user-mode emulator for bare-metal systems.
- ``qemu-system-arm`` can be used, but this is significantly more
diff icult
+- There is not a `qemu-arm` user-mode emulator for bare-metal systems.
+ `qemu-system-arm` can be used, but this is significantly more
diff icult
to setup. This document does not explain how to do this.
-* The targets to compile compiler-rt have the suffix ``-none-eabi``. This uses
+- The targets to compile compiler-rt have the suffix `-none-eabi`. This uses
the BareMetal driver in clang and by default will not find the libraries
needed to pass the cmake compiler check.
As the Armv6-M, Armv7-M and Armv7E-M builds of compiler-rt only use instructions
that are supported on Armv7-A we can still get most of the value of running the
-tests using the same ``qemu-arm`` that we used for Armv7-A by building and
+tests using the same `qemu-arm` that we used for Armv7-A by building and
running the test cases for Armv7-A but using the builtins compiled for
Armv6-M, Armv7-M or Armv7E-M. This will test that the builtins can be linked
into a binary and execute the tests correctly, but it will not catch if the
builtins use instructions that are supported on Armv7-A but not on Armv6-M,
Armv7-M and Armv7E-M.
-This requires a second ``arm-none-eabi`` toolchain for building the builtins.
+This requires a second `arm-none-eabi` toolchain for building the builtins.
Using a bare-metal toolchain ensures that the target and C library details are
specific to bare-metal instead of using Linux settings. This means that some
tests may behave
diff erently compared to real hardware, but at least the content
@@ -264,67 +269,70 @@ of the builtins library is correct.
Below is an example that builds the builtins for Armv7-M, but runs the tests
as Armv7-A. It is presented in full, but is very similar to the earlier
-command for Armv7-A build and test::
-
- LLVM_TOOLCHAIN=<path to llvm install>/
-
- # For the builtins.
- TARGET_TRIPLE=arm-none-eabi
- GCC_TOOLCHAIN=<path to arm-none-eabi toolchain>/
- SYSROOT=${GCC_TOOLCHAIN}/${TARGET_TRIPLE}/libc
- COMPILE_FLAGS="-march=armv7-m -mfpu=vfpv2"
-
- # For the test cases.
- A_PROFILE_TARGET_TRIPLE=arm-none-linux-gnueabihf
- A_PROFILE_GCC_TOOLCHAIN=<path to arm-none-linux-gnueabihf toolchain>/
- A_PROFILE_SYSROOT=${A_PROFILE_GCC_TOOLCHAIN}/${A_PROFILE_TARGET_TRIPLE}/libc
-
- cmake ../llvm-project/compiler-rt \
- -G Ninja \
- -DCMAKE_AR=${LLVM_TOOLCHAIN}/bin/llvm-ar \
- -DCMAKE_NM=${LLVM_TOOLCHAIN}/bin/llvm-nm \
- -DCMAKE_RANLIB=${LLVM_TOOLCHAIN}/bin/llvm-ranlib \
- -DLLVM_CMAKE_DIR="${LLVM_TOOLCHAIN}/lib/cmake/llvm" \
- -DCMAKE_SYSROOT="${SYSROOT}" \
- -DCMAKE_ASM_COMPILER_TARGET="${TARGET_TRIPLE}" \
- -DCMAKE_ASM_FLAGS="${COMPILE_FLAGS}" \
- -DCMAKE_C_COMPILER_TARGET="${TARGET_TRIPLE}" \
- -DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \
- -DCMAKE_C_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
- -DCMAKE_C_FLAGS="${COMPILE_FLAGS}" \
- -DCMAKE_CXX_COMPILER_TARGET="${TARGET_TRIPLE}" \
- -DCMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \
- -DCMAKE_CXX_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
- -DCMAKE_CXX_FLAGS="${COMPILE_FLAGS}" \
- -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
- -DCOMPILER_RT_BUILD_BUILTINS=ON \
- -DCOMPILER_RT_BUILD_LIBFUZZER=OFF \
- -DCOMPILER_RT_BUILD_MEMPROF=OFF \
- -DCOMPILER_RT_BUILD_PROFILE=OFF \
- -DCOMPILER_RT_BUILD_CTX_PROFILE=OFF \
- -DCOMPILER_RT_BUILD_SANITIZERS=OFF \
- -DCOMPILER_RT_BUILD_XRAY=OFF \
- -DCOMPILER_RT_BUILD_ORC=OFF \
- -DCOMPILER_RT_BUILD_CRT=OFF \
- -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \
- -DCOMPILER_RT_EMULATOR="qemu-arm -L ${A_PROFILE_SYSROOT}" \
- -DCOMPILER_RT_INCLUDE_TESTS=ON \
- -DCOMPILER_RT_TEST_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
- -DCOMPILER_RT_TEST_COMPILER_CFLAGS="--target=${A_PROFILE_TARGET_TRIPLE} -march=armv7-a --gcc-toolchain=${A_PROFILE_GCC_TOOLCHAIN} --sysroot=${A_PROFILE_SYSROOT} -fuse-ld=lld" \
- -DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY \
- -DCOMPILER_RT_OS_DIR="baremetal" \
- -DCOMPILER_RT_BAREMETAL_BUILD=ON
-
-.. note::
- The sysroot used for compiling the tests is ``arm-linux-gnueabihf``, not
- ``arm-none-eabi`` which is used when compiling the builtins.
+command for Armv7-A build and test:
+
+```
+LLVM_TOOLCHAIN=<path to llvm install>/
+
+# For the builtins.
+TARGET_TRIPLE=arm-none-eabi
+GCC_TOOLCHAIN=<path to arm-none-eabi toolchain>/
+SYSROOT=${GCC_TOOLCHAIN}/${TARGET_TRIPLE}/libc
+COMPILE_FLAGS="-march=armv7-m -mfpu=vfpv2"
+
+# For the test cases.
+A_PROFILE_TARGET_TRIPLE=arm-none-linux-gnueabihf
+A_PROFILE_GCC_TOOLCHAIN=<path to arm-none-linux-gnueabihf toolchain>/
+A_PROFILE_SYSROOT=${A_PROFILE_GCC_TOOLCHAIN}/${A_PROFILE_TARGET_TRIPLE}/libc
+
+cmake ../llvm-project/compiler-rt \
+ -G Ninja \
+ -DCMAKE_AR=${LLVM_TOOLCHAIN}/bin/llvm-ar \
+ -DCMAKE_NM=${LLVM_TOOLCHAIN}/bin/llvm-nm \
+ -DCMAKE_RANLIB=${LLVM_TOOLCHAIN}/bin/llvm-ranlib \
+ -DLLVM_CMAKE_DIR="${LLVM_TOOLCHAIN}/lib/cmake/llvm" \
+ -DCMAKE_SYSROOT="${SYSROOT}" \
+ -DCMAKE_ASM_COMPILER_TARGET="${TARGET_TRIPLE}" \
+ -DCMAKE_ASM_FLAGS="${COMPILE_FLAGS}" \
+ -DCMAKE_C_COMPILER_TARGET="${TARGET_TRIPLE}" \
+ -DCMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \
+ -DCMAKE_C_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
+ -DCMAKE_C_FLAGS="${COMPILE_FLAGS}" \
+ -DCMAKE_CXX_COMPILER_TARGET="${TARGET_TRIPLE}" \
+ -DCMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN=${GCC_TOOLCHAIN} \
+ -DCMAKE_CXX_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
+ -DCMAKE_CXX_FLAGS="${COMPILE_FLAGS}" \
+ -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld" \
+ -DCOMPILER_RT_BUILD_BUILTINS=ON \
+ -DCOMPILER_RT_BUILD_LIBFUZZER=OFF \
+ -DCOMPILER_RT_BUILD_MEMPROF=OFF \
+ -DCOMPILER_RT_BUILD_PROFILE=OFF \
+ -DCOMPILER_RT_BUILD_CTX_PROFILE=OFF \
+ -DCOMPILER_RT_BUILD_SANITIZERS=OFF \
+ -DCOMPILER_RT_BUILD_XRAY=OFF \
+ -DCOMPILER_RT_BUILD_ORC=OFF \
+ -DCOMPILER_RT_BUILD_CRT=OFF \
+ -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON \
+ -DCOMPILER_RT_EMULATOR="qemu-arm -L ${A_PROFILE_SYSROOT}" \
+ -DCOMPILER_RT_INCLUDE_TESTS=ON \
+ -DCOMPILER_RT_TEST_COMPILER=${LLVM_TOOLCHAIN}/bin/clang \
+ -DCOMPILER_RT_TEST_COMPILER_CFLAGS="--target=${A_PROFILE_TARGET_TRIPLE} -march=armv7-a --gcc-toolchain=${A_PROFILE_GCC_TOOLCHAIN} --sysroot=${A_PROFILE_SYSROOT} -fuse-ld=lld" \
+ -DCMAKE_TRY_COMPILE_TARGET_TYPE=STATIC_LIBRARY \
+ -DCOMPILER_RT_OS_DIR="baremetal" \
+ -DCOMPILER_RT_BAREMETAL_BUILD=ON
+```
+
+:::{note}
+The sysroot used for compiling the tests is `arm-linux-gnueabihf`, not
+`arm-none-eabi` which is used when compiling the builtins.
+:::
The Armv6-M builtins will use the soft-float ABI. When compiling the tests for
-Armv7-A we must include ``"-mthumb -mfloat-abi=soft -mfpu=none"`` in the
-test-c-flags. We must use an Armv7-A soft-float ABI sysroot for ``qemu-arm``.
+Armv7-A we must include `"-mthumb -mfloat-abi=soft -mfpu=none"` in the
+test-c-flags. We must use an Armv7-A soft-float ABI sysroot for `qemu-arm`.
Depending on the linker used for the test cases, you may encounter BuildAttribute
mismatches between the M-profile objects from compiler-rt and the A-profile
objects from the test. The lld linker does not check the profile
-BuildAttribute so it can be used to link the tests by adding ``-fuse-ld=lld`` to the
-``COMPILER_RT_TEST_COMPILER_CFLAGS``.
+BuildAttribute so it can be used to link the tests by adding `-fuse-ld=lld` to the
+`COMPILER_RT_TEST_COMPILER_CFLAGS`.
diff --git a/llvm/docs/HowToUpdateDebugInfo.md b/llvm/docs/HowToUpdateDebugInfo.md
index 428e3c1407a26..1e985f264685d 100644
--- a/llvm/docs/HowToUpdateDebugInfo.md
+++ b/llvm/docs/HowToUpdateDebugInfo.md
@@ -1,12 +1,10 @@
-=======================================================
-How to Update Debug Info: A Guide for LLVM Pass Authors
-=======================================================
+# How to Update Debug Info: A Guide for LLVM Pass Authors
-.. contents::
- :local:
+```{contents}
+:local: true
+```
-Introduction
-============
+## Introduction
Certain kinds of code transformations can inadvertently result in a loss of
debug info, or worse, make debug info misrepresent the state of a program. Debug
@@ -17,20 +15,18 @@ code transformations, and offers suggestions for how to create targeted debug
info tests for arbitrary transformations.
For more on the philosophy behind LLVM debugging information, see
-:doc:`SourceLevelDebugging`.
+{doc}`SourceLevelDebugging`.
-Rules for updating debug locations
-==================================
+## Rules for updating debug locations
-.. _WhenToPreserveLocation:
+(WhenToPreserveLocation)=
-When to preserve an instruction location
-----------------------------------------
+### When to preserve an instruction location
A transformation should preserve the debug location of an instruction if the
instruction either remains in its basic block, or if its basic block is folded
into a predecessor that branches unconditionally. The APIs to use are
-``IRBuilder``, or ``Instruction::setDebugLoc``.
+`IRBuilder`, or `Instruction::setDebugLoc`.
The purpose of this rule is to ensure that common block-local optimizations
preserve the ability to set breakpoints on source locations corresponding to
@@ -39,26 +35,23 @@ would be severely impacted if that ability were lost.
Examples of transformations that should follow this rule include:
-* Instruction scheduling. Block-local instruction reordering should not drop
+- Instruction scheduling. Block-local instruction reordering should not drop
source locations, even though this may lead to jumpy single-stepping
behavior.
-
-* Simple jump threading. For example, if block ``B1`` unconditionally jumps to
- ``B2``, *and* is its unique predecessor, instructions from ``B2`` can be
- hoisted into ``B1``. Source locations from ``B2`` should be preserved.
-
-* Peephole optimizations that replace or expand an instruction, like ``(add X
- X) => (shl X 1)``. The location of the ``shl`` instruction should be the same
- as the location of the ``add`` instruction.
-
-* Tail duplication. For example, if blocks ``B1`` and ``B2`` both
- unconditionally branch to ``B3`` and ``B3`` can be folded into its
- predecessors, source locations from ``B3`` should be preserved.
+- Simple jump threading. For example, if block `B1` unconditionally jumps to
+ `B2`, *and* is its unique predecessor, instructions from `B2` can be
+ hoisted into `B1`. Source locations from `B2` should be preserved.
+- Peephole optimizations that replace or expand an instruction, like `(add X
+ X) => (shl X 1)`. The location of the `shl` instruction should be the same
+ as the location of the `add` instruction.
+- Tail duplication. For example, if blocks `B1` and `B2` both
+ unconditionally branch to `B3` and `B3` can be folded into its
+ predecessors, source locations from `B3` should be preserved.
Examples of transformations for which this rule *does not* apply include:
-* LICM. E.g., if an instruction is moved from the loop body to the preheader,
- the rule for :ref:`dropping locations<WhenToDropLocation>` applies.
+- LICM. E.g., if an instruction is moved from the loop body to the preheader,
+ the rule for {ref}`dropping locations <WhenToDropLocation>` applies.
In addition to the rule above, a transformation should also preserve the debug
location of an instruction that is moved between basic blocks, if the
@@ -67,19 +60,18 @@ location.
Examples of transformations that should follow this rule include:
-* Moving instructions between basic blocks. For example, if instruction ``I1``
- in ``BB1`` is moved before ``I2`` in ``BB2``, the source location of ``I1``
- can be preserved if it has the same source location as ``I2``.
+- Moving instructions between basic blocks. For example, if instruction `I1`
+ in `BB1` is moved before `I2` in `BB2`, the source location of `I1`
+ can be preserved if it has the same source location as `I2`.
-.. _WhenToMergeLocation:
+(WhenToMergeLocation)=
-When to merge instruction locations
------------------------------------
+### When to merge instruction locations
A transformation should merge instruction locations if it replaces multiple
instructions with one or more new instructions, *and* the new instruction(s)
produce the output of more than one of the original instructions. The API to use
-is ``Instruction::applyMergedLocation``. For each new instruction I, its new
+is `Instruction::applyMergedLocation`. For each new instruction I, its new
location should be a merge of the locations of all instructions whose output is
produced by I. Typically, this includes any instruction being RAUWed by a new
instruction, and excludes any instruction that only produces an intermediate
@@ -101,60 +93,54 @@ representing merged instructions in the line table is implemented.
Examples of transformations that should follow this rule include:
-* Hoisting identical instructions from all successors of a conditional branch
+- Hoisting identical instructions from all successors of a conditional branch
or sinking those from all paths to a postdominating block. For example,
merging identical loads/stores which occur on both sides of a CFG diamond
- (see the ``MergedLoadStoreMotion`` pass). For each group of identical
+ (see the `MergedLoadStoreMotion` pass). For each group of identical
instructions being hoisted/sunk, the merge of all their locations should be
applied to the merged instruction.
-
-* Merging identical loop-invariant stores (see the LICM utility
- ``llvm::promoteLoopAccessesToScalars``).
-
-* Scalar instructions being combined into a vector instruction, like
- ``(add A1, B1), (add A2, B2) => (add (A1, A2), (B1, B2))``. As the new vector
- ``add`` computes the result of both original ``add`` instructions
+- Merging identical loop-invariant stores (see the LICM utility
+ `llvm::promoteLoopAccessesToScalars`).
+- Scalar instructions being combined into a vector instruction, like
+ `(add A1, B1), (add A2, B2) => (add (A1, A2), (B1, B2))`. As the new vector
+ `add` computes the result of both original `add` instructions
simultaneously, it should use a merge of the two locations. Similarly, if
- prior optimizations have already produced vectors ``(A1, A2)`` and
- ``(B2, B1)``, then we might create a ``(shufflevector (1, 0), (B2, B1))``
- instruction to produce ``(B1, B2)`` for the vector ``add``; in this case we've
- created two instructions to replace the original ``adds``, so both new
+ prior optimizations have already produced vectors `(A1, A2)` and
+ `(B2, B1)`, then we might create a `(shufflevector (1, 0), (B2, B1))`
+ instruction to produce `(B1, B2)` for the vector `add`; in this case we've
+ created two instructions to replace the original `adds`, so both new
instructions should use the merged location.
Examples of transformations for which this rule *does not* apply include:
-* Block-local peepholes which delete redundant instructions, like
- ``(sext (zext i8 %x to i16) to i32) => (zext i8 %x to i32)``. The inner
- ``zext`` is modified but remains in its block, so the rule for
- :ref:`preserving locations<WhenToPreserveLocation>` should apply.
-
-* Peephole optimizations which combine multiple instructions together, like
- ``(add (mul A B) C) => llvm.fma.f32(A, B, C)``. Note that the result of the
- ``mul`` no longer appears in the program, while the result of the ``add`` is
- now produced by the ``fma``, so the ``add``'s location should be used.
-
-* Converting an if-then-else CFG diamond into a ``select``. Preserving the
+- Block-local peepholes which delete redundant instructions, like
+ `(sext (zext i8 %x to i16) to i32) => (zext i8 %x to i32)`. The inner
+ `zext` is modified but remains in its block, so the rule for
+ {ref}`preserving locations <WhenToPreserveLocation>` should apply.
+- Peephole optimizations which combine multiple instructions together, like
+ `(add (mul A B) C) => llvm.fma.f32(A, B, C)`. Note that the result of the
+ `mul` no longer appears in the program, while the result of the `add` is
+ now produced by the `fma`, so the `add`'s location should be used.
+- Converting an if-then-else CFG diamond into a `select`. Preserving the
debug locations of speculated instructions can make it seem like a condition
is true when it's not (or vice versa), which leads to a confusing
single-stepping experience. The rule for
- :ref:`dropping locations<WhenToDropLocation>` should apply here.
-
-* Hoisting/sinking that would make a location reachable when it previously
+ {ref}`dropping locations <WhenToDropLocation>` should apply here.
+- Hoisting/sinking that would make a location reachable when it previously
wasn't. Consider hoisting two identical instructions with the same location
from first two cases of a switch that has three cases. Merging their
locations would make the location from the first two cases reachable when the
third case is taken. The rule for
- :ref:`dropping locations<WhenToDropLocation>` applies.
+ {ref}`dropping locations <WhenToDropLocation>` applies.
-.. _WhenToDropLocation:
+(WhenToDropLocation)=
-When to drop an instruction location
-------------------------------------
+### When to drop an instruction location
A transformation should drop debug locations if the rules for
-:ref:`preserving<WhenToPreserveLocation>` and
-:ref:`merging<WhenToMergeLocation>` debug locations do not apply. The API to
-use is ``Instruction::dropLocation()``.
+{ref}`preserving <WhenToPreserveLocation>` and
+{ref}`merging <WhenToMergeLocation>` debug locations do not apply. The API to
+use is `Instruction::dropLocation()`.
The purpose of this rule is to prevent erratic or misleading single-stepping
behavior in situations in which an instruction has no clear, unambiguous
@@ -166,51 +152,46 @@ to setting a line 0 location with viable scope information if no previous
location is available.
See the discussion in the section about
-:ref:`merging locations<WhenToMergeLocation>` for examples of when the rule for
+{ref}`merging locations <WhenToMergeLocation>` for examples of when the rule for
dropping locations applies.
-When to remap a debug location
-------------------------------
+### When to remap a debug location
When code paths are duplicated, during passes such as loop unrolling or jump
threading, `DILocation` attachments need to be remapped using `mapAtomInstance`
and `RemapSourceAtom`. This is to support the Key Instructions debug info feature.
-See :doc:`KeyInstructionsDebugInfo` for information.
+See {doc}`KeyInstructionsDebugInfo` for information.
-.. _NewInstLocations:
+(NewInstLocations)=
-Setting locations for new instructions
---------------------------------------
+### Setting locations for new instructions
Whenever a new instruction is created and there is no suitable location for that
instruction, that instruction should be annotated accordingly. There are a set
-of special ``DebugLoc`` values that can be set on an instruction to annotate the
+of special `DebugLoc` values that can be set on an instruction to annotate the
reason that it does not have a valid location. These are as follows:
-* ``DebugLoc::getCompilerGenerated()``: This indicates that the instruction is a
+- `DebugLoc::getCompilerGenerated()`: This indicates that the instruction is a
compiler-generated instruction, i.e. it is not associated with any user source
code.
-
-* ``DebugLoc::getDropped()``: This indicates that the instruction has
+- `DebugLoc::getDropped()`: This indicates that the instruction has
intentionally had its source location removed, according to the rules for
- :ref:`dropping locations<WhenToDropLocation>`; this is set automatically by
- ``Instruction::dropLocation()``.
-
-* ``DebugLoc::getUnknown()``: This indicates that the instruction does not have
+ {ref}`dropping locations <WhenToDropLocation>`; this is set automatically by
+ `Instruction::dropLocation()`.
+- `DebugLoc::getUnknown()`: This indicates that the instruction does not have
a known or currently knowable source location, e.g. that it is infeasible to
determine the correct source location, or that the source location is
ambiguous in a way that LLVM cannot currently represent.
-
-* ``DebugLoc::getTemporary()``: This is used for instructions that we don't
- expect to be emitted (e.g. ``UnreachableInst``), and so should not need a
+- `DebugLoc::getTemporary()`: This is used for instructions that we don't
+ expect to be emitted (e.g. `UnreachableInst`), and so should not need a
valid location; if we ever try to emit a temporary location into an object/asm
file, this indicates that something has gone wrong.
Where applicable, these should be used instead of leaving an instruction without
-an assigned location or explicitly setting the location as ``DebugLoc()``.
+an assigned location or explicitly setting the location as `DebugLoc()`.
Ordinarily these special locations are identical to an absent location, but LLVM
built with coverage-tracking
-(``-DLLVM_ENABLE_DEBUGLOC_COVERAGE_TRACKING="COVERAGE"``) will keep track of
+(`-DLLVM_ENABLE_DEBUGLOC_COVERAGE_TRACKING="COVERAGE"`) will keep track of
these special locations in order to detect unintentionally-missing locations;
for this reason, the most important rule is to *not* apply any of these if it
isn't clear which, if any, is appropriate - an absent location can be detected
@@ -218,106 +199,97 @@ and fixed, while an incorrectly annotated instruction is much harder to detect.
On the other hand, if any of these clearly apply, then they should be used to
prevent false positives from being flagged up.
-Rules for updating debug values
-===============================
+## Rules for updating debug values
-Deleting an IR-level Instruction
---------------------------------
+### Deleting an IR-level Instruction
-When an ``Instruction`` is deleted, its debug uses change to ``undef``. This is
+When an `Instruction` is deleted, its debug uses change to `undef`. This is
a loss of debug info: the value of one or more source variables becomes
-unavailable, starting with the ``#dbg_value(undef, ...)``. When there is no
+unavailable, starting with the `#dbg_value(undef, ...)`. When there is no
way to reconstitute the value of the lost instruction, this is the best
possible outcome. However, it's often possible to do better:
-* If the dying instruction can be RAUW'd, do so. The
- ``Value::replaceAllUsesWith`` API transparently updates debug uses of the
+- If the dying instruction can be RAUW'd, do so. The
+ `Value::replaceAllUsesWith` API transparently updates debug uses of the
dying instruction to point to the replacement value.
-
-* If the dying instruction cannot be RAUW'd, call ``llvm::salvageDebugInfo`` on
+- If the dying instruction cannot be RAUW'd, call `llvm::salvageDebugInfo` on
it. This makes a best-effort attempt to rewrite debug uses of the dying
- instruction by describing its effect as a ``DIExpression``.
-
-* If one of the **operands** of a dying instruction would become trivially
- dead, use ``llvm::replaceAllDbgUsesWith`` to rewrite the debug uses of that
+ instruction by describing its effect as a `DIExpression`.
+- If one of the **operands** of a dying instruction would become trivially
+ dead, use `llvm::replaceAllDbgUsesWith` to rewrite the debug uses of that
operand. Consider the following example function:
-.. code-block:: llvm
-
- define i16 @foo(i16 %a) {
- %b = sext i16 %a to i32
- %c = and i32 %b, 15
- #dbg_value(i32 %c, ...)
- %d = trunc i32 %c to i16
- ret i16 %d
- }
-
-Now, here's what happens after the unnecessary truncation instruction ``%d`` is
+```llvm
+define i16 @foo(i16 %a) {
+ %b = sext i16 %a to i32
+ %c = and i32 %b, 15
+ #dbg_value(i32 %c, ...)
+ %d = trunc i32 %c to i16
+ ret i16 %d
+}
+```
+
+Now, here's what happens after the unnecessary truncation instruction `%d` is
replaced with a simplified instruction:
-.. code-block:: llvm
+```llvm
+define i16 @foo(i16 %a) {
+ #dbg_value(i32 undef, ...)
+ %simplified = and i16 %a, 15
+ ret i16 %simplified
+}
+```
- define i16 @foo(i16 %a) {
- #dbg_value(i32 undef, ...)
- %simplified = and i16 %a, 15
- ret i16 %simplified
- }
-
-Note that after deleting ``%d``, all uses of its operand ``%c`` become
-trivially dead. The debug use which used to point to ``%c`` is now ``undef``,
+Note that after deleting `%d`, all uses of its operand `%c` become
+trivially dead. The debug use which used to point to `%c` is now `undef`,
and debug info is needlessly lost.
To solve this problem, do:
-.. code-block:: cpp
-
- llvm::replaceAllDbgUsesWith(%c, theSimplifiedAndInstruction, ...)
+```cpp
+llvm::replaceAllDbgUsesWith(%c, theSimplifiedAndInstruction, ...)
+```
-This results in better debug info because the debug use of ``%c`` is preserved:
+This results in better debug info because the debug use of `%c` is preserved:
-.. code-block:: llvm
+```llvm
+define i16 @foo(i16 %a) {
+ %simplified = and i16 %a, 15
+ #dbg_value(i16 %simplified, ...)
+ ret i16 %simplified
+}
+```
- define i16 @foo(i16 %a) {
- %simplified = and i16 %a, 15
- #dbg_value(i16 %simplified, ...)
- ret i16 %simplified
- }
-
-You may have noticed that ``%simplified`` is narrower than ``%c``: this is not
-a problem, because ``llvm::replaceAllDbgUsesWith`` takes care of inserting the
+You may have noticed that `%simplified` is narrower than `%c`: this is not
+a problem, because `llvm::replaceAllDbgUsesWith` takes care of inserting the
necessary conversion operations into the DIExpressions of updated debug uses.
-Deleting a MIR-level MachineInstr
----------------------------------
+### Deleting a MIR-level MachineInstr
TODO
-Rules for updating ``DIAssignID`` Attachments
-=============================================
+## Rules for updating `DIAssignID` Attachments
-``DIAssignID`` metadata attachments are used by Assignment Tracking, which is
+`DIAssignID` metadata attachments are used by Assignment Tracking, which is
currently an experimental debug mode.
-See :doc:`AssignmentTracking` for how to update them and for more info on
+See {doc}`AssignmentTracking` for how to update them and for more info on
Assignment Tracking.
-How to automatically convert tests into debug info tests
-========================================================
+## How to automatically convert tests into debug info tests
-.. _IRDebugify:
+(IRDebugify)=
-Mutation testing for IR-level transformations
----------------------------------------------
+### Mutation testing for IR-level transformations
An IR test case for a transformation can, in many cases, be automatically
mutated to test debug info handling within that transformation. This is a
simple way to test for proper debug info handling.
-The ``debugify`` utility pass
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### The `debugify` utility pass
-The ``debugify`` testing utility is just a pair of passes: ``debugify`` and
-``check-debugify``.
+The `debugify` testing utility is just a pair of passes: `debugify` and
+`check-debugify`.
The first applies synthetic debug information to every instruction of the
module, and the second checks that this DI is still available after an
@@ -328,130 +300,127 @@ immediately used by debug value records everywhere possible.
For example, here is a module before:
-.. code-block:: llvm
-
- define void @f(i32* %x) {
- entry:
- %x.addr = alloca i32*, align 8
- store i32* %x, i32** %x.addr, align 8
- %0 = load i32*, i32** %x.addr, align 8
- store i32 10, i32* %0, align 4
- ret void
- }
-
-and after running ``opt -debugify``:
-
-.. code-block:: llvm
-
- define void @f(i32* %x) !dbg !6 {
- entry:
- %x.addr = alloca i32*, align 8, !dbg !12
- #dbg_value(i32** %x.addr, !9, !DIExpression(), !12)
- store i32* %x, i32** %x.addr, align 8, !dbg !13
- %0 = load i32*, i32** %x.addr, align 8, !dbg !14
- #dbg_value(i32* %0, !11, !DIExpression(), !14)
- store i32 10, i32* %0, align 4, !dbg !15
- ret void, !dbg !16
- }
-
- !llvm.dbg.cu = !{!0}
- !llvm.debugify = !{!3, !4}
- !llvm.module.flags = !{!5}
-
- !0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1, producer: "debugify", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2)
- !1 = !DIFile(filename: "debugify-sample.ll", directory: "/")
- !2 = !{}
- !3 = !{i32 5}
- !4 = !{i32 2}
- !5 = !{i32 2, !"Debug Info Version", i32 3}
- !6 = distinct !DISubprogram(name: "f", linkageName: "f", scope: null, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, scopeLine: 1, isOptimized: true, unit: !0, retainedNodes: !8)
- !7 = !DISubroutineType(types: !2)
- !8 = !{!9, !11}
- !9 = !DILocalVariable(name: "1", scope: !6, file: !1, line: 1, type: !10)
- !10 = !DIBasicType(name: "ty64", size: 64, encoding: DW_ATE_unsigned)
- !11 = !DILocalVariable(name: "2", scope: !6, file: !1, line: 3, type: !10)
- !12 = !DILocation(line: 1, column: 1, scope: !6)
- !13 = !DILocation(line: 2, column: 1, scope: !6)
- !14 = !DILocation(line: 3, column: 1, scope: !6)
- !15 = !DILocation(line: 4, column: 1, scope: !6)
- !16 = !DILocation(line: 5, column: 1, scope: !6)
-
-Using ``debugify``
-^^^^^^^^^^^^^^^^^^
-
-A simple way to use ``debugify`` is as follows:
-
-.. code-block:: bash
-
- $ opt -debugify -pass-to-test -check-debugify sample.ll
-
-This will inject synthetic DI to ``sample.ll`` run the ``pass-to-test`` and
-then check for missing DI. The ``-check-debugify`` step can of course be
+```llvm
+define void @f(i32* %x) {
+entry:
+ %x.addr = alloca i32*, align 8
+ store i32* %x, i32** %x.addr, align 8
+ %0 = load i32*, i32** %x.addr, align 8
+ store i32 10, i32* %0, align 4
+ ret void
+}
+```
+
+and after running `opt -debugify`:
+
+```llvm
+define void @f(i32* %x) !dbg !6 {
+entry:
+ %x.addr = alloca i32*, align 8, !dbg !12
+ #dbg_value(i32** %x.addr, !9, !DIExpression(), !12)
+ store i32* %x, i32** %x.addr, align 8, !dbg !13
+ %0 = load i32*, i32** %x.addr, align 8, !dbg !14
+ #dbg_value(i32* %0, !11, !DIExpression(), !14)
+ store i32 10, i32* %0, align 4, !dbg !15
+ ret void, !dbg !16
+}
+
+!llvm.dbg.cu = !{!0}
+!llvm.debugify = !{!3, !4}
+!llvm.module.flags = !{!5}
+
+!0 = distinct !DICompileUnit(language: DW_LANG_C, file: !1, producer: "debugify", isOptimized: true, runtimeVersion: 0, emissionKind: FullDebug, enums: !2)
+!1 = !DIFile(filename: "debugify-sample.ll", directory: "/")
+!2 = !{}
+!3 = !{i32 5}
+!4 = !{i32 2}
+!5 = !{i32 2, !"Debug Info Version", i32 3}
+!6 = distinct !DISubprogram(name: "f", linkageName: "f", scope: null, file: !1, line: 1, type: !7, isLocal: false, isDefinition: true, scopeLine: 1, isOptimized: true, unit: !0, retainedNodes: !8)
+!7 = !DISubroutineType(types: !2)
+!8 = !{!9, !11}
+!9 = !DILocalVariable(name: "1", scope: !6, file: !1, line: 1, type: !10)
+!10 = !DIBasicType(name: "ty64", size: 64, encoding: DW_ATE_unsigned)
+!11 = !DILocalVariable(name: "2", scope: !6, file: !1, line: 3, type: !10)
+!12 = !DILocation(line: 1, column: 1, scope: !6)
+!13 = !DILocation(line: 2, column: 1, scope: !6)
+!14 = !DILocation(line: 3, column: 1, scope: !6)
+!15 = !DILocation(line: 4, column: 1, scope: !6)
+!16 = !DILocation(line: 5, column: 1, scope: !6)
+```
+
+#### Using `debugify`
+
+A simple way to use `debugify` is as follows:
+
+```bash
+$ opt -debugify -pass-to-test -check-debugify sample.ll
+```
+
+This will inject synthetic DI to `sample.ll` run the `pass-to-test` and
+then check for missing DI. The `-check-debugify` step can of course be
omitted in favor of more customizable FileCheck directives.
Some other ways to run debugify are available:
-.. code-block:: bash
-
- # Same as the above example.
- $ opt -enable-debugify -pass-to-test sample.ll
+```bash
+# Same as the above example.
+$ opt -enable-debugify -pass-to-test sample.ll
- # Suppresses verbose debugify output.
- $ opt -enable-debugify -debugify-quiet -pass-to-test sample.ll
+# Suppresses verbose debugify output.
+$ opt -enable-debugify -debugify-quiet -pass-to-test sample.ll
- # Prepend -debugify before and append -check-debugify -strip after
- # each pass on the pipeline (similar to -verify-each).
- $ opt -debugify-each -O2 sample.ll
+# Prepend -debugify before and append -check-debugify -strip after
+# each pass on the pipeline (similar to -verify-each).
+$ opt -debugify-each -O2 sample.ll
+```
-In order for ``check-debugify`` to work, the DI must be coming from
-``debugify``. Thus, modules with existing DI will be skipped.
+In order for `check-debugify` to work, the DI must be coming from
+`debugify`. Thus, modules with existing DI will be skipped.
-``debugify`` can be used to test a backend, e.g:
+`debugify` can be used to test a backend, e.g:
-.. code-block:: bash
-
- $ opt -debugify < sample.ll | llc -o -
+```bash
+$ opt -debugify < sample.ll | llc -o -
+```
There is also a MIR-level debugify pass that can be run before each backend
pass, see:
-:ref:`Mutation testing for MIR-level transformations<MIRDebugify>`.
+{ref}`Mutation testing for MIR-level transformations <MIRDebugify>`.
-``debugify`` in regression tests
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### `debugify` in regression tests
-The output of the ``debugify`` pass must be stable enough to use in regression
+The output of the `debugify` pass must be stable enough to use in regression
tests. Changes to this pass are not allowed to break existing tests.
-.. note::
-
- Regression tests must be robust. Avoid hardcoding line/variable numbers in
- check lines. In cases where this can't be avoided (say, if a test wouldn't
- be precise enough), moving the test to its own file is preferred.
+:::{note}
+Regression tests must be robust. Avoid hardcoding line/variable numbers in
+check lines. In cases where this can't be avoided (say, if a test wouldn't
+be precise enough), moving the test to its own file is preferred.
+:::
-Using Coverage Tracking to remove false positives
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### Using Coverage Tracking to remove false positives
-As described :ref:`above<WhenToDropLocation>`, there are valid reasons for
+As described {ref}`above <WhenToDropLocation>`, there are valid reasons for
instructions to not have source locations. Therefore, when detecting missing
source locations, it may be preferable to avoid detecting cases where the
missing source location is intentional. For this, you can use the "coverage
-tracking" feature in LLVM to prevent these from appearing in the ``debugify``
+tracking" feature in LLVM to prevent these from appearing in the `debugify`
output. This is enabled in a build of LLVM by setting the CMake flag
-``-DLLVM_ENABLE_DEBUGLOC_COVERAGE_TRACKING=COVERAGE``. When this has been set,
+`-DLLVM_ENABLE_DEBUGLOC_COVERAGE_TRACKING=COVERAGE`. When this has been set,
LLVM will enable runtime tracking of
-:ref:`DebugLoc annotations<NewInstLocations>`, allowing ``debugify`` to ignore
+{ref}`DebugLoc annotations <NewInstLocations>`, allowing `debugify` to ignore
instructions that have an explicitly recorded reason given for not having a
source location.
-For triaging source location bugs detected with ``debugify``, you may find it
+For triaging source location bugs detected with `debugify`, you may find it
helpful to instead set the CMake flag to enable "origin tracking",
-``-DLLVM_ENABLE_DEBUGLOC_COVERAGE_TRACKING=COVERAGE_AND_ORIGIN``. This flag
-allows more detail to be added to ``debugify``'s output, by including one or
+`-DLLVM_ENABLE_DEBUGLOC_COVERAGE_TRACKING=COVERAGE_AND_ORIGIN`. This flag
+allows more detail to be added to `debugify`'s output, by including one or
more stacktraces with every missing source location, capturing the point at
which the empty source location was created, and every point at which it was
copied to an instruction, making it trivial in most cases to find the origin of
the underlying bug. When origin tracking is enabled, the
-``--enable-origin-stacktraces`` flag must be passed to actually trigger the
+`--enable-origin-stacktraces` flag must be passed to actually trigger the
collecting of stacktraces; this flag can be passed as-is to collect stacktraces
all the time, or it can be passed with a comma-separated list of pass names (in
their internal PascalCase form) to enable collecting stacktraces during only
@@ -460,175 +429,172 @@ those passes.
If using origin tracking, it is recommended to also build LLVM with debug info
enabled, so that the stacktrace can be accurately symbolized.
-.. note::
+:::{note}
+The coverage tracking feature has been designed primarily for use with the
+{ref}`original debug info preservation <OriginalDI>` mode of `debugify`, and
+so may not be reliable in other settings. When using this mode, the
+stacktraces produced by the `COVERAGE_AND_ORIGIN` setting will be printed
+in an easy-to-read format as part of the reports generated by the
+`llvm-original-di-preservation.py` script.
+:::
- The coverage tracking feature has been designed primarily for use with the
- :ref:`original debug info preservation<OriginalDI>` mode of ``debugify``, and
- so may not be reliable in other settings. When using this mode, the
- stacktraces produced by the ``COVERAGE_AND_ORIGIN`` setting will be printed
- in an easy-to-read format as part of the reports generated by the
- ``llvm-original-di-preservation.py`` script.
+(OriginalDI)=
-.. _OriginalDI:
-
-Test original debug info preservation in optimizations
-------------------------------------------------------
+### Test original debug info preservation in optimizations
In addition to automatically generating debug info, the checks provided by
-the ``debugify`` utility pass can also be used to test the preservation of
+the `debugify` utility pass can also be used to test the preservation of
pre-existing debug info metadata. It could be run as follows:
-.. code-block:: bash
-
- # Run the pass by checking original Debug Info preservation.
- $ opt -verify-debuginfo-preserve -pass-to-test sample.ll
+```bash
+# Run the pass by checking original Debug Info preservation.
+$ opt -verify-debuginfo-preserve -pass-to-test sample.ll
- # Check the preservation of original Debug Info after each pass.
- $ opt -verify-each-debuginfo-preserve -O2 sample.ll
+# Check the preservation of original Debug Info after each pass.
+$ opt -verify-each-debuginfo-preserve -O2 sample.ll
+```
Limit number of observed functions to speed up the analysis:
-.. code-block:: bash
+```bash
+# Test up to 100 functions (per compile unit) per pass.
+$ opt -verify-each-debuginfo-preserve -O2 -debugify-func-limit=100 sample.ll
+```
- # Test up to 100 functions (per compile unit) per pass.
- $ opt -verify-each-debuginfo-preserve -O2 -debugify-func-limit=100 sample.ll
-
-Please do note that running ``-verify-each-debuginfo-preserve`` on big projects
+Please do note that running `-verify-each-debuginfo-preserve` on big projects
could be heavily time consuming. Therefore, we suggest using
-``-debugify-func-limit`` with a suitable limit number to prevent extremely long
+`-debugify-func-limit` with a suitable limit number to prevent extremely long
builds.
Furthermore, there is a way to export the issues that have been found into
a JSON file as follows:
-.. code-block:: bash
-
- $ opt -verify-debuginfo-preserve -verify-di-preserve-export=sample.json -pass-to-test sample.ll
+```bash
+$ opt -verify-debuginfo-preserve -verify-di-preserve-export=sample.json -pass-to-test sample.ll
+```
-and then use the ``llvm/utils/llvm-original-di-preservation.py`` script
+and then use the `llvm/utils/llvm-original-di-preservation.py` script
to generate an HTML page with the issues reported in a more human-readable form
as follows:
-.. code-block:: bash
-
- $ llvm-original-di-preservation.py sample.json --report-file sample.html
+```bash
+$ llvm-original-di-preservation.py sample.json --report-file sample.html
+```
Testing of original debug info preservation can be invoked from front-end level
as follows:
-.. code-block:: bash
-
- # Test each pass.
- $ clang -Xclang -fverify-debuginfo-preserve -g -O2 sample.c
+```bash
+# Test each pass.
+$ clang -Xclang -fverify-debuginfo-preserve -g -O2 sample.c
- # Test each pass and export the issues report into the JSON file.
- $ clang -Xclang -fverify-debuginfo-preserve -Xclang -fverify-debuginfo-preserve-export=sample.json -g -O2 sample.c
+# Test each pass and export the issues report into the JSON file.
+$ clang -Xclang -fverify-debuginfo-preserve -Xclang -fverify-debuginfo-preserve-export=sample.json -g -O2 sample.c
+```
Please do note that there are some known false positives, for source locations
and debug record checking, so that will be addressed as a future work.
-.. _MIRDebugify:
+(MIRDebugify)=
-Mutation testing for MIR-level transformations
-----------------------------------------------
+### Mutation testing for MIR-level transformations
-A variant of the ``debugify`` utility described in
-:ref:`Mutation testing for IR-level transformations<IRDebugify>` can be used
+A variant of the `debugify` utility described in
+{ref}`Mutation testing for IR-level transformations <IRDebugify>` can be used
for MIR-level transformations as well: much like the IR-level pass,
-``mir-debugify`` inserts sequentially increasing line locations to each
-``MachineInstr`` in a ``Module``. And the MIR-level ``mir-check-debugify`` is
-similar to IR-level ``check-debugify`` pass.
+`mir-debugify` inserts sequentially increasing line locations to each
+`MachineInstr` in a `Module`. And the MIR-level `mir-check-debugify` is
+similar to IR-level `check-debugify` pass.
For example, here is a snippet before:
-.. code-block:: llvm
-
- name: test
- body: |
- bb.1 (%ir-block.0):
- %0:_(s32) = IMPLICIT_DEF
- %1:_(s32) = IMPLICIT_DEF
- %2:_(s32) = G_CONSTANT i32 2
- %3:_(s32) = G_ADD %0, %2
- %4:_(s32) = G_SUB %3, %1
-
-and after running ``llc -run-pass=mir-debugify``:
-
-.. code-block:: llvm
-
- name: test
- body: |
- bb.0 (%ir-block.0):
- %0:_(s32) = IMPLICIT_DEF debug-location !12
- DBG_VALUE %0(s32), $noreg, !9, !DIExpression(), debug-location !12
- %1:_(s32) = IMPLICIT_DEF debug-location !13
- DBG_VALUE %1(s32), $noreg, !11, !DIExpression(), debug-location !13
- %2:_(s32) = G_CONSTANT i32 2, debug-location !14
- DBG_VALUE %2(s32), $noreg, !9, !DIExpression(), debug-location !14
- %3:_(s32) = G_ADD %0, %2, debug-location !DILocation(line: 4, column: 1, scope: !6)
- DBG_VALUE %3(s32), $noreg, !9, !DIExpression(), debug-location !DILocation(line: 4, column: 1, scope: !6)
- %4:_(s32) = G_SUB %3, %1, debug-location !DILocation(line: 5, column: 1, scope: !6)
- DBG_VALUE %4(s32), $noreg, !9, !DIExpression(), debug-location !DILocation(line: 5, column: 1, scope: !6)
-
-By default, ``mir-debugify`` inserts ``DBG_VALUE`` instructions **everywhere**
-it is legal to do so. In particular, every (non-PHI) machine instruction that
-defines a register must be followed by a ``DBG_VALUE`` use of that def. If
+```llvm
+name: test
+body: |
+ bb.1 (%ir-block.0):
+ %0:_(s32) = IMPLICIT_DEF
+ %1:_(s32) = IMPLICIT_DEF
+ %2:_(s32) = G_CONSTANT i32 2
+ %3:_(s32) = G_ADD %0, %2
+ %4:_(s32) = G_SUB %3, %1
+```
+
+and after running `llc -run-pass=mir-debugify`:
+
+```llvm
+name: test
+body: |
+ bb.0 (%ir-block.0):
+ %0:_(s32) = IMPLICIT_DEF debug-location !12
+ DBG_VALUE %0(s32), $noreg, !9, !DIExpression(), debug-location !12
+ %1:_(s32) = IMPLICIT_DEF debug-location !13
+ DBG_VALUE %1(s32), $noreg, !11, !DIExpression(), debug-location !13
+ %2:_(s32) = G_CONSTANT i32 2, debug-location !14
+ DBG_VALUE %2(s32), $noreg, !9, !DIExpression(), debug-location !14
+ %3:_(s32) = G_ADD %0, %2, debug-location !DILocation(line: 4, column: 1, scope: !6)
+ DBG_VALUE %3(s32), $noreg, !9, !DIExpression(), debug-location !DILocation(line: 4, column: 1, scope: !6)
+ %4:_(s32) = G_SUB %3, %1, debug-location !DILocation(line: 5, column: 1, scope: !6)
+ DBG_VALUE %4(s32), $noreg, !9, !DIExpression(), debug-location !DILocation(line: 5, column: 1, scope: !6)
+```
+
+By default, `mir-debugify` inserts `DBG_VALUE` instructions **everywhere**
+it is legal to do so. In particular, every (non-PHI) machine instruction that
+defines a register must be followed by a `DBG_VALUE` use of that def. If
an instruction does not define a register, but can be followed by a debug inst,
-MIRDebugify inserts a ``DBG_VALUE`` that references a constant. Insertion of
-``DBG_VALUE``'s can be disabled by setting ``-debugify-level=locations``.
+MIRDebugify inserts a `DBG_VALUE` that references a constant. Insertion of
+`DBG_VALUE`'s can be disabled by setting `-debugify-level=locations`.
-To run MIRDebugify once, simply insert ``mir-debugify`` into your ``llc``
+To run MIRDebugify once, simply insert `mir-debugify` into your `llc`
invocation, like:
-.. code-block:: bash
+```bash
+# Before some other pass.
+$ llc -run-pass=mir-debugify,other-pass ...
- # Before some other pass.
- $ llc -run-pass=mir-debugify,other-pass ...
-
- # After some other pass.
- $ llc -run-pass=other-pass,mir-debugify ...
+# After some other pass.
+$ llc -run-pass=other-pass,mir-debugify ...
+```
To run MIRDebugify before each pass in a pipeline, use
-``-debugify-and-strip-all-safe``. This can be combined with ``-start-before``
-and ``-start-after``. For example:
-
-.. code-block:: bash
+`-debugify-and-strip-all-safe`. This can be combined with `-start-before`
+and `-start-after`. For example:
- $ llc -debugify-and-strip-all-safe -run-pass=... <other llc args>
- $ llc -debugify-and-strip-all-safe -O1 <other llc args>
+```bash
+$ llc -debugify-and-strip-all-safe -run-pass=... <other llc args>
+$ llc -debugify-and-strip-all-safe -O1 <other llc args>
+```
If you want to check it after each pass in a pipeline, use
-``-debugify-check-and-strip-all-safe``. This can also be combined with
-``-start-before`` and ``-start-after``. For example:
+`-debugify-check-and-strip-all-safe`. This can also be combined with
+`-start-before` and `-start-after`. For example:
-.. code-block:: bash
+```bash
+$ llc -debugify-check-and-strip-all-safe -run-pass=... <other llc args>
+$ llc -debugify-check-and-strip-all-safe -O1 <other llc args>
+```
- $ llc -debugify-check-and-strip-all-safe -run-pass=... <other llc args>
- $ llc -debugify-check-and-strip-all-safe -O1 <other llc args>
+To check all debug info from a test, use `mir-check-debugify`, like:
-To check all debug info from a test, use ``mir-check-debugify``, like:
+```bash
+$ llc -run-pass=mir-debugify,other-pass,mir-check-debugify
+```
-.. code-block:: bash
+To strip out all debug info from a test, use `mir-strip-debug`, like:
- $ llc -run-pass=mir-debugify,other-pass,mir-check-debugify
+```bash
+$ llc -run-pass=mir-debugify,other-pass,mir-strip-debug
+```
-To strip out all debug info from a test, use ``mir-strip-debug``, like:
-
-.. code-block:: bash
-
- $ llc -run-pass=mir-debugify,other-pass,mir-strip-debug
-
-It can be useful to combine ``mir-debugify``, ``mir-check-debugify`` and/or
-``mir-strip-debug`` to identify backend transformations which break in
+It can be useful to combine `mir-debugify`, `mir-check-debugify` and/or
+`mir-strip-debug` to identify backend transformations which break in
the presence of debug info. For example, to run the AArch64 backend tests
with all normal passes "sandwiched" in between MIRDebugify and
MIRStripDebugify mutation passes, run:
-.. code-block:: bash
-
- $ llvm-lit test/CodeGen/AArch64 -Dllc="llc -debugify-and-strip-all-safe"
+```bash
+$ llvm-lit test/CodeGen/AArch64 -Dllc="llc -debugify-and-strip-all-safe"
+```
-Using LostDebugLocObserver
---------------------------
+### Using LostDebugLocObserver
TODO
diff --git a/llvm/docs/Instrumentor.md b/llvm/docs/Instrumentor.md
index 7c86437882ab5..4e4b7877c1470 100644
--- a/llvm/docs/Instrumentor.md
+++ b/llvm/docs/Instrumentor.md
@@ -1,12 +1,10 @@
-==================================
-The LLVM Instrumentor Pass
-==================================
+# The LLVM Instrumentor Pass
-.. contents::
- :local:
+```{contents}
+:local: true
+```
-Introduction
-============
+## Introduction
The **Instrumentor** is a highly configurable instrumentation pass for LLVM-IR
that allows users to insert custom runtime function calls at various program
@@ -33,30 +31,27 @@ To use the Instrumentor it is recommended to run the wizard script located at
create a configuration file and a stub runtime which is required to be linked
into the instrumented program.
-Key Features
-============
+## Key Features
-Configurable Instrumentation Opportunities
--------------------------------------------
+### Configurable Instrumentation Opportunities
The Instrumentor supports instrumentation at multiple levels:
**Instruction-level:**
- - **Load instructions**: Instrument memory reads with access to pointer, loaded value, alignment, size, atomicity, etc.
+: - **Load instructions**: Instrument memory reads with access to pointer, loaded value, alignment, size, atomicity, etc.
- **Store instructions**: Instrument memory writes with access to pointer, stored value, alignment, size, atomicity, etc.
- **Alloca instructions**: Instrument stack allocations with access to size, alignment, and allocated address
**Function-level:**
- - **Function entry**: Instrument at function start with access to function name, address, arguments, etc.
+: - **Function entry**: Instrument at function start with access to function name, address, arguments, etc.
- **Function exit**: Instrument at function return
**Future extensions:**
- - Basic block entry/exit
+: - Basic block entry/exit
- Module-level initialization
- Global variable access
-PRE and POST Instrumentation
------------------------------
+### PRE and POST Instrumentation
Each instrumentation opportunity supports two positions:
@@ -74,8 +69,7 @@ Each instrumentation opportunity supports two positions:
- For allocas: can inspect/modify the allocated address
- For functions: instrument at function exit
-Selective Argument Passing
----------------------------
+### Selective Argument Passing
For each instrumentation opportunity, users can individually enable/disable specific arguments to control:
@@ -96,8 +90,7 @@ For example, for load instrumentation, you can choose to pass:
- Volatility flag
- Unique instrumentation ID
-Value Replacement
------------------
+### Value Replacement
The Instrumentor supports **replacing** values returned from the runtime:
@@ -113,18 +106,16 @@ This enables use cases like:
- Fault injection
- Taint tracking
-Instrumentation Filtering
--------------------------
+### Instrumentation Filtering
The Instrumentor provides fine-grained control over what gets instrumented:
-- **Target regex**: Match against the target triple (e.g., ``x86_64-.*-linux``)
+- **Target regex**: Match against the target triple (e.g., `x86_64-.*-linux`)
- **Host/GPU toggle**: Separately enable/disable CPU and GPU instrumentation
- **Function filtering**: Exclude runtime functions from instrumentation via a regular expression
- **Property filters**: Filter individual instrumentation points based on static properties (see Property Filtering below)
-Property Filtering
-------------------
+### Property Filtering
The Instrumentor supports fine-grained filtering of individual instrumentation
opportunities based on their static properties. This allows you to instrument
@@ -135,33 +126,32 @@ only specific operations that meet certain criteria, such as:
- Functions with specific name patterns
- Allocations above a certain size threshold
-Property filters are specified using the ``filter`` field in the configuration JSON for each instrumentation opportunity.
+Property filters are specified using the `filter` field in the configuration JSON for each instrumentation opportunity.
-Filter Syntax
-^^^^^^^^^^^^^
+#### Filter Syntax
The filter expression language supports:
**Integer comparisons:**
- - ``==`` (equal)
- - ``!=`` (not equal)
- - ``<`` (less than)
- - ``>`` (greater than)
- - ``<=`` (less than or equal)
- - ``>=`` (greater than or equal)
+: - `==` (equal)
+ - `!=` (not equal)
+ - `<` (less than)
+ - `>` (greater than)
+ - `<=` (less than or equal)
+ - `>=` (greater than or equal)
**String comparisons:**
- - ``==`` (equal, with quoted string)
- - ``!=`` (not equal, with quoted string)
- - ``.startswith(\"prefix\")`` (prefix check, with quoted string)
+: - `==` (equal, with quoted string)
+ - `!=` (not equal, with quoted string)
+ - `.startswith("prefix")` (prefix check, with quoted string)
**Pointer comparisons:**
- - ``==null`` (null pointer check)
- - ``!=null`` (non-null pointer check)
+: - `==null` (null pointer check)
+ - `!=null` (non-null pointer check)
**Logical operators:**
- - ``&&`` (logical AND)
- - ``||`` (logical OR)
+: - `&&` (logical AND)
+ - `||` (logical OR)
**Important notes:**
@@ -170,94 +160,91 @@ The filter expression language supports:
- String literals must be enclosed in double quotes (with proper escaping)
- Property names are specific to each instrumentation opportunity (see Available Properties below)
-Filter Examples
-^^^^^^^^^^^^^^^
+#### Filter Examples
**Filter only atomic loads:**
-.. code-block:: json
-
- {
- "instruction_post": {
- "load": {
- "enabled": true,
- "filter": "atomicity_ordering>0",
- "pointer": true,
- "value": true
- }
- }
- }
+```json
+{
+ "instruction_post": {
+ "load": {
+ "enabled": true,
+ "filter": "atomicity_ordering>0",
+ "pointer": true,
+ "value": true
+ }
+ }
+}
+```
**Filter volatile stores or acquire and release operations:**
-.. code-block:: json
-
- {
- "instruction_post": {
- "store": {
- "enabled": true,
- "filter": "is_volatile==1 || atomicity_ordering==6",
- "pointer": true,
- "value": true
- }
- }
- }
+```json
+{
+ "instruction_post": {
+ "store": {
+ "enabled": true,
+ "filter": "is_volatile==1 || atomicity_ordering==6",
+ "pointer": true,
+ "value": true
+ }
+ }
+}
+```
**Filter functions by name prefix:**
-.. code-block:: json
-
- {
- "function_pre": {
- "function": {
- "enabled": true,
- "filter": "name.startswith(\"test_\")",
- "name": true
- }
- }
- }
+```json
+{
+ "function_pre": {
+ "function": {
+ "enabled": true,
+ "filter": "name.startswith(\"test_\")",
+ "name": true
+ }
+ }
+}
+```
**Complex filter with multiple conditions:**
-.. code-block:: json
-
- {
- "instruction_post": {
- "load": {
- "enabled": true,
- "filter": "(atomicity_ordering==4 || atomicity_ordering==7) && sync_scope_id==0",
- "pointer": true,
- "value": true,
- "atomicity_ordering": true
- }
- }
- }
-
-Available Properties
-^^^^^^^^^^^^^^^^^^^^
+```json
+{
+ "instruction_post": {
+ "load": {
+ "enabled": true,
+ "filter": "(atomicity_ordering==4 || atomicity_ordering==7) && sync_scope_id==0",
+ "pointer": true,
+ "value": true,
+ "atomicity_ordering": true
+ }
+ }
+}
+```
+
+#### Available Properties
The properties available for filtering depend on the instrumentation
opportunity type but generally include all values that can be passed to the
runtime, filtered using their respective name.
**Load/Store instructions:**
- - ``atomicity_ordering`` (integer): 0=non-atomic, 1=Unordered, 2=Monotonic, 4=Acquire, 5=Release, 6=AcquireRelease, 7=SequentiallyConsistent
- - ``sync_scope_id`` (integer): synchronization scope identifier
- - ``is_volatile`` (integer): 1 if volatile, 0 otherwise
- - ``alignment`` (integer): alignment in bytes
- - ``value_size`` (integer): size of loaded value in bytes
+: - `atomicity_ordering` (integer): 0=non-atomic, 1=Unordered, 2=Monotonic, 4=Acquire, 5=Release, 6=AcquireRelease, 7=SequentiallyConsistent
+ - `sync_scope_id` (integer): synchronization scope identifier
+ - `is_volatile` (integer): 1 if volatile, 0 otherwise
+ - `alignment` (integer): alignment in bytes
+ - `value_size` (integer): size of loaded value in bytes
**Function instrumentation:**
- - ``name`` (string): function name
- - ``num_arguments`` (integer): number of function arguments
- - ``is_main`` (integer): 1 if this is the main function, 0 otherwise
+: - `name` (string): function name
+ - `num_arguments` (integer): number of function arguments
+ - `is_main` (integer): 1 if this is the main function, 0 otherwise
**Alloca instructions:**
- - ``size`` (integer): allocation size in bytes (if constant)
- - ``alignment`` (integer): allocation alignment in bytes
+: - `size` (integer): allocation size in bytes (if constant)
+ - `alignment` (integer): allocation alignment in bytes
-Configuration System
-====================
+## Configuration System
The Instrumentor uses a JSON-based configuration system that allows users to:
@@ -266,115 +253,115 @@ The Instrumentor uses a JSON-based configuration system that allows users to:
3. Load and modify existing configurations
4. Generate runtime stub implementations
-Configuration File Format
--------------------------
+### Configuration File Format
The configuration file is a JSON document with the following structure:
-.. code-block:: json
-
- {
- "configuration": {
- "runtime_prefix": "__instrumentor_",
- "target_regex": "",
- "host_enabled": true,
- "gpu_enabled": true
- },
- "function_pre": {
- "function": {
- "enabled": true,
- "address": true,
- "name": true,
- "id": true
- }
- },
- "instruction_pre": {
- "load": {
- "enabled": true,
- "pointer": true,
- "pointer.replace": false,
- "value_size": true,
- "id": true
- },
- "store": {
- "enabled": true,
- "pointer": true,
- "value": true,
- "value_size": true
- }
- },
- "instruction_post": {
- "load": {
- "enabled": true,
- "value": true,
- "value.replace": false
- }
- }
- }
-
-Configuration Sections
-----------------------
+```json
+{
+ "configuration": {
+ "runtime_prefix": "__instrumentor_",
+ "target_regex": "",
+ "host_enabled": true,
+ "gpu_enabled": true
+ },
+ "function_pre": {
+ "function": {
+ "enabled": true,
+ "address": true,
+ "name": true,
+ "id": true
+ }
+ },
+ "instruction_pre": {
+ "load": {
+ "enabled": true,
+ "pointer": true,
+ "pointer.replace": false,
+ "value_size": true,
+ "id": true
+ },
+ "store": {
+ "enabled": true,
+ "pointer": true,
+ "value": true,
+ "value_size": true
+ }
+ },
+ "instruction_post": {
+ "load": {
+ "enabled": true,
+ "value": true,
+ "value.replace": false
+ }
+ }
+}
+```
+
+### Configuration Sections
**configuration**
- Global settings that apply to all instrumentation:
- - ``runtime_prefix``: Prefix for all runtime function names (default: ``__instrumentor_``)
- - ``target_regex``: Regular expression to filter targets (empty = all targets)
- - ``host_enabled``: Enable instrumentation for CPU targets (default: true)
- - ``gpu_enabled``: Enable instrumentation for GPU targets (default: true)
+: Global settings that apply to all instrumentation:
+
+ - `runtime_prefix`: Prefix for all runtime function names (default: `__instrumentor_`)
+ - `target_regex`: Regular expression to filter targets (empty = all targets)
+ - `host_enabled`: Enable instrumentation for CPU targets (default: true)
+ - `gpu_enabled`: Enable instrumentation for GPU targets (default: true)
**function_pre / function_post**
- Function-level instrumentation configuration.
+
+: Function-level instrumentation configuration.
**instruction_pre / instruction_post**
- Instruction-level instrumentation configuration, with subsections for each instruction type (``load``, ``store``, ``alloca``, etc.).
-Argument Configuration
-----------------------
+: Instruction-level instrumentation configuration, with subsections for each instruction type (`load`, `store`, `alloca`, etc.).
+
+### Argument Configuration
For each instrumentation opportunity, arguments are configured with:
- **enabled**: Boolean to enable/disable the entire opportunity
- **filter**: Optional string expression to filter instrumentation based on static properties (see Property Filtering)
-- **<argument_name>**: Boolean to enable/disable passing this argument
-- **<argument_name>.replace**: Boolean to enable value replacement (only for replaceable arguments)
-- **<argument_name>.description**: Human-readable description of the argument
+- **`<argument_name>`**: Boolean to enable/disable passing this argument
+- **`<argument_name>.replace`**: Boolean to enable value replacement (only for replaceable arguments)
+- **`<argument_name>.description`**: Human-readable description of the argument
-The Configuration Wizard
-=========================
+## The Configuration Wizard
The Instrumentor includes an interactive configuration wizard that simplifies the process of creating and modifying configurations.
-Running the Wizard
-------------------
-
-.. code-block:: bash
+### Running the Wizard
- # Run the wizard interactively
- ./llvm/utils/instrumentor-config-wizard.py
+```bash
+# Run the wizard interactively
+./llvm/utils/instrumentor-config-wizard.py
- # Specify output location
- ./llvm/utils/instrumentor-config-wizard.py -o my_config.json
+# Specify output location
+./llvm/utils/instrumentor-config-wizard.py -o my_config.json
- # Use specific opt binary
- ./llvm/utils/instrumentor-config-wizard.py --opt-path /path/to/opt
+# Use specific opt binary
+./llvm/utils/instrumentor-config-wizard.py --opt-path /path/to/opt
- # Load and modify existing configuration
- ./llvm/utils/instrumentor-config-wizard.py --input existing.json -o modified.json
+# Load and modify existing configuration
+./llvm/utils/instrumentor-config-wizard.py --input existing.json -o modified.json
+```
-Wizard Workflow
----------------
+### Wizard Workflow
The wizard guides you through five steps:
**Step 1: Select Instrumentation Types**
- Choose which types of operations to instrument (load, store, alloca, function, etc.). This is a high-level selection - you can configure individual arguments later.
+
+: Choose which types of operations to instrument (load, store, alloca, function, etc.). This is a high-level selection - you can configure individual arguments later.
**Step 2: PRE vs POST Configuration**
- Decide whether PRE and POST instrumentation should use the same configuration or
diff erent configurations. This saves time when you want both positions to have identical settings.
+
+: Decide whether PRE and POST instrumentation should use the same configuration or
diff erent configurations. This saves time when you want both positions to have identical settings.
**Step 3: Base Configuration**
- Configure global settings:
+
+: Configure global settings:
- Runtime prefix for function names
- Target regex for filtering
@@ -382,7 +369,8 @@ The wizard guides you through five steps:
- Enable/disable GPU instrumentation
**Step 4: Configure Arguments**
- For each enabled instrumentation type, select which arguments to pass to the runtime function. You can:
+
+: For each enabled instrumentation type, select which arguments to pass to the runtime function. You can:
- Toggle individual arguments on/off
- Enable value replacement for replaceable arguments
@@ -390,15 +378,15 @@ The wizard guides you through five steps:
- Configure PRE and POST separately (if selected in Step 2)
**Step 5: Review and Save**
- Review your configuration and optionally generate runtime stub implementations. The wizard displays a summary and provides commands for using the configuration with ``opt`` and ``clang``.
-Generating Runtime Stubs
--------------------------
+: Review your configuration and optionally generate runtime stub implementations. The wizard displays a summary and provides commands for using the configuration with `opt` and `clang`.
+
+### Generating Runtime Stubs
The wizard can automatically generate C stub implementations of your runtime functions:
1. In Step 5, select 'g' to generate stubs
-2. Specify the output file path (default: ``<config_name>_stubs.c``)
+2. Specify the output file path (default: `<config_name>_stubs.c`)
3. The wizard creates a C file with stub implementations that print their arguments
The generated stubs are useful as:
@@ -409,352 +397,339 @@ The generated stubs are useful as:
Example stub output:
-.. code-block:: c
-
- void __instrumentor_pre_load(void *pointer, int32_t pointer_as,
- uint64_t value_size, int32_t id) {
- printf("load pre -- pointer: %p, pointer_as: %i, "
- "value_size: %lu, id: %i\n",
- pointer, pointer_as, value_size, id);
- }
+```c
+void __instrumentor_pre_load(void *pointer, int32_t pointer_as,
+ uint64_t value_size, int32_t id) {
+ printf("load pre -- pointer: %p, pointer_as: %i, "
+ "value_size: %lu, id: %i\n",
+ pointer, pointer_as, value_size, id);
+}
+```
-Usage Examples
-==============
+## Usage Examples
-Basic Usage with opt
---------------------
+### Basic Usage with opt
**Step 1: (Optional) Generate a default configuration**
-.. code-block:: bash
+```bash
+opt -passes=instrumentor \
+ -instrumentor-write-config-file=config.json \
+ -disable-output \
+ input.ll
+```
- opt -passes=instrumentor \
- -instrumentor-write-config-file=config.json \
- -disable-output \
- input.ll
-
-This creates ``config.json`` with all available instrumentation opportunities and their arguments.
+This creates `config.json` with all available instrumentation opportunities and their arguments.
**Step 2: Customize the configuration**
-Edit ``config.json`` manually or use the wizard (no input needed):
-
-.. code-block:: bash
+Edit `config.json` manually or use the wizard (no input needed):
- ./llvm/utils/instrumentor-config-wizard.py --input config.json -o custom.json
+```bash
+./llvm/utils/instrumentor-config-wizard.py --input config.json -o custom.json
+```
**Step 3: Apply instrumentation**
-.. code-block:: bash
-
- opt -passes=instrumentor \
- -instrumentor-read-config-file=custom.json \
- input.ll -S -o instrumented.ll
+```bash
+opt -passes=instrumentor \
+ -instrumentor-read-config-file=custom.json \
+ input.ll -S -o instrumented.ll
+```
The instrumented output contains calls to your runtime functions at the configured program points.
-Using with Clang
-----------------
+### Using with Clang
To instrument during compilation:
-.. code-block:: bash
-
- clang -mllvm -enable-instrumentor \
- -mllvm -instrumentor-read-config-file=config.json \
- source.c -o program
+```bash
+clang -mllvm -enable-instrumentor \
+ -mllvm -instrumentor-read-config-file=config.json \
+ source.c -o program
+```
-Complete Workflow Example
---------------------------
+### Complete Workflow Example
Here's a complete example for creating a simple memory access profiler:
**1. Create configuration with the wizard:**
-.. code-block:: bash
-
- ./llvm/utils/instrumentor-config-wizard.py -o memory_profiler.json
+```bash
+./llvm/utils/instrumentor-config-wizard.py -o memory_profiler.json
- # In the wizard:
- # - Enable: load, store
- # - Use same config for PRE/POST: yes
- # - Base config: keep defaults
- # - For load/store: enable pointer, value_size, id
- # - Generate stubs: yes (memory_profiler_stubs.c)
+# In the wizard:
+# - Enable: load, store
+# - Use same config for PRE/POST: yes
+# - Base config: keep defaults
+# - For load/store: enable pointer, value_size, id
+# - Generate stubs: yes (memory_profiler_stubs.c)
+```
**2. Implement the runtime:**
-.. code-block:: c
-
- // memory_runtime.c
- #include <stdio.h>
- #include <stdint.h>
-
- static uint64_t load_count = 0;
- static uint64_t store_count = 0;
-
- void __instrumentor_pre_load(void *pointer, uint64_t value_size,
- int32_t id) {
- load_count++;
- printf("Load from %p (size: %lu, id: %d)\n",
- pointer, value_size, id);
- }
-
- void __instrumentor_pre_store(void *pointer, uint64_t value_size,
- int32_t id) {
- store_count++;
- printf("Store to %p (size: %lu, id: %d)\n",
- pointer, value_size, id);
- }
-
- __attribute__((destructor))
- void print_stats(void) {
- printf("Total loads: %lu\n", load_count);
- printf("Total stores: %lu\n", store_count);
- }
+```c
+// memory_runtime.c
+#include <stdio.h>
+#include <stdint.h>
+
+static uint64_t load_count = 0;
+static uint64_t store_count = 0;
+
+void __instrumentor_pre_load(void *pointer, uint64_t value_size,
+ int32_t id) {
+ load_count++;
+ printf("Load from %p (size: %lu, id: %d)\n",
+ pointer, value_size, id);
+}
+
+void __instrumentor_pre_store(void *pointer, uint64_t value_size,
+ int32_t id) {
+ store_count++;
+ printf("Store to %p (size: %lu, id: %d)\n",
+ pointer, value_size, id);
+}
+
+__attribute__((destructor))
+void print_stats(void) {
+ printf("Total loads: %lu\n", load_count);
+ printf("Total stores: %lu\n", store_count);
+}
+```
**3. Instrument and compile:**
-.. code-block:: bash
-
- # Instrument the program
- clang -emit-llvm -S -o program.ll program.c
- opt -passes=instrumentor \
- -instrumentor-read-config-file=memory_profiler.json \
- program.ll -S -o program_inst.ll
+```bash
+# Instrument the program
+clang -emit-llvm -S -o program.ll program.c
+opt -passes=instrumentor \
+ -instrumentor-read-config-file=memory_profiler.json \
+ program.ll -S -o program_inst.ll
- # Compile with runtime
- clang program_inst.ll memory_runtime.c -o program
+# Compile with runtime
+clang program_inst.ll memory_runtime.c -o program
+```
**4. Run and observe:**
-.. code-block:: bash
-
- ./program
- # Output includes:
- # Load from 0x7ffc12345678 (size: 4, id: 1)
- # Store to 0x7ffc12345680 (size: 8, id: 2)
- # ...
- # Total loads: 42
- # Total stores: 27
+```bash
+./program
+# Output includes:
+# Load from 0x7ffc12345678 (size: 4, id: 1)
+# Store to 0x7ffc12345680 (size: 8, id: 2)
+# ...
+# Total loads: 42
+# Total stores: 27
+```
-Advanced Use Cases
-==================
+## Advanced Use Cases
-Stack Usage Profiling
-----------------------
+### Stack Usage Profiling
Configure alloca instrumentation to track stack allocations:
-.. code-block:: json
-
- {
- "instruction_pre": {
- "alloca": {
- "enabled": true,
- "size": true,
- "alignment": true,
- "id": true
- }
- },
- "instruction_post": {
- "alloca": {
- "enabled": true,
- "address": true,
- "size": true
- }
- }
- }
+```json
+{
+ "instruction_pre": {
+ "alloca": {
+ "enabled": true,
+ "size": true,
+ "alignment": true,
+ "id": true
+ }
+ },
+ "instruction_post": {
+ "alloca": {
+ "enabled": true,
+ "address": true,
+ "size": true
+ }
+ }
+}
+```
Runtime implementation:
-.. code-block:: c
+```c
+static uint64_t total_stack_usage = 0;
+static uint64_t peak_stack_usage = 0;
+static uint64_t current_stack_usage = 0;
- static uint64_t total_stack_usage = 0;
- static uint64_t peak_stack_usage = 0;
- static uint64_t current_stack_usage = 0;
-
- void __instrumentor_post_alloca(void *address, uint64_t size,
- int32_t id) {
- current_stack_usage += size;
- total_stack_usage += size;
- if (current_stack_usage > peak_stack_usage) {
- peak_stack_usage = current_stack_usage;
- }
- }
+void __instrumentor_post_alloca(void *address, uint64_t size,
+ int32_t id) {
+ current_stack_usage += size;
+ total_stack_usage += size;
+ if (current_stack_usage > peak_stack_usage) {
+ peak_stack_usage = current_stack_usage;
+ }
+}
+```
-Value Replacement for Fault Injection
---------------------------------------
+### Value Replacement for Fault Injection
Use value replacement to inject faults:
-.. code-block:: json
-
- {
- "instruction_post": {
- "load": {
- "enabled": true,
- "value": true,
- "value.replace": true,
- "pointer": true
- }
- }
- }
+```json
+{
+ "instruction_post": {
+ "load": {
+ "enabled": true,
+ "value": true,
+ "value.replace": true,
+ "pointer": true
+ }
+ }
+}
+```
Runtime implementation:
-.. code-block:: c
-
- // Replace every 1000th loaded value with zero
- static uint64_t load_counter = 0;
+```c
+// Replace every 1000th loaded value with zero
+static uint64_t load_counter = 0;
- uint64_t __instrumentor_post_load(uint64_t value, void *pointer) {
- if (++load_counter % 1000 == 0) {
- printf("Injecting fault at %p\n", pointer);
- return 0; // Return fault value
- }
- return value; // Return original value
- }
+uint64_t __instrumentor_post_load(uint64_t value, void *pointer) {
+ if (++load_counter % 1000 == 0) {
+ printf("Injecting fault at %p\n", pointer);
+ return 0; // Return fault value
+ }
+ return value; // Return original value
+}
+```
-Function-Level Tracing
-----------------------
+### Function-Level Tracing
Instrument function entry and exit:
-.. code-block:: json
-
- {
- "function_pre": {
- "function": {
- "enabled": true,
- "name": true,
- "address": true,
- "num_arguments": true
- }
- },
- "function_post": {
- "function": {
- "enabled": true,
- "name": true
- }
- }
- }
+```json
+{
+ "function_pre": {
+ "function": {
+ "enabled": true,
+ "name": true,
+ "address": true,
+ "num_arguments": true
+ }
+ },
+ "function_post": {
+ "function": {
+ "enabled": true,
+ "name": true
+ }
+ }
+}
+```
Runtime implementation:
-.. code-block:: c
+```c
+static int call_depth = 0;
- static int call_depth = 0;
+void __instrumentor_pre_function(char *name, void *address,
+ int32_t num_args, int32_t id) {
+ printf("%*sEntering %s (%p) with %d args\n",
+ call_depth * 2, "", name, address, num_args);
+ call_depth++;
+}
- void __instrumentor_pre_function(char *name, void *address,
- int32_t num_args, int32_t id) {
- printf("%*sEntering %s (%p) with %d args\n",
- call_depth * 2, "", name, address, num_args);
- call_depth++;
- }
+void __instrumentor_post_function(char *name, int32_t id) {
+ call_depth--;
+ printf("%*sExiting %s\n", call_depth * 2, "", name);
+}
+```
- void __instrumentor_post_function(char *name, int32_t id) {
- call_depth--;
- printf("%*sExiting %s\n", call_depth * 2, "", name);
- }
-
-GPU Instrumentation
--------------------
+### GPU Instrumentation
The Instrumentor supports GPU targets (AMDGPU and NVPTX). Configure GPU-specific instrumentation:
-.. code-block:: json
-
- {
- "configuration": {
- "runtime_prefix": "__gpu_runtime_",
- "target_regex": "(amdgcn|nvptx).*",
- "host_enabled": false,
- "gpu_enabled": true
- },
- "instruction_pre": {
- "load": {
- "enabled": true,
- "pointer": true,
- "pointer_as": true
- }
- }
- }
+```json
+{
+ "configuration": {
+ "runtime_prefix": "__gpu_runtime_",
+ "target_regex": "(amdgcn|nvptx).*",
+ "host_enabled": false,
+ "gpu_enabled": true
+ },
+ "instruction_pre": {
+ "load": {
+ "enabled": true,
+ "pointer": true,
+ "pointer_as": true
+ }
+ }
+}
+```
Note that GPU runtime functions must be implemented with appropriate device attributes.
-Implementation Details
-======================
+## Implementation Details
-Generated Runtime Function Signatures
---------------------------------------
+### Generated Runtime Function Signatures
The Instrumentor generates runtime function names following this pattern:
-.. code-block:: text
-
- <runtime_prefix><position>_<opportunity_name>[_ind]
+```text
+<runtime_prefix><position>_<opportunity_name>[_ind]
+```
Where:
-- ``<runtime_prefix>``: Configurable prefix (default: ``__instrumentor_``)
-- ``<position>``: Either ``pre`` or ``post``
-- ``<opportunity_name>``: Name of the instrumentation opportunity (``load``, ``store``, ``function``, etc.)
-- ``_ind``: Optional suffix when indirection is used (see below)
+- `<runtime_prefix>`: Configurable prefix (default: `__instrumentor_`)
+- `<position>`: Either `pre` or `post`
+- `<opportunity_name>`: Name of the instrumentation opportunity (`load`, `store`, `function`, etc.)
+- `_ind`: Optional suffix when indirection is used (see below)
Examples:
-- ``__instrumentor_pre_load``
-- ``__instrumentor_post_store``
-- ``__instrumentor_pre_function``
-- ``__instrumentor_pre_load_ind`` (with indirection)
+- `__instrumentor_pre_load`
+- `__instrumentor_post_store`
+- `__instrumentor_pre_function`
+- `__instrumentor_pre_load_ind` (with indirection)
-Direct vs Indirect Arguments
------------------------------
+### Direct vs Indirect Arguments
The Instrumentor uses two modes for passing arguments:
**Direct mode** (default):
- Arguments are passed by value. This is efficient but requires that all arguments fit in registers or can be passed through the stack efficiently.
+
+: Arguments are passed by value. This is efficient but requires that all arguments fit in registers or can be passed through the stack efficiently.
**Indirect mode**:
- Arguments are passed by pointer. This is used automatically when:
+
+: Arguments are passed by pointer. This is used automatically when:
- Multiple replaceable arguments are enabled (requires indirection for all replaceable args)
- An argument's value is too large (aggregate types, large values)
-When indirect mode is used, a separate function with the ``_ind`` suffix is generated:
-
-.. code-block:: c
+When indirect mode is used, a separate function with the `_ind` suffix is generated:
- // Direct mode
- void __instrumentor_pre_load(void *pointer, uint64_t value_size);
+```c
+// Direct mode
+void __instrumentor_pre_load(void *pointer, uint64_t value_size);
- // Indirect mode (automatically generated when needed)
- void __instrumentor_pre_load_ind(void **pointer, uint32_t pointer_size,
- void *value_size, uint32_t value_size_size);
+// Indirect mode (automatically generated when needed)
+void __instrumentor_pre_load_ind(void **pointer, uint32_t pointer_size,
+ void *value_size, uint32_t value_size_size);
+```
Users typically don't need to worry about this - the Instrumentor handles it automatically and the wizard-generated stubs show the correct signatures.
-Unique IDs
-----------
+### Unique IDs
-When the ``id`` argument is enabled, the Instrumentor assigns a unique 32-bit integer to each instrumentation call site:
+When the `id` argument is enabled, the Instrumentor assigns a unique 32-bit integer to each instrumentation call site:
- PRE positions get positive IDs (1, 2, 3, ...)
- POST positions get negative IDs (-1, -2, -3, ...)
- IDs are consistent across multiple runs
-Caching
--------
+### Caching
The Instrumentor caches certain argument values between PRE and POST calls when possible:
- Values computed in PRE are reused in POST (e.g., pointer value)
- This reduces overhead and ensures consistency
-Runtime Function Requirements
-------------------------------
+### Runtime Function Requirements
Runtime functions must be:
@@ -766,11 +741,9 @@ Runtime functions **must not**:
- Call back into instrumented code (to avoid infinite recursion)
-Performance Considerations
-==========================
+## Performance Considerations
-Overhead Factors
-----------------
+### Overhead Factors
Instrumentation overhead depends on:
@@ -779,20 +752,23 @@ Instrumentation overhead depends on:
3. **Runtime function complexity**: Complex runtime logic increases overhead
4. **Frequency of instrumented operations**: Instrumenting hot loops has high impact
-Optimization Tips
------------------
+### Optimization Tips
**Minimize arguments:**
- Only enable arguments you actually need. Passing fewer arguments reduces overhead.
+
+: Only enable arguments you actually need. Passing fewer arguments reduces overhead.
**Use PRE or POST, not both:**
- If you only need one position, disable the other.
+
+: If you only need one position, disable the other.
**Target filtering:**
- Use ``target_regex`` to instrument only specific targets or modules.
+
+: Use `target_regex` to instrument only specific targets or modules.
**Efficient runtime:**
- Keep runtime functions simple and fast. Consider:
+
+: Keep runtime functions simple and fast. Consider:
- Lock-free data structures
- Thread-local storage
@@ -800,101 +776,107 @@ Optimization Tips
- Sampling (instrument 1 in N calls)
**Build with optimizations:**
- Use ``-O2`` or ``-O3`` when compiling instrumented code. LLVM can optimize away some overhead.
-Troubleshooting
-===============
+: Use `-O2` or `-O3` when compiling instrumented code. LLVM can optimize away some overhead.
-Common Issues
--------------
+## Troubleshooting
+
+### Common Issues
**"Could not find 'opt' binary"**
- The wizard can't locate the opt binary.
- - Specify the path: ``--opt-path /path/to/opt``
+: The wizard can't locate the opt binary.
+
+ - Specify the path: `--opt-path /path/to/opt`
**"Indirection needed but not indicated"**
- An argument value is too large for direct passing. The Instrumentor handles this automatically, but you might see this warning. It's usually harmless - the indirect version of the function will be generated.
+
+: An argument value is too large for direct passing. The Instrumentor handles this automatically, but you might see this warning. It's usually harmless - the indirect version of the function will be generated.
**Infinite recursion / stack overflow**
- Your runtime function is calling back into instrumented code. Solutions:
+
+: Your runtime function is calling back into instrumented code. Solutions:
- Ensure runtime functions don't trigger more instrumentation
**Linking errors**
- Runtime functions are undefined. You must:
+
+: Runtime functions are undefined. You must:
- Implement all enabled runtime functions
- Link the runtime implementation with your program
- Use the exact function signatures (check generated stubs)
**Unexpected instrumentation**
- More instrumentation than expected. Check:
- - The ``enabled`` flag for each opportunity
- - ``host_enabled`` / ``gpu_enabled`` settings
- - ``target_regex`` matches your target
+: More instrumentation than expected. Check:
+
+ - The `enabled` flag for each opportunity
+ - `host_enabled` / `gpu_enabled` settings
+ - `target_regex` matches your target
- Runtime functions aren't being instrumented (they should be automatically excluded)
- - Property filters (``filter`` field) are correctly specified
+ - Property filters (`filter` field) are correctly specified
**Less instrumentation than expected**
- Property filters may be excluding instrumentation points:
- - Check if the ``filter`` field is set for the instrumentation opportunity
+: Property filters may be excluding instrumentation points:
+
+ - Check if the `filter` field is set for the instrumentation opportunity
- Remember that filters only apply to static (compile-time constant) properties
- Dynamic values always pass the filter
- - Use empty filter (``"filter": ""``) or remove the field to disable filtering
+ - Use empty filter (`"filter": ""`) or remove the field to disable filtering
- Test your filter expression by examining the IR properties
**Filter syntax errors**
- Invalid filter expressions will be reported as errors:
- - Ensure string literals are quoted: ``"name==\"foo\""`` not ``"name==foo"``
- - Use correct operators: ``&&`` for AND, ``||`` for OR
+: Invalid filter expressions will be reported as errors:
+
+ - Ensure string literals are quoted: `"name==\"foo\""` not `"name==foo"`
+ - Use correct operators: `&&` for AND, `||` for OR
- Property names must match exactly
- Close all parentheses and quotes
-Debugging Instrumented Code
-----------------------------
+### Debugging Instrumented Code
**View instrumented IR:**
-.. code-block:: bash
-
- opt -passes=instrumentor \
- -instrumentor-read-config-file=config.json \
- input.ll -S -o output.ll
+```bash
+opt -passes=instrumentor \
+ -instrumentor-read-config-file=config.json \
+ input.ll -S -o output.ll
- # Examine output.ll to see inserted calls
+# Examine output.ll to see inserted calls
+```
**Print configuration:**
-.. code-block:: bash
+```bash
+opt -passes=instrumentor \
+ -instrumentor-write-config-file=debug_config.json \
+ input.ll -disable-output
- opt -passes=instrumentor \
- -instrumentor-write-config-file=debug_config.json \
- input.ll -disable-output
-
- # Examine debug_config.json to see all options
+# Examine debug_config.json to see all options
+```
**Verify IR:**
- The Instrumentor automatically verifies the module after instrumentation. If verification fails, there's a bug in the Instrumentor or the configuration is invalid.
+
+: The Instrumentor automatically verifies the module after instrumentation. If verification fails, there's a bug in the Instrumentor or the configuration is invalid.
**Use debug builds:**
- Build LLVM with assertions enabled (``-DLLVM_ENABLE_ASSERTIONS=ON``) to catch issues early.
-Extending the Instrumentor
-===========================
+: Build LLVM with assertions enabled (`-DLLVM_ENABLE_ASSERTIONS=ON`) to catch issues early.
+
+## Extending the Instrumentor
The Instrumentor is designed to be extensible. To add new instrumentation opportunities:
-1. **Define the opportunity class** inheriting from ``InstrumentationOpportunity``
+1. **Define the opportunity class** inheriting from `InstrumentationOpportunity`
2. **Implement getter/setter functions** for the arguments
3. **Add initialization** to populate the opportunity with arguments
-4. **Register** the opportunity in ``InstrumentationConfig::populate()``
-5. **Add tests** in ``llvm/test/Transforms/Instrumentor/``
+4. **Register** the opportunity in `InstrumentationConfig::populate()`
+5. **Add tests** in `llvm/test/Transforms/Instrumentor/`
-See ``llvm/lib/Transforms/IPO/Instrumentor.cpp`` and ``llvm/include/llvm/Transforms/IPO/Instrumentor.h`` for examples (``LoadIO``, ``StoreIO``).
+See `llvm/lib/Transforms/IPO/Instrumentor.cpp` and `llvm/include/llvm/Transforms/IPO/Instrumentor.h` for examples (`LoadIO`, `StoreIO`).
Future instrumentation opportunities being considered:
@@ -906,20 +888,17 @@ Future instrumentation opportunities being considered:
- Exception handling
- Global variable access
-Reference
-=========
+## Reference
-Command-Line Options
---------------------
+### Command-Line Options
-**-instrumentor-read-config-file=<path>**
- Load instrumentation configuration from the specified JSON file.
+**`-instrumentor-read-config-file=<path>`**
+: Load instrumentation configuration from the specified JSON file.
-**-instrumentor-write-config-file=<path>**
- Write the default instrumentation configuration to the specified JSON file (useful for generating templates).
+**`-instrumentor-write-config-file=<path>`**
+: Write the default instrumentation configuration to the specified JSON file (useful for generating templates).
-Related Passes
---------------
+### Related Passes
The Instrumentor is more flexible but related to:
@@ -931,9 +910,8 @@ The Instrumentor is more flexible but related to:
The Instrumentor can implement similar functionality with custom runtime code, but specialized passes may have better performance for their specific use cases.
-Further Reading
----------------
+### Further Reading
-- Source code: ``llvm/lib/Transforms/IPO/Instrumentor.cpp``
-- Header: ``llvm/include/llvm/Transforms/IPO/Instrumentor.h``
-- Configuration wizard: ``llvm/utils/instrumentor-config-wizard.py``
+- Source code: `llvm/lib/Transforms/IPO/Instrumentor.cpp`
+- Header: `llvm/include/llvm/Transforms/IPO/Instrumentor.h`
+- Configuration wizard: `llvm/utils/instrumentor-config-wizard.py`
diff --git a/llvm/docs/JITLink.md b/llvm/docs/JITLink.md
index f3059b12984a8..403398c958d1b 100644
--- a/llvm/docs/JITLink.md
+++ b/llvm/docs/JITLink.md
@@ -1,25 +1,24 @@
-====================================
-JITLink and ORC's ObjectLinkingLayer
-====================================
+# JITLink and ORC's ObjectLinkingLayer
-.. contents::
- :local:
+```{contents}
+:local: true
+```
-Introduction
-============
+## Introduction
This document aims to provide a high-level overview of the design and API
of the JITLink library. It assumes some familiarity with linking and
relocatable object files, but should not require deep expertise. If you know
what a section, symbol, and relocation are then you should find this document
-accessible. If it is not, please submit a patch (:doc:`Contributing`) or file a
-bug (:doc:`HowToSubmitABug`).
-
-JITLink is a library for :ref:`jit_linking`. It was built to support the :doc:`ORC JIT
-APIs<ORCv2>` and is most commonly accessed via ORC's ObjectLinkingLayer API. JITLink was
-developed with the aim of supporting the full set of features provided by each
-object format; including static initializers, exception handling, thread local
-variables, and language runtime registration. Supporting these features enables
+accessible. If it is not, please submit a patch ({doc}`Contributing`) or file a
+bug ({doc}`HowToSubmitABug`).
+
+JITLink is a library for {ref}`jit_linking`. It was built to support the
+{doc}`ORC JIT APIs <ORCv2>` and is most commonly accessed via ORC's
+ObjectLinkingLayer API. JITLink was developed with the aim of supporting the
+full set of features provided by each object format; including static
+initializers, exception handling, thread local variables, and language runtime
+registration. Supporting these features enables
ORC to execute code generated from source languages which rely on these features
(e.g. C++ requires object format support for static initializers to support
static constructors, eh-frame registration for exceptions, and TLV support for
@@ -33,155 +32,150 @@ development:
1. Cross-process and cross-architecture linking of single relocatable objects
into a target *executor* process.
-
2. Support for all object format features.
+3. Open linker data structures (`LinkGraph`) and pass system.
-3. Open linker data structures (``LinkGraph``) and pass system.
-
-JITLink and ObjectLinkingLayer
-==============================
+## JITLink and ObjectLinkingLayer
-``ObjectLinkingLayer`` is ORCs wrapper for JITLink. It is an ORC layer that
-allows objects to be added to a ``JITDylib``, or emitted from some higher level
-program representation. When an object is emitted, ``ObjectLinkingLayer`` uses
-JITLink to construct a ``LinkGraph`` (see :ref:`constructing_linkgraphs`) and
-calls JITLink's ``link`` function to link the graph into the executor process.
+`ObjectLinkingLayer` is ORCs wrapper for JITLink. It is an ORC layer that
+allows objects to be added to a `JITDylib`, or emitted from some higher level
+program representation. When an object is emitted, `ObjectLinkingLayer` uses
+JITLink to construct a `LinkGraph` (see {ref}`constructing_linkgraphs`) and
+calls JITLink's `link` function to link the graph into the executor process.
-The ``ObjectLinkingLayer`` class provides a plugin API,
-``ObjectLinkingLayer::Plugin``, which users can subclass in order to inspect and
-modify ``LinkGraph`` instances at link time, and react to important JIT events
+The `ObjectLinkingLayer` class provides a plugin API,
+`ObjectLinkingLayer::Plugin`, which users can subclass in order to inspect and
+modify `LinkGraph` instances at link time, and react to important JIT events
(such as an object being emitted into target memory). This enables many features
and optimizations that were not possible under MCJIT or RuntimeDyld.
-ObjectLinkingLayer Plugins
---------------------------
+### ObjectLinkingLayer Plugins
-The ``ObjectLinkingLayer::Plugin`` class provides the following methods:
+The `ObjectLinkingLayer::Plugin` class provides the following methods:
-* ``modifyPassConfig`` is called each time a LinkGraph is about to be linked. It
+- `modifyPassConfig` is called each time a LinkGraph is about to be linked. It
can be overridden to install JITLink *Passes* to run during the link process.
- .. code-block:: c++
-
- void modifyPassConfig(MaterializationResponsibility &MR,
- jitlink::LinkGraph &G,
- jitlink::PassConfiguration &Config)
+ ```c++
+ void modifyPassConfig(MaterializationResponsibility &MR,
+ jitlink::LinkGraph &G,
+ jitlink::PassConfiguration &Config)
+ ```
-* ``notifyLoaded`` is called before the link begins, and can be overridden to
- set up any initial state for the given ``MaterializationResponsibility`` if
+- `notifyLoaded` is called before the link begins, and can be overridden to
+ set up any initial state for the given `MaterializationResponsibility` if
needed.
- .. code-block:: c++
+ ```c++
+ void notifyLoaded(MaterializationResponsibility &MR)
+ ```
- void notifyLoaded(MaterializationResponsibility &MR)
-
-* ``notifyEmitted`` is called after the link is complete and code has been
+- `notifyEmitted` is called after the link is complete and code has been
emitted to the executor process. It can be overridden to finalize state
- for the ``MaterializationResponsibility`` if needed.
-
- .. code-block:: c++
+ for the `MaterializationResponsibility` if needed.
- Error notifyEmitted(MaterializationResponsibility &MR)
+ ```c++
+ Error notifyEmitted(MaterializationResponsibility &MR)
+ ```
-* ``notifyFailed`` is called if the link fails at any point. It can be
+- `notifyFailed` is called if the link fails at any point. It can be
overridden to react to the failure (e.g. to deallocate any already allocated
resources).
- .. code-block:: c++
-
- Error notifyFailed(MaterializationResponsibility &MR)
+ ```c++
+ Error notifyFailed(MaterializationResponsibility &MR)
+ ```
-* ``notifyRemovingResources`` is called when a request is made to remove any
- resources associated with the ``ResourceKey`` *K* for the
- ``MaterializationResponsibility``.
+- `notifyRemovingResources` is called when a request is made to remove any
+ resources associated with the `ResourceKey` *K* for the
+ `MaterializationResponsibility`.
- .. code-block:: c++
+ ```c++
+ Error notifyRemovingResources(JITDylib &JD, ResourceKey K)
+ ```
- Error notifyRemovingResources(JITDylib &JD, ResourceKey K)
-
-* ``notifyTransferringResources`` is called if/when a request is made to
- transfer tracking of any resources associated with ``ResourceKey``
+- `notifyTransferringResources` is called if/when a request is made to
+ transfer tracking of any resources associated with `ResourceKey`
*SrcKey* to *DstKey*.
- .. code-block:: c++
-
- void notifyTransferringResources(JITDylib &JD, ResourceKey DstKey,
- ResourceKey SrcKey)
+ ```c++
+ void notifyTransferringResources(JITDylib &JD, ResourceKey DstKey,
+ ResourceKey SrcKey)
+ ```
-Plugin authors are required to implement the ``notifyFailed``,
-``notifyRemovingResources``, and ``notifyTransferringResources`` methods in
+Plugin authors are required to implement the `notifyFailed`,
+`notifyRemovingResources`, and `notifyTransferringResources` methods in
order to safely manage resources in the case of resource removal or transfer,
or link failure. If no resources are managed by the plugin then these methods
-can be implemented as no-ops returning ``Error::success()``.
-
-Plugin instances are added to an ``ObjectLinkingLayer`` by
-calling the ``addPlugin`` method [1]_. E.g.
-
-.. code-block:: c++
-
- // Plugin class to print the set of defined symbols in an object when that
- // object is linked.
- class MyPlugin : public ObjectLinkingLayer::Plugin {
- public:
-
- // Add passes to print the set of defined symbols after dead-stripping.
- void modifyPassConfig(MaterializationResponsibility &MR,
- jitlink::LinkGraph &G,
- jitlink::PassConfiguration &Config) override {
- Config.PostPrunePasses.push_back([this](jitlink::LinkGraph &G) {
- return printAllSymbols(G);
- });
- }
-
- // Implement mandatory overrides:
- Error notifyFailed(MaterializationResponsibility &MR) override {
- return Error::success();
- }
- Error notifyRemovingResources(JITDylib &JD, ResourceKey K) override {
- return Error::success();
- }
- void notifyTransferringResources(JITDylib &JD, ResourceKey DstKey,
- ResourceKey SrcKey) override {}
-
- // JITLink pass to print all defined symbols in G.
- Error printAllSymbols(LinkGraph &G) {
- for (auto *Sym : G.defined_symbols())
- if (Sym->hasName())
- dbgs() << Sym->getName() << "\n";
- return Error::success();
- }
- };
+can be implemented as no-ops returning `Error::success()`.
+
+Plugin instances are added to an `ObjectLinkingLayer` by
+calling the `addPlugin` method [^1]. E.g.
+
+```c++
+// Plugin class to print the set of defined symbols in an object when that
+// object is linked.
+class MyPlugin : public ObjectLinkingLayer::Plugin {
+public:
+
+ // Add passes to print the set of defined symbols after dead-stripping.
+ void modifyPassConfig(MaterializationResponsibility &MR,
+ jitlink::LinkGraph &G,
+ jitlink::PassConfiguration &Config) override {
+ Config.PostPrunePasses.push_back([this](jitlink::LinkGraph &G) {
+ return printAllSymbols(G);
+ });
+ }
- // Create our LLJIT instance using a custom object linking layer setup.
- // This gives us a chance to install our plugin.
- auto J = ExitOnErr(LLJITBuilder()
- .setObjectLinkingLayerCreator(
- [](ExecutionSession &ES, const Triple &T) {
- // Manually set up the ObjectLinkingLayer for our LLJIT
- // instance.
- auto OLL = std::make_unique<ObjectLinkingLayer>(
- ES, std::make_unique<jitlink::InProcessMemoryManager>());
+ // Implement mandatory overrides:
+ Error notifyFailed(MaterializationResponsibility &MR) override {
+ return Error::success();
+ }
+ Error notifyRemovingResources(JITDylib &JD, ResourceKey K) override {
+ return Error::success();
+ }
+ void notifyTransferringResources(JITDylib &JD, ResourceKey DstKey,
+ ResourceKey SrcKey) override {}
+
+ // JITLink pass to print all defined symbols in G.
+ Error printAllSymbols(LinkGraph &G) {
+ for (auto *Sym : G.defined_symbols())
+ if (Sym->hasName())
+ dbgs() << Sym->getName() << "\n";
+ return Error::success();
+ }
+};
+
+// Create our LLJIT instance using a custom object linking layer setup.
+// This gives us a chance to install our plugin.
+auto J = ExitOnErr(LLJITBuilder()
+ .setObjectLinkingLayerCreator(
+ [](ExecutionSession &ES, const Triple &T) {
+ // Manually set up the ObjectLinkingLayer for our LLJIT
+ // instance.
+ auto OLL = std::make_unique<ObjectLinkingLayer>(
+ ES, std::make_unique<jitlink::InProcessMemoryManager>());
- // Install our plugin:
- OLL->addPlugin(std::make_unique<MyPlugin>());
+ // Install our plugin:
+ OLL->addPlugin(std::make_unique<MyPlugin>());
- return OLL;
- })
- .create());
+ return OLL;
+ })
+ .create());
- // Add an object to the JIT. Nothing happens here: linking isn't triggered
- // until we look up some symbol in our object.
- ExitOnErr(J->addObject(loadFromDisk("main.o")));
+// Add an object to the JIT. Nothing happens here: linking isn't triggered
+// until we look up some symbol in our object.
+ExitOnErr(J->addObject(loadFromDisk("main.o")));
- // Plugin triggers here when our lookup of main triggers linking of main.o
- auto MainSym = J->lookup("main");
+// Plugin triggers here when our lookup of main triggers linking of main.o
+auto MainSym = J->lookup("main");
+```
-LinkGraph
-=========
+## LinkGraph
-JITLink maps all relocatable object formats to a generic ``LinkGraph`` type
-that is designed to make linking fast and easy (``LinkGraph`` instances can
-also be created manually. See :ref:`constructing_linkgraphs`).
+JITLink maps all relocatable object formats to a generic `LinkGraph` type
+that is designed to make linking fast and easy (`LinkGraph` instances can
+also be created manually. See {ref}`constructing_linkgraphs`).
Relocatable object formats (e.g. COFF, ELF, MachO)
diff er in their details,
but share a common goal: to represent machine level code and data with
@@ -192,257 +186,217 @@ or subsections, depending on the format), and annotations describing how to
patch content based on the final address of some target symbol/section
(relocations).
-At a high level, the ``LinkGraph`` type represents these concepts as a decorated
+At a high level, the `LinkGraph` type represents these concepts as a decorated
graph. Nodes in the graph represent symbols and content, and edges represent
relocations. Each of the elements of the graph is listed here:
-* ``Addressable`` -- A node in the link graph that can be assigned an address
+- `Addressable` -- A node in the link graph that can be assigned an address
in the executor process's virtual address space.
- Absolute and external symbols are represented using plain ``Addressable``
+ Absolute and external symbols are represented using plain `Addressable`
instances. Content defined inside the object file is represented using the
- ``Block`` subclass.
+ `Block` subclass.
-* ``Block`` -- An ``Addressable`` node that has ``Content`` (or is marked as
- zero-filled), a parent ``Section``, a ``Size``, an ``Alignment`` (and an
- ``AlignmentOffset``), and a list of ``Edge`` instances.
+- `Block` -- An `Addressable` node that has `Content` (or is marked as
+ zero-filled), a parent `Section`, a `Size`, an `Alignment` (and an
+ `AlignmentOffset`), and a list of `Edge` instances.
Blocks provide a container for binary content which must remain contiguous in
the target address space (a *layout unit*). Many interesting low level
- operations on ``LinkGraph`` instances involve inspecting or mutating block
+ operations on `LinkGraph` instances involve inspecting or mutating block
content or edges.
- * ``Content`` is represented as an ``llvm::StringRef``, and accessible via
- the ``getContent`` method. Content is only available for content blocks,
- and not for zero-fill blocks (use ``isZeroFill`` to check, and prefer
- ``getSize`` when only the block size is needed as it works for both
+ - `Content` is represented as an `llvm::StringRef`, and accessible via
+ the `getContent` method. Content is only available for content blocks,
+ and not for zero-fill blocks (use `isZeroFill` to check, and prefer
+ `getSize` when only the block size is needed as it works for both
zero-fill and content blocks).
-
- * ``Section`` is represented as a ``Section&`` reference, and accessible via
- the ``getSection`` method. The ``Section`` class is described in more detail
+ - `Section` is represented as a `Section&` reference, and accessible via
+ the `getSection` method. The `Section` class is described in more detail
below.
-
- * ``Size`` is represented as a ``size_t``, and is accessible via the
- ``getSize`` method for both content and zero-filled blocks.
-
- * ``Alignment`` is represented as a ``uint64_t``, and available via the
- ``getAlignment`` method. It represents the minimum alignment requirement (in
+ - `Size` is represented as a `size_t`, and is accessible via the
+ `getSize` method for both content and zero-filled blocks.
+ - `Alignment` is represented as a `uint64_t`, and available via the
+ `getAlignment` method. It represents the minimum alignment requirement (in
bytes) of the start of the block.
-
- * ``AlignmentOffset`` is represented as a ``uint64_t``, and accessible via the
- ``getAlignmentOffset`` method. It represents the offset from the alignment
+ - `AlignmentOffset` is represented as a `uint64_t`, and accessible via the
+ `getAlignmentOffset` method. It represents the offset from the alignment
required for the start of the block. This is required to support blocks
whose minimum alignment requirement comes from data at some non-zero offset
inside the block. E.g. if a block consists of a single byte (with byte
alignment) followed by a uint64_t (with 8-byte alignment), then the block
will have 8-byte alignment with an alignment offset of 7.
+ - list of `Edge` instances. An iterator range for this list is returned by
+ the `edges` method. The `Edge` class is described in more detail below.
- * list of ``Edge`` instances. An iterator range for this list is returned by
- the ``edges`` method. The ``Edge`` class is described in more detail below.
-
-* ``Symbol`` -- An offset from an ``Addressable`` (often a ``Block``), with an
- optional ``Name``, a ``Linkage``, a ``Scope``, a ``Callable`` flag, and a
- ``Live`` flag.
+- `Symbol` -- An offset from an `Addressable` (often a `Block`), with an
+ optional `Name`, a `Linkage`, a `Scope`, a `Callable` flag, and a
+ `Live` flag.
Symbols make it possible to name content (blocks and addressables are
- anonymous), or target content with an ``Edge``.
-
- * ``Name`` is represented as an ``llvm::StringRef`` (equal to
- ``llvm::StringRef()`` if the symbol has no name), and accessible via the
- ``getName`` method.
+ anonymous), or target content with an `Edge`.
- * ``Linkage`` is one of *Strong* or *Weak*, and is accessible via the
- ``getLinkage`` method. The ``JITLinkContext`` can use this flag to determine
+ - `Name` is represented as an `llvm::StringRef` (equal to
+ `llvm::StringRef()` if the symbol has no name), and accessible via the
+ `getName` method.
+ - `Linkage` is one of *Strong* or *Weak*, and is accessible via the
+ `getLinkage` method. The `JITLinkContext` can use this flag to determine
whether this symbol definition should be kept or dropped.
-
- * ``Scope`` is one of *Default*, *Hidden*, or *Local*, and is accessible via
- the ``getScope`` method. The ``JITLinkContext`` can use this to determine
+ - `Scope` is one of *Default*, *Hidden*, or *Local*, and is accessible via
+ the `getScope` method. The `JITLinkContext` can use this to determine
who should be able to see the symbol. A symbol with default scope should be
globally visible. A symbol with hidden scope should be visible to other
- definitions within the same simulated dylib (e.g. ORC ``JITDylib``) or
+ definitions within the same simulated dylib (e.g. ORC `JITDylib`) or
executable, but not from elsewhere. A symbol with local scope should only be
- visible within the current ``LinkGraph``.
-
- * ``Callable`` is a boolean which is set to true if this symbol can be called,
- and is accessible via the ``isCallable`` method. This can be used to
+ visible within the current `LinkGraph`.
+ - `Callable` is a boolean which is set to true if this symbol can be called,
+ and is accessible via the `isCallable` method. This can be used to
automate the introduction of call-stubs for lazy compilation.
-
- * ``Live`` is a boolean that can be set to mark this symbol as root for
- dead-stripping purposes (see :ref:`generic_link_algorithm`). JITLink's
+ - `Live` is a boolean that can be set to mark this symbol as root for
+ dead-stripping purposes (see {ref}`generic_link_algorithm`). JITLink's
dead-stripping algorithm will propagate liveness flags through the graph to
all reachable symbols before deleting any symbols (and blocks) that are not
marked live.
-* ``Edge`` -- A quad of an ``Offset`` (implicitly from the start of the
- containing ``Block``), a ``Kind`` (describing the relocation type), a
- ``Target``, and an ``Addend``.
+- `Edge` -- A quad of an `Offset` (implicitly from the start of the
+ containing `Block`), a `Kind` (describing the relocation type), a
+ `Target`, and an `Addend`.
Edges represent relocations, and occasionally other relationships, between
blocks and symbols.
- * ``Offset``, accessible via ``getOffset``, is an offset from the start of the
- ``Block`` containing the ``Edge``.
-
- * ``Kind``, accessible via ``getKind`` is a relocation type -- it describes
+ - `Offset`, accessible via `getOffset`, is an offset from the start of the
+ `Block` containing the `Edge`.
+ - `Kind`, accessible via `getKind` is a relocation type -- it describes
what kinds of changes (if any) should be made to block content at the given
- ``Offset`` based on the address of the ``Target``.
-
- * ``Target``, accessible via ``getTarget``, is a pointer to a ``Symbol``,
+ `Offset` based on the address of the `Target`.
+ - `Target`, accessible via `getTarget`, is a pointer to a `Symbol`,
representing whose address is relevant to the fixup calculation specified by
- the edge's ``Kind``.
-
- * ``Addend``, accessible via ``getAddend``, is a constant whose interpretation
- is determined by the edge's ``Kind``.
+ the edge's `Kind`.
+ - `Addend`, accessible via `getAddend`, is a constant whose interpretation
+ is determined by the edge's `Kind`.
-* ``Section`` -- A set of ``Symbol`` instances, plus a set of ``Block``
- instances, with a ``Name``, a set of ``ProtectionFlags``, and an ``Ordinal``.
+- `Section` -- A set of `Symbol` instances, plus a set of `Block`
+ instances, with a `Name`, a set of `ProtectionFlags`, and an `Ordinal`.
Sections make it easy to iterate over the symbols or blocks associated with
a particular section in the source object file.
- * ``blocks()`` returns an iterator over the set of blocks defined in the
- section (as ``Block*`` pointers).
-
- * ``symbols()`` returns an iterator over the set of symbols defined in the
- section (as ``Symbol*`` pointers).
-
- * ``Name`` is represented as an ``llvm::StringRef``, and is accessible via the
- ``getName`` method.
-
- * ``ProtectionFlags`` are represented as a sys::Memory::ProtectionFlags enum,
- and accessible via the ``getProtectionFlags`` method. These flags describe
+ - `blocks()` returns an iterator over the set of blocks defined in the
+ section (as `Block*` pointers).
+ - `symbols()` returns an iterator over the set of symbols defined in the
+ section (as `Symbol*` pointers).
+ - `Name` is represented as an `llvm::StringRef`, and is accessible via the
+ `getName` method.
+ - `ProtectionFlags` are represented as a sys::Memory::ProtectionFlags enum,
+ and accessible via the `getProtectionFlags` method. These flags describe
whether the section is readable, writable, executable, or some combination
- of these. The most common combinations are ``RW-`` for writable data,
- ``R--`` for constant data, and ``R-X`` for code.
-
- * ``SectionOrdinal``, accessible via ``getOrdinal``, is a number used to order
- the section relative to others. It is usually used to preserve section
+ of these. The most common combinations are `RW-` for writable data,
+ `R--` for constant data, and `R-X` for code.
+ - `SectionOrdinal`, accessible via `getOrdinal`, is a number used to order
+ the section relative to others. It is usually used to preserve section
order within a segment (a set of sections with the same memory protections)
when laying out memory.
-For the graph-theorists: The ``LinkGraph`` is bipartite, with one set of
-``Symbol`` nodes and one set of ``Addressable`` nodes. Each ``Symbol`` node has
-one (implicit) edge to its target ``Addressable``. Each ``Block`` has a set of
-edges (possibly empty, represented as ``Edge`` instances) back to elements of
-the ``Symbol`` set. For convenience and performance of common algorithms,
-symbols and blocks are further grouped into ``Sections``.
+For the graph-theorists: The `LinkGraph` is bipartite, with one set of
+`Symbol` nodes and one set of `Addressable` nodes. Each `Symbol` node has
+one (implicit) edge to its target `Addressable`. Each `Block` has a set of
+edges (possibly empty, represented as `Edge` instances) back to elements of
+the `Symbol` set. For convenience and performance of common algorithms,
+symbols and blocks are further grouped into `Sections`.
-The ``LinkGraph`` itself provides operations for constructing, removing, and
+The `LinkGraph` itself provides operations for constructing, removing, and
iterating over sections, symbols, and blocks. It also provides metadata
and utilities relevant to the linking process:
-* Graph element operations
-
- * ``sections`` returns an iterator over all sections in the graph.
+- Graph element operations
- * ``findSectionByName`` returns a pointer to the section with the given
- name (as a ``Section*``) if it exists, otherwise returns a nullptr.
-
- * ``blocks`` returns an iterator over all blocks in the graph (across all
+ - `sections` returns an iterator over all sections in the graph.
+ - `findSectionByName` returns a pointer to the section with the given
+ name (as a `Section*`) if it exists, otherwise returns a nullptr.
+ - `blocks` returns an iterator over all blocks in the graph (across all
sections).
-
- * ``defined_symbols`` returns an iterator over all defined symbols in the
+ - `defined_symbols` returns an iterator over all defined symbols in the
graph (across all sections).
-
- * ``external_symbols`` returns an iterator over all external symbols in the
+ - `external_symbols` returns an iterator over all external symbols in the
graph.
-
- * ``absolute_symbols`` returns an iterator over all absolute symbols in the
+ - `absolute_symbols` returns an iterator over all absolute symbols in the
graph.
-
- * ``createSection`` creates a section with a given name and protection flags.
-
- * ``createContentBlock`` creates a block with the given initial content,
+ - `createSection` creates a section with a given name and protection flags.
+ - `createContentBlock` creates a block with the given initial content,
parent section, address, alignment, and alignment offset.
-
- * ``createZeroFillBlock`` creates a zero-fill block with the given size,
+ - `createZeroFillBlock` creates a zero-fill block with the given size,
parent section, address, alignment, and alignment offset.
-
- * ``addExternalSymbol`` creates a new addressable and symbol with a given
+ - `addExternalSymbol` creates a new addressable and symbol with a given
name, size, and linkage.
-
- * ``addAbsoluteSymbol`` creates a new addressable and symbol with a given
+ - `addAbsoluteSymbol` creates a new addressable and symbol with a given
name, address, size, linkage, scope, and liveness.
-
- * ``addCommonSymbol`` convenience function for creating a zero-filled block
+ - `addCommonSymbol` convenience function for creating a zero-filled block
and weak symbol with a given name, scope, section, initial address, size,
alignment and liveness.
-
- * ``addAnonymousSymbol`` creates a new anonymous symbol for a given block,
+ - `addAnonymousSymbol` creates a new anonymous symbol for a given block,
offset, size, callable-ness, and liveness.
-
- * ``addDefinedSymbol`` creates a new symbol for a given block with a name,
+ - `addDefinedSymbol` creates a new symbol for a given block with a name,
offset, size, linkage, scope, callable-ness and liveness.
-
- * ``makeExternal`` transforms a formerly defined symbol into an external one
+ - `makeExternal` transforms a formerly defined symbol into an external one
by creating a new addressable and pointing the symbol at it. The existing
block is not deleted, but can be manually removed (if unreferenced) by
- calling ``removeBlock``. All edges to the symbol remain valid, but the
- symbol must now be defined outside this ``LinkGraph``.
-
- * ``removeExternalSymbol`` removes an external symbol and its target
+ calling `removeBlock`. All edges to the symbol remain valid, but the
+ symbol must now be defined outside this `LinkGraph`.
+ - `removeExternalSymbol` removes an external symbol and its target
addressable. The target addressable must not be referenced by any other
symbols.
-
- * ``removeAbsoluteSymbol`` removes an absolute symbol and its target
+ - `removeAbsoluteSymbol` removes an absolute symbol and its target
addressable. The target addressable must not be referenced by any other
symbols.
-
- * ``removeDefinedSymbol`` removes a defined symbol, but *does not* remove
+ - `removeDefinedSymbol` removes a defined symbol, but *does not* remove
its target block.
-
- * ``removeBlock`` removes the given block.
-
- * ``splitBlock`` split a given block in two at a given index (useful where
+ - `removeBlock` removes the given block.
+ - `splitBlock` split a given block in two at a given index (useful where
it is known that a block contains decomposable records, e.g. CFI records
in an eh-frame section).
-* Graph utility operations
+- Graph utility operations
- * ``getName`` returns the name of this graph, which is usually based on the
+ - `getName` returns the name of this graph, which is usually based on the
name of the input object file.
-
- * ``getTargetTriple`` returns an `llvm::Triple` for the executor process.
-
- * ``getPointerSize`` returns the size of a pointer (in bytes) in the executor
+ - `getTargetTriple` returns an `llvm::Triple` for the executor process.
+ - `getPointerSize` returns the size of a pointer (in bytes) in the executor
process.
-
- * ``getEndianness`` returns the endianness of the executor process.
-
- * ``allocateString`` copies data from a given ``llvm::Twine`` into the
+ - `getEndianness` returns the endianness of the executor process.
+ - `allocateString` copies data from a given `llvm::Twine` into the
link graph's internal allocator. This can be used to ensure that content
created inside a pass outlives that pass's execution.
-.. _generic_link_algorithm:
+(generic_link_algorithm)=
-Generic Link Algorithm
-======================
+## Generic Link Algorithm
JITLink provides a generic link algorithm which can be extended / modified at
-certain points by the introduction of JITLink :ref:`passes`.
+certain points by the introduction of JITLink {ref}`passes`.
At the end of each phase the linker packages its state into a *continuation*
-and calls the ``JITLinkContext`` object to perform a (potentially high-latency)
+and calls the `JITLinkContext` object to perform a (potentially high-latency)
asynchronous operation: allocating memory, resolving external symbols, and
finally transferring linked memory to the executing process.
-#. Phase 1
+1. Phase 1
- This phase is called immediately by the ``link`` function as soon as the
+ This phase is called immediately by the `link` function as soon as the
initial configuration (including the pass pipeline setup) is complete.
- #. Run pre-prune passes.
+ 1. Run pre-prune passes.
These passes are called on the graph before it is pruned. At this stage
- ``LinkGraph`` nodes still have their original vmaddrs. A mark-live pass
- (supplied by the ``JITLinkContext``) will be run at the end of this
+ `LinkGraph` nodes still have their original vmaddrs. A mark-live pass
+ (supplied by the `JITLinkContext`) will be run at the end of this
sequence to mark the initial set of live symbols.
Notable use cases: marking nodes live, accessing/copying graph data that
will be pruned (e.g. metadata that's important for the JIT, but not needed
for the link process).
- #. Prune (dead-strip) the ``LinkGraph``.
+ 2. Prune (dead-strip) the `LinkGraph`.
Removes all symbols and blocks not reachable from the initial set of live
symbols.
@@ -450,7 +404,7 @@ finally transferring linked memory to the executing process.
This allows JITLink to remove unreachable symbols / content, including
overridden weak and redundant ODR definitions.
- #. Run post-prune passes.
+ 3. Run post-prune passes.
These passes are run on the graph after dead-stripping, but before memory
is allocated or nodes assigned their final target vmaddrs.
@@ -462,54 +416,54 @@ finally transferring linked memory to the executing process.
Notable use cases: Building Global Offset Table (GOT), Procedure Linkage
Table (PLT), and Thread Local Variable (TLV) entries.
- #. Asynchronously allocate memory.
+ 4. Asynchronously allocate memory.
- Calls the ``JITLinkContext``'s ``JITLinkMemoryManager`` to allocate both
+ Calls the `JITLinkContext`'s `JITLinkMemoryManager` to allocate both
working and target memory for the graph. As part of this process the
- ``JITLinkMemoryManager`` will update the addresses of all nodes
+ `JITLinkMemoryManager` will update the addresses of all nodes
defined in the graph to their assigned target address.
Note: This step only updates the addresses of nodes defined in this graph.
External symbols will still have null addresses.
-#. Phase 2
+2. Phase 2
- #. Run post-allocation passes.
+ 1. Run post-allocation passes.
These passes are run on the graph after working and target memory have
- been allocated, but before the ``JITLinkContext`` is notified of the
+ been allocated, but before the `JITLinkContext` is notified of the
final addresses of the symbols in the graph. This gives these passes a
chance to set up data structures associated with target addresses before
any JITLink clients (especially ORC queries for symbol resolution) can
attempt to access them.
Notable use cases: Setting up mappings between target addresses and
- JIT data structures, such as a mapping between ``__dso_handle`` and
- ``JITDylib*``.
+ JIT data structures, such as a mapping between `__dso_handle` and
+ `JITDylib*`.
- #. Notify the ``JITLinkContext`` of the assigned symbol addresses.
+ 2. Notify the `JITLinkContext` of the assigned symbol addresses.
- Calls ``JITLinkContext::notifyResolved`` on the link graph, allowing
+ Calls `JITLinkContext::notifyResolved` on the link graph, allowing
clients to react to the symbol address assignments made for this graph.
In ORC this is used to notify any pending queries for *resolved* symbols,
including pending queries from concurrently running JITLink instances that
have reached the next step and are waiting on the address of a symbol in
this graph to proceed with their link.
- #. Identify external symbols and resolve their addresses asynchronously.
+ 3. Identify external symbols and resolve their addresses asynchronously.
- Calls the ``JITLinkContext`` to resolve the target address of any external
+ Calls the `JITLinkContext` to resolve the target address of any external
symbols in the graph.
-#. Phase 3
+3. Phase 3
- #. Apply external symbol resolution results.
+ 1. Apply external symbol resolution results.
This updates the addresses of all external symbols. At this point all
nodes in the graph have their final target addresses, however node
content still points back to the original data in the object file.
- #. Run pre-fixup passes.
+ 2. Run pre-fixup passes.
These passes are called on the graph after all nodes have been assigned
their final target addresses, but before node content is copied into
@@ -520,13 +474,13 @@ finally transferring linked memory to the executing process.
bypassed for fixup targets that are directly accessible under the assigned
memory layout.
- #. Copy block content to working memory and apply fixups.
+ 3. Copy block content to working memory and apply fixups.
Copies all block content into allocated working memory (following the
target layout) and applies fixups. Graph blocks are updated to point at
the fixed up content.
- #. Run post-fixup passes.
+ 4. Run post-fixup passes.
These passes are called on the graph after fixups have been applied and
blocks updated to point to the fixed up content.
@@ -534,43 +488,42 @@ finally transferring linked memory to the executing process.
Post-fixup passes can inspect blocks contents to see the exact bytes that
will be copied to the assigned target addresses.
- #. Finalize memory asynchronously.
+ 5. Finalize memory asynchronously.
- Calls the ``JITLinkMemoryManager`` to copy working memory to the executor
+ Calls the `JITLinkMemoryManager` to copy working memory to the executor
process and apply the requested permissions.
-#. Phase 3.
+4. Phase 3.
- #. Notify the context that the graph has been emitted.
+ 1. Notify the context that the graph has been emitted.
- Calls ``JITLinkContext::notifyFinalized`` and hands off the
- ``JITLinkMemoryManager::FinalizedAlloc`` object for this graph's memory
+ Calls `JITLinkContext::notifyFinalized` and hands off the
+ `JITLinkMemoryManager::FinalizedAlloc` object for this graph's memory
allocation. This allows the context to track/hold memory allocations and
react to the newly emitted definitions. In ORC this is used to update the
- ``ExecutionSession`` instance's dependence graph, which may result in
+ `ExecutionSession` instance's dependence graph, which may result in
these symbols (and possibly others) becoming *Ready* if all of their
dependencies have also been emitted.
-.. _passes:
+(passes)=
-Passes
-------
+### Passes
-JITLink passes are ``std::function<Error(LinkGraph&)>`` instances. They are free
-to inspect and modify the given ``LinkGraph`` subject to the constraints of
-whatever phase they are running in (see :ref:`generic_link_algorithm`). If a
-pass returns ``Error::success()`` then linking continues. If a pass returns
-a failure value then linking is stopped and the ``JITLinkContext`` is notified
+JITLink passes are `std::function<Error(LinkGraph&)>` instances. They are free
+to inspect and modify the given `LinkGraph` subject to the constraints of
+whatever phase they are running in (see {ref}`generic_link_algorithm`). If a
+pass returns `Error::success()` then linking continues. If a pass returns
+a failure value then linking is stopped and the `JITLinkContext` is notified
that the link failed.
Passes may be used by both JITLink backends (e.g. MachO/x86-64 implements GOT
and PLT construction as a pass), and external clients like
-``ObjectLinkingLayer::Plugin``.
+`ObjectLinkingLayer::Plugin`.
-In combination with the open ``LinkGraph`` API, JITLink passes enable the
+In combination with the open `LinkGraph` API, JITLink passes enable the
implementation of powerful new features. For example:
-* Relaxation optimizations -- A pre-fixup pass can inspect GOT accesses and PLT
+- Relaxation optimizations -- A pre-fixup pass can inspect GOT accesses and PLT
calls and identify situations where the addresses of the entry target and the
access are close enough to be accessed directly. In this case the pass can
rewrite the instruction stream of the containing block and update the fixup
@@ -578,66 +531,65 @@ implementation of powerful new features. For example:
Code for this looks like:
-.. code-block:: c++
-
- Error relaxGOTEdges(LinkGraph &G) {
- for (auto *B : G.blocks())
- for (auto &E : B->edges())
- if (E.getKind() == x86_64::GOTLoad) {
- auto &GOTTarget = getGOTEntryTarget(E.getTarget());
- if (isInRange(B.getFixupAddress(E), GOTTarget)) {
- // Rewrite B.getContent() at fixup address from
- // MOVQ to LEAQ
-
- // Update edge target and kind.
- E.setTarget(GOTTarget);
- E.setKind(x86_64::PCRel32);
- }
+```c++
+Error relaxGOTEdges(LinkGraph &G) {
+ for (auto *B : G.blocks())
+ for (auto &E : B->edges())
+ if (E.getKind() == x86_64::GOTLoad) {
+ auto &GOTTarget = getGOTEntryTarget(E.getTarget());
+ if (isInRange(B.getFixupAddress(E), GOTTarget)) {
+ // Rewrite B.getContent() at fixup address from
+ // MOVQ to LEAQ
+
+ // Update edge target and kind.
+ E.setTarget(GOTTarget);
+ E.setKind(x86_64::PCRel32);
}
+ }
- return Error::success();
- }
+ return Error::success();
+}
+```
-* Metadata registration -- Post allocation passes can be used to record the
+- Metadata registration -- Post allocation passes can be used to record the
address range of sections in the target. This can be used to register the
metadata (e.g exception handling frames, language metadata) in the target
once memory has been finalized.
-.. code-block:: c++
-
- Error registerEHFrameSection(LinkGraph &G) {
- if (auto *Sec = G.findSectionByName("__eh_frame")) {
- SectionRange SR(*Sec);
- registerEHFrameSection(SR.getStart(), SR.getEnd());
- }
-
- return Error::success();
+```c++
+Error registerEHFrameSection(LinkGraph &G) {
+ if (auto *Sec = G.findSectionByName("__eh_frame")) {
+ SectionRange SR(*Sec);
+ registerEHFrameSection(SR.getStart(), SR.getEnd());
}
-* Record call sites for later mutation -- A post-allocation pass can record
+ return Error::success();
+}
+```
+
+- Record call sites for later mutation -- A post-allocation pass can record
the call sites of all calls to a particular function, allowing those call
sites to be updated later at runtime (e.g. for instrumentation, or to
enable the function to be lazily compiled but still called directly after
compilation).
-.. code-block:: c++
-
- StringRef FunctionName = "foo";
- std::vector<ExecutorAddr> CallSitesForFunction;
+```c++
+StringRef FunctionName = "foo";
+std::vector<ExecutorAddr> CallSitesForFunction;
- auto RecordCallSites =
- [&](LinkGraph &G) -> Error {
- for (auto *B : G.blocks())
- for (auto &E : B.edges())
- if (E.getKind() == CallEdgeKind &&
- E.getTarget().hasName() &&
- E.getTraget().getName() == FunctionName)
- CallSitesForFunction.push_back(B.getFixupAddress(E));
- return Error::success();
- };
+auto RecordCallSites =
+ [&](LinkGraph &G) -> Error {
+ for (auto *B : G.blocks())
+ for (auto &E : B.edges())
+ if (E.getKind() == CallEdgeKind &&
+ E.getTarget().hasName() &&
+ E.getTraget().getName() == FunctionName)
+ CallSitesForFunction.push_back(B.getFixupAddress(E));
+ return Error::success();
+ };
+```
-Memory Management with JITLinkMemoryManager
--------------------------------------------
+### Memory Management with JITLinkMemoryManager
JIT linking requires allocation of two kinds of memory: working memory in the
JIT process and target memory in the execution process (these processes and
@@ -652,88 +604,87 @@ since sharing and protection assignment can often be efficiently managed (in
the common case of running across processes on the same machine for security)
via the host operating system's virtual memory management APIs.
-To satisfy these requirements ``JITLinkMemoryManager`` adopts the following
+To satisfy these requirements `JITLinkMemoryManager` adopts the following
design: The memory manager itself has just two virtual methods for asynchronous
operations (each with convenience overloads for calling synchronously):
-.. code-block:: c++
-
- /// Called when allocation has been completed.
- using OnAllocatedFunction =
- unique_function<void(Expected<std::unique_ptr<InFlightAlloc>)>;
+```c++
+/// Called when allocation has been completed.
+using OnAllocatedFunction =
+ unique_function<void(Expected<std::unique_ptr<InFlightAlloc>)>;
- /// Called when deallocation has completed.
- using OnDeallocatedFunction = unique_function<void(Error)>;
+/// Called when deallocation has completed.
+using OnDeallocatedFunction = unique_function<void(Error)>;
- /// Call to allocate memory.
- virtual void allocate(const JITLinkDylib *JD, LinkGraph &G,
- OnAllocatedFunction OnAllocated) = 0;
+/// Call to allocate memory.
+virtual void allocate(const JITLinkDylib *JD, LinkGraph &G,
+ OnAllocatedFunction OnAllocated) = 0;
- /// Call to deallocate memory.
- virtual void deallocate(std::vector<FinalizedAlloc> Allocs,
- OnDeallocatedFunction OnDeallocated) = 0;
+/// Call to deallocate memory.
+virtual void deallocate(std::vector<FinalizedAlloc> Allocs,
+ OnDeallocatedFunction OnDeallocated) = 0;
+```
-The ``allocate`` method takes a ``JITLinkDylib*`` representing the target
-simulated dylib, a reference to the ``LinkGraph`` that must be allocated for,
-and a callback to run once an ``InFlightAlloc`` has been constructed.
-``JITLinkMemoryManager`` implementations can (optionally) use the ``JD``
+The `allocate` method takes a `JITLinkDylib*` representing the target
+simulated dylib, a reference to the `LinkGraph` that must be allocated for,
+and a callback to run once an `InFlightAlloc` has been constructed.
+`JITLinkMemoryManager` implementations can (optionally) use the `JD`
argument to manage a per-simulated-dylib memory pool (since code model
constraints are typically imposed on a per-dylib basis, and not across
-dylibs) [2]_. The ``LinkGraph`` describes the object file that we need to
+dylibs) [^2]. The `LinkGraph` describes the object file that we need to
allocate memory for. The allocator must allocate working memory for all of
the Blocks defined in the graph, assign address space for each Block within the
executing processes memory, and update the Blocks' addresses to reflect this
assignment. Block content should be copied to working memory, but does not need
to be transferred to executor memory yet (that will be done once the content is
-fixed up). ``JITLinkMemoryManager`` implementations can take full
-responsibility for these steps, or use the ``BasicLayout`` utility to reduce
+fixed up). `JITLinkMemoryManager` implementations can take full
+responsibility for these steps, or use the `BasicLayout` utility to reduce
the task to allocating working and executor memory for *segments*: chunks of
memory defined by permissions, alignments, content sizes, and zero-fill sizes.
Once the allocation step is complete the memory manager should construct an
-``InFlightAlloc`` object to represent the allocation, and then pass this object
-to the ``OnAllocated`` callback.
-
-The ``InFlightAlloc`` object has two virtual methods:
+`InFlightAlloc` object to represent the allocation, and then pass this object
+to the `OnAllocated` callback.
-.. code-block:: c++
+The `InFlightAlloc` object has two virtual methods:
- using OnFinalizedFunction = unique_function<void(Expected<FinalizedAlloc>)>;
- using OnAbandonedFunction = unique_function<void(Error)>;
+```c++
+using OnFinalizedFunction = unique_function<void(Expected<FinalizedAlloc>)>;
+using OnAbandonedFunction = unique_function<void(Error)>;
- /// Called prior to finalization if the allocation should be abandoned.
- virtual void abandon(OnAbandonedFunction OnAbandoned) = 0;
+/// Called prior to finalization if the allocation should be abandoned.
+virtual void abandon(OnAbandonedFunction OnAbandoned) = 0;
- /// Called to transfer working memory to the target and apply finalization.
- virtual void finalize(OnFinalizedFunction OnFinalized) = 0;
+/// Called to transfer working memory to the target and apply finalization.
+virtual void finalize(OnFinalizedFunction OnFinalized) = 0;
+```
-The linking process will call the ``finalize`` method on the ``InFlightAlloc``
+The linking process will call the `finalize` method on the `InFlightAlloc`
object if linking succeeds up to the finalization step, otherwise it will call
-``abandon`` to indicate that some error occurred during linking. A call to the
-``InFlightAlloc::finalize`` method should cause content for the allocation to be
+`abandon` to indicate that some error occurred during linking. A call to the
+`InFlightAlloc::finalize` method should cause content for the allocation to be
transferred from working to executor memory, and permissions to be run. A call
-to ``abandon`` should result in both kinds of memory being deallocated.
+to `abandon` should result in both kinds of memory being deallocated.
-On successful finalization, the ``InFlightAlloc::finalize`` method should
-construct a ``FinalizedAlloc`` object (an opaque uint64_t id that the
-``JITLinkMemoryManager`` can use to identify executor memory for deallocation)
-and pass it to the ``OnFinalized`` callback.
+On successful finalization, the `InFlightAlloc::finalize` method should
+construct a `FinalizedAlloc` object (an opaque uint64_t id that the
+`JITLinkMemoryManager` can use to identify executor memory for deallocation)
+and pass it to the `OnFinalized` callback.
-Finalized allocations (represented by ``FinalizedAlloc`` objects) can be
-deallocated by calling the ``JITLinkMemoryManager::dealloc`` method. This method
-takes a vector of ``FinalizedAlloc`` objects, since it is common to deallocate
+Finalized allocations (represented by `FinalizedAlloc` objects) can be
+deallocated by calling the `JITLinkMemoryManager::dealloc` method. This method
+takes a vector of `FinalizedAlloc` objects, since it is common to deallocate
multiple objects at the same time and this allows us to batch these requests for
transmission to the executing process.
JITLink provides a simple in-process implementation of this interface:
-``InProcessMemoryManager``. It allocates pages once and re-uses them as both
+`InProcessMemoryManager`. It allocates pages once and re-uses them as both
working and target memory.
-ORC provides a cross-process-capable ``MapperJITLinkMemoryManager`` that can use
+ORC provides a cross-process-capable `MapperJITLinkMemoryManager` that can use
shared memory or ORC-RPC-based communication to transfer content to the executing
process.
-JITLinkMemoryManager and Security
----------------------------------
+### JITLinkMemoryManager and Security
JITLink's ability to link JIT'd code for a separate executor process can be
used to improve the security of a JIT system: The executor process can be
@@ -752,12 +703,11 @@ as RW- in the JITLink process and R-X in the executor process, allowing
modification from the JITLink process but not from the executor (at the cost of
extra administrative overhead for the dual mapping).
-Error Handling
---------------
+### Error Handling
-JITLink makes extensive use of the ``llvm::Error`` type (see the error handling
-section of :doc:`ProgrammersManual` for details). The link process itself, all
-passes, the memory manager interface, and operations on the ``JITLinkContext``
+JITLink makes extensive use of the `llvm::Error` type (see the error handling
+section of {doc}`ProgrammersManual` for details). The link process itself, all
+passes, the memory manager interface, and operations on the `JITLinkContext`
are all permitted to fail. Link graph construction utilities (especially parsers
for object formats) are encouraged to validate input, and validate fixups
(e.g. with range checks) before application.
@@ -767,10 +717,9 @@ reported failures are propagated to queries pending on definitions provided by
the failing link, and also through edges of the dependence graph to any queries
waiting on dependent symbols.
-.. _connection_to_orc_runtime:
+(connection_to_orc_runtime)=
-Connection to the ORC Runtime
-=============================
+## Connection to the ORC Runtime
The ORC Runtime (currently under development) aims to provide runtime support
for advanced JIT features, including object format features that require
@@ -781,41 +730,36 @@ ORC Runtime support for object format features typically requires cooperation
between the runtime (which executes in the executor process) and JITLink (which
runs in the JIT process and can inspect LinkGraphs to determine what actions
must be taken in the executor). For example: Execution of MachO static
-initializers in the ORC runtime is performed by the ``jit_dlopen`` function,
+initializers in the ORC runtime is performed by the `jit_dlopen` function,
which calls back to the JIT process to ask for the list of address ranges of
-``__mod_init`` sections to walk. This list is collated by the
-``MachOPlatformPlugin``, which installs a pass to record this information for
+`__mod_init` sections to walk. This list is collated by the
+`MachOPlatformPlugin`, which installs a pass to record this information for
each object as it is linked into the target.
-.. _constructing_linkgraphs:
-
-Constructing LinkGraphs
-=======================
-
-Clients usually access and manipulate ``LinkGraph`` instances that were created
-for them by an ``ObjectLinkingLayer`` instance, but they can be created manually:
-
-#. By directly constructing and populating a ``LinkGraph`` instance.
+(constructing_linkgraphs)=
-#. By using the ``createLinkGraph`` family of functions to create a
- ``LinkGraph`` from an in-memory buffer containing an object file. This is how
- ``ObjectLinkingLayer`` usually creates ``LinkGraphs``.
+## Constructing LinkGraphs
- #. ``createLinkGraph_<Object-Format>_<Architecture>`` can be used when
- both the object format and architecture are known ahead of time.
+Clients usually access and manipulate `LinkGraph` instances that were created
+for them by an `ObjectLinkingLayer` instance, but they can be created manually:
- #. ``createLinkGraph_<Object-Format>`` can be used when the object format is
- known ahead of time, but the architecture is not. In this case the
- architecture will be determined by inspection of the object header.
+1. By directly constructing and populating a `LinkGraph` instance.
+2. By using the `createLinkGraph` family of functions to create a
+ `LinkGraph` from an in-memory buffer containing an object file. This is how
+ `ObjectLinkingLayer` usually creates `LinkGraphs`.
- #. ``createLinkGraph`` can be used when neither the object format nor
- the architecture are known ahead of time. In this case the object header
- will be inspected to determine both the format and architecture.
+ 1. `createLinkGraph_<Object-Format>_<Architecture>` can be used when
+ both the object format and architecture are known ahead of time.
+ 2. `createLinkGraph_<Object-Format>` can be used when the object format is
+ known ahead of time, but the architecture is not. In this case the
+ architecture will be determined by inspection of the object header.
+ 3. `createLinkGraph` can be used when neither the object format nor
+ the architecture are known ahead of time. In this case the object header
+ will be inspected to determine both the format and architecture.
-.. _jit_linking:
+(jit_linking)=
-JIT Linking
-===========
+## JIT Linking
The JIT linker concept was introduced in LLVM's earlier generation of JIT APIs,
MCJIT. In MCJIT the *RuntimeDyld* component enabled re-use of LLVM as an
@@ -836,19 +780,16 @@ A *JIT linker* takes a single relocatable object file at a time and links it
into a target process, usually using a context object to allow the linked code
to resolve symbols in the target.
-RuntimeDyld
------------
+### RuntimeDyld
In order to keep RuntimeDyld's implementation simple MCJIT imposed some
restrictions on compiled code:
-#. It had to use the Large code model, and often restricted available relocation
+1. It had to use the Large code model, and often restricted available relocation
models in order to limit the kinds of relocations that had to be supported.
-
-#. It required strong linkage and default visibility on all symbols -- behavior
+2. It required strong linkage and default visibility on all symbols -- behavior
for other linkages/visibilities was not well defined.
-
-#. It constrained and/or prohibited the use of features requiring runtime
+3. It constrained and/or prohibited the use of features requiring runtime
support, e.g. static initializers or thread local storage.
As a result of these restrictions not all language features supported by LLVM
@@ -865,205 +806,197 @@ internal object representations.
Eliminating these restrictions and limitations was one of the primary motivations
for the development of JITLink.
-The llvm-jitlink tool
-=====================
+## The llvm-jitlink tool
-The ``llvm-jitlink`` tool is a command line wrapper for the JITLink library.
+The `llvm-jitlink` tool is a command line wrapper for the JITLink library.
It loads some set of relocatable object files and then links them using
JITLink. Depending on the options used it will then execute them, or validate
the linked memory.
-The ``llvm-jitlink`` tool was originally designed to aid JITLink development by
+The `llvm-jitlink` tool was originally designed to aid JITLink development by
providing a simple environment for testing.
-Basic usage
------------
+### Basic usage
-By default, ``llvm-jitlink`` will link the set of objects passed on the command
+By default, `llvm-jitlink` will link the set of objects passed on the command
line, then search for a "main" function and execute it:
-.. code-block:: sh
+```sh
+% cat hello-world.c
+#include <stdio.h>
- % cat hello-world.c
- #include <stdio.h>
+int main(int argc, char *argv[]) {
+ printf("hello, world!\n");
+ return 0;
+}
- int main(int argc, char *argv[]) {
- printf("hello, world!\n");
- return 0;
- }
-
- % clang -c -o hello-world.o hello-world.c
- % llvm-jitlink hello-world.o
- Hello, World!
+% clang -c -o hello-world.o hello-world.c
+% llvm-jitlink hello-world.o
+Hello, World!
+```
Multiple objects may be specified, and arguments may be provided to the JIT'd
main function using the -args option:
-.. code-block:: sh
-
- % cat print-args.c
- #include <stdio.h>
+```sh
+% cat print-args.c
+#include <stdio.h>
- void print_args(int argc, char *argv[]) {
- for (int i = 0; i != argc; ++i)
- printf("arg %i is \"%s\"\n", i, argv[i]);
- }
+void print_args(int argc, char *argv[]) {
+ for (int i = 0; i != argc; ++i)
+ printf("arg %i is \"%s\"\n", i, argv[i]);
+}
- % cat print-args-main.c
- void print_args(int argc, char *argv[]);
+% cat print-args-main.c
+void print_args(int argc, char *argv[]);
- int main(int argc, char *argv[]) {
- print_args(argc, argv);
- return 0;
- }
+int main(int argc, char *argv[]) {
+ print_args(argc, argv);
+ return 0;
+}
- % clang -c -o print-args.o print-args.c
- % clang -c -o print-args-main.o print-args-main.c
- % llvm-jitlink print-args.o print-args-main.o -args a b c
- arg 0 is "a"
- arg 1 is "b"
- arg 2 is "c"
+% clang -c -o print-args.o print-args.c
+% clang -c -o print-args-main.o print-args-main.c
+% llvm-jitlink print-args.o print-args-main.o -args a b c
+arg 0 is "a"
+arg 1 is "b"
+arg 2 is "c"
+```
-Alternative entry points may be specified using the ``-entry <entry point
-name>`` option.
+Alternative entry points may be specified using the `-entry <entry point
+name>` option.
-Other options can be found by calling ``llvm-jitlink -help``.
+Other options can be found by calling `llvm-jitlink -help`.
-llvm-jitlink as a regression testing utility
---------------------------------------------
+### llvm-jitlink as a regression testing utility
-One of the primary aims of ``llvm-jitlink`` was to enable readable regression
+One of the primary aims of `llvm-jitlink` was to enable readable regression
tests for JITLink. To do this it supports two options:
-The ``-noexec`` option tells llvm-jitlink to stop after looking up the entry
+The `-noexec` option tells llvm-jitlink to stop after looking up the entry
point, and before attempting to execute it. Since the linked code is not
executed, this can be used to link for other targets even if you do not have
-access to the target being linked (the ``-define-abs`` or ``-phony-externals``
+access to the target being linked (the `-define-abs` or `-phony-externals`
options can be used to supply any missing definitions in this case).
-The ``-check <check-file>`` option can be used to run a set of ``jitlink-check``
+The `-check <check-file>` option can be used to run a set of `jitlink-check`
expressions against working memory. It is typically used in conjunction with
-``-noexec``, since the aim is to validate JIT'd memory rather than to run the
-code and ``-noexec`` allows us to link for any supported target architecture
-from the current process. In ``-check`` mode, ``llvm-jitlink`` will scan the
-given check-file for lines of the form ``# jitlink-check: <expr>``. See
-examples of this usage in ``llvm/test/ExecutionEngine/JITLink``.
+`-noexec`, since the aim is to validate JIT'd memory rather than to run the
+code and `-noexec` allows us to link for any supported target architecture
+from the current process. In `-check` mode, `llvm-jitlink` will scan the
+given check-file for lines of the form `# jitlink-check: <expr>`. See
+examples of this usage in `llvm/test/ExecutionEngine/JITLink`.
-Remote execution via llvm-jitlink-executor
-------------------------------------------
+### Remote execution via llvm-jitlink-executor
-By default ``llvm-jitlink`` will link the given objects into its own process,
+By default `llvm-jitlink` will link the given objects into its own process,
but this can be overridden by two options:
-The ``-oop-executor[=/path/to/executor]`` option tells ``llvm-jitlink`` to
-execute the given executor (which defaults to ``llvm-jitlink-executor``) and
+The `-oop-executor[=/path/to/executor]` option tells `llvm-jitlink` to
+execute the given executor (which defaults to `llvm-jitlink-executor`) and
communicate with it via file descriptors which it passes to the executor
-as the first argument with the format ``filedescs=<in-fd>,<out-fd>``.
+as the first argument with the format `filedescs=<in-fd>,<out-fd>`.
-The ``-oop-executor-connect=<host>:<port>`` option tells ``llvm-jitlink`` to
+The `-oop-executor-connect=<host>:<port>` option tells `llvm-jitlink` to
connect to an already running executor via TCP on the given host and port. To
-use this option you will need to start ``llvm-jitlink-executor`` manually with
-``listen=<host>:<port>`` as the first argument.
+use this option you will need to start `llvm-jitlink-executor` manually with
+`listen=<host>:<port>` as the first argument.
-Harness mode
-------------
+### Harness mode
-The ``-harness`` option allows a set of input objects to be designated as a test
+The `-harness` option allows a set of input objects to be designated as a test
harness, with the regular object files implicitly treated as objects to be
tested. Definitions of symbols in the harness set override definitions in the
test set, and external references from the harness cause automatic scope
promotion of local symbols in the test set (these modifications to the usual
-linker rules are accomplished via an ``ObjectLinkingLayer::Plugin`` installed by
-``llvm-jitlink`` when it sees the ``-harness`` option).
+linker rules are accomplished via an `ObjectLinkingLayer::Plugin` installed by
+`llvm-jitlink` when it sees the `-harness` option).
With these modifications in place we can selectively test functions in an object
file by mocking those function's callees. For example, suppose we have an object
-file, ``test_code.o``, compiled from the following C source (which we need not
+file, `test_code.o`, compiled from the following C source (which we need not
have access to):
-.. code-block:: c
-
- void irrelevant_function() { irrelevant_external(); }
+```c
+void irrelevant_function() { irrelevant_external(); }
- int function_to_mock(int X) {
- return /* some function of X */;
- }
+int function_to_mock(int X) {
+ return /* some function of X */;
+}
- static void function_to_test() {
- ...
- int Y = function_to_mock();
- printf("Y is %i\n", Y);
- }
+static void function_to_test() {
+ ...
+ int Y = function_to_mock();
+ printf("Y is %i\n", Y);
+}
+```
-If we want to know how ``function_to_test`` behaves when we change the behavior
-of ``function_to_mock`` we can test it by writing a test harness:
+If we want to know how `function_to_test` behaves when we change the behavior
+of `function_to_mock` we can test it by writing a test harness:
-.. code-block:: c
+```c
+void function_to_test();
- void function_to_test();
+int function_to_mock(int X) {
+ printf("used mock utility function\n");
+ return 42;
+}
- int function_to_mock(int X) {
- printf("used mock utility function\n");
- return 42;
- }
-
- int main(int argc, char *argv[]) {
- function_to_test():
- return 0;
- }
+int main(int argc, char *argv[]) {
+ function_to_test():
+ return 0;
+}
+```
Under normal circumstances these objects could not be linked together:
-``function_to_test`` is static and could not be resolved outside
-``test_code.o``, the two ``function_to_mock`` functions would result in a
-duplicate definition error, and ``irrelevant_external`` is undefined.
-However, using ``-harness`` and ``-phony-externals`` we can run this code
+`function_to_test` is static and could not be resolved outside
+`test_code.o`, the two `function_to_mock` functions would result in a
+duplicate definition error, and `irrelevant_external` is undefined.
+However, using `-harness` and `-phony-externals` we can run this code
with:
-.. code-block:: sh
-
- % clang -c -o test_code_harness.o test_code_harness.c
- % llvm-jitlink -phony-externals test_code.o -harness test_code_harness.o
- used mock utility function
- Y is 42
+```sh
+% clang -c -o test_code_harness.o test_code_harness.c
+% llvm-jitlink -phony-externals test_code.o -harness test_code_harness.o
+used mock utility function
+Y is 42
+```
-The ``-harness`` option may be of interest to people who want to perform some
+The `-harness` option may be of interest to people who want to perform some
very late testing on build products to verify that compiled code behaves as
expected. On basic C test cases this is relatively straightforward. Mocks for
more complicated languages (e.g. C++) are much trickier: Any code involving
classes tends to have a lot of non-trivial surface area (e.g. vtables) that
would require great care to mock.
-Tips for JITLink backend developers
------------------------------------
+### Tips for JITLink backend developers
-#. Make liberal use of assert and ``llvm::Error``. Do *not* assume that the input
+1. Make liberal use of assert and `llvm::Error`. Do *not* assume that the input
object is well formed: Return any errors produced by libObject (or your own
object parsing code) and validate as you construct. Think carefully about the
distinction between contract (which should be validated with asserts and
llvm_unreachable) and environmental errors (which should generate
- ``llvm::Error`` instances).
+ `llvm::Error` instances).
+2. Don't assume you're linking in-process. Use libSupport's sized,
+ endian-specific types when reading/writing content in the `LinkGraph`.
-#. Don't assume you're linking in-process. Use libSupport's sized,
- endian-specific types when reading/writing content in the ``LinkGraph``.
-
-As a "minimum viable" JITLink wrapper, the ``llvm-jitlink`` tool is an
+As a "minimum viable" JITLink wrapper, the `llvm-jitlink` tool is an
invaluable resource for developers bringing in a new JITLink backend. A standard
workflow is to start by throwing an unsupported object at the tool and seeing
what error is returned, then fixing that (you can often make a reasonable guess
at what should be done based on existing code for other formats or
architectures).
-In debug builds of LLVM, the ``-debug-only=jitlink`` option dumps logs from the
+In debug builds of LLVM, the `-debug-only=jitlink` option dumps logs from the
JITLink library during the link process. These can be useful for spotting some bugs at
-a glance. The ``-debug-only=llvm_jitlink`` option dumps logs from the ``llvm-jitlink``
+a glance. The `-debug-only=llvm_jitlink` option dumps logs from the `llvm-jitlink`
tool, which can be useful for debugging both testcases (it is often less verbose than
-``-debug-only=jitlink``) and the tool itself.
+`-debug-only=jitlink`) and the tool itself.
-The ``-oop-executor`` and ``-oop-executor-connect`` options are helpful for testing
+The `-oop-executor` and `-oop-executor-connect` options are helpful for testing
handling of cross-process and cross-architecture use cases.
-Roadmap
-=======
+## Roadmap
JITLink is under active development. The MachO and ELF backends are mature, with
ELF support spanning x86-64, arm64, RISC-V, LoongArch, PowerPC 64, arm32,
@@ -1072,44 +1005,43 @@ support for PowerPC 64 is available but not yet usable for general JIT compilati
Major outstanding projects include:
-* Improve XCOFF support.
+- Improve XCOFF support.
The XCOFF/ppc64 backend exists but does not yet implement the relocation
handling needed for general JIT use. Completing this would enable JITLink on
AIX.
-* Continue improving early-stage backends.
+- Continue improving early-stage backends.
Some backends (arm32, Hexagon) support common relocations but are not yet
ready for general use. Contributions to extend relocation coverage are welcome.
-* Implement support for other new architectures and formats.
+- Implement support for other new architectures and formats.
-JITLink Availability and Feature Status
----------------------------------------
+### JITLink Availability and Feature Status
The following table describes the status of the JITlink backends for various
format / architecture combinations (as of May 2026).
Support levels:
-* None: No backend. JITLink will return an "architecture not supported" error.
+- None: No backend. JITLink will return an "architecture not supported" error.
Represented by empty cells in the table below.
-* Skeleton: A backend exists, but does not support commonly used relocations.
+- Skeleton: A backend exists, but does not support commonly used relocations.
Even simple programs are likely to trigger an "unsupported relocation" error.
Backends in this state may be easy to improve by implementing new relocations.
Consider getting involved!
-* Basic: The backend supports simple programs, isn't ready for general use yet.
-* Usable: The backend is useable for general use for at least one code and
+- Basic: The backend supports simple programs, isn't ready for general use yet.
+- Usable: The backend is useable for general use for at least one code and
relocation model.
-* Good: The backend supports almost all relocations. Advanced features like
+- Good: The backend supports almost all relocations. Advanced features like
native thread local storage may not be available yet.
-* Complete: The backend supports all relocations and object format features.
+- Complete: The backend supports all relocations and object format features.
-.. list-table:: Availability and Status
- :widths: 10 22 22 22 22
- :header-rows: 1
- :stub-columns: 1
+```{list-table} Availability and Status
+:widths: 10 22 22 22 22
+:header-rows: 1
+:stub-columns: 1
* - Architecture
- ELF
@@ -1161,17 +1093,18 @@ Support levels:
- Usable
- Good
-
-
-.. [1] See ``llvm/examples/OrcV2Examples/LLJITWithObjectLinkingLayerPlugin`` for
- a full worked example.
-
-.. [2] If not for *hidden* scoped symbols we could eliminate the
- ``JITLinkDylib*`` argument to ``JITLinkMemoryManager::allocate`` and
- treat every object as a separate simulated dylib for the purposes of
- memory layout. Hidden symbols break this by generating in-range accesses
- to external symbols, requiring the access and symbol to be allocated
- within range of one another. That said, providing a pre-reserved address
- range pool for each simulated dylib guarantees that the relaxation
- optimizations will kick in for all intra-dylib references, which is good
- for performance (at the cost of whatever overhead is introduced by
- reserving the address-range up-front).
+```
+
+[^1]: See `llvm/examples/OrcV2Examples/LLJITWithObjectLinkingLayerPlugin` for
+ a full worked example.
+
+[^2]: If not for *hidden* scoped symbols we could eliminate the
+ `JITLinkDylib*` argument to `JITLinkMemoryManager::allocate` and
+ treat every object as a separate simulated dylib for the purposes of
+ memory layout. Hidden symbols break this by generating in-range accesses
+ to external symbols, requiring the access and symbol to be allocated
+ within range of one another. That said, providing a pre-reserved address
+ range pool for each simulated dylib guarantees that the relaxation
+ optimizations will kick in for all intra-dylib references, which is good
+ for performance (at the cost of whatever overhead is introduced by
+ reserving the address-range up-front).
diff --git a/llvm/docs/MCJITDesignAndImplementation.md b/llvm/docs/MCJITDesignAndImplementation.md
index ca38cbac030f8..b3c495b79788c 100644
--- a/llvm/docs/MCJITDesignAndImplementation.md
+++ b/llvm/docs/MCJITDesignAndImplementation.md
@@ -1,71 +1,69 @@
-===============================
-MCJIT Design and Implementation
-===============================
+# MCJIT Design and Implementation
-Introduction
-============
+## Introduction
This document describes the internal workings of the MCJIT execution
-engine and the RuntimeDyld component. It is intended as a high level
+engine and the RuntimeDyld component. It is intended as a high level
overview of the implementation, showing the flow and interactions of
objects throughout the code generation and dynamic loading process.
-Engine Creation
-===============
+## Engine Creation
In most cases, an EngineBuilder object is used to create an instance of
-the MCJIT execution engine. The EngineBuilder takes an llvm::Module
-object as an argument to its constructor. The client may then set various
+the MCJIT execution engine. The EngineBuilder takes an llvm::Module
+object as an argument to its constructor. The client may then set various
options that we control the later be passed along to the MCJIT engine,
including the selection of MCJIT as the engine type to be created.
Of particular interest is the EngineBuilder::setMCJITMemoryManager
-function. If the client does not explicitly create a memory manager at
+function. If the client does not explicitly create a memory manager at
this time, a default memory manager (specifically SectionMemoryManager)
will be created when the MCJIT engine is instantiated.
Once the options have been set, a client calls EngineBuilder::create to
-create an instance of the MCJIT engine. If the client does not use the
+create an instance of the MCJIT engine. If the client does not use the
form of this function that takes a TargetMachine as a parameter, a new
TargetMachine will be created based on the target triple associated with
the Module that was used to create the EngineBuilder.
-.. image:: MCJIT-engine-builder.png
+```{image} MCJIT-engine-builder.png
+```
EngineBuilder::create will call the static MCJIT::createJIT function,
passing in its pointers to the module, memory manager and target machine
objects, all of which will subsequently be owned by the MCJIT object.
The MCJIT class has a member variable, Dyld, which contains an instance of
-the RuntimeDyld wrapper class. This member will be used for
+the RuntimeDyld wrapper class. This member will be used for
communications between MCJIT and the actual RuntimeDyldImpl object that
gets created when an object is loaded.
-.. image:: MCJIT-creation.png
+```{image} MCJIT-creation.png
+```
Upon creation, MCJIT holds a pointer to the Module object that it received
from EngineBuilder but it does not immediately generate code for this
-module. Code generation is deferred until either the
+module. Code generation is deferred until either the
MCJIT::finalizeObject method is called explicitly or a function such as
MCJIT::getPointerToFunction is called which requires the code to have been
generated.
-Code Generation
-===============
+## Code Generation
When code generation is triggered, as described above, MCJIT will first
attempt to retrieve an object image from its ObjectCache member, if one
-has been set. If a cached object image cannot be retrieved, MCJIT will
-call its emitObject method. MCJIT::emitObject uses a local PassManager
+has been set. If a cached object image cannot be retrieved, MCJIT will
+call its emitObject method. MCJIT::emitObject uses a local PassManager
instance and creates a new ObjectBufferStream instance, both of which it
passes to TargetMachine::addPassesToEmitMC before calling PassManager::run
on the Module with which it was created.
-.. image:: MCJIT-load.png
+```{image} MCJIT-load.png
+```
The PassManager::run call causes the MC code generation mechanisms to emit
a complete relocatable binary object image (either in either ELF or MachO
format, depending on the target) into the ObjectBufferStream object, which
-is flushed to complete the process. If an ObjectCache is being used, the
+is flushed to complete the process. If an ObjectCache is being used, the
image will be passed to the ObjectCache here.
At this point, the ObjectBufferStream contains the raw object image.
@@ -73,39 +71,40 @@ Before the code can be executed, the code and data sections from this
image must be loaded into suitable memory, relocations must be applied and
memory permission and code cache invalidation (if required) must be completed.
-Object Loading
-==============
+## Object Loading
Once an object image has been obtained, either through code generation or
having been retrieved from an ObjectCache, it is passed to RuntimeDyld to
-be loaded. The RuntimeDyld wrapper class examines the object to determine
+be loaded. The RuntimeDyld wrapper class examines the object to determine
its file format and creates an instance of either RuntimeDyldELF or
RuntimeDyldMachO (both of which derive from the RuntimeDyldImpl base
class) and calls the RuntimeDyldImpl::loadObject method to perform that
actual loading.
-.. image:: MCJIT-dyld-load.png
+```{image} MCJIT-dyld-load.png
+```
RuntimeDyldImpl::loadObject begins by creating an ObjectImage instance
-from the ObjectBuffer it received. ObjectImage, which wraps the
+from the ObjectBuffer it received. ObjectImage, which wraps the
ObjectFile class, is a helper class which parses the binary object image
and provides access to the information contained in the format-specific
headers, including section, symbol and relocation information.
RuntimeDyldImpl::loadObject then iterates through the symbols in the
-image. Information about common symbols is collected for later use. For
+image. Information about common symbols is collected for later use. For
each function or data symbol, the associated section is loaded into memory
-and the symbol is stored in a symbol table map data structure. When the
+and the symbol is stored in a symbol table map data structure. When the
iteration is complete, a section is emitted for the common symbols.
Next, RuntimeDyldImpl::loadObject iterates through the sections in the
object image and for each section iterates through the relocations for
-that sections. For each relocation, it calls the format-specific
+that sections. For each relocation, it calls the format-specific
processRelocationRef method, which will examine the relocation and store
it in one of two data structures, a section-based relocation list map and
an external symbol relocation map.
-.. image:: MCJIT-load-object.png
+```{image} MCJIT-load-object.png
+```
When RuntimeDyldImpl::loadObject returns, all of the code and data
sections for the object will have been loaded into memory allocated by the
@@ -114,43 +113,41 @@ relocations have not yet been applied and the generated code is still not
ready to be executed.
[Currently (as of August 2013) the MCJIT engine will immediately apply
-relocations when loadObject completes. However, this shouldn't be
-happening. Because the code may have been generated for a remote target,
+relocations when loadObject completes. However, this shouldn't be
+happening. Because the code may have been generated for a remote target,
the client should be given a chance to re-map the section addresses before
-relocations are applied. It is possible to apply relocations multiple
+relocations are applied. It is possible to apply relocations multiple
times, but in the case where addresses are to be re-mapped, this first
application is wasted effort.]
-Address Remapping
-=================
+## Address Remapping
At any time after initial code has been generated and before
finalizeObject is called, the client can remap the address of sections in
-the object. Typically this is done because the code was generated for an
+the object. Typically this is done because the code was generated for an
external process and is being mapped into that process' address space.
The client remaps the section address by calling MCJIT::mapSectionAddress.
This should happen before the section memory is copied to its new
location.
When MCJIT::mapSectionAddress is called, MCJIT passes the call on to
-RuntimeDyldImpl (via its Dyld member). RuntimeDyldImpl stores the new
+RuntimeDyldImpl (via its Dyld member). RuntimeDyldImpl stores the new
address in an internal data structure but does not update the code at this
time, since other sections are likely to change.
When the client is finished remapping section addresses, it will call
MCJIT::finalizeObject to complete the remapping process.
-Final Preparations
-==================
+## Final Preparations
When MCJIT::finalizeObject is called, MCJIT calls
-RuntimeDyld::resolveRelocations. This function will attempt to locate any
+RuntimeDyld::resolveRelocations. This function will attempt to locate any
external symbols and then apply all relocations for the object.
External symbols are resolved by calling the memory manager's
-getPointerToNamedFunction method. The memory manager will return the
-address of the requested symbol in the target address space. (Note, this
-may not be a valid pointer in the host process.) RuntimeDyld will then
+getPointerToNamedFunction method. The memory manager will return the
+address of the requested symbol in the target address space. (Note, this
+may not be a valid pointer in the host process.) RuntimeDyld will then
iterate through the list of relocations it has stored which are associated
with this symbol and invoke the resolveRelocation method which, through an
format-specific implementation, will apply the relocation to the loaded
@@ -159,13 +156,14 @@ section memory.
Next, RuntimeDyld::resolveRelocations iterates through the list of
sections and for each section iterates through a list of relocations that
have been saved which reference that symbol and call resolveRelocation for
-each entry in this list. The relocation list here is a list of
+each entry in this list. The relocation list here is a list of
relocations for which the symbol associated with the relocation is located
-in the section associated with the list. Each of these locations will
+in the section associated with the list. Each of these locations will
have a target location at which the relocation will be applied that is
likely located in a
diff erent section.
-.. image:: MCJIT-resolve-relocations.png
+```{image} MCJIT-resolve-relocations.png
+```
Once relocations have been applied as described above, MCJIT calls
RuntimeDyld::getEHFrameSection, and if a non-zero result is returned
@@ -173,7 +171,7 @@ passes the section data to the memory manager's registerEHFrames method.
This allows the memory manager to call any desired target-specific
functions, such as registering the EH frame information with a debugger.
-Finally, MCJIT calls the memory manager's finalizeMemory method. In this
+Finally, MCJIT calls the memory manager's finalizeMemory method. In this
method, the memory manager will invalidate the target code cache, if
necessary, and apply final permissions to the memory pages it has
allocated for code and data memory.
diff --git a/llvm/docs/NVPTXUsage.md b/llvm/docs/NVPTXUsage.md
index d6ca9154a1be9..923a2b10f6be9 100644
--- a/llvm/docs/NVPTXUsage.md
+++ b/llvm/docs/NVPTXUsage.md
@@ -1,14 +1,11 @@
-=============================
-User Guide for NVPTX Back-end
-=============================
+# User Guide for NVPTX Back-end
-.. contents::
- :local:
- :depth: 3
+```{contents}
+:depth: 3
+:local: true
+```
-
-Introduction
-============
+## Introduction
To support GPU programming, the NVPTX back-end supports a subset of LLVM IR
along with a defined set of conventions used to represent GPU programming
@@ -16,276 +13,260 @@ concepts. This document provides an overview of the general usage of the back-
end, including a description of the conventions used and the set of accepted
LLVM IR.
-.. note::
-
- This document assumes a basic familiarity with CUDA and the PTX
- assembly language. Information about the CUDA Driver API and the PTX assembly
- language can be found in the `CUDA documentation
- <http://docs.nvidia.com/cuda/index.html>`__.
-
-
+:::{note}
+This document assumes a basic familiarity with CUDA and the PTX
+assembly language. Information about the CUDA Driver API and the PTX assembly
+language can be found in the [CUDA documentation](http://docs.nvidia.com/cuda/index.html).
+:::
-Conventions
-===========
+## Conventions
-Marking Functions as Kernels
-----------------------------
+### Marking Functions as Kernels
In PTX, there are two types of functions: *device functions*, which are only
callable by device code, and *kernel functions*, which are callable by host
-code. By default, the back-end will emit device functions. The ``ptx_kernel``
+code. By default, the back-end will emit device functions. The `ptx_kernel`
calling convention is used to declare a function as a kernel function.
The following example shows a kernel function calling a device function in LLVM
-IR. The function ``@my_kernel`` is callable from host code, but ``@my_fmad`` is
+IR. The function `@my_kernel` is callable from host code, but `@my_fmad` is
not.
-.. code-block:: llvm
+```llvm
+define float @my_fmad(float %x, float %y, float %z) {
+ %mul = fmul float %x, %y
+ %add = fadd float %mul, %z
+ ret float %add
+}
+
+define ptx_kernel void @my_kernel(ptr %ptr) {
+ %val = load float, ptr %ptr
+ %ret = call float @my_fmad(float %val, float %val, float %val)
+ store float %ret, ptr %ptr
+ ret void
+}
+```
- define float @my_fmad(float %x, float %y, float %z) {
- %mul = fmul float %x, %y
- %add = fadd float %mul, %z
- ret float %add
- }
+When compiled, the PTX kernel functions are callable by host-side code.
- define ptx_kernel void @my_kernel(ptr %ptr) {
- %val = load float, ptr %ptr
- %ret = call float @my_fmad(float %val, float %val, float %val)
- store float %ret, ptr %ptr
- ret void
- }
+### Parameter Attributes
-When compiled, the PTX kernel functions are callable by host-side code.
+`"nvvm.grid_constant"`
+
+: This attribute may be attached to a `byval` parameter of a kernel function
+ to indicate that the parameter should be lowered as a direct reference to
+ the grid-constant memory of the parameter, as opposed to a copy of the
+ parameter in local memory. Writing to a grid-constant parameter is
+ undefined behavior. Unlike a normal `byval` parameter, the address of a
+ grid-constant parameter is not unique to a given function invocation but
+ instead is shared by all kernels in the grid.
+(nvptx-fnattrs)=
-Parameter Attributes
---------------------
+### Function Attributes
-``"nvvm.grid_constant"``
- This attribute may be attached to a ``byval`` parameter of a kernel function
- to indicate that the parameter should be lowered as a direct reference to
- the grid-constant memory of the parameter, as opposed to a copy of the
- parameter in local memory. Writing to a grid-constant parameter is
- undefined behavior. Unlike a normal ``byval`` parameter, the address of a
- grid-constant parameter is not unique to a given function invocation but
- instead is shared by all kernels in the grid.
+`"nvvm.maxclusterrank"="<n>"`
-.. _nvptx_fnattrs:
+: This attribute specifies the maximum number of blocks per cluster. Must be
+ non-zero. Only supported for Hopper+.
-Function Attributes
--------------------
+`"nvvm.minctasm"="<n>"`
-``"nvvm.maxclusterrank"="<n>"``
- This attribute specifies the maximum number of blocks per cluster. Must be
- non-zero. Only supported for Hopper+.
+: This indicates a hint/directive to the compiler/driver, asking it to put at
+ least these many CTAs on an SM.
-``"nvvm.minctasm"="<n>"``
- This indicates a hint/directive to the compiler/driver, asking it to put at
- least these many CTAs on an SM.
+`"nvvm.maxnreg"="<n>"`
-``"nvvm.maxnreg"="<n>"``
- This attribute indicates the maximum number of registers to be used for the
- kernel function.
+: This attribute indicates the maximum number of registers to be used for the
+ kernel function.
-``"nvvm.maxntid"="<x>[,<y>[,<z>]]"``
- This attribute declares the maximum number of threads in the thread block
- (CTA). The maximum number of threads is the product of the maximum extent in
- each dimension. Exceeding the maximum number of threads results in a runtime
- error or kernel launch failure.
+`"nvvm.maxntid"="<x>[,<y>[,<z>]]"`
-``"nvvm.reqntid"="<x>[,<y>[,<z>]]"``
- This attribute declares the exact number of threads in the thread block
- (CTA). The number of threads is the product of the value in each dimension.
- Specifying a
diff erent CTA dimension at launch will result in a runtime
- error or kernel launch failure.
+: This attribute declares the maximum number of threads in the thread block
+ (CTA). The maximum number of threads is the product of the maximum extent in
+ each dimension. Exceeding the maximum number of threads results in a runtime
+ error or kernel launch failure.
-``"nvvm.cluster_dim"="<x>[,<y>[,<z>]]"``
- This attribute declares the number of thread blocks (CTAs) in the cluster.
- The total number of CTAs is the product of the number of CTAs in each
- dimension. Specifying a
diff erent cluster dimension at launch will result in
- a runtime error or kernel launch failure. Only supported for Hopper+.
+`"nvvm.reqntid"="<x>[,<y>[,<z>]]"`
-``"nvvm.blocksareclusters"``
- This attribute implies that the grid launch configuration for the
- corresponding kernel function is specifying the number of clusters instead
- of the number of thread blocks. This attribute is only allowed for kernel
- functions and requires ``nvvm.reqntid`` and ``nvvm.cluster_dim`` attributes.
+: This attribute declares the exact number of threads in the thread block
+ (CTA). The number of threads is the product of the value in each dimension.
+ Specifying a
diff erent CTA dimension at launch will result in a runtime
+ error or kernel launch failure.
-.. _address_spaces:
+`"nvvm.cluster_dim"="<x>[,<y>[,<z>]]"`
-Address Spaces
---------------
+: This attribute declares the number of thread blocks (CTAs) in the cluster.
+ The total number of CTAs is the product of the number of CTAs in each
+ dimension. Specifying a
diff erent cluster dimension at launch will result in
+ a runtime error or kernel launch failure. Only supported for Hopper+.
+
+`"nvvm.blocksareclusters"`
+
+: This attribute implies that the grid launch configuration for the
+ corresponding kernel function is specifying the number of clusters instead
+ of the number of thread blocks. This attribute is only allowed for kernel
+ functions and requires `nvvm.reqntid` and `nvvm.cluster_dim` attributes.
+
+(address-spaces)=
+
+### Address Spaces
The NVPTX back-end uses the following address space mapping:
- ============= ======================
- Address Space Memory Space
- ============= ======================
- 0 Generic
- 1 Global
- 2 Internal Use
- 3 Shared
- 4 Constant
- 5 Local
- 7 Shared Cluster
- ============= ======================
+| Address Space | Memory Space |
+| ------------- | -------------- |
+| 0 | Generic |
+| 1 | Global |
+| 2 | Internal Use |
+| 3 | Shared |
+| 4 | Constant |
+| 5 | Local |
+| 7 | Shared Cluster |
Every global variable and pointer type is assigned to one of these address
spaces, with 0 being the default address space. Intrinsics are provided which
can be used to convert pointers between the generic and non-generic address
spaces.
-As an example, the following IR will define an array ``@g`` that resides in
+As an example, the following IR will define an array `@g` that resides in
global device memory.
-.. code-block:: llvm
-
- @g = internal addrspace(1) global [4 x i32] [ i32 0, i32 1, i32 2, i32 3 ]
+```llvm
+ at g = internal addrspace(1) global [4 x i32] [ i32 0, i32 1, i32 2, i32 3 ]
+```
LLVM IR functions can read and write to this array, and host-side code can
copy data to it by name with the CUDA Driver API.
Note that since address space 0 is the generic space, it is illegal to have
-global variables in address space 0. Address space 0 is the default address
-space in LLVM, so the ``addrspace(N)`` annotation is *required* for global
+global variables in address space 0. Address space 0 is the default address
+space in LLVM, so the `addrspace(N)` annotation is *required* for global
variables.
-
-Triples
--------
+### Triples
The NVPTX target uses the module triple to select between 32/64-bit code
generation and the driver-compiler interface to use. The triple architecture
-can be one of ``nvptx`` (32-bit PTX) or ``nvptx64`` (64-bit PTX). The
-operating system should be one of ``cuda`` or ``nvcl``, which determines the
-interface used by the generated code to communicate with the driver. Most
-users will want to use ``cuda`` as the operating system, which makes the
+can be one of `nvptx` (32-bit PTX) or `nvptx64` (64-bit PTX). The
+operating system should be one of `cuda` or `nvcl`, which determines the
+interface used by the generated code to communicate with the driver. Most
+users will want to use `cuda` as the operating system, which makes the
generated PTX compatible with the CUDA Driver API.
-Example: 32-bit PTX for CUDA Driver API: ``nvptx-nvidia-cuda``
+Example: 32-bit PTX for CUDA Driver API: `nvptx-nvidia-cuda`
-Example: 64-bit PTX for CUDA Driver API: ``nvptx64-nvidia-cuda``
+Example: 64-bit PTX for CUDA Driver API: `nvptx64-nvidia-cuda`
-.. _nvptx_arch_hierarchy:
+(nvptx-arch-hierarchy)=
-NVPTX Architecture Hierarchy and Ordering
-=========================================
+## NVPTX Architecture Hierarchy and Ordering
GPU architectures: sm_2Y/sm_3Y/sm_5Y/sm_6Y/sm_7Y/sm_8Y/sm_9Y/sm_10Y/sm_12Y
('Y' represents version within the architecture). The architectures have name of
-the form ``sm_XYz`` where:
+the form `sm_XYz` where:
-* ``X`` represent the generation number
-* ``Y`` represents the version within the architecture, and
-* ``z`` represents the optional feature suffix.
+- `X` represent the generation number
+- `Y` represents the version within the architecture, and
+- `z` represents the optional feature suffix.
-If ``X1Y1 <= X2Y2``, then GPU capabilities of ``sm_X1Y1`` are included in
-``sm_X2Y2``. For example, take ``sm_90`` (9 represents ``X``, 0 represents
-``Y``, and no feature suffix) and ``sm_103`` architectures (10 represents ``X``,
-3 represents ``Y``, and no feature suffix). Since 90 <= 103, ``sm_90`` is
-compatible with ``sm_103``.
+If `X1Y1 <= X2Y2`, then GPU capabilities of `sm_X1Y1` are included in
+`sm_X2Y2`. For example, take `sm_90` (9 represents `X`, 0 represents
+`Y`, and no feature suffix) and `sm_103` architectures (10 represents `X`,
+3 represents `Y`, and no feature suffix). Since 90 <= 103, `sm_90` is
+compatible with `sm_103`.
-The family-specific variants have ``f`` feature suffix and they follow
+The family-specific variants have `f` feature suffix and they follow
following order:
-``sm_X{Y2}f > sm_X{Y1}f`` iff ``Y2 > Y1``
-``sm_XY{f} > sm_{XY}{}``
+`sm_X{Y2}f > sm_X{Y1}f` iff `Y2 > Y1`
+`sm_XY{f} > sm_{XY}{}`
-For example, take ``sm_100f`` (10 represents ``X``, 0 represents ``Y``, and
-``f`` represents ``z``) and ``sm_103f`` (10 represents ``X``, 3 represents
-``Y``, and ``f`` represents ``z``) architecture variants. Since ``Y1 < Y2``,
-``sm_100f`` is compatible with ``sm_103f``. Similarly based on the second rule,
-``sm_90`` is compatible with ``sm_103f``.
+For example, take `sm_100f` (10 represents `X`, 0 represents `Y`, and
+`f` represents `z`) and `sm_103f` (10 represents `X`, 3 represents
+`Y`, and `f` represents `z`) architecture variants. Since `Y1 < Y2`,
+`sm_100f` is compatible with `sm_103f`. Similarly based on the second rule,
+`sm_90` is compatible with `sm_103f`.
-Some counter examples, take ``sm_100f`` and ``sm_120f`` (12 represents ``X``, 0
-represents ``Y``, and ``f`` represents ``z``) architecture variants. Since both
-belongs to
diff erent family i.e. ``X1 != X2``, ``sm_100f`` is not compatible
-with ``sm_120f``.
+Some counter examples, take `sm_100f` and `sm_120f` (12 represents `X`, 0
+represents `Y`, and `f` represents `z`) architecture variants. Since both
+belongs to
diff erent family i.e. `X1 != X2`, `sm_100f` is not compatible
+with `sm_120f`.
-The architecture-specific variants have ``a`` feature suffix and they follow
+The architecture-specific variants have `a` feature suffix and they follow
following order:
-``sm_XY{a} > sm_XY{f} > sm_{XY}{}``
+`sm_XY{a} > sm_XY{f} > sm_{XY}{}`
-For example, take ``sm_103a`` (10 represents ``X``, 3 represents ``Y``, and
-``a`` represents ``z``), ``sm_103f``, and ``sm_103`` architecture variants. The
-``sm_103`` is compatible with ``sm_103a`` and ``sm_103f``, and ``sm_103f`` is
-compatible with ``sm_103a``.
+For example, take `sm_103a` (10 represents `X`, 3 represents `Y`, and
+`a` represents `z`), `sm_103f`, and `sm_103` architecture variants. The
+`sm_103` is compatible with `sm_103a` and `sm_103f`, and `sm_103f` is
+compatible with `sm_103a`.
Encoding := Arch * 10 + 2 (for 'f') + 1 (for 'a')
Arch := X * 10 + Y
-For example, ``sm_103f`` is encoded as 1032 (103 * 10 + 2) and ``sm_103a`` is
+For example, `sm_103f` is encoded as 1032 (103 * 10 + 2) and `sm_103a` is
encoded as 1033 (103 * 10 + 2 + 1).
This encoding allows simple partial ordering of the architectures.
-* Compare Family and Arch by dividing FullSMVersion by 100 and 10
+- Compare Family and Arch by dividing FullSMVersion by 100 and 10
respectively before the comparison.
-* Compare within the family by comparing FullSMVersion, given both belongs to
+- Compare within the family by comparing FullSMVersion, given both belongs to
the same family.
-* Detect ``a`` variants by checking FullSMVersion & 1.
-
-.. _nvptx_intrinsics:
+- Detect `a` variants by checking FullSMVersion & 1.
-NVPTX Intrinsics
-================
+(nvptx-intrinsics)=
-Reading PTX Special Registers
------------------------------
+## NVPTX Intrinsics
-'``llvm.nvvm.read.ptx.sreg.*``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+### Reading PTX Special Registers
-Syntax:
-"""""""
+#### '`llvm.nvvm.read.ptx.sreg.*`'
-.. code-block:: llvm
+##### Syntax:
- declare i32 @llvm.nvvm.read.ptx.sreg.tid.x()
- declare i32 @llvm.nvvm.read.ptx.sreg.tid.y()
- declare i32 @llvm.nvvm.read.ptx.sreg.tid.z()
- declare i32 @llvm.nvvm.read.ptx.sreg.ntid.x()
- declare i32 @llvm.nvvm.read.ptx.sreg.ntid.y()
- declare i32 @llvm.nvvm.read.ptx.sreg.ntid.z()
- declare i32 @llvm.nvvm.read.ptx.sreg.ctaid.x()
- declare i32 @llvm.nvvm.read.ptx.sreg.ctaid.y()
- declare i32 @llvm.nvvm.read.ptx.sreg.ctaid.z()
- declare i32 @llvm.nvvm.read.ptx.sreg.nctaid.x()
- declare i32 @llvm.nvvm.read.ptx.sreg.nctaid.y()
- declare i32 @llvm.nvvm.read.ptx.sreg.nctaid.z()
- declare i32 @llvm.nvvm.read.ptx.sreg.warpsize()
+```llvm
+declare i32 @llvm.nvvm.read.ptx.sreg.tid.x()
+declare i32 @llvm.nvvm.read.ptx.sreg.tid.y()
+declare i32 @llvm.nvvm.read.ptx.sreg.tid.z()
+declare i32 @llvm.nvvm.read.ptx.sreg.ntid.x()
+declare i32 @llvm.nvvm.read.ptx.sreg.ntid.y()
+declare i32 @llvm.nvvm.read.ptx.sreg.ntid.z()
+declare i32 @llvm.nvvm.read.ptx.sreg.ctaid.x()
+declare i32 @llvm.nvvm.read.ptx.sreg.ctaid.y()
+declare i32 @llvm.nvvm.read.ptx.sreg.ctaid.z()
+declare i32 @llvm.nvvm.read.ptx.sreg.nctaid.x()
+declare i32 @llvm.nvvm.read.ptx.sreg.nctaid.y()
+declare i32 @llvm.nvvm.read.ptx.sreg.nctaid.z()
+declare i32 @llvm.nvvm.read.ptx.sreg.warpsize()
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.read.ptx.sreg.*``' intrinsics provide access to the PTX
-special registers, in particular the kernel launch bounds. These registers
+The '`@llvm.nvvm.read.ptx.sreg.*`' intrinsics provide access to the PTX
+special registers, in particular the kernel launch bounds. These registers
map in the following way to CUDA builtins:
- ============ =====================================
- CUDA Builtin PTX Special Register Intrinsic
- ============ =====================================
- ``threadId`` ``@llvm.nvvm.read.ptx.sreg.tid.*``
- ``blockIdx`` ``@llvm.nvvm.read.ptx.sreg.ctaid.*``
- ``blockDim`` ``@llvm.nvvm.read.ptx.sreg.ntid.*``
- ``gridDim`` ``@llvm.nvvm.read.ptx.sreg.nctaid.*``
- ============ =====================================
+| CUDA Builtin | PTX Special Register Intrinsic |
+| ------------ | ----------------------------------- |
+| `threadId` | `@llvm.nvvm.read.ptx.sreg.tid.*` |
+| `blockIdx` | `@llvm.nvvm.read.ptx.sreg.ctaid.*` |
+| `blockDim` | `@llvm.nvvm.read.ptx.sreg.ntid.*` |
+| `gridDim` | `@llvm.nvvm.read.ptx.sreg.nctaid.*` |
-'``llvm.nvvm.read.ptx.sreg.*_smem_size``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.read.ptx.sreg.*_smem_size`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare i32 @llvm.nvvm.read.ptx.sreg.total_smem_size()
+declare i32 @llvm.nvvm.read.ptx.sreg.aggr_smem_size()
+declare i32 @llvm.nvvm.read.ptx.sreg.dynamic_smem_size()
+```
- declare i32 @llvm.nvvm.read.ptx.sreg.total_smem_size()
- declare i32 @llvm.nvvm.read.ptx.sreg.aggr_smem_size()
- declare i32 @llvm.nvvm.read.ptx.sreg.dynamic_smem_size()
+##### Overview:
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.read.ptx.sreg.total_smem_size``' intrinsic reads the PTX
+The '`@llvm.nvvm.read.ptx.sreg.total_smem_size`' intrinsic reads the PTX
special register that holds the total amount of shared memory allocated per CTA
for the kernel at launch.
@@ -294,617 +275,557 @@ shared memory, but excludes any shared memory reserved for system use. The size
is expressed in units of the architecture-specific shared memory allocation
granularity. For targets sm_8x and newer, this granularity is 128 bytes.
-The '``aggr_smem_size``' variant returns the aggregate shared memory size,
+The '`aggr_smem_size`' variant returns the aggregate shared memory size,
including the portion reserved for system software use.
-The '``dynamic_smem_size``' variant returns the amount of dynamic shared
+The '`dynamic_smem_size`' variant returns the amount of dynamic shared
memory allocated per CTA for the kernel at launch time.
-'``llvm.nvvm.read.ptx.sreg.reserved_smem_offset_*``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
+#### '`llvm.nvvm.read.ptx.sreg.reserved_smem_offset_*`'
-.. code-block:: llvm
+##### Syntax:
- declare i32 @llvm.nvvm.read.ptx.sreg.reserved_smem_offset_begin()
- declare i32 @llvm.nvvm.read.ptx.sreg.reserved_smem_offset_end()
- declare i32 @llvm.nvvm.read.ptx.sreg.reserved_smem_offset_cap()
- declare i32 @llvm.nvvm.read.ptx.sreg.reserved_smem_offset_0()
- declare i32 @llvm.nvvm.read.ptx.sreg.reserved_smem_offset_1()
+```llvm
+declare i32 @llvm.nvvm.read.ptx.sreg.reserved_smem_offset_begin()
+declare i32 @llvm.nvvm.read.ptx.sreg.reserved_smem_offset_end()
+declare i32 @llvm.nvvm.read.ptx.sreg.reserved_smem_offset_cap()
+declare i32 @llvm.nvvm.read.ptx.sreg.reserved_smem_offset_0()
+declare i32 @llvm.nvvm.read.ptx.sreg.reserved_smem_offset_1()
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.read.ptx.sreg.reserved_smem_offset_*``' intrinsics provide
+The '`@llvm.nvvm.read.ptx.sreg.reserved_smem_offset_*`' intrinsics provide
access to PTX special registers that hold information about reserved shared
-memory offsets.
+memory offsets.
-The '``reserved_smem_offset_begin``' intrinsic reads the start offset of
+The '`reserved_smem_offset_begin`' intrinsic reads the start offset of
the reserved shared memory region.
-The '``reserved_smem_offset_end``' intrinsic reads the end offset of the
+The '`reserved_smem_offset_end`' intrinsic reads the end offset of the
reserved shared memory region.
-The '``reserved_smem_offset_cap``' intrinsic reads the capacity limit of
+The '`reserved_smem_offset_cap`' intrinsic reads the capacity limit of
the reserved shared memory region.
-The '``reserved_smem_offset_0``' and '``reserved_smem_offset_1``' intrinsics
+The '`reserved_smem_offset_0`' and '`reserved_smem_offset_1`' intrinsics
read additional offsets in the reserved shared memory region.
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#special-registers-reserved-smem>`__.
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#special-registers-reserved-smem).
-Barriers
---------
+### Barriers
-'``llvm.nvvm.barrier.cta.*``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.barrier.cta.*`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.barrier.cta.sync.count(i32 %id, i32 %n)
+declare void @llvm.nvvm.barrier.cta.sync.all(i32 %id)
+declare void @llvm.nvvm.barrier.cta.sync.aligned.count(i32 %id, i32 %n)
+declare void @llvm.nvvm.barrier.cta.sync.aligned.all(i32 %id)
- declare void @llvm.nvvm.barrier.cta.sync.count(i32 %id, i32 %n)
- declare void @llvm.nvvm.barrier.cta.sync.all(i32 %id)
- declare void @llvm.nvvm.barrier.cta.sync.aligned.count(i32 %id, i32 %n)
- declare void @llvm.nvvm.barrier.cta.sync.aligned.all(i32 %id)
+declare void @llvm.nvvm.barrier.cta.arrive.count(i32 %id, i32 %n)
+declare void @llvm.nvvm.barrier.cta.arrive.aligned.count(i32 %id, i32 %n)
- declare void @llvm.nvvm.barrier.cta.arrive.count(i32 %id, i32 %n)
- declare void @llvm.nvvm.barrier.cta.arrive.aligned.count(i32 %id, i32 %n)
+declare i32 @llvm.nvvm.barrier.cta.red.popc.count(i32 %id, i32 %n, i1 %pred)
+declare i32 @llvm.nvvm.barrier.cta.red.popc.all(i32 %id, i1 %pred)
+declare i32 @llvm.nvvm.barrier.cta.red.popc.aligned.count(i32 %id, i32 %n, i1 %pred)
+declare i32 @llvm.nvvm.barrier.cta.red.popc.aligned.all(i32 %id, i1 %pred)
+declare i32 @llvm.nvvm.barrier.cta.red.and.count(i32 %id, i32 %n, i1 %pred)
+declare i32 @llvm.nvvm.barrier.cta.red.and.all(i32 %id, i1 %pred)
+declare i32 @llvm.nvvm.barrier.cta.red.and.aligned.count(i32 %id, i32 %n, i1 %pred)
+declare i32 @llvm.nvvm.barrier.cta.red.and.aligned.all(i32 %id, i1 %pred)
+declare i32 @llvm.nvvm.barrier.cta.red.or.count(i32 %id, i32 %n, i1 %pred)
+declare i32 @llvm.nvvm.barrier.cta.red.or.all(i32 %id, i1 %pred)
+declare i32 @llvm.nvvm.barrier.cta.red.or.aligned.count(i32 %id, i32 %n, i1 %pred)
+declare i32 @llvm.nvvm.barrier.cta.red.or.aligned.all(i32 %id, i1 %pred)
+```
- declare i32 @llvm.nvvm.barrier.cta.red.popc.count(i32 %id, i32 %n, i1 %pred)
- declare i32 @llvm.nvvm.barrier.cta.red.popc.all(i32 %id, i1 %pred)
- declare i32 @llvm.nvvm.barrier.cta.red.popc.aligned.count(i32 %id, i32 %n, i1 %pred)
- declare i32 @llvm.nvvm.barrier.cta.red.popc.aligned.all(i32 %id, i1 %pred)
- declare i32 @llvm.nvvm.barrier.cta.red.and.count(i32 %id, i32 %n, i1 %pred)
- declare i32 @llvm.nvvm.barrier.cta.red.and.all(i32 %id, i1 %pred)
- declare i32 @llvm.nvvm.barrier.cta.red.and.aligned.count(i32 %id, i32 %n, i1 %pred)
- declare i32 @llvm.nvvm.barrier.cta.red.and.aligned.all(i32 %id, i1 %pred)
- declare i32 @llvm.nvvm.barrier.cta.red.or.count(i32 %id, i32 %n, i1 %pred)
- declare i32 @llvm.nvvm.barrier.cta.red.or.all(i32 %id, i1 %pred)
- declare i32 @llvm.nvvm.barrier.cta.red.or.aligned.count(i32 %id, i32 %n, i1 %pred)
- declare i32 @llvm.nvvm.barrier.cta.red.or.aligned.all(i32 %id, i1 %pred)
+##### Overview:
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.barrier.cta.*``' family of intrinsics perform barrier
+The '`@llvm.nvvm.barrier.cta.*`' family of intrinsics perform barrier
synchronization and communication within a CTA. They can be used by the threads
within the CTA for synchronization and communication.
-Semantics:
-""""""""""
+##### Semantics:
Operand %id specifies a logical barrier resource and must fall within the range
0 through 15. When present, operand %n specifies the number of threads
participating in the barrier. When specifying a thread count, the value must be
-a multiple of the warp size. With the '``@llvm.nvvm.barrier.cta.sync.*``'
-variants, the '``.all``' suffix indicates that all threads in the CTA should
-participate in the barrier while the '``.count``' suffix indicates that only
+a multiple of the warp size. With the '`@llvm.nvvm.barrier.cta.sync.*`'
+variants, the '`.all`' suffix indicates that all threads in the CTA should
+participate in the barrier while the '`.count`' suffix indicates that only
the threads specified by the %n operand should participate in the barrier.
-All forms of the '``@llvm.nvvm.barrier.cta.*``' intrinsic cause the executing
+All forms of the '`@llvm.nvvm.barrier.cta.*`' intrinsic cause the executing
thread to wait for all non-exited threads from its warp and then marks the
warp's arrival at the barrier. In addition to signaling its arrival at the
-barrier, the '``@llvm.nvvm.barrier.cta.red.*``' and
-'``@llvm.nvvm.barrier.cta.sync.*``' intrinsics cause the executing thread to
+barrier, the '`@llvm.nvvm.barrier.cta.red.*`' and
+'`@llvm.nvvm.barrier.cta.sync.*`' intrinsics cause the executing thread to
wait for non-exited threads of all other warps participating in the barrier to
-arrive. On the other hand, the '``@llvm.nvvm.barrier.cta.arrive.*``' intrinsic
+arrive. On the other hand, the '`@llvm.nvvm.barrier.cta.arrive.*`' intrinsic
does not cause the executing thread to wait for threads of other participating
warps.
When a barrier completes, the waiting threads are restarted without delay,
and the barrier is reinitialized so that it can be immediately reused.
-The '``@llvm.nvvm.barrier.cta.red.*``' intrinsics perform a reduction operation
+The '`@llvm.nvvm.barrier.cta.red.*`' intrinsics perform a reduction operation
across threads. The %pred operands from all threads in the CTA are combined
using the specified reduction operator. Once the barrier count is reached, the
final value is returned in all threads waiting at the barrier.
-The reduction operations for '``@llvm.nvvm.barrier.cta.red.*``' are
-population-count ('``.popc``'), all-threads-true ('``.and``'),
-and any-thread-true ('``.or``'). The result of '``.popc``' is the number of
-threads with a true predicate, while '``.and``' and '``.or``' indicate if all
+The reduction operations for '`@llvm.nvvm.barrier.cta.red.*`' are
+population-count ('`.popc`'), all-threads-true ('`.and`'),
+and any-thread-true ('`.or`'). The result of '`.popc`' is the number of
+threads with a true predicate, while '`.and`' and '`.or`' indicate if all
the threads had a true predicate or if any of the threads had a true predicate.
-The '``@llvm.nvvm.barrier.cta.*``' intrinsic has an optional '``.aligned``'
+The '`@llvm.nvvm.barrier.cta.*`' intrinsic has an optional '`.aligned`'
modifier to indicate textual alignment of the barrier. When specified, it
indicates that all threads in the CTA will execute the same
-'``@llvm.nvvm.barrier.cta.*``' instruction. In conditionally executed code, an
-aligned '``@llvm.nvvm.barrier.cta.*``' instruction should only be used if it is
+'`@llvm.nvvm.barrier.cta.*`' instruction. In conditionally executed code, an
+aligned '`@llvm.nvvm.barrier.cta.*`' instruction should only be used if it is
known that all threads in the CTA evaluate the condition identically, otherwise
behavior is undefined.
-MBarrier family of Intrinsics
------------------------------
+### MBarrier family of Intrinsics
-Overview:
-^^^^^^^^^
+#### Overview:
-An ``mbarrier`` is a barrier created in shared memory that supports:
+An `mbarrier` is a barrier created in shared memory that supports:
-* Synchronizing any subset of threads within a CTA.
-* One-way synchronization of threads across CTAs of a cluster.
- Threads can perform only ``arrive`` operations but not ``*_wait`` on an
+- Synchronizing any subset of threads within a CTA.
+- One-way synchronization of threads across CTAs of a cluster.
+ Threads can perform only `arrive` operations but not `*_wait` on an
mbarrier located in shared::cluster space.
-* Waiting for completion of asynchronous memory operations initiated by a
+- Waiting for completion of asynchronous memory operations initiated by a
thread and making them visible to other threads.
-Unlike ``bar{.cta}/barrier{.cta}`` instructions which can access a limited
-number of barriers per CTA, ``mbarrier`` objects are user-defined and are
+Unlike `bar{.cta}/barrier{.cta}` instructions which can access a limited
+number of barriers per CTA, `mbarrier` objects are user-defined and are
only limited by the total shared memory size available.
An mbarrier object is an opaque object in shared memory with an
alignment of 8-bytes. It keeps track of:
-* Current phase of the mbarrier object
-* Count of pending arrivals for the current phase of the mbarrier object
-* Count of expected arrivals for the next phase of the mbarrier object
-* Count of pending asynchronous memory operations (or transactions)
+- Current phase of the mbarrier object
+- Count of pending arrivals for the current phase of the mbarrier object
+- Count of expected arrivals for the next phase of the mbarrier object
+- Count of pending asynchronous memory operations (or transactions)
tracked by the current phase of the mbarrier object. This is also
- referred to as ``tx-count``. The unit of ``tx-count`` is specified
+ referred to as `tx-count`. The unit of `tx-count` is specified
by the asynchronous memory operation (for example,
- ``llvm.nvvm.cp.async.bulk.tensor.g2s.*``).
+ `llvm.nvvm.cp.async.bulk.tensor.g2s.*`).
-The ``phase`` of an mbarrier object is the number of times the mbarrier
+The `phase` of an mbarrier object is the number of times the mbarrier
object has been used to synchronize threads/track async operations.
In each phase, threads perform:
-* arrive/expect-tx/complete-tx operations to progress the current phase.
-* test_wait/try_wait operations to check for completion of the current phase.
+- arrive/expect-tx/complete-tx operations to progress the current phase.
+- test_wait/try_wait operations to check for completion of the current phase.
An mbarrier object completes the current phase when:
-* The count of the pending arrivals has reached zero AND
-* The tx-count has reached zero.
+- The count of the pending arrivals has reached zero AND
+- The tx-count has reached zero.
When an mbarrier object completes the current phase, below
-actions are performed ``atomically``:
-
-* The mbarrier object transitions to the next phase.
-* The pending arrival count is reinitialized to the expected arrival count.
+actions are performed `atomically`:
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#parallel-synchronization-and-communication-instructions-mbarrier>`__.
+- The mbarrier object transitions to the next phase.
+- The pending arrival count is reinitialized to the expected arrival count.
-'``llvm.nvvm.mbarrier.init``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#parallel-synchronization-and-communication-instructions-mbarrier).
-Syntax:
-"""""""
+#### '`llvm.nvvm.mbarrier.init`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.mbarrier.init(ptr %addr, i32 %count)
- declare void @llvm.nvvm.mbarrier.init.shared(ptr addrspace(3) %addr, i32 %count)
+```llvm
+declare void @llvm.nvvm.mbarrier.init(ptr %addr, i32 %count)
+declare void @llvm.nvvm.mbarrier.init.shared(ptr addrspace(3) %addr, i32 %count)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.mbarrier.init.*``' intrinsics are used to initialize
-an mbarrier object located at ``addr`` with the value ``count``.
-``count`` is a 32-bit unsigned integer value and must be within
+The '`@llvm.nvvm.mbarrier.init.*`' intrinsics are used to initialize
+an mbarrier object located at `addr` with the value `count`.
+`count` is a 32-bit unsigned integer value and must be within
the range [1...2^20-1]. During initialization:
-* The tx-count and the current phase of the mbarrier object are set to 0.
-* The expected and pending arrival counts are set to ``count``.
+- The tx-count and the current phase of the mbarrier object are set to 0.
+- The expected and pending arrival counts are set to `count`.
-Semantics:
-""""""""""
+##### Semantics:
-The ``.shared`` variant explicitly uses shared memory address space for
-the ``addr`` operand. If the ``addr`` does not fall within the
+The `.shared` variant explicitly uses shared memory address space for
+the `addr` operand. If the `addr` does not fall within the
shared::cta space, then the behavior of this intrinsic is undefined.
-Performing ``mbarrier.init`` on a valid mbarrier object is undefined;
-use ``mbarrier.inval`` before reusing the memory for another mbarrier
+Performing `mbarrier.init` on a valid mbarrier object is undefined;
+use `mbarrier.inval` before reusing the memory for another mbarrier
or any other purpose.
-'``llvm.nvvm.mbarrier.inval``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.mbarrier.inval`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.mbarrier.inval(ptr %addr)
+declare void @llvm.nvvm.mbarrier.inval.shared(ptr addrspace(3) %addr)
+```
- declare void @llvm.nvvm.mbarrier.inval(ptr %addr)
- declare void @llvm.nvvm.mbarrier.inval.shared(ptr addrspace(3) %addr)
+##### Overview:
-Overview:
-"""""""""
+The '`@llvm.nvvm.mbarrier.inval.*`' intrinsics invalidate the mbarrier
+object at the address specified by `addr`.
-The '``@llvm.nvvm.mbarrier.inval.*``' intrinsics invalidate the mbarrier
-object at the address specified by ``addr``.
+##### Semantics:
-Semantics:
-""""""""""
-
-The ``.shared`` variant explicitly uses shared memory address space for
-the ``addr`` operand. If the ``addr`` does not fall within the
+The `.shared` variant explicitly uses shared memory address space for
+the `addr` operand. If the `addr` does not fall within the
shared::cta space, then the behavior of this intrinsic is undefined.
-It is expected that ``addr`` was previously initialized using
-``mbarrier.init``; otherwise, the behavior is undefined.
-
-'``llvm.nvvm.mbarrier.expect.tx``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+It is expected that `addr` was previously initialized using
+`mbarrier.init`; otherwise, the behavior is undefined.
-Syntax:
-"""""""
+#### '`llvm.nvvm.mbarrier.expect.tx`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.mbarrier.expect.tx.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
- declare void @llvm.nvvm.mbarrier.expect.tx.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
- declare void @llvm.nvvm.mbarrier.expect.tx.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
- declare void @llvm.nvvm.mbarrier.expect.tx.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
+```llvm
+declare void @llvm.nvvm.mbarrier.expect.tx.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
+declare void @llvm.nvvm.mbarrier.expect.tx.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
+declare void @llvm.nvvm.mbarrier.expect.tx.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
+declare void @llvm.nvvm.mbarrier.expect.tx.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.mbarrier.expect.tx.*``' intrinsics increase the transaction
-count of the mbarrier object at ``%addr`` by ``%tx_count``. The ``%tx_count``
+The '`@llvm.nvvm.mbarrier.expect.tx.*`' intrinsics increase the transaction
+count of the mbarrier object at `%addr` by `%tx_count`. The `%tx_count`
is a 32-bit unsigned integer value.
-Semantics:
-""""""""""
+##### Semantics:
-The ``.space.{cta/cluster}`` indicates the address space where the mbarrier
+The `.space.{cta/cluster}` indicates the address space where the mbarrier
object resides.
-The ``.scope.{cta/cluster}`` denotes the set of threads that can directly
+The `.scope.{cta/cluster}` denotes the set of threads that can directly
observe the synchronizing effect of the mbarrier operation. When scope is
"cta", all threads executing in the same CTA (as the current thread) can
-directly observe the effect of the ``expect.tx`` operation. Similarly,
+directly observe the effect of the `expect.tx` operation. Similarly,
when scope is "cluster", all threads executing in the same Cluster
(as the current thread) can directly observe the effect of the operation.
-If the ``addr`` does not fall within shared::cta or shared::cluster space,
+If the `addr` does not fall within shared::cta or shared::cluster space,
then the behavior of this intrinsic is undefined. This intrinsic has
-``relaxed`` semantics and hence does not provide any memory ordering
+`relaxed` semantics and hence does not provide any memory ordering
or visibility guarantees.
-'``llvm.nvvm.mbarrier.complete.tx``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
+#### '`llvm.nvvm.mbarrier.complete.tx`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.mbarrier.complete.tx.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
- declare void @llvm.nvvm.mbarrier.complete.tx.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
- declare void @llvm.nvvm.mbarrier.complete.tx.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
- declare void @llvm.nvvm.mbarrier.complete.tx.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
+```llvm
+declare void @llvm.nvvm.mbarrier.complete.tx.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
+declare void @llvm.nvvm.mbarrier.complete.tx.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
+declare void @llvm.nvvm.mbarrier.complete.tx.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
+declare void @llvm.nvvm.mbarrier.complete.tx.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.mbarrier.complete.tx.*``' intrinsics decrease the transaction
-count of the mbarrier object at ``%addr`` by ``%tx_count``. The ``%tx_count``
+The '`@llvm.nvvm.mbarrier.complete.tx.*`' intrinsics decrease the transaction
+count of the mbarrier object at `%addr` by `%tx_count`. The `%tx_count`
is a 32-bit unsigned integer value. As a result of this decrement,
the mbarrier can potentially complete its current phase and transition
to the next phase.
-Semantics:
-""""""""""
+##### Semantics:
The semantics of these intrinsics are identical to those of the
-``llvm.nvvm.mbarrier.expect.tx.*`` intrinsics described above.
+`llvm.nvvm.mbarrier.expect.tx.*` intrinsics described above.
-'``llvm.nvvm.mbarrier.arrive``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.mbarrier.arrive`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare i64 @llvm.nvvm.mbarrier.arrive.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %count)
+declare i64 @llvm.nvvm.mbarrier.arrive.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %count)
+declare void @llvm.nvvm.mbarrier.arrive.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %count)
+declare void @llvm.nvvm.mbarrier.arrive.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %count)
- declare i64 @llvm.nvvm.mbarrier.arrive.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %count)
- declare i64 @llvm.nvvm.mbarrier.arrive.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %count)
- declare void @llvm.nvvm.mbarrier.arrive.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %count)
- declare void @llvm.nvvm.mbarrier.arrive.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %count)
+declare i64 @llvm.nvvm.mbarrier.arrive.relaxed.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %count)
+declare i64 @llvm.nvvm.mbarrier.arrive.relaxed.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %count)
+declare void @llvm.nvvm.mbarrier.arrive.relaxed.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %count)
+declare void @llvm.nvvm.mbarrier.arrive.relaxed.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %count)
+```
- declare i64 @llvm.nvvm.mbarrier.arrive.relaxed.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %count)
- declare i64 @llvm.nvvm.mbarrier.arrive.relaxed.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %count)
- declare void @llvm.nvvm.mbarrier.arrive.relaxed.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %count)
- declare void @llvm.nvvm.mbarrier.arrive.relaxed.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %count)
+##### Overview:
-Overview:
-"""""""""
-
-The ``@llvm.nvvm.mbarrier.arrive.*`` intrinsics signal the arrival of the
+The `@llvm.nvvm.mbarrier.arrive.*` intrinsics signal the arrival of the
executing thread or completion of an asynchronous instruction associated with
-an arrive operation on the mbarrier object at ``%addr``. This operation
-decrements the pending arrival count by ``%count``, a 32-bit unsigned integer,
+an arrive operation on the mbarrier object at `%addr`. This operation
+decrements the pending arrival count by `%count`, a 32-bit unsigned integer,
potentially completing the current phase and triggering a transition to the
next phase.
-Semantics:
-""""""""""
+##### Semantics:
-The ``.space.{cta/cluster}`` indicates the address space where the mbarrier
+The `.space.{cta/cluster}` indicates the address space where the mbarrier
object resides. When the mbarrier is in shared::cta space, the intrinsics
return an opaque 64-bit value capturing the phase of the mbarrier object
-_prior_ to this arrive operation. This value can be used with a try_wait
+\_prior\_ to this arrive operation. This value can be used with a try_wait
or test_wait operation to check for the completion of the mbarrier.
-The ``.scope.{cta/cluster}`` denotes the set of threads that can directly
+The `.scope.{cta/cluster}` denotes the set of threads that can directly
observe the synchronizing effect of the mbarrier operation. When scope is
"cta", all threads executing in the same CTA (as the current thread) can
-directly observe the effect of the ``arrive`` operation. Similarly,
+directly observe the effect of the `arrive` operation. Similarly,
when scope is "cluster", all threads executing in the same Cluster
(as the current thread) can directly observe the effect of the operation.
-If the ``addr`` does not fall within shared::cta or shared::cluster space,
+If the `addr` does not fall within shared::cta or shared::cluster space,
then the behavior of this intrinsic is undefined.
-These intrinsics have ``release`` semantics by default. The release semantics
-ensure ordering of operations that occur in program order _before_ this arrive
+These intrinsics have `release` semantics by default. The release semantics
+ensure ordering of operations that occur in program order \_before\_ this arrive
instruction, making their effects visible to subsequent operations in other
threads of the CTA (or cluster, depending on scope). Threads performing
corresponding acquire operations (such as mbarrier.test.wait) synchronize
-with this release. The ``relaxed`` variants of these intrinsics do not
+with this release. The `relaxed` variants of these intrinsics do not
provide any memory ordering or visibility guarantees.
-'``llvm.nvvm.mbarrier.arrive.expect.tx``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
+#### '`llvm.nvvm.mbarrier.arrive.expect.tx`'
-.. code-block:: llvm
+##### Syntax:
- declare i64 @llvm.nvvm.mbarrier.arrive.expect.tx.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
- declare i64 @llvm.nvvm.mbarrier.arrive.expect.tx.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
- declare void @llvm.nvvm.mbarrier.arrive.expect.tx.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
- declare void @llvm.nvvm.mbarrier.arrive.expect.tx.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
+```llvm
+declare i64 @llvm.nvvm.mbarrier.arrive.expect.tx.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
+declare i64 @llvm.nvvm.mbarrier.arrive.expect.tx.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
+declare void @llvm.nvvm.mbarrier.arrive.expect.tx.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
+declare void @llvm.nvvm.mbarrier.arrive.expect.tx.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
- declare i64 @llvm.nvvm.mbarrier.arrive.expect.tx.relaxed.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
- declare i64 @llvm.nvvm.mbarrier.arrive.expect.tx.relaxed.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
- declare void @llvm.nvvm.mbarrier.arrive.expect.tx.relaxed.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
- declare void @llvm.nvvm.mbarrier.arrive.expect.tx.relaxed.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
+declare i64 @llvm.nvvm.mbarrier.arrive.expect.tx.relaxed.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
+declare i64 @llvm.nvvm.mbarrier.arrive.expect.tx.relaxed.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
+declare void @llvm.nvvm.mbarrier.arrive.expect.tx.relaxed.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
+declare void @llvm.nvvm.mbarrier.arrive.expect.tx.relaxed.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
+```
-Overview:
-"""""""""
+##### Overview:
-The ``@llvm.nvvm.mbarrier.arrive.expect.tx.*`` intrinsics are similar to
-the ``@llvm.nvvm.mbarrier.arrive`` intrinsics except that they also
-perform an ``expect-tx`` operation _prior_ to the ``arrive`` operation.
-The ``%tx_count`` specifies the transaction count for the ``expect-tx``
-operation and the count for the ``arrive`` operation is assumed to be 1.
+The `@llvm.nvvm.mbarrier.arrive.expect.tx.*` intrinsics are similar to
+the `@llvm.nvvm.mbarrier.arrive` intrinsics except that they also
+perform an `expect-tx` operation \_prior\_ to the `arrive` operation.
+The `%tx_count` specifies the transaction count for the `expect-tx`
+operation and the count for the `arrive` operation is assumed to be 1.
-Semantics:
-""""""""""
+##### Semantics:
The semantics of these intrinsics are identical to those of the
-``llvm.nvvm.mbarrier.arrive.*`` intrinsics described above.
+`llvm.nvvm.mbarrier.arrive.*` intrinsics described above.
-'``llvm.nvvm.mbarrier.arrive.drop``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.mbarrier.arrive.drop`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare i64 @llvm.nvvm.mbarrier.arrive.drop.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %count)
+declare i64 @llvm.nvvm.mbarrier.arrive.drop.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %count)
+declare void @llvm.nvvm.mbarrier.arrive.drop.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %count)
+declare void @llvm.nvvm.mbarrier.arrive.drop.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %count)
- declare i64 @llvm.nvvm.mbarrier.arrive.drop.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %count)
- declare i64 @llvm.nvvm.mbarrier.arrive.drop.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %count)
- declare void @llvm.nvvm.mbarrier.arrive.drop.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %count)
- declare void @llvm.nvvm.mbarrier.arrive.drop.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %count)
+declare i64 @llvm.nvvm.mbarrier.arrive.drop.relaxed.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %count)
+declare i64 @llvm.nvvm.mbarrier.arrive.drop.relaxed.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %count)
+declare void @llvm.nvvm.mbarrier.arrive.drop.relaxed.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %count)
+declare void @llvm.nvvm.mbarrier.arrive.drop.relaxed.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %count)
+```
- declare i64 @llvm.nvvm.mbarrier.arrive.drop.relaxed.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %count)
- declare i64 @llvm.nvvm.mbarrier.arrive.drop.relaxed.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %count)
- declare void @llvm.nvvm.mbarrier.arrive.drop.relaxed.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %count)
- declare void @llvm.nvvm.mbarrier.arrive.drop.relaxed.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %count)
+##### Overview:
-Overview:
-"""""""""
+The `@llvm.nvvm.mbarrier.arrive.drop.*` intrinsics decrement the
+expected arrival count of the mbarrier object at `%addr` by
+`%count` and then perform an `arrive` operation with `%count`.
+The `%count` is a 32-bit integer.
-The ``@llvm.nvvm.mbarrier.arrive.drop.*`` intrinsics decrement the
-expected arrival count of the mbarrier object at ``%addr`` by
-``%count`` and then perform an ``arrive`` operation with ``%count``.
-The ``%count`` is a 32-bit integer.
-
-Semantics:
-""""""""""
+##### Semantics:
The semantics of these intrinsics are identical to those of the
-``llvm.nvvm.mbarrier.arrive.*`` intrinsics described above.
-
-'``llvm.nvvm.mbarrier.arrive.drop.expect.tx``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+`llvm.nvvm.mbarrier.arrive.*` intrinsics described above.
-Syntax:
-"""""""
+#### '`llvm.nvvm.mbarrier.arrive.drop.expect.tx`'
-.. code-block:: llvm
+##### Syntax:
- declare i64 @llvm.nvvm.mbarrier.arrive.drop.expect.tx.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
- declare i64 @llvm.nvvm.mbarrier.arrive.drop.expect.tx.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
- declare void @llvm.nvvm.mbarrier.arrive.drop.expect.tx.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
- declare void @llvm.nvvm.mbarrier.arrive.drop.expect.tx.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
+```llvm
+declare i64 @llvm.nvvm.mbarrier.arrive.drop.expect.tx.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
+declare i64 @llvm.nvvm.mbarrier.arrive.drop.expect.tx.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
+declare void @llvm.nvvm.mbarrier.arrive.drop.expect.tx.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
+declare void @llvm.nvvm.mbarrier.arrive.drop.expect.tx.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
- declare i64 @llvm.nvvm.mbarrier.arrive.drop.expect.tx.relaxed.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
- declare i64 @llvm.nvvm.mbarrier.arrive.drop.expect.tx.relaxed.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
- declare void @llvm.nvvm.mbarrier.arrive.drop.expect.tx.relaxed.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
- declare void @llvm.nvvm.mbarrier.arrive.drop.expect.tx.relaxed.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
+declare i64 @llvm.nvvm.mbarrier.arrive.drop.expect.tx.relaxed.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
+declare i64 @llvm.nvvm.mbarrier.arrive.drop.expect.tx.relaxed.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %tx_count)
+declare void @llvm.nvvm.mbarrier.arrive.drop.expect.tx.relaxed.scope.cta.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
+declare void @llvm.nvvm.mbarrier.arrive.drop.expect.tx.relaxed.scope.cluster.space.cluster(ptr addrspace(7) %addr, i32 %tx_count)
+```
-Overview:
-"""""""""
+##### Overview:
-The ``@llvm.nvvm.mbarrier.arrive.drop.expect.tx.*`` intrinsics perform
-the below operations on the mbarrier located at ``%addr``.
+The `@llvm.nvvm.mbarrier.arrive.drop.expect.tx.*` intrinsics perform
+the below operations on the mbarrier located at `%addr`.
-* Perform an ``expect-tx`` operation i.e. increase the transaction count
- of the mbarrier by ``%tx_count``, a 32-bit unsigned integer value.
-* Decrement the expected arrival count of the mbarrier by 1.
-* Perform an ``arrive`` operation on the mbarrier with a value of 1.
+- Perform an `expect-tx` operation i.e. increase the transaction count
+ of the mbarrier by `%tx_count`, a 32-bit unsigned integer value.
+- Decrement the expected arrival count of the mbarrier by 1.
+- Perform an `arrive` operation on the mbarrier with a value of 1.
-Semantics:
-""""""""""
+##### Semantics:
The semantics of these intrinsics are identical to those of the
-``llvm.nvvm.mbarrier.arrive.*`` intrinsics described above.
+`llvm.nvvm.mbarrier.arrive.*` intrinsics described above.
-'``llvm.nvvm.mbarrier.test.wait``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.mbarrier.test.wait`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare i1 @llvm.nvvm.mbarrier.test.wait.scope.cta.space.cta(ptr addrspace(3) %addr, i64 %state)
+declare i1 @llvm.nvvm.mbarrier.test.wait.scope.cluster.space.cta(ptr addrspace(3) %addr, i64 %state)
+declare i1 @llvm.nvvm.mbarrier.test.wait.parity.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %phase)
+declare i1 @llvm.nvvm.mbarrier.test.wait.parity.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %phase)
- declare i1 @llvm.nvvm.mbarrier.test.wait.scope.cta.space.cta(ptr addrspace(3) %addr, i64 %state)
- declare i1 @llvm.nvvm.mbarrier.test.wait.scope.cluster.space.cta(ptr addrspace(3) %addr, i64 %state)
- declare i1 @llvm.nvvm.mbarrier.test.wait.parity.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %phase)
- declare i1 @llvm.nvvm.mbarrier.test.wait.parity.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %phase)
+declare i1 @llvm.nvvm.mbarrier.test.wait.relaxed.scope.cta.space.cta(ptr addrspace(3) %addr, i64 %state)
+declare i1 @llvm.nvvm.mbarrier.test.wait.relaxed.scope.cluster.space.cta(ptr addrspace(3) %addr, i64 %state)
+declare i1 @llvm.nvvm.mbarrier.test.wait.parity.relaxed.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %phase)
+declare i1 @llvm.nvvm.mbarrier.test.wait.parity.relaxed.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %phase)
+```
- declare i1 @llvm.nvvm.mbarrier.test.wait.relaxed.scope.cta.space.cta(ptr addrspace(3) %addr, i64 %state)
- declare i1 @llvm.nvvm.mbarrier.test.wait.relaxed.scope.cluster.space.cta(ptr addrspace(3) %addr, i64 %state)
- declare i1 @llvm.nvvm.mbarrier.test.wait.parity.relaxed.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %phase)
- declare i1 @llvm.nvvm.mbarrier.test.wait.parity.relaxed.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %phase)
+##### Overview:
-Overview:
-"""""""""
-
-The ``@llvm.nvvm.mbarrier.test.wait.*`` intrinsics test for the completion
+The `@llvm.nvvm.mbarrier.test.wait.*` intrinsics test for the completion
of the current or the immediately preceding phase of an mbarrier object at
-``%addr``. The test for completion can be done with either the ``state`` or
-the ``phase-parity`` of the mbarrier object.
+`%addr`. The test for completion can be done with either the `state` or
+the `phase-parity` of the mbarrier object.
-* When done through the ``i64 %state`` operand, the state must be
- returned by an ``llvm.nvvm.mbarrier.arrive.*`` on the _same_
+- When done through the `i64 %state` operand, the state must be
+ returned by an `llvm.nvvm.mbarrier.arrive.*` on the \_same\_
mbarrier object.
-* The ``.parity`` variant of these intrinsics test for completion
- of the phase indicated by the operand ``i32 %phase``, which is
+- The `.parity` variant of these intrinsics test for completion
+ of the phase indicated by the operand `i32 %phase`, which is
the integer parity of either the current phase or the immediately
preceding phase of the mbarrier object. An even phase has integer
parity 0 and an odd phase has integer parity of 1. So the valid
values for phase-parity are 0 and 1.
-Semantics:
-""""""""""
+##### Semantics:
-The ``.scope.{cta/cluster}`` denotes the set of threads that the
+The `.scope.{cta/cluster}` denotes the set of threads that the
test_wait operation can directly synchronize with.
-If the ``addr`` does not fall within shared::cta space, then the
+If the `addr` does not fall within shared::cta space, then the
the behavior of this intrinsic is undefined.
-These intrinsics have ``acquire`` semantics by default. This acquire
+These intrinsics have `acquire` semantics by default. This acquire
pattern establishes memory ordering for operations occurring in program
-order after this ``test_wait`` instruction by making operations from
+order after this `test_wait` instruction by making operations from
other threads in the CTA (or cluster, depending on scope) visible to
subsequent operations in the current thread. When this wait completes,
it synchronizes with the corresponding release pattern from the
-``mbarrier.arrive`` operation. The ``relaxed`` variants of these intrinsics
+`mbarrier.arrive` operation. The `relaxed` variants of these intrinsics
do not provide any memory ordering or visibility guarantees.
-This ``test.wait`` intrinsic is non-blocking and immediately returns
+This `test.wait` intrinsic is non-blocking and immediately returns
the completion status without suspending the executing thread.
The boolean return value indicates:
-* True: The immediately preceding phase has completed
-* False: The current phase is still incomplete
+- True: The immediately preceding phase has completed
+- False: The current phase is still incomplete
When this wait returns true, the following ordering guarantees hold:
-* All memory accesses (except async operations) requested prior to
- ``mbarrier.arrive`` having release semantics by participating
+- All memory accesses (except async operations) requested prior to
+ `mbarrier.arrive` having release semantics by participating
threads of a CTA (or cluster, depending on scope) are visible to
the executing thread.
-* All ``cp.async`` operations requested prior to ``cp.async.mbarrier.arrive``
+- All `cp.async` operations requested prior to `cp.async.mbarrier.arrive`
by participating threads of a CTA are visible to the executing thread.
-* All ``cp.async.bulk`` operations using the same mbarrier object requested
- prior to ``mbarrier.arrive`` having release semantics by participating CTA
+- All `cp.async.bulk` operations using the same mbarrier object requested
+ prior to `mbarrier.arrive` having release semantics by participating CTA
threads are visible to the executing thread.
-* Memory accesses requested after this wait are not visible to memory
- accesses performed prior to ``mbarrier.arrive`` by other participating
+- Memory accesses requested after this wait are not visible to memory
+ accesses performed prior to `mbarrier.arrive` by other participating
threads.
-* No ordering guarantee exists for memory accesses by the same thread
- between an ``mbarrier.arrive`` and this wait.
-
-'``llvm.nvvm.mbarrier.try.wait``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+- No ordering guarantee exists for memory accesses by the same thread
+ between an `mbarrier.arrive` and this wait.
-Syntax:
-"""""""
+#### '`llvm.nvvm.mbarrier.try.wait`'
-.. code-block:: llvm
+##### Syntax:
- declare i1 @llvm.nvvm.mbarrier.try.wait{.relaxed}.scope.cta.space.cta(ptr addrspace(3) %addr, i64 %state)
- declare i1 @llvm.nvvm.mbarrier.try.wait{.relaxed}.scope.cluster.space.cta(ptr addrspace(3) %addr, i64 %state)
+```llvm
+declare i1 @llvm.nvvm.mbarrier.try.wait{.relaxed}.scope.cta.space.cta(ptr addrspace(3) %addr, i64 %state)
+declare i1 @llvm.nvvm.mbarrier.try.wait{.relaxed}.scope.cluster.space.cta(ptr addrspace(3) %addr, i64 %state)
- declare i1 @llvm.nvvm.mbarrier.try.wait.parity{.relaxed}.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %phase)
- declare i1 @llvm.nvvm.mbarrier.try.wait.parity{.relaxed}.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %phase)
+declare i1 @llvm.nvvm.mbarrier.try.wait.parity{.relaxed}.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %phase)
+declare i1 @llvm.nvvm.mbarrier.try.wait.parity{.relaxed}.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %phase)
- declare i1 @llvm.nvvm.mbarrier.try.wait.tl{.relaxed}.scope.cta.space.cta(ptr addrspace(3) %addr, i64 %state, i32 %timelimit)
- declare i1 @llvm.nvvm.mbarrier.try.wait.tl{.relaxed}.scope.cluster.space.cta(ptr addrspace(3) %addr, i64 %state, i32 %timelimit)
+declare i1 @llvm.nvvm.mbarrier.try.wait.tl{.relaxed}.scope.cta.space.cta(ptr addrspace(3) %addr, i64 %state, i32 %timelimit)
+declare i1 @llvm.nvvm.mbarrier.try.wait.tl{.relaxed}.scope.cluster.space.cta(ptr addrspace(3) %addr, i64 %state, i32 %timelimit)
- declare i1 @llvm.nvvm.mbarrier.try.wait.parity.tl{.relaxed}.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %phase, i32 %timelimit)
- declare i1 @llvm.nvvm.mbarrier.try.wait.parity.tl{.relaxed}.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %phase, i32 %timelimit)
+declare i1 @llvm.nvvm.mbarrier.try.wait.parity.tl{.relaxed}.scope.cta.space.cta(ptr addrspace(3) %addr, i32 %phase, i32 %timelimit)
+declare i1 @llvm.nvvm.mbarrier.try.wait.parity.tl{.relaxed}.scope.cluster.space.cta(ptr addrspace(3) %addr, i32 %phase, i32 %timelimit)
+```
-Overview:
-"""""""""
+##### Overview:
-The ``@llvm.nvvm.mbarrier.try.wait.*`` intrinsics test for the completion of
-the current or immediately preceding phase of an mbarrier object at ``%addr``.
-Unlike the ``test.wait`` intrinsics, which perform a non-blocking test, these
+The `@llvm.nvvm.mbarrier.try.wait.*` intrinsics test for the completion of
+the current or immediately preceding phase of an mbarrier object at `%addr`.
+Unlike the `test.wait` intrinsics, which perform a non-blocking test, these
intrinsics may block the executing thread until the specified phase completes
or a system-dependent time limit expires. Suspended threads resume execution
when the phase completes or the time limit elapses. This time limit is
-configurable through the ``.tl`` variants of these intrinsics, where the
-``%timelimit`` operand (an unsigned integer) specifies the limit in
-nanoseconds. Other semantics are identical to those of the ``test.wait``
+configurable through the `.tl` variants of these intrinsics, where the
+`%timelimit` operand (an unsigned integer) specifies the limit in
+nanoseconds. Other semantics are identical to those of the `test.wait`
intrinsics described above.
-Electing a thread
------------------
-
-'``llvm.nvvm.elect.sync``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+### Electing a thread
-Syntax:
-"""""""
+#### '`llvm.nvvm.elect.sync`'
-.. code-block:: llvm
+##### Syntax:
- declare {i32, i1} @llvm.nvvm.elect.sync(i32 %membermask)
+```llvm
+declare {i32, i1} @llvm.nvvm.elect.sync(i32 %membermask)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.elect.sync``' intrinsic generates the ``elect.sync``
+The '`@llvm.nvvm.elect.sync`' intrinsic generates the `elect.sync`
PTX instruction, which elects one predicated active leader thread from
-a set of threads specified by ``membermask``. The behavior is undefined
-if the executing thread is not in ``membermask``. The laneid of the
+a set of threads specified by `membermask`. The behavior is undefined
+if the executing thread is not in `membermask`. The laneid of the
elected thread is captured in the i32 return value. The i1 return
-value is set to ``True`` for the leader thread and ``False`` for all
+value is set to `True` for the leader thread and `False` for all
the other threads. Election of a leader thread happens deterministically,
-i.e. the same leader thread is elected for the same ``membermask``
-every time. For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-elect-sync>`__.
+i.e. the same leader thread is elected for the same `membermask`
+every time. For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-elect-sync).
-Membar/Fences
--------------
+### Membar/Fences
-'``llvm.nvvm.fence.acquire/release.sync_restrict.*``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.fence.acquire/release.sync_restrict.*`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.fence.acquire.sync_restrict.space.cluster.scope.cluster()
+declare void @llvm.nvvm.fence.release.sync_restrict.space.cta.scope.cluster()
+```
- declare void @llvm.nvvm.fence.acquire.sync_restrict.space.cluster.scope.cluster()
- declare void @llvm.nvvm.fence.release.sync_restrict.space.cta.scope.cluster()
-
-Overview:
-"""""""""
+##### Overview:
The `nvvm.fence.{semantics}.sync_restrict.*` restrict the class of memory
operations for which the fence instruction provides the memory ordering
@@ -914,79 +835,66 @@ to operations performed on objects in `shared_cta` space. Likewise, when
`sync_restrict` is restricted to `shared_cluster`, then memory semantics must be
`acquire` and the effect of the fence operation only applies to operations
performed on objects in `shared_cluster` memory space. The scope for both
-operations is `cluster`. For more details, please refer the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-membar>`__.
-
-'``llvm.nvvm.fence.mbarrier_init.release.cluster``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+operations is `cluster`. For more details, please refer the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-membar).
-Syntax:
-"""""""
+#### '`llvm.nvvm.fence.mbarrier_init.release.cluster`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.fence.mbarrier_init.release.cluster()
+```llvm
+declare void @llvm.nvvm.fence.mbarrier_init.release.cluster()
+```
-Overview:
-"""""""""
+##### Overview:
`nvvm.fence.mbarrier_init.release.cluster` intrinsic restrict the class of
memory operations for which the fence instruction provides the memory ordering
guarantees. The `mbarrier_init` modifiers restricts the synchronizing effect to
the prior `mbarrier_init` operation executed by the same thread on mbarrier
-objects in `shared_cta` memory space. For more details, please refer the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-membar>`__
+objects in `shared_cta` memory space. For more details, please refer the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-membar)
-'``llvm.nvvm.fence.proxy.async_generic.acquire/release.sync_restrict``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.fence.proxy.async_generic.acquire/release.sync_restrict`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.fence.proxy.async_generic.acquire.sync_restrict.space.cluster.scope.cluster()
+declare void @llvm.nvvm.fence.proxy.async_generic.release.sync_restrict.space.cta.scope.cluster()
+```
- declare void @llvm.nvvm.fence.proxy.async_generic.acquire.sync_restrict.space.cluster.scope.cluster()
- declare void @llvm.nvvm.fence.proxy.async_generic.release.sync_restrict.space.cta.scope.cluster()
-
-Overview:
-"""""""""
+##### Overview:
`nvvm.fence.proxy.async_generic.{semantics}.sync_restrict` are used to establish
-ordering between a prior memory access performed via the `async proxy
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#proxies>`__
+ordering between a prior memory access performed via the [async proxy](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#proxies)
and a subsequent memory access performed via the generic proxy.
-``nvvm.fence.proxy.async_generic.release.sync_restrict`` can form a release
+`nvvm.fence.proxy.async_generic.release.sync_restrict` can form a release
sequence that synchronizes with an acquire sequence that contains the
-``nvvm.fence.proxy.async_generic.acquire.sync_restrict`` proxy fence. When
+`nvvm.fence.proxy.async_generic.acquire.sync_restrict` proxy fence. When
`.sync_restrict` is restricted to `shared_cta`, then memory semantics must
be `release` and the effect of the fence operation only applies to operations
performed on objects in `shared_cta` space. Likewise, when `sync_restrict` is
restricted to `shared_cluster`, then memory semantics must be `acquire` and the
effect of the fence operation only applies to operations performed on objects in
`shared_cluster` memory space. The scope for both operations is `cluster`.
-For more details, please refer the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-membar>`__
-
-'``llvm.nvvm.fence.proxy.<proxykind>``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more details, please refer the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-membar)
-Syntax:
-"""""""
+#### '`llvm.nvvm.fence.proxy.<proxykind>`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.fence.proxy.alias()
- declare void @llvm.nvvm.fence.proxy.async()
- declare void @llvm.nvvm.fence.proxy.async.global()
- declare void @llvm.nvvm.fence.proxy.async.shared_cluster()
- declare void @llvm.nvvm.fence.proxy.async.shared_cta()
+```llvm
+declare void @llvm.nvvm.fence.proxy.alias()
+declare void @llvm.nvvm.fence.proxy.async()
+declare void @llvm.nvvm.fence.proxy.async.global()
+declare void @llvm.nvvm.fence.proxy.async.shared_cluster()
+declare void @llvm.nvvm.fence.proxy.async.shared_cta()
+```
-Overview:
-"""""""""
+##### Overview:
`nvvm.fence.proxy.{proxykind}` intrinsics represent a fence with bi-directional
proxy ordering that is established between the memory accesses done between the
-`generic proxy <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#proxies>`__
+[generic proxy](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#proxies)
and the proxy specified by `proxykind`. A `bi-directional proxy` ordering between
two proxykinds establishes two `uni-directional` proxy orderings: one from the
first proxykind to the second proxykind and the other from the second proxykind
@@ -1000,716 +908,626 @@ addresses to the same memory location
operations performed on objects in the state space specified (`generic`, `global`,
`shared_cluster`, `shared_cta`). If no state space is specified, then the memory
ordering applies on all state spaces. For more details, please refer the
-`PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-membar>`__
+[PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-membar)
-'``llvm.nvvm.fence.proxy.tensormap_generic.*``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.fence.proxy.tensormap_generic.*`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.fence.proxy.tensormap_generic.release.cta()
+declare void @llvm.nvvm.fence.proxy.tensormap_generic.release.cluster()
+declare void @llvm.nvvm.fence.proxy.tensormap_generic.release.gpu()
+declare void @llvm.nvvm.fence.proxy.tensormap_generic.release.sys()
- declare void @llvm.nvvm.fence.proxy.tensormap_generic.release.cta()
- declare void @llvm.nvvm.fence.proxy.tensormap_generic.release.cluster()
- declare void @llvm.nvvm.fence.proxy.tensormap_generic.release.gpu()
- declare void @llvm.nvvm.fence.proxy.tensormap_generic.release.sys()
+declare void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.cta(ptr %addr, i32 %size)
+declare void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.cluster(ptr %addr, i32 %size)
+declare void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.gpu(ptr %addr, i32 %size)
+declare void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.sys(ptr %addr, i32 %size)
+```
- declare void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.cta(ptr %addr, i32 %size)
- declare void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.cluster(ptr %addr, i32 %size)
- declare void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.gpu(ptr %addr, i32 %size)
- declare void @llvm.nvvm.fence.proxy.tensormap_generic.acquire.sys(ptr %addr, i32 %size)
+##### Overview:
-Overview:
-"""""""""
-
-The ``@llvm.nvvm.fence.proxy.tensormap_generic.*`` is a uni-directional fence
+The `@llvm.nvvm.fence.proxy.tensormap_generic.*` is a uni-directional fence
used to establish ordering between a prior memory access performed via the
-generic `proxy <https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#proxies>`_
+generic [proxy](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#proxies)
and a subsequent memory access performed via the tensormap proxy.
-``nvvm.fence.proxy.tensormap_generic.release`` can form a release sequence that
+`nvvm.fence.proxy.tensormap_generic.release` can form a release sequence that
synchronizes with an acquire sequence that contains the
-``nvvm.fence.proxy.tensormap_generic.acquire`` proxy fence. The following table
+`nvvm.fence.proxy.tensormap_generic.acquire` proxy fence. The following table
describes the mapping between LLVM Intrinsic and the PTX instruction:
- ====================================================== =========================================================
- NVVM Intrinsic PTX Instruction
- ====================================================== =========================================================
- ``@llvm.nvvm.fence.proxy.tensormap_generic.release.*`` ``fence.proxy.tensormap::generic.release.*``
- ``@llvm.nvvm.fence.proxy.tensormap_generic.acquire.*`` ``fence.proxy.tensormap::generic.acquire.* [addr], size``
- ====================================================== =========================================================
+| NVVM Intrinsic | PTX Instruction |
+| ---------------------------------------------------- | ------------------------------------------------------- |
+| `@llvm.nvvm.fence.proxy.tensormap_generic.release.*` | `fence.proxy.tensormap::generic.release.*` |
+| `@llvm.nvvm.fence.proxy.tensormap_generic.acquire.*` | `fence.proxy.tensormap::generic.acquire.* [addr], size` |
-The address operand ``addr`` and the operand ``size`` together specify the
-memory range ``[addr, addr+size)`` on which the ordering guarantees on the
+The address operand `addr` and the operand `size` together specify the
+memory range `[addr, addr+size)` on which the ordering guarantees on the
memory accesses across the proxies is to be provided. The only supported value
-for the ``size`` operand is ``128`` and must be an immediate. Generic Addressing
+for the `size` operand is `128` and must be an immediate. Generic Addressing
is used unconditionally, and the address specified by the operand addr must fall
-within the ``.global`` state space. Otherwise, the behavior is undefined. For
-more information, see `PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/#parallel-synchronization-and-communication-instructions-membar>`__.
-
-Address Space Intrinsics
-------------------------
+within the `.global` state space. Otherwise, the behavior is undefined. For
+more information, see [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#parallel-synchronization-and-communication-instructions-membar).
-'``llvm.nvvm.isspacep.*``' Intrinsics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+### Address Space Intrinsics
-Syntax:
-"""""""
+#### '`llvm.nvvm.isspacep.*`' Intrinsics
-.. code-block:: llvm
+##### Syntax:
- declare i1 @llvm.nvvm.isspacep.const(ptr %p)
- declare i1 @llvm.nvvm.isspacep.global(ptr %p)
- declare i1 @llvm.nvvm.isspacep.local(ptr %p)
- declare i1 @llvm.nvvm.isspacep.shared(ptr %p)
- declare i1 @llvm.nvvm.isspacep.shared.cluster(ptr %p)
+```llvm
+declare i1 @llvm.nvvm.isspacep.const(ptr %p)
+declare i1 @llvm.nvvm.isspacep.global(ptr %p)
+declare i1 @llvm.nvvm.isspacep.local(ptr %p)
+declare i1 @llvm.nvvm.isspacep.shared(ptr %p)
+declare i1 @llvm.nvvm.isspacep.shared.cluster(ptr %p)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``llvm.nvvm.isspacep.*``' intrinsics determine whether the provided generic
+The '`llvm.nvvm.isspacep.*`' intrinsics determine whether the provided generic
pointer references memory which falls within a particular address space.
-Semantics:
-""""""""""
+##### Semantics:
If the given pointer in the generic address space refers to memory which falls
within the state space of the intrinsic (and therefore could be safely address
space casted to this space), 1 is returned, otherwise 0 is returned.
-'``llvm.nvvm.mapa.*``' Intrinsics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
+#### '`llvm.nvvm.mapa.*`' Intrinsics
-.. code-block:: llvm
+##### Syntax:
- declare ptr @llvm.nvvm.mapa(ptr %p, i32 %rank)
- declare ptr addrspace(7) @llvm.nvvm.mapa.shared.cluster(ptr addrspace(3) %p, i32 %rank)
+```llvm
+declare ptr @llvm.nvvm.mapa(ptr %p, i32 %rank)
+declare ptr addrspace(7) @llvm.nvvm.mapa.shared.cluster(ptr addrspace(3) %p, i32 %rank)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``llvm.nvvm.mapa.*``' intrinsics map a shared memory pointer ``p`` of
-another CTA with ``%rank`` to the current CTA. The ``llvm.nvvm.mapa`` form
+The '`llvm.nvvm.mapa.*`' intrinsics map a shared memory pointer `p` of
+another CTA with `%rank` to the current CTA. The `llvm.nvvm.mapa` form
expects a generic pointer to shared memory and returns a generic pointer to
-shared cluster memory. The ``llvm.nvvm.mapa.shared.cluster`` form expects a
+shared cluster memory. The `llvm.nvvm.mapa.shared.cluster` form expects a
pointer to shared memory and returns a pointer to shared cluster memory. They
-corresponds directly to the ``mapa`` and ``mapa.shared.cluster`` PTX
+corresponds directly to the `mapa` and `mapa.shared.cluster` PTX
instructions.
-Semantics:
-""""""""""
+##### Semantics:
If the given pointer in the generic address space refers to memory which falls
within the state space of the intrinsic (and therefore could be safely address
space casted to this space), 1 is returned, otherwise 0 is returned.
-Narrow Floating-Point Conversion intrinsics
--------------------------------------------
+### Narrow Floating-Point Conversion intrinsics
These intrinsics perform conversions involving narrow floating-point formats.
The following table describes the rounding modes used across these intrinsics:
-.. _narrow-fp-rounding-modes:
-
-.. table:: Narrow Floating-Point Conversion Rounding Modes
- :widths: 30 60
-
- +-----------------------+---------------------------------------------------+
- | Rounding Mode | Description |
- +=======================+===================================================+
- |``rn`` (default) | Round to nearest, with ties to even |
- +-----------------------+---------------------------------------------------+
- |``rz`` | Round towards zero |
- +-----------------------+---------------------------------------------------+
- |``rp`` | Round towards positive infinity |
- +-----------------------+---------------------------------------------------+
- |``rs`` | Stochastic rounding which is achieved through the |
- | | use of the supplied random bits (``%rnd_bits``). |
- | | The result s rounded in the direction towards |
- | | zero or away from zero based on the carry out of |
- | | the integer addition of the of mantissa from |
- | | the input. |
- +-----------------------+---------------------------------------------------+
-
-.. _scale-factor:
-
-Some conversions involve a scale factor which is provided as a packed 16-bit
-integer containing two scaling factors of type ``ue8m0``, one for each input.
-For down conversion, inputs are divided by ``scale_factor`` and then the
-conversion is performed. For up-conversion, inputs are converted to destination
-type and then multiplied by ``scale_factor``.
-
-``fp8`` Conversion Intrinsics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
-
-.. code-block:: llvm
-
- declare i16 @llvm.nvvm.ff.to{.e4m3x2, .e5m2x2}.rn{.relu}(float %a, float %b)
- declare i16 @llvm.nvvm.ff.to.ue8m0x2{.rz, .rp}{.satfinite}(float %a, float %b)
- declare i16 @llvm.f16x2.to{.e4m3x2, .e5m2x2}.rn{.relu}(<2 x half> %a)
- declare i16 @llvm.bf16x2.to{.e4m3x2, .e5m2x2}.rn{.relu}.satfinite(<2 x bfloat> %a)
- declare i16 @llvm.bf16x2.to.ue8m0x2{.rz, .rp}{.satfinite}(<2 x bfloat> %a)
- declare <2 x half> @llvm.nvvm{.e4m3x2, .e5m2x2}.to.f16x2.rn{.relu}(i16 %a)
- declare <2 x bfloat> @llvm.nvvm{.e4m3x2, .e5m2x2}.to.bf16x2.rn{.relu}{.satfinite}.scale.n2.ue8m0(i16 %a, i16 %scale_factor)
- declare <2 x bfloat> @llvm.nvvm.ue8m0x2.to.bf16x2(i16 %a)
- declare <4 x i8> @llvm.nvvm.f32x4.to{.e4m3x4, .e5m2x4}.rs{.relu}.satfinite(<4 x f32> %a, i32 %rnd_bits)
-
-Overview:
-"""""""""
-
-These intrinsics perform conversions involving the ``e4m3`` and ``e5m2`` narrow
-floating-point formats. In case of two inputs, the value converted from input
-``%a`` is stored in the upper 8-bits of the result, and the value converted
-from input ``%b`` is stored in the lower 8-bits of the result.
-
-For rounding modes, see :ref:`narrow-fp-rounding-modes`.
-
-The ``relu`` modifier clamps negative results to 0.
-
-When ``satfinite`` is specified, if the absolute value of input (ignoring sign)
-is greater than ``MAX_NORM`` of the specified destination format, then the
-result is sign-preserved ``MAX_NORM`` of the destination format and a positive
-``MAX_NORM`` in ``.ue8m0x2`` for which the destination sign is not supported.
-Also, if the input value is ``NaN``, then the result is ``NaN`` in the
-specified destination format. The ``satfinite`` modifier is assumed to be
-present for conversions involving ``e4m3`` and ``e5m2`` types as the
+(narrow-fp-rounding-modes)=
+
+```{list-table} Narrow Floating-Point Conversion Rounding Modes
+:widths: 30 60
+:header-rows: 1
+
+ * - Rounding Mode
+ - Description
+ * - `rn` (default)
+ - Round to nearest, with ties to even
+ * - `rz`
+ - Round towards zero
+ * - `rp`
+ - Round towards positive infinity
+ * - `rs`
+ - Stochastic rounding which is achieved through the use of the supplied
+ random bits (`%rnd_bits`). The result s rounded in the direction towards
+ zero or away from zero based on the carry out of the integer addition of
+ the of mantissa from the input.
+```
+
+(scale-factor)=
+
+Some conversions involve a scale factor which is provided as a packed 16-bit
+integer containing two scaling factors of type `ue8m0`, one for each input.
+For down conversion, inputs are divided by `scale_factor` and then the
+conversion is performed. For up-conversion, inputs are converted to destination
+type and then multiplied by `scale_factor`.
+
+#### `fp8` Conversion Intrinsics
+
+##### Syntax:
+
+```llvm
+declare i16 @llvm.nvvm.ff.to{.e4m3x2, .e5m2x2}.rn{.relu}(float %a, float %b)
+declare i16 @llvm.nvvm.ff.to.ue8m0x2{.rz, .rp}{.satfinite}(float %a, float %b)
+declare i16 @llvm.f16x2.to{.e4m3x2, .e5m2x2}.rn{.relu}(<2 x half> %a)
+declare i16 @llvm.bf16x2.to{.e4m3x2, .e5m2x2}.rn{.relu}.satfinite(<2 x bfloat> %a)
+declare i16 @llvm.bf16x2.to.ue8m0x2{.rz, .rp}{.satfinite}(<2 x bfloat> %a)
+declare <2 x half> @llvm.nvvm{.e4m3x2, .e5m2x2}.to.f16x2.rn{.relu}(i16 %a)
+declare <2 x bfloat> @llvm.nvvm{.e4m3x2, .e5m2x2}.to.bf16x2.rn{.relu}{.satfinite}.scale.n2.ue8m0(i16 %a, i16 %scale_factor)
+declare <2 x bfloat> @llvm.nvvm.ue8m0x2.to.bf16x2(i16 %a)
+declare <4 x i8> @llvm.nvvm.f32x4.to{.e4m3x4, .e5m2x4}.rs{.relu}.satfinite(<4 x f32> %a, i32 %rnd_bits)
+```
+
+##### Overview:
+
+These intrinsics perform conversions involving the `e4m3` and `e5m2` narrow
+floating-point formats. In case of two inputs, the value converted from input
+`%a` is stored in the upper 8-bits of the result, and the value converted
+from input `%b` is stored in the lower 8-bits of the result.
+
+For rounding modes, see {ref}`narrow-fp-rounding-modes`.
+
+The `relu` modifier clamps negative results to 0.
+
+When `satfinite` is specified, if the absolute value of input (ignoring sign)
+is greater than `MAX_NORM` of the specified destination format, then the
+result is sign-preserved `MAX_NORM` of the destination format and a positive
+`MAX_NORM` in `.ue8m0x2` for which the destination sign is not supported.
+Also, if the input value is `NaN`, then the result is `NaN` in the
+specified destination format. The `satfinite` modifier is assumed to be
+present for conversions involving `e4m3` and `e5m2` types as the
destination.
-For scale factor, see :ref:`scale-factor <scale-factor>`.
+For scale factor, see {ref}`scale-factor <scale-factor>`.
-For more information, see `PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-cvt>`__.
+For more information, see [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-cvt).
-``s2f6`` Conversion Intrinsics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### `s2f6` Conversion Intrinsics
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare i16 @llvm.nvvm.ff.to.s2f6x2.rn{.relu}.satfinite.scale.n2.ue8m0(float %a, float %b, i16 %scale_factor)
+declare i16 @llvm.nvvm.bf16x2.to.s2f6x2.rn{.relu}.satfinite.scale.n2.ue8m0(<2 x bfloat> %a, i16 %scale_factor)
+declare <2 x bfloat> @llvm.nvvm.s2f6x2.to.bf16x2.rn{.relu}{.satfinite}.scale.n2.ue8m0(i16 %a, i16 %scale_factor)
+```
- declare i16 @llvm.nvvm.ff.to.s2f6x2.rn{.relu}.satfinite.scale.n2.ue8m0(float %a, float %b, i16 %scale_factor)
- declare i16 @llvm.nvvm.bf16x2.to.s2f6x2.rn{.relu}.satfinite.scale.n2.ue8m0(<2 x bfloat> %a, i16 %scale_factor)
- declare <2 x bfloat> @llvm.nvvm.s2f6x2.to.bf16x2.rn{.relu}{.satfinite}.scale.n2.ue8m0(i16 %a, i16 %scale_factor)
+##### Overview:
-Overview:
-"""""""""
+These intrinsics perform conversions involving the `s2f6` narrow
+floating-point format. In case of two inputs, the value converted from input
+`%a` is stored in the upper 8-bits of the result, and the value converted
+from input `%b` is stored in the lower 8-bits of the result.
-These intrinsics perform conversions involving the ``s2f6`` narrow
-floating-point format. In case of two inputs, the value converted from input
-``%a`` is stored in the upper 8-bits of the result, and the value converted
-from input ``%b`` is stored in the lower 8-bits of the result.
+For rounding modes, see {ref}`narrow-fp-rounding-modes`.
-For rounding modes, see :ref:`narrow-fp-rounding-modes`.
+The `relu` modifier clamps negative results to 0.
-The ``relu`` modifier clamps negative results to 0.
-
-When ``satfinite`` is specified, if the absolute value of input (ignoring sign)
-is greater than ``MAX_NORM`` of the specified destination format, then the
-result is sign-preserved ``MAX_NORM`` of the destination format. Also, if the
-input is ``NaN``, then the result is the positive ``MAX_NORM`` of the
+When `satfinite` is specified, if the absolute value of input (ignoring sign)
+is greater than `MAX_NORM` of the specified destination format, then the
+result is sign-preserved `MAX_NORM` of the destination format. Also, if the
+input is `NaN`, then the result is the positive `MAX_NORM` of the
destination format.
-For scale factor, see :ref:`scale-factor <scale-factor>`.
+For scale factor, see {ref}`scale-factor <scale-factor>`.
-For more information, see `PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-cvt>`__.
+For more information, see [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-cvt).
-``fp6`` Conversion Intrinsics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### `fp6` Conversion Intrinsics
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare i16 @llvm.nvvm.ff.to{.e2m3x2, .e3m2x2}.rn{.relu}.satfinite(float %a, float %b)
+declare i16 @llvm.nvvm.f16x2.to{.e2m3x2, .e3m2x2}.rn{.relu}.satfinite(<2 x half> %a)
+declare i16 @llvm.nvvm.bf16x2.to{.e2m3x2, .e3m2x2}.rn{.relu}.satfinite(<2 x bfloat> %a)
+declare <2 x half> @llvm.nvvm{.e2m3x2, .e3m2x2}.to.f16x2.rn{.relu}(i16 %a)
+declare <2 x bfloat> @llvm.nvvm{.e2m3x2, .e3m2x2}.to.bf16x2.rn{.relu}{.satfinite}.scale.n2.ue8m0(i16 %a, i16 %scale_factor)
+declare <4 x i8> @llvm.nvvm.f32x4.to{.e2m3x4, .e3m2x4}.rs{.relu}.satfinite(<4 x f32> %a, i32 %rnd_bits)
+```
- declare i16 @llvm.nvvm.ff.to{.e2m3x2, .e3m2x2}.rn{.relu}.satfinite(float %a, float %b)
- declare i16 @llvm.nvvm.f16x2.to{.e2m3x2, .e3m2x2}.rn{.relu}.satfinite(<2 x half> %a)
- declare i16 @llvm.nvvm.bf16x2.to{.e2m3x2, .e3m2x2}.rn{.relu}.satfinite(<2 x bfloat> %a)
- declare <2 x half> @llvm.nvvm{.e2m3x2, .e3m2x2}.to.f16x2.rn{.relu}(i16 %a)
- declare <2 x bfloat> @llvm.nvvm{.e2m3x2, .e3m2x2}.to.bf16x2.rn{.relu}{.satfinite}.scale.n2.ue8m0(i16 %a, i16 %scale_factor)
- declare <4 x i8> @llvm.nvvm.f32x4.to{.e2m3x4, .e3m2x4}.rs{.relu}.satfinite(<4 x f32> %a, i32 %rnd_bits)
-
-Overview:
-"""""""""
+##### Overview:
-These intrinsics perform conversions involving the ``e2m3`` and ``e3m2`` narrow
-floating-point formats. In case of two inputs, the value converted from input
-``%a`` is stored in the upper 8-bits of the result, and the value converted
-from input ``%b`` is stored in the lower 8-bits of the result with 2 MSBs
+These intrinsics perform conversions involving the `e2m3` and `e3m2` narrow
+floating-point formats. In case of two inputs, the value converted from input
+`%a` is stored in the upper 8-bits of the result, and the value converted
+from input `%b` is stored in the lower 8-bits of the result with 2 MSBs
padded with 0s in both cases.
-For rounding modes, see :ref:`narrow-fp-rounding-modes`.
+For rounding modes, see {ref}`narrow-fp-rounding-modes`.
-The ``relu`` modifier clamps negative results to 0.
+The `relu` modifier clamps negative results to 0.
-When ``satfinite`` is specified, if the absolute value of input (ignoring sign)
-is greater than ``MAX_NORM`` of the specified destination format, then the
-result is sign-preserved ``MAX_NORM`` of the destination format. Also, if the
-input is ``NaN``, then the result is the positive ``MAX_NORM`` of the
+When `satfinite` is specified, if the absolute value of input (ignoring sign)
+is greater than `MAX_NORM` of the specified destination format, then the
+result is sign-preserved `MAX_NORM` of the destination format. Also, if the
+input is `NaN`, then the result is the positive `MAX_NORM` of the
destination format.
-For scale factor, see :ref:`scale-factor <scale-factor>`.
-
-For more information, see `PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-cvt>`__.
+For scale factor, see {ref}`scale-factor <scale-factor>`.
-``fp4`` Conversion Intrinsics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information, see [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-cvt).
-Syntax:
-"""""""
+#### `fp4` Conversion Intrinsics
-.. code-block:: llvm
+##### Syntax:
- declare i16 @llvm.nvvm.ff.to.e2m1x2.rn{.relu}.satfinite(float %a, float %b)
- declare i16 @llvm.nvvm.f16x2.to.e2m1x2.rn{.relu}.satfinite(<2 x half> %a)
- declare i16 @llvm.nvvm.bf16x2.to.e2m1x2.rn{.relu}.satfinite(<2 x bfloat> %a)
- declare <2 x half> @llvm.nvvm.e2m1x2.to.f16x2.rn{.relu}(i16 %a)
- declare <2 x bfloat> @llvm.nvvm.e2m1x2.to.bf16x2.rn{.relu}{.satfinite}.scale.n2.ue8m0(i16 %a, i16 %scale_factor)
- declare i16 @llvm.nvvm.f32x4.to.e2m1x4.rs{.relu}.satfinite(<4 x f32> %a, i32 %rnd_bits)
+```llvm
+declare i16 @llvm.nvvm.ff.to.e2m1x2.rn{.relu}.satfinite(float %a, float %b)
+declare i16 @llvm.nvvm.f16x2.to.e2m1x2.rn{.relu}.satfinite(<2 x half> %a)
+declare i16 @llvm.nvvm.bf16x2.to.e2m1x2.rn{.relu}.satfinite(<2 x bfloat> %a)
+declare <2 x half> @llvm.nvvm.e2m1x2.to.f16x2.rn{.relu}(i16 %a)
+declare <2 x bfloat> @llvm.nvvm.e2m1x2.to.bf16x2.rn{.relu}{.satfinite}.scale.n2.ue8m0(i16 %a, i16 %scale_factor)
+declare i16 @llvm.nvvm.f32x4.to.e2m1x4.rs{.relu}.satfinite(<4 x f32> %a, i32 %rnd_bits)
+```
-Overview:
-"""""""""
+##### Overview:
-These intrinsics perform conversions involving the ``e2m1`` narrow
-floating-point format. For conversions involving ``e2m1x2``, the packed
-``e2m1x2`` value is stored in the lower byte of the ``i16`` argument or result.
-In case of two inputs, the value converted from input
-``%a`` is stored in the upper 4-bits of the result, and the value converted
-from input ``%b`` is stored in the lower 4-bits of the result.
+These intrinsics perform conversions involving the `e2m1` narrow
+floating-point format. For conversions involving `e2m1x2`, the packed
+`e2m1x2` value is stored in the lower byte of the `i16` argument or result.
+In case of two inputs, the value converted from input
+`%a` is stored in the upper 4-bits of the result, and the value converted
+from input `%b` is stored in the lower 4-bits of the result.
-For rounding modes, see :ref:`narrow-fp-rounding-modes`.
+For rounding modes, see {ref}`narrow-fp-rounding-modes`.
-The ``relu`` modifier clamps negative results to 0.
+The `relu` modifier clamps negative results to 0.
-When ``satfinite`` is specified, if the absolute value of input (ignoring sign)
-is greater than ``MAX_NORM`` of the specified destination format, then the
-result is sign-preserved ``MAX_NORM`` of the destination format. Also, if the
-input is ``NaN``, then the result is the positive ``MAX_NORM`` of the
+When `satfinite` is specified, if the absolute value of input (ignoring sign)
+is greater than `MAX_NORM` of the specified destination format, then the
+result is sign-preserved `MAX_NORM` of the destination format. Also, if the
+input is `NaN`, then the result is the positive `MAX_NORM` of the
destination format.
-For scale factor, see :ref:`scale-factor <scale-factor>`.
+For scale factor, see {ref}`scale-factor <scale-factor>`.
-For more information, see `PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-cvt>`__.
+For more information, see [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-cvt).
-Arithmetic Intrinsics
----------------------
+### Arithmetic Intrinsics
-'``llvm.nvvm.fabs.*``' Intrinsic
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.fabs.*`' Intrinsic
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare float @llvm.nvvm.fabs.f32(float %a)
+declare double @llvm.nvvm.fabs.f64(double %a)
+declare half @llvm.nvvm.fabs.f16(half %a)
+declare <2 x half> @llvm.nvvm.fabs.v2f16(<2 x half> %a)
+declare bfloat @llvm.nvvm.fabs.bf16(bfloat %a)
+declare <2 x bfloat> @llvm.nvvm.fabs.v2bf16(<2 x bfloat> %a)
+```
- declare float @llvm.nvvm.fabs.f32(float %a)
- declare double @llvm.nvvm.fabs.f64(double %a)
- declare half @llvm.nvvm.fabs.f16(half %a)
- declare <2 x half> @llvm.nvvm.fabs.v2f16(<2 x half> %a)
- declare bfloat @llvm.nvvm.fabs.bf16(bfloat %a)
- declare <2 x bfloat> @llvm.nvvm.fabs.v2bf16(<2 x bfloat> %a)
+##### Overview:
-Overview:
-"""""""""
+The '`llvm.nvvm.fabs.*`' intrinsics return the absolute value of the operand.
-The '``llvm.nvvm.fabs.*``' intrinsics return the absolute value of the operand.
+##### Semantics:
-Semantics:
-""""""""""
-
-Unlike, '``llvm.fabs.*``', these intrinsics do not perfectly preserve NaN
+Unlike, '`llvm.fabs.*`', these intrinsics do not perfectly preserve NaN
values. Instead, a NaN input yields an unspecified NaN output.
+#### '`llvm.nvvm.fabs.ftz.*`' Intrinsic
-'``llvm.nvvm.fabs.ftz.*``' Intrinsic
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare float @llvm.nvvm.fabs.ftz.f32(float %a)
+declare half @llvm.nvvm.fabs.ftz.f16(half %a)
+declare <2 x half> @llvm.nvvm.fabs.ftz.v2f16(<2 x half> %a)
+```
- declare float @llvm.nvvm.fabs.ftz.f32(float %a)
- declare half @llvm.nvvm.fabs.ftz.f16(half %a)
- declare <2 x half> @llvm.nvvm.fabs.ftz.v2f16(<2 x half> %a)
+##### Overview:
-Overview:
-"""""""""
-
-The '``llvm.nvvm.fabs.ftz.*``' intrinsics return the absolute value of the
+The '`llvm.nvvm.fabs.ftz.*`' intrinsics return the absolute value of the
operand, flushing subnormals to sign preserving zero.
-Semantics:
-""""""""""
+##### Semantics:
Before the absolute value is taken, the input is flushed to sign preserving
-zero if it is a subnormal. In addition, unlike '``llvm.fabs.*``', a NaN input
+zero if it is a subnormal. In addition, unlike '`llvm.fabs.*`', a NaN input
yields an unspecified NaN output.
+#### '`llvm.nvvm.idp2a.[us].[us]`' Intrinsics
-'``llvm.nvvm.idp2a.[us].[us]``' Intrinsics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
-
-.. code-block:: llvm
+##### Syntax:
- declare i32 @llvm.nvvm.idp2a.s.s(i32 %a, i32 %b, i1 immarg %is.hi, i32 %c)
- declare i32 @llvm.nvvm.idp2a.s.u(i32 %a, i32 %b, i1 immarg %is.hi, i32 %c)
- declare i32 @llvm.nvvm.idp2a.u.s(i32 %a, i32 %b, i1 immarg %is.hi, i32 %c)
- declare i32 @llvm.nvvm.idp2a.u.u(i32 %a, i32 %b, i1 immarg %is.hi, i32 %c)
+```llvm
+declare i32 @llvm.nvvm.idp2a.s.s(i32 %a, i32 %b, i1 immarg %is.hi, i32 %c)
+declare i32 @llvm.nvvm.idp2a.s.u(i32 %a, i32 %b, i1 immarg %is.hi, i32 %c)
+declare i32 @llvm.nvvm.idp2a.u.s(i32 %a, i32 %b, i1 immarg %is.hi, i32 %c)
+declare i32 @llvm.nvvm.idp2a.u.u(i32 %a, i32 %b, i1 immarg %is.hi, i32 %c)
+```
+##### Overview:
-Overview:
-"""""""""
-
-The '``llvm.nvvm.idp2a.[us].[us]``' intrinsics performs a 2-element vector dot
-product followed by addition. They corresponds directly to the ``dp2a`` PTX
+The '`llvm.nvvm.idp2a.[us].[us]`' intrinsics performs a 2-element vector dot
+product followed by addition. They corresponds directly to the `dp2a` PTX
instruction.
-Semantics:
-""""""""""
+##### Semantics:
-The 32-bit value in ``%a`` is broken into 2 16-bit values which are extended to
-32 bits. For the '``llvm.nvvm.idp2a.u.[us]``' variants zero-extension is used,
-while for the '``llvm.nvvm.idp2a.s.[us]``' sign-extension is used. Two bytes are
-selected from ``%b``, if ``%is.hi`` is true, the most significant bytes are
+The 32-bit value in `%a` is broken into 2 16-bit values which are extended to
+32 bits. For the '`llvm.nvvm.idp2a.u.[us]`' variants zero-extension is used,
+while for the '`llvm.nvvm.idp2a.s.[us]`' sign-extension is used. Two bytes are
+selected from `%b`, if `%is.hi` is true, the most significant bytes are
selected, otherwise the least significant bytes are selected. These bytes are
-then extended to 32-bits. For the '``llvm.nvvm.idp2a.[us].u``' variants
-zero-extension is used, while for the '``llvm.nvvm.idp2a.[us].s``'
+then extended to 32-bits. For the '`llvm.nvvm.idp2a.[us].u`' variants
+zero-extension is used, while for the '`llvm.nvvm.idp2a.[us].s`'
sign-extension is used. The dot product of these 2-element vectors is added to
-``%c`` to produce the return.
-
+`%c` to produce the return.
-'``llvm.nvvm.idp4a.[us].[us]``' Intrinsics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.idp4a.[us].[us]`' Intrinsics
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare i32 @llvm.nvvm.idp4a.s.s(i32 %a, i32 %b, i32 %c)
+declare i32 @llvm.nvvm.idp4a.s.u(i32 %a, i32 %b, i32 %c)
+declare i32 @llvm.nvvm.idp4a.u.s(i32 %a, i32 %b, i32 %c)
+declare i32 @llvm.nvvm.idp4a.u.u(i32 %a, i32 %b, i32 %c)
+```
- declare i32 @llvm.nvvm.idp4a.s.s(i32 %a, i32 %b, i32 %c)
- declare i32 @llvm.nvvm.idp4a.s.u(i32 %a, i32 %b, i32 %c)
- declare i32 @llvm.nvvm.idp4a.u.s(i32 %a, i32 %b, i32 %c)
- declare i32 @llvm.nvvm.idp4a.u.u(i32 %a, i32 %b, i32 %c)
+##### Overview:
-Overview:
-"""""""""
-
-The '``llvm.nvvm.idp4a.[us].[us]``' intrinsics perform a 4-element vector dot
-product followed by addition. They corresponds directly to the ``dp4a`` PTX
+The '`llvm.nvvm.idp4a.[us].[us]`' intrinsics perform a 4-element vector dot
+product followed by addition. They corresponds directly to the `dp4a` PTX
instruction.
-Semantics:
-""""""""""
-
-Each of the 4 bytes in both ``%a`` and ``%b`` are extended to 32-bit integers
-forming 2 ``<4 x i32>``. For ``%a``, zero-extension is used in the
-'``llvm.nvvm.idp4a.u.[us]``' variants, while sign-extension is used with
-'``llvm.nvvm.idp4a.s.[us]``' variants. Similarly, for ``%b``, zero-extension is
-used in the '``llvm.nvvm.idp4a.[us].u``' variants, while sign-extension is used
-with '``llvm.nvvm.idp4a.[us].s``' variants. The dot product of these 4-element
-vectors is added to ``%c`` to produce the return.
+##### Semantics:
-'``llvm.nvvm.add.*``' Half-precision Intrinsics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Each of the 4 bytes in both `%a` and `%b` are extended to 32-bit integers
+forming 2 `<4 x i32>`. For `%a`, zero-extension is used in the
+'`llvm.nvvm.idp4a.u.[us]`' variants, while sign-extension is used with
+'`llvm.nvvm.idp4a.s.[us]`' variants. Similarly, for `%b`, zero-extension is
+used in the '`llvm.nvvm.idp4a.[us].u`' variants, while sign-extension is used
+with '`llvm.nvvm.idp4a.[us].s`' variants. The dot product of these 4-element
+vectors is added to `%c` to produce the return.
-Syntax:
-"""""""
+#### '`llvm.nvvm.add.*`' Half-precision Intrinsics
-.. code-block:: llvm
+##### Syntax:
- declare half @llvm.nvvm.add.rn.sat.f16(half %a, half %b)
- declare <2 x half> @llvm.nvvm.add.rn.sat.v2f16(<2 x half> %a, <2 x half> %b)
+```llvm
+declare half @llvm.nvvm.add.rn.sat.f16(half %a, half %b)
+declare <2 x half> @llvm.nvvm.add.rn.sat.v2f16(<2 x half> %a, <2 x half> %b)
- declare half @llvm.nvvm.add.rn.ftz.sat.f16(half %a, half %b)
- declare <2 x half> @llvm.nvvm.add.rn.ftz.sat.v2f16(<2 x half> %a, <2 x half> %b)
+declare half @llvm.nvvm.add.rn.ftz.sat.f16(half %a, half %b)
+declare <2 x half> @llvm.nvvm.add.rn.ftz.sat.v2f16(<2 x half> %a, <2 x half> %b)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``llvm.nvvm.add.*``' intrinsics perform an addition operation with the
-specified rounding mode and modifiers.
+The '`llvm.nvvm.add.*`' intrinsics perform an addition operation with the
+specified rounding mode and modifiers.
-Semantics:
-""""""""""
+##### Semantics:
-The '``.sat``' modifier performs a saturating addition where the result is
-clamped to ``[0.0, 1.0]`` and ``NaN`` results are flushed to ``+0.0f``.
-The '``.ftz``' modifier flushes subnormal inputs and results to sign-preserving
+The '`.sat`' modifier performs a saturating addition where the result is
+clamped to `[0.0, 1.0]` and `NaN` results are flushed to `+0.0f`.
+The '`.ftz`' modifier flushes subnormal inputs and results to sign-preserving
zero.
-'``llvm.nvvm.mul.*``' Half-precision Intrinsics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.mul.*`' Half-precision Intrinsics
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare half @llvm.nvvm.mul.rn.sat.f16(half %a, half %b)
+declare <2 x half> @llvm.nvvm.mul.rn.sat.v2f16(<2 x half> %a, <2 x half> %b)
- declare half @llvm.nvvm.mul.rn.sat.f16(half %a, half %b)
- declare <2 x half> @llvm.nvvm.mul.rn.sat.v2f16(<2 x half> %a, <2 x half> %b)
+declare half @llvm.nvvm.mul.rn.ftz.sat.f16(half %a, half %b)
+declare <2 x half> @llvm.nvvm.mul.rn.ftz.sat.v2f16(<2 x half> %a, <2 x half> %b)
+```
- declare half @llvm.nvvm.mul.rn.ftz.sat.f16(half %a, half %b)
- declare <2 x half> @llvm.nvvm.mul.rn.ftz.sat.v2f16(<2 x half> %a, <2 x half> %b)
+##### Overview:
-Overview:
-"""""""""
+The '`llvm.nvvm.mul.*`' intrinsics perform a multiplication operation with
+the specified rounding mode and modifiers.
-The '``llvm.nvvm.mul.*``' intrinsics perform a multiplication operation with
-the specified rounding mode and modifiers.
+##### Semantics:
-Semantics:
-""""""""""
-
-The '``.sat``' modifier performs a saturating multiplication where the result is
-clamped to ``[0.0, 1.0]`` and ``NaN`` results are flushed to ``+0.0f``.
-The '``.ftz``' modifier flushes subnormal inputs and results to sign-preserving
+The '`.sat`' modifier performs a saturating multiplication where the result is
+clamped to `[0.0, 1.0]` and `NaN` results are flushed to `+0.0f`.
+The '`.ftz`' modifier flushes subnormal inputs and results to sign-preserving
zero.
-'``llvm.nvvm.fma.*``' Half-precision Intrinsics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
+#### '`llvm.nvvm.fma.*`' Half-precision Intrinsics
-.. code-block:: llvm
+##### Syntax:
- declare half @llvm.nvvm.fma.rn{.ftz}.f16(half %a, half %b, half %c)
- declare <2 x half> @llvm.nvvm.fma.rn{.ftz}.f16x2(<2 x half> %a, <2 x half> %b, <2 x half> %c)
- declare bfloat @llvm.nvvm.fma.rn.bf16(bfloat %a, bfloat %b, bfloat %c)
- declare <2 x bfloat> @llvm.nvvm.fma.rn.bf16x2(<2 x bfloat> %a, <2 x bfloat> %b, <2 x bfloat> %c)
+```llvm
+declare half @llvm.nvvm.fma.rn{.ftz}.f16(half %a, half %b, half %c)
+declare <2 x half> @llvm.nvvm.fma.rn{.ftz}.f16x2(<2 x half> %a, <2 x half> %b, <2 x half> %c)
+declare bfloat @llvm.nvvm.fma.rn.bf16(bfloat %a, bfloat %b, bfloat %c)
+declare <2 x bfloat> @llvm.nvvm.fma.rn.bf16x2(<2 x bfloat> %a, <2 x bfloat> %b, <2 x bfloat> %c)
- declare half @llvm.nvvm.fma.rn{.ftz}.sat.f16(half %a, half %b, half %c)
- declare <2 x half> @llvm.nvvm.fma.rn{.ftz}.sat.f16x2(<2 x half> %a, <2 x half> %b, <2 x half> %c)
+declare half @llvm.nvvm.fma.rn{.ftz}.sat.f16(half %a, half %b, half %c)
+declare <2 x half> @llvm.nvvm.fma.rn{.ftz}.sat.f16x2(<2 x half> %a, <2 x half> %b, <2 x half> %c)
- declare half @llvm.nvvm.fma.rn{.ftz}.relu.f16(half %a, half %b, half %c)
- declare <2 x half> @llvm.nvvm.fma.rn{.ftz}.relu.f16x2(<2 x half> %a, <2 x half> %b, <2 x half> %c)
- declare bfloat @llvm.nvvm.fma.rn.relu.bf16(bfloat %a, bfloat %b, bfloat %c)
- declare <2 x bfloat> @llvm.nvvm.fma.rn.relu.bf16x2(<2 x bfloat> %a, <2 x bfloat> %b, <2 x bfloat> %c)
+declare half @llvm.nvvm.fma.rn{.ftz}.relu.f16(half %a, half %b, half %c)
+declare <2 x half> @llvm.nvvm.fma.rn{.ftz}.relu.f16x2(<2 x half> %a, <2 x half> %b, <2 x half> %c)
+declare bfloat @llvm.nvvm.fma.rn.relu.bf16(bfloat %a, bfloat %b, bfloat %c)
+declare <2 x bfloat> @llvm.nvvm.fma.rn.relu.bf16x2(<2 x bfloat> %a, <2 x bfloat> %b, <2 x bfloat> %c)
- declare half @llvm.nvvm.fma.rn.oob{.relu}.f16(half %a, half %b, half %c)
- declare <2 x half> @llvm.nvvm.fma.rn.oob{.relu}.v2f16(<2 x half> %a, <2 x half> %b, <2 x half> %c)
- declare bfloat @llvm.nvvm.fma.rn.oob{.relu}.bf16(bfloat %a, bfloat %b, bfloat %c)
- declare <2 x bfloat> @llvm.nvvm.fma.rn.oob{.relu}.v2bf16(<2 x bfloat> %a, <2 x bfloat> %b, <2 x bfloat> %c)
+declare half @llvm.nvvm.fma.rn.oob{.relu}.f16(half %a, half %b, half %c)
+declare <2 x half> @llvm.nvvm.fma.rn.oob{.relu}.v2f16(<2 x half> %a, <2 x half> %b, <2 x half> %c)
+declare bfloat @llvm.nvvm.fma.rn.oob{.relu}.bf16(bfloat %a, bfloat %b, bfloat %c)
+declare <2 x bfloat> @llvm.nvvm.fma.rn.oob{.relu}.v2bf16(<2 x bfloat> %a, <2 x bfloat> %b, <2 x bfloat> %c)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``llvm.nvvm.fma.*``' intrinsics perform a fused multiply-add with no loss
+The '`llvm.nvvm.fma.*`' intrinsics perform a fused multiply-add with no loss
of precision in the intermediate product and addition.
-Semantics:
-""""""""""
+##### Semantics:
-The '``.sat``' modifier performs a saturating operation where the result is
-clamped to ``[0.0, 1.0]`` and ``NaN`` results are flushed to ``+0.0f``.
-The '``.ftz``' modifier flushes subnormal inputs and results to sign-preserving
+The '`.sat`' modifier performs a saturating operation where the result is
+clamped to `[0.0, 1.0]` and `NaN` results are flushed to `+0.0f`.
+The '`.ftz`' modifier flushes subnormal inputs and results to sign-preserving
zero.
-The '``.relu``' modifier clamps the result to ``0`` if negative and ``NaN``
-results are flushed to canonical ``NaN``.
-The '``.oob``' modifier clamps the result to ``0`` if either of the operands is
-an ``OOB NaN`` (defined under `Tensors <https://docs.nvidia.com/cuda/parallel-thread-execution/#tensors>`__) value.
-
-Bit Manipulation Intrinsics
----------------------------
+The '`.relu`' modifier clamps the result to `0` if negative and `NaN`
+results are flushed to canonical `NaN`.
+The '`.oob`' modifier clamps the result to `0` if either of the operands is
+an `OOB NaN` (defined under [Tensors](https://docs.nvidia.com/cuda/parallel-thread-execution/#tensors)) value.
-'``llvm.nvvm.fshl.clamp.*``' Intrinsic
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+### Bit Manipulation Intrinsics
-Syntax:
-"""""""
+#### '`llvm.nvvm.fshl.clamp.*`' Intrinsic
-.. code-block:: llvm
+##### Syntax:
- declare i32 @llvm.nvvm.fshl.clamp.i32(i32 %hi, i32 %lo, i32 %n)
+```llvm
+declare i32 @llvm.nvvm.fshl.clamp.i32(i32 %hi, i32 %lo, i32 %n)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``llvm.nvvm.fshl.clamp``' family of intrinsics performs a clamped funnel
-shift left. These intrinsics are very similar to '``llvm.fshl``', except the
+The '`llvm.nvvm.fshl.clamp`' family of intrinsics performs a clamped funnel
+shift left. These intrinsics are very similar to '`llvm.fshl`', except the
shift amount is clamped at the integer width (instead of modulo it). Currently,
-only ``i32`` is supported.
+only `i32` is supported.
-Semantics:
-""""""""""
+##### Semantics:
-The '``llvm.nvvm.fshl.clamp``' family of intrinsic functions performs a clamped
+The '`llvm.nvvm.fshl.clamp`' family of intrinsic functions performs a clamped
funnel shift left: the first two values are concatenated as { %hi : %lo } (%hi
is the most significant bits of the wide value), the combined value is shifted
left, and the most significant bits are extracted to produce a result that is
the same size as the original arguments. The shift amount is the minimum of the
value of %n and the bit width of the integer type.
-'``llvm.nvvm.fshr.clamp.*``' Intrinsic
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.fshr.clamp.*`' Intrinsic
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare i32 @llvm.nvvm.fshr.clamp.i32(i32 %hi, i32 %lo, i32 %n)
+```
- declare i32 @llvm.nvvm.fshr.clamp.i32(i32 %hi, i32 %lo, i32 %n)
+##### Overview:
-Overview:
-"""""""""
-
-The '``llvm.nvvm.fshr.clamp``' family of intrinsics perform a clamped funnel
-shift right. These intrinsics are very similar to '``llvm.fshr``', except the
+The '`llvm.nvvm.fshr.clamp`' family of intrinsics perform a clamped funnel
+shift right. These intrinsics are very similar to '`llvm.fshr`', except the
shift amount is clamped at the integer width (instead of modulo it). Currently,
-only ``i32`` is supported.
+only `i32` is supported.
-Semantics:
-""""""""""
+##### Semantics:
-The '``llvm.nvvm.fshr.clamp``' family of intrinsic functions performs a clamped
+The '`llvm.nvvm.fshr.clamp`' family of intrinsic functions performs a clamped
funnel shift right: the first two values are concatenated as { %hi : %lo } (%hi
is the most significant bits of the wide value), the combined value is shifted
right, and the least significant bits are extracted to produce a result that is
the same size as the original arguments. The shift amount is the minimum of the
value of %n and the bit width of the integer type.
-'``llvm.nvvm.flo.u.*``' Intrinsic
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
+#### '`llvm.nvvm.flo.u.*`' Intrinsic
-.. code-block:: llvm
+##### Syntax:
- declare i32 @llvm.nvvm.flo.u.i32(i32 %a, i1 %shiftamt)
- declare i32 @llvm.nvvm.flo.u.i64(i64 %a, i1 %shiftamt)
+```llvm
+declare i32 @llvm.nvvm.flo.u.i32(i32 %a, i1 %shiftamt)
+declare i32 @llvm.nvvm.flo.u.i64(i64 %a, i1 %shiftamt)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``llvm.nvvm.flo.u``' family of intrinsics identifies the bit position of
+The '`llvm.nvvm.flo.u`' family of intrinsics identifies the bit position of
the leading one, returning either it's offset from the most or least significant
bit.
-Semantics:
-""""""""""
+##### Semantics:
-The '``llvm.nvvm.flo.u``' family of intrinsics returns the bit position of the
+The '`llvm.nvvm.flo.u`' family of intrinsics returns the bit position of the
most significant 1. If %shiftamt is true, The result is the shift amount needed
to left-shift the found bit into the most-significant bit position, otherwise
the result is the shift amount needed to right-shift the found bit into the
least-significant bit position. 0xffffffff is returned if no 1 bit is found.
-'``llvm.nvvm.flo.s.*``' Intrinsic
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
+#### '`llvm.nvvm.flo.s.*`' Intrinsic
-.. code-block:: llvm
+##### Syntax:
- declare i32 @llvm.nvvm.flo.s.i32(i32 %a, i1 %shiftamt)
- declare i32 @llvm.nvvm.flo.s.i64(i64 %a, i1 %shiftamt)
+```llvm
+declare i32 @llvm.nvvm.flo.s.i32(i32 %a, i1 %shiftamt)
+declare i32 @llvm.nvvm.flo.s.i64(i64 %a, i1 %shiftamt)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``llvm.nvvm.flo.s``' family of intrinsics identifies the bit position of
+The '`llvm.nvvm.flo.s`' family of intrinsics identifies the bit position of
the leading non-sign bit, returning either it's offset from the most or least
significant bit.
-Semantics:
-""""""""""
+##### Semantics:
-The '``llvm.nvvm.flo.s``' family of intrinsics returns the bit position of the
-most significant 0 for negative inputs and the most significant 1 for
+The '`llvm.nvvm.flo.s`' family of intrinsics returns the bit position of the
+most significant 0 for negative inputs and the most significant 1 for
non-negative inputs. If %shiftamt is true, The result is the shift amount needed
to left-shift the found bit into the most-significant bit position, otherwise
the result is the shift amount needed to right-shift the found bit into the
least-significant bit position. 0xffffffff is returned if no 1 bit is found.
-'``llvm.nvvm.{zext,sext}.{wrap,clamp}``' Intrinsics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.{zext,sext}.{wrap,clamp}`' Intrinsics
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare i32 @llvm.nvvm.zext.wrap(i32 %a, i32 %b)
+declare i32 @llvm.nvvm.zext.clamp(i32 %a, i32 %b)
+declare i32 @llvm.nvvm.sext.wrap(i32 %a, i32 %b)
+declare i32 @llvm.nvvm.sext.clamp(i32 %a, i32 %b)
+```
- declare i32 @llvm.nvvm.zext.wrap(i32 %a, i32 %b)
- declare i32 @llvm.nvvm.zext.clamp(i32 %a, i32 %b)
- declare i32 @llvm.nvvm.sext.wrap(i32 %a, i32 %b)
- declare i32 @llvm.nvvm.sext.clamp(i32 %a, i32 %b)
+##### Overview:
-Overview:
-"""""""""
-
-The '``llvm.nvvm.{zext,sext}.{wrap,clamp}``' family of intrinsics extracts the
+The '`llvm.nvvm.{zext,sext}.{wrap,clamp}`' family of intrinsics extracts the
low bits of the input value, and zero- or sign-extends them back to the original
width.
-Semantics:
-""""""""""
+##### Semantics:
-The '``llvm.nvvm.{zext,sext}.{wrap,clamp}``' family of intrinsics returns
-extension of N lowest bits of operand %a. For the '``wrap``' variants, N is the
-value of operand %b modulo 32. For the '``clamp``' variants, N is the value of
+The '`llvm.nvvm.{zext,sext}.{wrap,clamp}`' family of intrinsics returns
+extension of N lowest bits of operand %a. For the '`wrap`' variants, N is the
+value of operand %b modulo 32. For the '`clamp`' variants, N is the value of
operand %b clamped to the range [0, 32]. The N lowest bits are then
-zero-extended the case of the '``zext``' variants, or sign-extended the case of
-the '``sext``' variants. If N is 0, the result is 0.
-
-'``llvm.nvvm.bmsk.{wrap,clamp}``' Intrinsic
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+zero-extended the case of the '`zext`' variants, or sign-extended the case of
+the '`sext`' variants. If N is 0, the result is 0.
-Syntax:
-"""""""
+#### '`llvm.nvvm.bmsk.{wrap,clamp}`' Intrinsic
-.. code-block:: llvm
+##### Syntax:
- declare i32 @llvm.nvvm.bmsk.wrap(i32 %a, i32 %b)
- declare i32 @llvm.nvvm.bmsk.clamp(i32 %a, i32 %b)
+```llvm
+declare i32 @llvm.nvvm.bmsk.wrap(i32 %a, i32 %b)
+declare i32 @llvm.nvvm.bmsk.clamp(i32 %a, i32 %b)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``llvm.nvvm.bmsk.{wrap,clamp}``' family of intrinsics creates a bit mask
+The '`llvm.nvvm.bmsk.{wrap,clamp}`' family of intrinsics creates a bit mask
given a starting bit position and a bit width.
-Semantics:
-""""""""""
+##### Semantics:
-The '``llvm.nvvm.bmsk.{wrap,clamp}``' family of intrinsics returns a value with
+The '`llvm.nvvm.bmsk.{wrap,clamp}`' family of intrinsics returns a value with
all bits set to 0 except for %b bits starting at bit position %a. For the
-'``wrap``' variants, the values of %a and %b modulo 32 are used. For the
-'``clamp``' variants, the values of %a and %b are clamped to the range [0, 32],
+'`wrap`' variants, the values of %a and %b modulo 32 are used. For the
+'`clamp`' variants, the values of %a and %b are clamped to the range [0, 32],
which in practice is equivalent to using them as is.
-'``llvm.nvvm.prmt``' Intrinsic
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
+#### '`llvm.nvvm.prmt`' Intrinsic
-.. code-block:: llvm
+##### Syntax:
- declare i32 @llvm.nvvm.prmt(i32 %lo, i32 %hi, i32 %selector)
+```llvm
+declare i32 @llvm.nvvm.prmt(i32 %lo, i32 %hi, i32 %selector)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``llvm.nvvm.prmt``' constructs a permutation of the bytes of the first two
+The '`llvm.nvvm.prmt`' constructs a permutation of the bytes of the first two
operands, selecting based on the third operand.
-Semantics:
-""""""""""
+##### Semantics:
The bytes in the first two source operands are numbered from 0 to 7:
-{%hi, %lo} = {{b7, b6, b5, b4}, {b3, b2, b1, b0}}. For each byte in the target
-register, a 4-bit selection value is defined.
+`{%hi, %lo} = {{b7, b6, b5, b4}, {b3, b2, b1, b0}}`. For each byte in the
+target register, a 4-bit selection value is defined.
The 3 lsbs of the selection value specify which of the 8 source bytes should be
moved into the target position. The msb defines if the byte value should be
@@ -1721,45 +1539,41 @@ These 4-bit selection values are pulled from the lower 16-bits of the %selector
operand, with the least significant selection value corresponding to the least
significant byte of the destination.
+#### '`llvm.nvvm.prmt.*`' Intrinsics
-'``llvm.nvvm.prmt.*``' Intrinsics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+##### Syntax:
-Syntax:
-"""""""
+```llvm
+declare i32 @llvm.nvvm.prmt.f4e(i32 %lo, i32 %hi, i32 %selector)
+declare i32 @llvm.nvvm.prmt.b4e(i32 %lo, i32 %hi, i32 %selector)
-.. code-block:: llvm
+declare i32 @llvm.nvvm.prmt.rc8(i32 %lo, i32 %selector)
+declare i32 @llvm.nvvm.prmt.ecl(i32 %lo, i32 %selector)
+declare i32 @llvm.nvvm.prmt.ecr(i32 %lo, i32 %selector)
+declare i32 @llvm.nvvm.prmt.rc16(i32 %lo, i32 %selector)
+```
- declare i32 @llvm.nvvm.prmt.f4e(i32 %lo, i32 %hi, i32 %selector)
- declare i32 @llvm.nvvm.prmt.b4e(i32 %lo, i32 %hi, i32 %selector)
+##### Overview:
- declare i32 @llvm.nvvm.prmt.rc8(i32 %lo, i32 %selector)
- declare i32 @llvm.nvvm.prmt.ecl(i32 %lo, i32 %selector)
- declare i32 @llvm.nvvm.prmt.ecr(i32 %lo, i32 %selector)
- declare i32 @llvm.nvvm.prmt.rc16(i32 %lo, i32 %selector)
-
-Overview:
-"""""""""
-
-The '``llvm.nvvm.prmt.*``' family of intrinsics constructs a permutation of the
+The '`llvm.nvvm.prmt.*`' family of intrinsics constructs a permutation of the
bytes of the first one or two operands, selecting based on the 2 least
significant bits of the final operand.
-Semantics:
-""""""""""
+##### Semantics:
-As with the generic '``llvm.nvvm.prmt``' intrinsic, the bytes in the first one
+As with the generic '`llvm.nvvm.prmt`' intrinsic, the bytes in the first one
or two source operands are numbered. The first source operand (%lo) is numbered
-{b3, b2, b1, b0}, in the case of the '``f4e``' and '``b4e``' variants, the
+{b3, b2, b1, b0}, in the case of the '`f4e`' and '`b4e`' variants, the
second source operand (%hi) is numbered {b7, b6, b5, b4}.
Depending on the 2 least significant bits of the %selector operand, the result
of the permutation is defined as follows:
+```{eval-rst}
+------------+----------------+--------------+
| Mode | %selector[1:0] | Output |
+------------+----------------+--------------+
-| '``f4e``' | 0 | {3, 2, 1, 0} |
+| ``f4e`` | 0 | {3, 2, 1, 0} |
| +----------------+--------------+
| | 1 | {4, 3, 2, 1} |
| +----------------+--------------+
@@ -1767,7 +1581,7 @@ of the permutation is defined as follows:
| +----------------+--------------+
| | 3 | {6, 5, 4, 3} |
+------------+----------------+--------------+
-| '``b4e``' | 0 | {5, 6, 7, 0} |
+| ``b4e`` | 0 | {5, 6, 7, 0} |
| +----------------+--------------+
| | 1 | {6, 7, 0, 1} |
| +----------------+--------------+
@@ -1775,7 +1589,7 @@ of the permutation is defined as follows:
| +----------------+--------------+
| | 3 | {0, 1, 2, 3} |
+------------+----------------+--------------+
-| '``rc8``' | 0 | {0, 0, 0, 0} |
+| ``rc8`` | 0 | {0, 0, 0, 0} |
| +----------------+--------------+
| | 1 | {1, 1, 1, 1} |
| +----------------+--------------+
@@ -1783,7 +1597,7 @@ of the permutation is defined as follows:
| +----------------+--------------+
| | 3 | {3, 3, 3, 3} |
+------------+----------------+--------------+
-| '``ecl``' | 0 | {3, 2, 1, 0} |
+| ``ecl`` | 0 | {3, 2, 1, 0} |
| +----------------+--------------+
| | 1 | {3, 2, 1, 1} |
| +----------------+--------------+
@@ -1791,7 +1605,7 @@ of the permutation is defined as follows:
| +----------------+--------------+
| | 3 | {3, 3, 3, 3} |
+------------+----------------+--------------+
-| '``ecr``' | 0 | {0, 0, 0, 0} |
+| ``ecr`` | 0 | {0, 0, 0, 0} |
| +----------------+--------------+
| | 1 | {1, 1, 1, 0} |
| +----------------+--------------+
@@ -1799,7 +1613,7 @@ of the permutation is defined as follows:
| +----------------+--------------+
| | 3 | {3, 2, 1, 0} |
+------------+----------------+--------------+
-| '``rc16``' | 0 | {1, 0, 1, 0} |
+| ``rc16`` | 0 | {1, 0, 1, 0} |
| +----------------+--------------+
| | 1 | {3, 2, 3, 2} |
| +----------------+--------------+
@@ -1807,1362 +1621,1172 @@ of the permutation is defined as follows:
| +----------------+--------------+
| | 3 | {3, 2, 3, 2} |
+------------+----------------+--------------+
+```
-TMA family of Intrinsics
-------------------------
+### TMA family of Intrinsics
-'``llvm.nvvm.cp.async.bulk.global.to.shared.cluster``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.cp.async.bulk.global.to.shared.cluster`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.cp.async.bulk.global.to.shared.cluster(ptr addrspace(7) %dst, ptr addrspace(3) %mbar, ptr addrspace(1) %src, i32 %size, i16 %mc, i64 %ch, i1 %flag_mc, i1 %flag_ch)
+```
- declare void @llvm.nvvm.cp.async.bulk.global.to.shared.cluster(ptr addrspace(7) %dst, ptr addrspace(3) %mbar, ptr addrspace(1) %src, i32 %size, i16 %mc, i64 %ch, i1 %flag_mc, i1 %flag_ch)
+##### Overview:
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.cp.async.bulk.global.to.shared.cluster``' intrinsic
-corresponds to the ``cp.async.bulk.shared::cluster.global.*`` family of PTX
+The '`@llvm.nvvm.cp.async.bulk.global.to.shared.cluster`' intrinsic
+corresponds to the `cp.async.bulk.shared::cluster.global.*` family of PTX
instructions. These instructions initiate an asynchronous copy of bulk data from
-global memory to shared::cluster memory. The 32-bit operand ``%size`` specifies
+global memory to shared::cluster memory. The 32-bit operand `%size` specifies
the amount of memory to be copied and it must be a multiple of 16.
-* The last two arguments to these intrinsics are boolean flags indicating
+- The last two arguments to these intrinsics are boolean flags indicating
support for cache_hint and/or multicast modifiers. These flag arguments must
be compile-time constants. The backend looks through these flags and lowers
the intrinsics appropriately.
-
-* The Nth argument (denoted by ``i1 %flag_ch``) when set, indicates a valid
- cache_hint (``i64 %ch``) and generates the ``.L2::cache_hint`` variant of the
+- The Nth argument (denoted by `i1 %flag_ch`) when set, indicates a valid
+ cache_hint (`i64 %ch`) and generates the `.L2::cache_hint` variant of the
PTX instruction.
+- The [N-1]th argument (denoted by `i1 %flag_mc`) when set, indicates the
+ presence of a multicast mask (`i16 %mc`) and generates the PTX instruction
+ with the `.multicast::cluster` modifier.
-* The [N-1]th argument (denoted by ``i1 %flag_mc``) when set, indicates the
- presence of a multicast mask (``i16 %mc``) and generates the PTX instruction
- with the ``.multicast::cluster`` modifier.
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk).
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk>`__.
+#### '`llvm.nvvm.cp.async.bulk.global.to.shared.cta`'
-'``llvm.nvvm.cp.async.bulk.global.to.shared.cta``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+##### Syntax:
-Syntax:
-"""""""
+```llvm
+declare void @llvm.nvvm.cp.async.bulk.global.to.shared.cta(ptr addrspace(3) %dst, ptr addrspace(3) %mbar, ptr addrspace(1) %src, i32 %size, i64 %ch, i1 %flag_ch)
+```
-.. code-block:: llvm
+##### Overview:
- declare void @llvm.nvvm.cp.async.bulk.global.to.shared.cta(ptr addrspace(3) %dst, ptr addrspace(3) %mbar, ptr addrspace(1) %src, i32 %size, i64 %ch, i1 %flag_ch)
-
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.cp.async.bulk.global.to.shared.cta``' intrinsic corresponds to
-the ``cp.async.bulk.shared::cta.global.*`` family of PTX instructions. These
+The '`@llvm.nvvm.cp.async.bulk.global.to.shared.cta`' intrinsic corresponds to
+the `cp.async.bulk.shared::cta.global.*` family of PTX instructions. These
instructions initiate an asynchronous copy of bulk data from global memory to
-shared::cta memory. The 32-bit operand ``%size`` specifies the amount of memory
+shared::cta memory. The 32-bit operand `%size` specifies the amount of memory
to be copied and it must be a multiple of 16. The last argument (denoted by
-``i1 %flag_ch``) is a compile-time constant. When set, it indicates a valid
-cache_hint (``i64 %ch``) and generates the ``.L2::cache_hint`` variant of the
+`i1 %flag_ch`) is a compile-time constant. When set, it indicates a valid
+cache_hint (`i64 %ch`) and generates the `.L2::cache_hint` variant of the
PTX instruction.
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk>`__.
-
-'``llvm.nvvm.cp.async.bulk.shared.cta.to.global``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk).
-Syntax:
-"""""""
+#### '`llvm.nvvm.cp.async.bulk.shared.cta.to.global`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.cp.async.bulk.shared.cta.to.global(ptr addrspace(1) %dst, ptr addrspace(3) %src, i32 %size, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.shared.cta.to.global.bytemask(..., i32 %size, i64 %ch, i1 %flag_ch, i16 %mask)
+```llvm
+declare void @llvm.nvvm.cp.async.bulk.shared.cta.to.global(ptr addrspace(1) %dst, ptr addrspace(3) %src, i32 %size, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.shared.cta.to.global.bytemask(..., i32 %size, i64 %ch, i1 %flag_ch, i16 %mask)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.cp.async.bulk.shared.cta.to.global``' intrinsic corresponds to
-the ``cp.async.bulk.global.shared::cta.*`` set of PTX instructions. These
+The '`@llvm.nvvm.cp.async.bulk.shared.cta.to.global`' intrinsic corresponds to
+the `cp.async.bulk.global.shared::cta.*` set of PTX instructions. These
instructions initiate an asynchronous copy from shared::cta to global memory.
-The 32-bit operand ``%size`` specifies the amount of memory to be copied
-(in bytes) and it must be a multiple of 16. For the ``.bytemask`` variant, the
+The 32-bit operand `%size` specifies the amount of memory to be copied
+(in bytes) and it must be a multiple of 16. For the `.bytemask` variant, the
16-bit wide mask operand specifies whether the i-th byte of each 16-byte wide
chunk of source data is copied to the destination.
-* The ``i1 %flag_ch`` argument to these intrinsics is a boolean flag indicating
+- The `i1 %flag_ch` argument to these intrinsics is a boolean flag indicating
support for cache_hint. This flag argument must be a compile-time constant.
- When set, it indicates a valid cache_hint (``i64 %ch``) and generates the
- ``.L2::cache_hint`` variant of the PTX instruction.
+ When set, it indicates a valid cache_hint (`i64 %ch`) and generates the
+ `.L2::cache_hint` variant of the PTX instruction.
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk>`__.
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk).
-'``llvm.nvvm.cp.async.bulk.shared.cta.to.cluster``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.cp.async.bulk.shared.cta.to.cluster`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.cp.async.bulk.shared.cta.to.cluster(ptr addrspace(7) %dst, ptr addrspace(3) %mbar, ptr addrspace(3) %src, i32 %size)
+```
- declare void @llvm.nvvm.cp.async.bulk.shared.cta.to.cluster(ptr addrspace(7) %dst, ptr addrspace(3) %mbar, ptr addrspace(3) %src, i32 %size)
+##### Overview:
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.cp.async.bulk.shared.cta.to.cluster``' intrinsic corresponds
-to the ``cp.async.bulk.shared::cluster.shared::cta.*`` PTX instruction. This
+The '`@llvm.nvvm.cp.async.bulk.shared.cta.to.cluster`' intrinsic corresponds
+to the `cp.async.bulk.shared::cluster.shared::cta.*` PTX instruction. This
instruction initiates an asynchronous copy from shared::cta to shared::cluster
memory. The destination has to be in the shared memory of a
diff erent CTA within
-the cluster. The 32-bit operand ``%size`` specifies the amount of memory to be
+the cluster. The 32-bit operand `%size` specifies the amount of memory to be
copied and it must be a multiple of 16.
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk>`__.
-
-'``llvm.nvvm.cp.async.bulk.prefetch.L2``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk).
-Syntax:
-"""""""
+#### '`llvm.nvvm.cp.async.bulk.prefetch.L2`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.cp.async.bulk.prefetch.L2(ptr addrspace(1) %src, i32 %size, i64 %ch, i1 %flag_ch)
+```llvm
+declare void @llvm.nvvm.cp.async.bulk.prefetch.L2(ptr addrspace(1) %src, i32 %size, i64 %ch, i1 %flag_ch)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.cp.async.bulk.prefetch.L2``' intrinsic corresponds to the
-``cp.async.bulk.prefetch.L2.*`` family of PTX instructions. These instructions
+The '`@llvm.nvvm.cp.async.bulk.prefetch.L2`' intrinsic corresponds to the
+`cp.async.bulk.prefetch.L2.*` family of PTX instructions. These instructions
initiate an asynchronous prefetch of bulk data from global memory to the L2
-cache. The 32-bit operand ``%size`` specifies the amount of memory to be
+cache. The 32-bit operand `%size` specifies the amount of memory to be
prefetched in terms of bytes and it must be a multiple of 16.
-* The last argument to these intrinsics is boolean flag indicating support for
+- The last argument to these intrinsics is boolean flag indicating support for
cache_hint. These flag argument must be compile-time constant. When set, it
- indicates a valid cache_hint (``i64 %ch``) and generates the
- ``.L2::cache_hint`` variant of the PTX instruction.
-
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-cp-async-bulk-prefetch>`__.
-
-'``llvm.nvvm.prefetch.*``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
-
-.. code-block:: llvm
-
- declare void @llvm.nvvm.prefetch.global.L1(ptr addrspace(1) %global_ptr)
- declare void @llvm.nvvm.prefetch.global.L2(ptr addrspace(1) %global_ptr)
- declare void @llvm.nvvm.prefetch.local.L1(ptr addrspace(5) %local_ptr)
- declare void @llvm.nvvm.prefetch.local.L2(ptr addrspace(5) %local_ptr)
-
- declare void @llvm.nvvm.prefetch.L1(ptr %ptr)
- declare void @llvm.nvvm.prefetch.L2(ptr %ptr)
-
- declare void @llvm.nvvm.prefetch.tensormap.p0(ptr %ptr)
- declare void @llvm.nvvm.prefetch.tensormap.p4(ptr addrspace(4) %const_ptr)
- declare void @llvm.nvvm.prefetch.tensormap.p101(ptr addrspace(101) %param_ptr)
-
- declare void @llvm.nvvm.prefetch.global.L2.evict.normal(ptr addrspace(1) %global_ptr)
- declare void @llvm.nvvm.prefetch.global.L2.evict.last(ptr addrspace(1) %global_ptr)
-
- declare void @llvm.nvvm.prefetchu.L1(ptr %ptr)
-
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.prefetch.*``' and '``@llvm.nvvm.prefetchu.*``' intrinsic
-correspond to the '``prefetch.*``;' and '``prefetchu.*``' family of PTX
-instructions. The '``prefetch.*``' instructions bring the cache line containing
+ indicates a valid cache_hint (`i64 %ch`) and generates the
+ `.L2::cache_hint` variant of the PTX instruction.
+
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-cp-async-bulk-prefetch).
+
+#### '`llvm.nvvm.prefetch.*`'
+
+##### Syntax:
+
+```llvm
+declare void @llvm.nvvm.prefetch.global.L1(ptr addrspace(1) %global_ptr)
+declare void @llvm.nvvm.prefetch.global.L2(ptr addrspace(1) %global_ptr)
+declare void @llvm.nvvm.prefetch.local.L1(ptr addrspace(5) %local_ptr)
+declare void @llvm.nvvm.prefetch.local.L2(ptr addrspace(5) %local_ptr)
+
+declare void @llvm.nvvm.prefetch.L1(ptr %ptr)
+declare void @llvm.nvvm.prefetch.L2(ptr %ptr)
+
+declare void @llvm.nvvm.prefetch.tensormap.p0(ptr %ptr)
+declare void @llvm.nvvm.prefetch.tensormap.p4(ptr addrspace(4) %const_ptr)
+declare void @llvm.nvvm.prefetch.tensormap.p101(ptr addrspace(101) %param_ptr)
+
+declare void @llvm.nvvm.prefetch.global.L2.evict.normal(ptr addrspace(1) %global_ptr)
+declare void @llvm.nvvm.prefetch.global.L2.evict.last(ptr addrspace(1) %global_ptr)
+
+declare void @llvm.nvvm.prefetchu.L1(ptr %ptr)
+```
+
+##### Overview:
+
+The '`@llvm.nvvm.prefetch.*`' and '`@llvm.nvvm.prefetchu.*`' intrinsic
+correspond to the '`prefetch.*`;' and '`prefetchu.*`' family of PTX
+instructions. The '`prefetch.*`' instructions bring the cache line containing
the specified address in global or local memory address space into the specified
-cache level (L1 or L2). If the '``.tensormap``' qualifier is specified then the
+cache level (L1 or L2). If the '`.tensormap`' qualifier is specified then the
prefetch instruction brings the cache line containing the specified address in
-the '``.const``' or '``.param memory``' state space for subsequent use by the
-'``cp.async.bulk.tensor``' instruction. The '``prefetchu.*``' instruction brings
+the '`.const`' or '`.param memory`' state space for subsequent use by the
+'`cp.async.bulk.tensor`' instruction. The '`prefetchu.*`' instruction brings
the cache line containing the specified generic address into the specified
uniform cache level. If no address space is specified, it is assumed to be
generic address. The intrinsic uses and eviction priority which can be accessed
-by the '``.level::eviction_priority``' modifier.
+by the '`.level::eviction_priority`' modifier.
-* A prefetch to a shared memory location performs no operation.
-* A prefetch into the uniform cache requires a generic address, and no operation
+- A prefetch to a shared memory location performs no operation.
+- A prefetch into the uniform cache requires a generic address, and no operation
occurs if the address maps to a const, local, or shared memory location.
-For more information, refer to the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-prefetch-prefetchu>`__.
-
-'``llvm.nvvm.applypriority.*``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-prefetch-prefetchu).
-Syntax:
-"""""""
+#### '`llvm.nvvm.applypriority.*`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.applypriority.global.L2.evict.normal(ptr addrspace(1) %global_ptr, i64 %size)
- declare void @llvm.nvvm.applypriority.L2.evict.normal(ptr %ptr, i64 %size)
+```llvm
+declare void @llvm.nvvm.applypriority.global.L2.evict.normal(ptr addrspace(1) %global_ptr, i64 %size)
+declare void @llvm.nvvm.applypriority.L2.evict.normal(ptr %ptr, i64 %size)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.applypriority.*``' applies the cache eviction priority
+The '`@llvm.nvvm.applypriority.*`' applies the cache eviction priority
specified by the .level::eviction_priority qualifier to the address range
-[a..a+size) in the specified cache level. If no state space is specified then
+\[a..a+size) in the specified cache level. If no state space is specified then
generic addressing is used. If the specified address does not fall within the
address window of .global state space then the behavior is undefined. The
operand size is an integer constant that specifies the amount of data, in bytes,
in the specified cache level on which the priority is to be applied. The only
supported value for the size operand is 128.
-For more information, refer to the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-applypriority>`__.
-
-``llvm.nvvm.discard.*``'
-^^^^^^^^^^^^^^^^^^^^^^^^
+For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-applypriority).
-Syntax:
-"""""""
+#### `llvm.nvvm.discard.*`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.discard.global.L2(ptr addrspace(1) %global_ptr, i64 immarg)
- declare void @llvm.nvvm.discard.L2(ptr %ptr, i64 immarg)
+```llvm
+declare void @llvm.nvvm.discard.global.L2(ptr addrspace(1) %global_ptr, i64 immarg)
+declare void @llvm.nvvm.discard.L2(ptr %ptr, i64 immarg)
+```
-Overview:
-"""""""""
+##### Overview:
-The *effects* of the ``@llvm.nvvm.discard.L2*`` intrinsics are those of a
-non-atomic non-volatile ``llvm.memset`` that writes ``undef`` to the destination
-address range ``[%ptr, %ptr + immarg)``. The ``%ptr`` must be aligned by 128
-bytes. Subsequent reads from the address range may read ``undef`` until the
+The *effects* of the `@llvm.nvvm.discard.L2*` intrinsics are those of a
+non-atomic non-volatile `llvm.memset` that writes `undef` to the destination
+address range `[%ptr, %ptr + immarg)`. The `%ptr` must be aligned by 128
+bytes. Subsequent reads from the address range may read `undef` until the
memory is overwritten with a
diff erent value. These operations *hint* the
implementation that data in the L2 cache can be destructively discarded without
-writing it back to memory. The operand ``immarg`` is an integer constant that
-specifies the length in bytes of the address range ``[%ptr, %ptr + immarg)`` to
-write ``undef`` into. The only supported value for the ``immarg`` operand is
-``128``. If generic addressing is used and the specified address does not fall
-within the address window of global memory (``addrspace(1)``) the behavior is
+writing it back to memory. The operand `immarg` is an integer constant that
+specifies the length in bytes of the address range `[%ptr, %ptr + immarg)` to
+write `undef` into. The only supported value for the `immarg` operand is
+`128`. If generic addressing is used and the specified address does not fall
+within the address window of global memory (`addrspace(1)`) the behavior is
undefined.
-.. code-block:: llvm
-
- call void @llvm.nvvm.discard.L2(ptr %p, i64 128) ;; writes `undef` to [p, p+128)
- %a = load i64, ptr %p. ;; loads 8 bytes containing undef
- %b = load i64, ptr %p ;; loads 8 bytes containing undef
- ;; comparing %a and %b compares `undef` values!
- %fa = freeze i64 %a ;; freezes undef to stable bit-pattern
- %fb = freeze i64 %b ;; freezes undef to stable bit-pattern
- ;; %fa may compare
diff erent to %fb!
-
-For more information, refer to the `CUDA C++ discard documentation
-<https://nvidia.github.io/cccl/libcudacxx/extended_api/memory_access_properties/discard_memory.html>`__
-and to the `PTX ISA discard documentation
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-discard>`__ .
-
-'``llvm.nvvm.cp.async.bulk.tensor.g2s.tile.[1-5]d``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
-
-.. code-block:: llvm
-
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %d0, i16 %mc, i64 %ch, i1 %flag_mc, i1 %flag_ch, i32 %flag_cta_group)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.2d(..., i32 %d0, i32 %d1, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.3d(..., i32 %d0, i32 %d1, i32 %d2, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
-
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.gather4.2d(ptr addrspace(7) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %x0, i32 %y0, i32 %y1, i32 %y2, i32 %y3, i16 %mc, i64 %ch, i1 %flag_mc, i1 %flag_ch, i32 %flag_cta_group)
-
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.cp.async.bulk.tensor.g2s.tile.[1-5]d``' intrinsics correspond
-to the ``cp.async.bulk.tensor.[1-5]d.*`` set of PTX instructions. These
+```llvm
+call void @llvm.nvvm.discard.L2(ptr %p, i64 128) ;; writes `undef` to [p, p+128)
+%a = load i64, ptr %p. ;; loads 8 bytes containing undef
+%b = load i64, ptr %p ;; loads 8 bytes containing undef
+;; comparing %a and %b compares `undef` values!
+%fa = freeze i64 %a ;; freezes undef to stable bit-pattern
+%fb = freeze i64 %b ;; freezes undef to stable bit-pattern
+;; %fa may compare
diff erent to %fb!
+```
+
+For more information, refer to the [CUDA C++ discard documentation](https://nvidia.github.io/cccl/libcudacxx/extended_api/memory_access_properties/discard_memory.html)
+and to the [PTX ISA discard documentation](https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-discard) .
+
+#### '`llvm.nvvm.cp.async.bulk.tensor.g2s.tile.[1-5]d`'
+
+##### Syntax:
+
+```llvm
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.1d(ptr addrspace(7) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %d0, i16 %mc, i64 %ch, i1 %flag_mc, i1 %flag_ch, i32 %flag_cta_group)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.2d(..., i32 %d0, i32 %d1, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.3d(..., i32 %d0, i32 %d1, i32 %d2, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.tile.gather4.2d(ptr addrspace(7) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %x0, i32 %y0, i32 %y1, i32 %y2, i32 %y3, i16 %mc, i64 %ch, i1 %flag_mc, i1 %flag_ch, i32 %flag_cta_group)
+```
+
+##### Overview:
+
+The '`@llvm.nvvm.cp.async.bulk.tensor.g2s.tile.[1-5]d`' intrinsics correspond
+to the `cp.async.bulk.tensor.[1-5]d.*` set of PTX instructions. These
instructions initiate an asynchronous copy of tensor data from global memory to
-shared::cluster memory (indicated by the ``g2s`` prefix) in ``tile`` mode. In
+shared::cluster memory (indicated by the `g2s` prefix) in `tile` mode. In
tile mode, the multi-dimensional layout of the source tensor is preserved at the
destination. The dimension of the tensor data ranges from 1d to 5d with the
-coordinates specified by the ``i32 %d0 ... i32 %d4`` arguments. In
-``tile.gather4`` mode, four rows in a 2D tensor are combined to form a single 2D
-destination tensor. The first coordinate ``i32 %x0`` denotes the column index
+coordinates specified by the `i32 %d0 ... i32 %d4` arguments. In
+`tile.gather4` mode, four rows in a 2D tensor are combined to form a single 2D
+destination tensor. The first coordinate `i32 %x0` denotes the column index
followed by four coordinates indicating the four row-indices. So, this mode
takes a total of 5 coordinates as input arguments. For more information on
-``gather4`` mode, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-tiled-scatter4-gather4-modes>`__.
+`gather4` mode, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-tiled-scatter4-gather4-modes).
-* The last three arguments to these intrinsics are flags indicating support for
+- The last three arguments to these intrinsics are flags indicating support for
multicast, cache_hint and cta_group::1/2 modifiers. These flag arguments must
be compile-time constants. The backend looks through these flags and lowers
the intrinsics appropriately.
-
-* The argument denoted by ``i1 %flag_ch`` when set, indicates a valid cache_hint
- (``i64 %ch``) and generates the ``.L2::cache_hint`` variant of the PTX
+- The argument denoted by `i1 %flag_ch` when set, indicates a valid cache_hint
+ (`i64 %ch`) and generates the `.L2::cache_hint` variant of the PTX
instruction.
-
-* The argument denoted by ``i1 %flag_mc`` when set, indicates the presence of a
- multicast mask (``i16 %mc``) and generates the PTX instruction with the
- ``.multicast::cluster`` modifier.
-
-* The argument denoted by ``i32 %flag_cta_group`` takes values within the range
- [0, 3) i.e. {0,1,2}. When the value of ``%flag_cta_group`` is not within the
+- The argument denoted by `i1 %flag_mc` when set, indicates the presence of a
+ multicast mask (`i16 %mc`) and generates the PTX instruction with the
+ `.multicast::cluster` modifier.
+- The argument denoted by `i32 %flag_cta_group` takes values within the range
+ \[0, 3) i.e. {0,1,2}. When the value of `%flag_cta_group` is not within the
range, it may raise an error from the Verifier. The default value is '0' with
no cta_group modifier in the instruction. The values of '1' and '2' lower to
- ``cta_group::1`` and ``cta_group::2`` variants of the PTX instruction
+ `cta_group::1` and `cta_group::2` variants of the PTX instruction
respectively.
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor>`__.
-
-'``llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.[3-5]d``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor).
-Syntax:
-"""""""
+#### '`llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.[3-5]d`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.3d(ptr addrspace(7) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %im2col0, i16 %mc, i64 %ch, i1 %flag_mc, i1 %flag_ch, i32 %flag_cta_group)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i16 %im2col0, i16 %im2col1, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, i16 %im2col0, i16 %im2col1, i16 %im2col2, ...)
+```llvm
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.3d(ptr addrspace(7) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %im2col0, i16 %mc, i64 %ch, i1 %flag_mc, i1 %flag_ch, i32 %flag_cta_group)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i16 %im2col0, i16 %im2col1, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, i16 %im2col0, i16 %im2col1, i16 %im2col2, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.w.3d(ptr addrspace(7) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %wHalo, i16 %wOffset, i16 %mc, i64 %ch, i1 %flag_mc, i1 %flag_ch, i32 %flag_cta_group)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.w.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.w.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.w.3d(ptr addrspace(7) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %wHalo, i16 %wOffset, i16 %mc, i64 %ch, i1 %flag_mc, i1 %flag_ch, i32 %flag_cta_group)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.w.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.w.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.w.128.3d(ptr addrspace(7) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %wHalo, i16 %wOffset, i16 %mc, i64 %ch, i1 %flag_mc, i1 %flag_ch, i32 %flag_cta_group)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.w.128.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.w.128.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.w.128.3d(ptr addrspace(7) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %wHalo, i16 %wOffset, i16 %mc, i64 %ch, i1 %flag_mc, i1 %flag_ch, i32 %flag_cta_group)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.w.128.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.w.128.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.[3-5]d``' intrinsics
-correspond to the ``cp.async.bulk.tensor.[1-5]d.*`` set of PTX instructions.
+The '`@llvm.nvvm.cp.async.bulk.tensor.g2s.im2col.[3-5]d`' intrinsics
+correspond to the `cp.async.bulk.tensor.[1-5]d.*` set of PTX instructions.
These instructions initiate an asynchronous copy of tensor data from global
-memory to shared::cluster memory (indicated by the ``g2s`` prefix) in ``im2col``
+memory to shared::cluster memory (indicated by the `g2s` prefix) in `im2col`
mode. In im2col mode, some dimensions of the source tensor are unrolled into a
single dimensional column at the destination. In this mode, the tensor has to be
at least three-dimensional. Along with the tensor coordinates, im2col offsets
-are also specified (denoted by ``i16 im2col0...i16 %im2col2``). For the
-``im2col`` mode, the number of offsets is two less than the number of dimensions
-of the tensor operation. For the ``im2col.w`` and ``im2col.w.128`` mode, the
-number of offsets is always 2, denoted by ``i16 %wHalo`` and ``i16 %wOffset``
-arguments. For more information on ``im2col.w`` and ``im2col.w.128`` modes,
-refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-im2col-w-w128-modes>`__.
+are also specified (denoted by `i16 im2col0...i16 %im2col2`). For the
+`im2col` mode, the number of offsets is two less than the number of dimensions
+of the tensor operation. For the `im2col.w` and `im2col.w.128` mode, the
+number of offsets is always 2, denoted by `i16 %wHalo` and `i16 %wOffset`
+arguments. For more information on `im2col.w` and `im2col.w.128` modes,
+refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-im2col-w-w128-modes).
The last three arguments to these intrinsics are flags, with the same
-functionality as described in the ``tile`` mode intrinsics above.
+functionality as described in the `tile` mode intrinsics above.
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor>`__.
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor).
-'``llvm.nvvm.cp.async.bulk.tensor.g2s.cta.tile.[1-5]d``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.cp.async.bulk.tensor.g2s.cta.tile.[1-5]d`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.tile.1d(ptr addrspace(3) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.tile.2d(..., i32 %d0, i32 %d1, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.tile.3d(..., i32 %d0, i32 %d1, i32 %d2, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.tile.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.tile.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.tile.1d(ptr addrspace(3) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.tile.2d(..., i32 %d0, i32 %d1, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.tile.3d(..., i32 %d0, i32 %d1, i32 %d2, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.tile.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.tile.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.tile.gather4.2d(ptr addrspace(3) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %x0, i32 %y0, i32 %y1, i32 %y2, i32 %y3, i64 %ch, i1 %flag_ch)
+```
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.tile.gather4.2d(ptr addrspace(3) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %x0, i32 %y0, i32 %y1, i32 %y2, i32 %y3, i64 %ch, i1 %flag_ch)
+##### Overview:
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.cp.async.bulk.tensor.g2s.cta.tile.[1-5]d``' intrinsics
-correspond to the ``cp.async.bulk.tensor.[1-5]d.shared::cta.global.*`` set of
+The '`@llvm.nvvm.cp.async.bulk.tensor.g2s.cta.tile.[1-5]d`' intrinsics
+correspond to the `cp.async.bulk.tensor.[1-5]d.shared::cta.global.*` set of
PTX instructions. These instructions initiate an asynchronous copy of tensor
-data from global memory to shared::cta memory in ``tile`` mode. In tile mode,
+data from global memory to shared::cta memory in `tile` mode. In tile mode,
the multi-dimensional layout of the source tensor is preserved at the
destination. The dimension of the tensor data ranges from 1d to 5d with the
-coordinates specified by the ``i32 %d0 ... i32 %d4`` arguments. In
-``tile.gather4`` mode, four rows in a 2D tensor are combined to form a single 2D
-destination tensor. The first coordinate ``i32 %x0`` denotes the column index
+coordinates specified by the `i32 %d0 ... i32 %d4` arguments. In
+`tile.gather4` mode, four rows in a 2D tensor are combined to form a single 2D
+destination tensor. The first coordinate `i32 %x0` denotes the column index
followed by four coordinates indicating the four row-indices. So, this mode
takes a total of 5 coordinates as input arguments. For more information on
-``gather4`` mode, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-tiled-scatter4-gather4-modes>`__.
+`gather4` mode, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-tiled-scatter4-gather4-modes).
-* The last argument to these intrinsics is a boolean flag indicating support for
+- The last argument to these intrinsics is a boolean flag indicating support for
cache_hint. This flag argument must be a compile-time constant. When set, it
- indicates a valid cache_hint (``i64 %ch``) and generates the
- ``.L2::cache_hint`` variant of the PTX instruction.
-
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor>`__.
+ indicates a valid cache_hint (`i64 %ch`) and generates the
+ `.L2::cache_hint` variant of the PTX instruction.
-'``llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.[3-5]d``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor).
-Syntax:
-"""""""
+#### '`llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.[3-5]d`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.3d(ptr addrspace(3) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %im2col0, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i16 %im2col0, i16 %im2col1, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, i16 %im2col0, i16 %im2col1, i16 %im2col2, ...)
+```llvm
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.3d(ptr addrspace(3) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %im2col0, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i16 %im2col0, i16 %im2col1, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, i16 %im2col0, i16 %im2col1, i16 %im2col2, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.w.3d(ptr addrspace(3) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %wHalo, i16 %wOffset, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.w.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.w.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.w.3d(ptr addrspace(3) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %wHalo, i16 %wOffset, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.w.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.w.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.w.128.3d(ptr addrspace(3) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %wHalo, i16 %wOffset, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.w.128.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.w.128.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.w.128.3d(ptr addrspace(3) %dst, ptr addrspace(3) %bar, ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %wHalo, i16 %wOffset, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.w.128.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.w.128.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.[3-5]d``' intrinsics
-correspond to the ``cp.async.bulk.tensor.[1-5]d.shared::cta.global.*`` set of
+The '`@llvm.nvvm.cp.async.bulk.tensor.g2s.cta.im2col.[3-5]d`' intrinsics
+correspond to the `cp.async.bulk.tensor.[1-5]d.shared::cta.global.*` set of
PTX instructions. These instructions initiate an asynchronous copy of tensor
-data from global memory to shared::cta memory in ``im2col`` mode. In im2col
+data from global memory to shared::cta memory in `im2col` mode. In im2col
mode, some dimensions of the source tensor are unrolled into a single
dimensional column at the destination. In this mode, the tensor has to be at
least three-dimensional. Along with the tensor coordinates, im2col offsets are
-also specified (denoted by ``i16 im2col0...i16 %im2col2``). For the ``im2col``
+also specified (denoted by `i16 im2col0...i16 %im2col2`). For the `im2col`
mode, the number of offsets is two less than the number of dimensions of the
-tensor operation. For the ``im2col.w`` and ``im2col.w.128`` mode, the number of
-offsets is always 2, denoted by ``i16 %wHalo`` and ``i16 %wOffset`` arguments.
-For more information on ``im2col.w`` and ``im2col.w.128`` modes, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-im2col-w-w128-modes>`__.
+tensor operation. For the `im2col.w` and `im2col.w.128` mode, the number of
+offsets is always 2, denoted by `i16 %wHalo` and `i16 %wOffset` arguments.
+For more information on `im2col.w` and `im2col.w.128` modes, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-im2col-w-w128-modes).
-* The last argument to these intrinsics is a boolean flag indicating support
+- The last argument to these intrinsics is a boolean flag indicating support
for cache_hint. This flag argument must be a compile-time constant. When set,
- it indicates a valid cache_hint (``i64 %ch``) and generates the
- ``.L2::cache_hint`` variant of the PTX instruction.
-
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor>`__.
+ it indicates a valid cache_hint (`i64 %ch`) and generates the
+ `.L2::cache_hint` variant of the PTX instruction.
-'``llvm.nvvm.cp.async.bulk.tensor.s2g.tile.[1-5]d``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor).
-Syntax:
-"""""""
+#### '`llvm.nvvm.cp.async.bulk.tensor.s2g.tile.[1-5]d`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.tile.2d(..., i32 %d0, i32 %d1, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.tile.3d(..., i32 %d0, i32 %d1, i32 %d2, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.tile.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.tile.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+```llvm
+declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.tile.2d(..., i32 %d0, i32 %d1, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.tile.3d(..., i32 %d0, i32 %d1, i32 %d2, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.tile.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.tile.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.tile.scatter4.2d(ptr addrspace(3) %src, ptr %tensor_map, i32 %x0, i32 %y0, i32 %y1, i32 %y2, i32 %y3, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.tile.scatter4.2d(ptr addrspace(3) %src, ptr %tensor_map, i32 %x0, i32 %y0, i32 %y1, i32 %y2, i32 %y3, i64 %ch, i1 %flag_ch)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.cp.async.bulk.tensor.s2g.tile.[1-5]d``' intrinsics correspond
-to the ``cp.async.bulk.tensor.[1-5]d.*`` set of PTX instructions. These
+The '`@llvm.nvvm.cp.async.bulk.tensor.s2g.tile.[1-5]d`' intrinsics correspond
+to the `cp.async.bulk.tensor.[1-5]d.*` set of PTX instructions. These
instructions initiate an asynchronous copy of tensor data from shared::cta to
-global memory (indicated by the ``s2g`` prefix) in ``tile`` mode. The dimension
+global memory (indicated by the `s2g` prefix) in `tile` mode. The dimension
of the tensor data ranges from 1d to 5d with the coordinates specified by the
-``i32 %d0 ... i32 %d4`` arguments. In ``tile.scatter4`` mode, a single 2D source
+`i32 %d0 ... i32 %d4` arguments. In `tile.scatter4` mode, a single 2D source
tensor is divided into four rows in the 2D destination tensor. The first
-coordinate ``i32 %x0`` denotes the column index followed by four coordinates
+coordinate `i32 %x0` denotes the column index followed by four coordinates
indicating the four row-indices. So, this mode takes a total of 5 coordinates as
-input arguments. For more information on ``scatter4`` mode, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-tiled-scatter4-gather4-modes>`__.
+input arguments. For more information on `scatter4` mode, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-tiled-scatter4-gather4-modes).
-* The last argument to these intrinsics is a boolean flag indicating support for
+- The last argument to these intrinsics is a boolean flag indicating support for
cache_hint. This flag argument must be a compile-time constant. When set, it
- indicates a valid cache_hint (``i64 %ch``) and generates the
- ``.L2::cache_hint`` variant of the PTX instruction.
+ indicates a valid cache_hint (`i64 %ch`) and generates the
+ `.L2::cache_hint` variant of the PTX instruction.
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor>`__.
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor).
-'``llvm.nvvm.cp.async.bulk.tensor.s2g.im2col.[3-5]d``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.cp.async.bulk.tensor.s2g.im2col.[3-5]d`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.im2col.3d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.im2col.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.im2col.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+```
- declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.im2col.3d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.im2col.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.s2g.im2col.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+##### Overview:
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.cp.async.bulk.tensor.s2g.im2col.[1-5]d``' intrinsics
-correspond to the ``cp.async.bulk.tensor.[1-5]d.*`` set of PTX instructions.
+The '`@llvm.nvvm.cp.async.bulk.tensor.s2g.im2col.[1-5]d`' intrinsics
+correspond to the `cp.async.bulk.tensor.[1-5]d.*` set of PTX instructions.
These instructions initiate an asynchronous copy of tensor data from
-shared::cta to global memory (indicated by the ``s2g`` prefix) in ``im2col``
+shared::cta to global memory (indicated by the `s2g` prefix) in `im2col`
mode. In this mode, the tensor has to be at least three-dimensional. Unlike the
-``g2s`` variants, there are no im2col_offsets for these intrinsics. The last
+`g2s` variants, there are no im2col_offsets for these intrinsics. The last
argument to these intrinsics is a boolean flag, with the same functionality as
-described in the ``s2g.tile`` mode intrinsics above.
-
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor>`__.
+described in the `s2g.tile` mode intrinsics above.
-'``llvm.nvvm.cp.async.bulk.tensor.prefetch.tile.[1-5]d``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-async-bulk-tensor).
-Syntax:
-"""""""
+#### '`llvm.nvvm.cp.async.bulk.tensor.prefetch.tile.[1-5]d`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.tile.1d(ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.tile.2d(..., i32 %d0, i32 %d1, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.tile.3d(..., i32 %d0, i32 %d1, i32 %d2, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.tile.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.tile.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+```llvm
+declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.tile.1d(ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.tile.2d(..., i32 %d0, i32 %d1, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.tile.3d(..., i32 %d0, i32 %d1, i32 %d2, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.tile.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.tile.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.tile.gather4.2d(ptr %tensor_map, i32 %x0, i32 %y0, i32 %y1, i32 %y2, i32 %y3, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.tile.gather4.2d(ptr %tensor_map, i32 %x0, i32 %y0, i32 %y1, i32 %y2, i32 %y3, i64 %ch, i1 %flag_ch)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.cp.async.bulk.tensor.prefetch.tile.[1-5]d``' intrinsics
-correspond to the ``cp.async.bulk.prefetch.tensor.[1-5]d.L2.global*`` set of
+The '`@llvm.nvvm.cp.async.bulk.tensor.prefetch.tile.[1-5]d`' intrinsics
+correspond to the `cp.async.bulk.prefetch.tensor.[1-5]d.L2.global*` set of
PTX instructions. These instructions initiate an asynchronous prefetch of tensor
data from global memory to the L2 cache. In tile mode, the multi-dimensional
layout of the source tensor is preserved at the destination. The dimension of
the tensor data ranges from 1d to 5d with the coordinates specified by the
-``i32 %d0 ... i32 %d4`` arguments.
+`i32 %d0 ... i32 %d4` arguments.
-In ``tile.gather4`` mode, four rows in the 2-dimnesional source tensor are
-fetched to the L2 cache. The first coordinate ``i32 %x0`` denotes the column
+In `tile.gather4` mode, four rows in the 2-dimnesional source tensor are
+fetched to the L2 cache. The first coordinate `i32 %x0` denotes the column
index followed by four coordinates indicating the four row-indices. So, this
mode takes a total of 5 coordinates as input arguments. For more information
-on ``gather4`` mode, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-tiled-scatter4-gather4-modes>`__.
+on `gather4` mode, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-tiled-scatter4-gather4-modes).
-* The last argument to these intrinsics is a boolean flag indicating support
+- The last argument to these intrinsics is a boolean flag indicating support
for cache_hint. This flag argument must be a compile-time constant. When set,
- it indicates a valid cache_hint (``i64 %ch``) and generates the
- ``.L2::cache_hint`` variant of the PTX instruction.
+ it indicates a valid cache_hint (`i64 %ch`) and generates the
+ `.L2::cache_hint` variant of the PTX instruction.
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-cp-async-bulk-prefetch-tensor>`__.
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-cp-async-bulk-prefetch-tensor).
-'``llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.[3-5]d``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.[3-5]d`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.3d(ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %im2col0, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i16 %im2col0, i16 %im2col1, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, i16 %im2col0, i16 %im2col1, i16 %im2col2, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.3d(ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %im2col0, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i16 %im2col0, i16 %im2col1, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, i16 %im2col0, i16 %im2col1, i16 %im2col2, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.w.3d(ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %wHalo, i16 %wOffset, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.w.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.w.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.w.3d(ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %wHalo, i16 %wOffset, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.w.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.w.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.w.128.3d(ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %wHalo, i16 %wOffset, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.w.128.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.w.128.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+```
- declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.w.128.3d(ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i16 %wHalo, i16 %wOffset, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.w.128.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.w.128.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+##### Overview:
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.[3-5]d``' intrinsics
-correspond to the ``cp.async.bulk.prefetch.tensor.[1-5]d.L2.global*`` set of PTX
+The '`@llvm.nvvm.cp.async.bulk.tensor.prefetch.im2col.[3-5]d`' intrinsics
+correspond to the `cp.async.bulk.prefetch.tensor.[1-5]d.L2.global*` set of PTX
instructions. These instructions initiate an asynchronous prefetch of tensor
data from global memory to the L2 cache. In im2col mode, some dimensions of the
source tensor are unrolled into a single dimensional column at the destination.
In this mode, the tensor has to be at least three-dimensional. Along with the
tensor coordinates, im2col offsets are also specified (denoted by
-``i16 im2col0...i16 %im2col2``). For ``im2col`` mode, the number of offsets is
+`i16 im2col0...i16 %im2col2`). For `im2col` mode, the number of offsets is
two less than the number of dimensions of the tensor operation. For the
-``im2col.w`` and ``im2col.w.128`` modes, the number of offsets is always 2,
-denoted by ``i16 %wHalo`` and ``i16 %wOffset`` arguments. For more information
-on ``im2col.w`` and ``im2col.w.128`` modes, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-im2col-w-w128-modes>`__.
-
+`im2col.w` and `im2col.w.128` modes, the number of offsets is always 2,
+denoted by `i16 %wHalo` and `i16 %wOffset` arguments. For more information
+on `im2col.w` and `im2col.w.128` modes, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-im2col-w-w128-modes).
The last argument to these intrinsics is a boolean flag, with the same
-functionality as described in the ``tile`` mode intrinsics above.
+functionality as described in the `tile` mode intrinsics above.
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-cp-async-bulk-prefetch-tensor>`__.
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-cp-async-bulk-prefetch-tensor).
-'``llvm.nvvm.cp.async.bulk.tensor.reduce.[red_op].tile.[1-5]d``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.cp.async.bulk.tensor.reduce.[red_op].tile.[1-5]d`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.add.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.min.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.max.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.inc.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.dec.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.and.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.or.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.xor.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.add.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.min.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.max.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.inc.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.dec.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.and.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.or.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.xor.tile.1d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.tile.2d(..., i32 %d0, i32 %d1, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.tile.3d(..., i32 %d0, i32 %d1, i32 %d2, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.tile.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.tile.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+```
- declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.tile.2d(..., i32 %d0, i32 %d1, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.tile.3d(..., i32 %d0, i32 %d1, i32 %d2, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.tile.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.tile.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+##### Overview:
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.tile.[1-5]d``' intrinsics
-correspond to the ``cp.reduce.async.bulk.tensor.[1-5]d.*`` set of PTX
+The '`@llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.tile.[1-5]d`' intrinsics
+correspond to the `cp.reduce.async.bulk.tensor.[1-5]d.*` set of PTX
instructions. These instructions initiate an asynchronous reduction operation of
-tensor data in global memory with the tensor data in shared{::cta} memory, using
-``tile`` mode. The dimension of the tensor data ranges from 1d to 5d with the
-coordinates specified by the ``i32 %d0 ... i32 %d4`` arguments. The supported
+tensor data in global memory with the tensor data in shared\{::cta} memory, using
+`tile` mode. The dimension of the tensor data ranges from 1d to 5d with the
+coordinates specified by the `i32 %d0 ... i32 %d4` arguments. The supported
reduction operations are {add, min, max, inc, dec, and, or, xor} as described in
-the ``tile.1d`` intrinsics.
+the `tile.1d` intrinsics.
-* The last argument to these intrinsics is a boolean flag indicating support for
+- The last argument to these intrinsics is a boolean flag indicating support for
cache_hint. This flag argument must be a compile-time constant. When set, it
- indicates a valid cache_hint (``i64 %ch``) and generates the
- ``.L2::cache_hint`` variant of the PTX instruction.
-
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-reduce-async-bulk-tensor>`__.
+ indicates a valid cache_hint (`i64 %ch`) and generates the
+ `.L2::cache_hint` variant of the PTX instruction.
-'``llvm.nvvm.cp.async.bulk.tensor.reduce.[red_op].im2col.[3-5]d``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-reduce-async-bulk-tensor).
-Syntax:
-"""""""
+#### '`llvm.nvvm.cp.async.bulk.tensor.reduce.[red_op].im2col.[3-5]d`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.im2col.3d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i64 %ch, i1 %flag_ch)
- declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.im2col.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
- declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.im2col.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+```llvm
+declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.im2col.3d(ptr addrspace(3) %src, ptr %tensor_map, i32 %d0, i32 %d1, i32 %d2, i64 %ch, i1 %flag_ch)
+declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.im2col.4d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, ...)
+declare void @llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.im2col.5d(..., i32 %d0, i32 %d1, i32 %d2, i32 %d3, i32 %d4, ...)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.im2col.[3-5]d``'
-intrinsics correspond to the ``cp.reduce.async.bulk.tensor.[3-5]d.*`` set of PTX
+The '`@llvm.nvvm.cp.async.bulk.tensor.reduce.<red_op>.im2col.[3-5]d`'
+intrinsics correspond to the `cp.reduce.async.bulk.tensor.[3-5]d.*` set of PTX
instructions. These instructions initiate an asynchronous reduction operation of
-tensor data in global memory with the tensor data in shared{::cta} memory, using
-``im2col`` mode. In this mode, the tensor has to be at least three-dimensional.
+tensor data in global memory with the tensor data in shared\{::cta} memory, using
+`im2col` mode. In this mode, the tensor has to be at least three-dimensional.
The supported reduction operations supported are the same as the ones in the
tile mode. The last argument to these intrinsics is a boolean flag, with the
-same functionality as described in the ``tile`` mode intrinsics above.
-
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-reduce-async-bulk-tensor>`__.
+same functionality as described in the `tile` mode intrinsics above.
-Warp Group Intrinsics
----------------------
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#data-movement-and-conversion-instructions-cp-reduce-async-bulk-tensor).
-'``llvm.nvvm.wgmma.fence.sync.aligned``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+### Warp Group Intrinsics
-Syntax:
-"""""""
+#### '`llvm.nvvm.wgmma.fence.sync.aligned`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.wgmma.fence.sync.aligned()
+```llvm
+declare void @llvm.nvvm.wgmma.fence.sync.aligned()
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.wgmma.fence.sync.aligned``' intrinsic generates the
-``wgmma.fence.sync.aligned`` PTX instruction, which establishes an ordering
+The '`@llvm.nvvm.wgmma.fence.sync.aligned`' intrinsic generates the
+`wgmma.fence.sync.aligned` PTX instruction, which establishes an ordering
between prior accesses to any warpgroup registers and subsequent accesses to
-the same registers by a ``wgmma.mma_async`` instruction.
+the same registers by a `wgmma.mma_async` instruction.
-The ``wgmma.fence`` instruction must be issued by all warps of the warpgroup in
+The `wgmma.fence` instruction must be issued by all warps of the warpgroup in
the following locations:
-* Before the first ``wgmma.mma_async`` operation in a warpgroup.
-* Between a register access by a thread in the warpgroup and any
- ``wgmma.mma_async`` instruction that accesses the same registers, except when
- these are accumulator register accesses across multiple ``wgmma.mma_async``
+- Before the first `wgmma.mma_async` operation in a warpgroup.
+- Between a register access by a thread in the warpgroup and any
+ `wgmma.mma_async` instruction that accesses the same registers, except when
+ these are accumulator register accesses across multiple `wgmma.mma_async`
instructions of the same shape in which case an ordering guarantee is
provided by default.
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#asynchronous-warpgroup-level-matrix-instructions-wgmma-fence>`__.
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#asynchronous-warpgroup-level-matrix-instructions-wgmma-fence).
-'``llvm.nvvm.wgmma.commit_group.sync.aligned``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.wgmma.commit_group.sync.aligned`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.wgmma.commit_group.sync.aligned()
+```
- declare void @llvm.nvvm.wgmma.commit_group.sync.aligned()
+##### Overview:
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.wgmma.commit_group.sync.aligned``' intrinsic generates the
-``wgmma.commit_group.sync.aligned`` PTX instruction, which creates a new
-wgmma-group per warpgroup and batches all prior ``wgmma.mma_async``
+The '`@llvm.nvvm.wgmma.commit_group.sync.aligned`' intrinsic generates the
+`wgmma.commit_group.sync.aligned` PTX instruction, which creates a new
+wgmma-group per warpgroup and batches all prior `wgmma.mma_async`
instructions initiated by the executing warp but not committed to any
-wgmma-group into the new wgmma-group. If there are no uncommitted ``wgmma
-mma_async`` instructions then, ``wgmma.commit_group`` results in an empty
+wgmma-group into the new wgmma-group. If there are no uncommitted `wgmma
+mma_async` instructions then, `wgmma.commit_group` results in an empty
wgmma-group.
-An executing thread can wait for the completion of all ``wgmma.mma_async``
-operations in a wgmma-group by using ``wgmma.wait_group``.
-
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#asynchronous-warpgroup-level-matrix-instructions-wgmma-commit-group>`__.
+An executing thread can wait for the completion of all `wgmma.mma_async`
+operations in a wgmma-group by using `wgmma.wait_group`.
-'``llvm.nvvm.wgmma.wait_group.sync.aligned``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#asynchronous-warpgroup-level-matrix-instructions-wgmma-commit-group).
-Syntax:
-"""""""
+#### '`llvm.nvvm.wgmma.wait_group.sync.aligned`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.wgmma.wait_group.sync.aligned(i64 immarg N)
+```llvm
+declare void @llvm.nvvm.wgmma.wait_group.sync.aligned(i64 immarg N)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.wgmma.wait_group.sync.aligned``' intrinsic generates the
-``wgmma.commit_group.sync.aligned N`` PTX instruction, which will cause the
-executing thread to wait until only ``N`` or fewer of the most recent
+The '`@llvm.nvvm.wgmma.wait_group.sync.aligned`' intrinsic generates the
+`wgmma.commit_group.sync.aligned N` PTX instruction, which will cause the
+executing thread to wait until only `N` or fewer of the most recent
wgmma-groups are pending and all the prior wgmma-groups committed by the
-executing threads are complete. For example, when ``N`` is 0, the executing
-thread waits on all the prior wgmma-groups to complete. Operand ``N`` is an
+executing threads are complete. For example, when `N` is 0, the executing
+thread waits on all the prior wgmma-groups to complete. Operand `N` is an
integer constant.
Accessing the accumulator register or the input register containing the
-fragments of matrix A of a ``wgmma.mma_async`` instruction without first
-performing a ``wgmma.wait_group`` instruction that waits on a wgmma-group
-including that ``wgmma.mma_async`` instruction is undefined behavior.
+fragments of matrix A of a `wgmma.mma_async` instruction without first
+performing a `wgmma.wait_group` instruction that waits on a wgmma-group
+including that `wgmma.mma_async` instruction is undefined behavior.
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#asynchronous-warpgroup-level-matrix-instructions-wgmma-wait-group>`__.
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#asynchronous-warpgroup-level-matrix-instructions-wgmma-wait-group).
-'``llvm.nvvm.griddepcontrol.*``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.griddepcontrol.*`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.griddepcontrol.launch_dependents()
+declare void @llvm.nvvm.griddepcontrol.wait()
+```
- declare void @llvm.nvvm.griddepcontrol.launch_dependents()
- declare void @llvm.nvvm.griddepcontrol.wait()
+##### Overview:
-Overview:
-"""""""""
-
-The ``griddepcontrol`` intrinsics allows the dependent grids and prerequisite
+The `griddepcontrol` intrinsics allows the dependent grids and prerequisite
grids as defined by the runtime, to control execution in the following way:
-``griddepcontrol.launch_dependents`` intrinsic signals that the dependents can
+`griddepcontrol.launch_dependents` intrinsic signals that the dependents can
be scheduled, before the current grid completes. The intrinsic can be invoked by
multiple threads in the current CTA and repeated invocations of the intrinsic
will have no additional side effects past that of the first invocation.
-``griddepcontrol.wait`` intrinsic causes the executing thread to wait until all
+`griddepcontrol.wait` intrinsic causes the executing thread to wait until all
prerequisite grids in flight have completed and all the memory operations from
the prerequisite grids are performed and made visible to the current grid.
-For more information, refer
-`PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/#parallel-synchronization-and-communication-instructions-griddepcontrol>`__.
-
-Tensormap Replace Intrinsics
-----------------------------
+For more information, refer
+[PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#parallel-synchronization-and-communication-instructions-griddepcontrol).
-These intrinsics modify the fields of the tensor-map object at ``%addr`` in
-``tile`` mode.
+### Tensormap Replace Intrinsics
-For more information, refer to the
-`PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-tensormap-replace>`__.
+These intrinsics modify the fields of the tensor-map object at `%addr` in
+`tile` mode.
-'``llvm.nvvm.tensormap.replace.global.address``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
+For more information, refer to the
+[PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-tensormap-replace).
-.. code-block:: llvm
+#### '`llvm.nvvm.tensormap.replace.global.address`'
- declare void @llvm.nvvm.tensormap.replace.global.address.p1(ptr addrspace(1) %addr, i64 %new_value)
- declare void @llvm.nvvm.tensormap.replace.global.address.p3(ptr addrspace(3) %addr, i64 %new_value)
+##### Syntax:
-Overview:
-"""""""""
+```llvm
+declare void @llvm.nvvm.tensormap.replace.global.address.p1(ptr addrspace(1) %addr, i64 %new_value)
+declare void @llvm.nvvm.tensormap.replace.global.address.p3(ptr addrspace(3) %addr, i64 %new_value)
+```
-The '``@llvm.nvvm.tensormap.replace.global.address.*``' intrinsics replace the
-``global_address`` field of the tensor-map object with ``%new_value``.
+##### Overview:
-'``llvm.nvvm.tensormap.replace.rank``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The '`@llvm.nvvm.tensormap.replace.global.address.*`' intrinsics replace the
+`global_address` field of the tensor-map object with `%new_value`.
-Syntax:
-"""""""
+#### '`llvm.nvvm.tensormap.replace.rank`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.tensormap.replace.rank.p1(ptr addrspace(1) %addr, i32 %new_value)
- declare void @llvm.nvvm.tensormap.replace.rank.p3(ptr addrspace(3) %addr, i32 %new_value)
+```llvm
+declare void @llvm.nvvm.tensormap.replace.rank.p1(ptr addrspace(1) %addr, i32 %new_value)
+declare void @llvm.nvvm.tensormap.replace.rank.p3(ptr addrspace(3) %addr, i32 %new_value)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.tensormap.replace.rank.*``' intrinsics replace the ``rank``
-field of the tensor-map object with ``%new_value`` which must be one less than
+The '`@llvm.nvvm.tensormap.replace.rank.*`' intrinsics replace the `rank`
+field of the tensor-map object with `%new_value` which must be one less than
the desired tensor rank as this field uses zero-based numbering.
-'``llvm.nvvm.tensormap.replace.global.stride``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
+#### '`llvm.nvvm.tensormap.replace.global.stride`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.tensormap.replace.global.stride.p1(ptr addrspace(1) %addr, i32 immarg %ord, i64 %new_value)
- declare void @llvm.nvvm.tensormap.replace.global.stride.p3(ptr addrspace(3) %addr, i32 immarg %ord, i64 %new_value)
+```llvm
+declare void @llvm.nvvm.tensormap.replace.global.stride.p1(ptr addrspace(1) %addr, i32 immarg %ord, i64 %new_value)
+declare void @llvm.nvvm.tensormap.replace.global.stride.p3(ptr addrspace(3) %addr, i32 immarg %ord, i64 %new_value)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.tensormap.replace.global.stride.*``' intrinsics replace the
-``%ord``-th element of the ``global_stride`` field of the tensor-map object
-with ``%new_value``. ``%ord`` must be in the range [0, 4].
+The '`@llvm.nvvm.tensormap.replace.global.stride.*`' intrinsics replace the
+`%ord`-th element of the `global_stride` field of the tensor-map object
+with `%new_value`. `%ord` must be in the range [0, 4].
-'``llvm.nvvm.tensormap.replace.element.stride``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.tensormap.replace.element.stride`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.tensormap.replace.element.stride.p1(ptr addrspace(1) %addr, i32 immarg %ord, i32 %new_value)
+declare void @llvm.nvvm.tensormap.replace.element.stride.p3(ptr addrspace(3) %addr, i32 immarg %ord, i32 %new_value)
+```
- declare void @llvm.nvvm.tensormap.replace.element.stride.p1(ptr addrspace(1) %addr, i32 immarg %ord, i32 %new_value)
- declare void @llvm.nvvm.tensormap.replace.element.stride.p3(ptr addrspace(3) %addr, i32 immarg %ord, i32 %new_value)
+##### Overview:
-Overview:
-"""""""""
+The '`@llvm.nvvm.tensormap.replace.element.stride.*`' intrinsics replace the
+`%ord`-th element of the `element_stride` field of the tensor-map object
+with `%new_value`. `%ord` must be in the range [0, 4].
-The '``@llvm.nvvm.tensormap.replace.element.stride.*``' intrinsics replace the
-``%ord``-th element of the ``element_stride`` field of the tensor-map object
-with ``%new_value``. ``%ord`` must be in the range [0, 4].
+#### '`llvm.nvvm.tensormap.replace.global.dim`'
-'``llvm.nvvm.tensormap.replace.global.dim``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+##### Syntax:
-Syntax:
-"""""""
+```llvm
+declare void @llvm.nvvm.tensormap.replace.global.dim.p1(ptr addrspace(1) %addr, i32 immarg %ord, i32 %new_value)
+declare void @llvm.nvvm.tensormap.replace.global.dim.p3(ptr addrspace(3) %addr, i32 immarg %ord, i32 %new_value)
+```
-.. code-block:: llvm
+##### Overview:
- declare void @llvm.nvvm.tensormap.replace.global.dim.p1(ptr addrspace(1) %addr, i32 immarg %ord, i32 %new_value)
- declare void @llvm.nvvm.tensormap.replace.global.dim.p3(ptr addrspace(3) %addr, i32 immarg %ord, i32 %new_value)
+The '`@llvm.nvvm.tensormap.replace.global.dim.*`' intrinsics replace the
+`%ord`-th element of the `global_dim` field of the tensor-map object
+with `%new_value`. `%ord` must be in the range [0, 4].
-Overview:
-"""""""""
+#### '`llvm.nvvm.tensormap.replace.box.dim`'
-The '``@llvm.nvvm.tensormap.replace.global.dim.*``' intrinsics replace the
-``%ord``-th element of the ``global_dim`` field of the tensor-map object
-with ``%new_value``. ``%ord`` must be in the range [0, 4].
+##### Syntax:
-'``llvm.nvvm.tensormap.replace.box.dim``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+```llvm
+declare void @llvm.nvvm.tensormap.replace.box.dim.p1(ptr addrspace(1) %addr, i32 immarg %ord, i32 %new_value)
+declare void @llvm.nvvm.tensormap.replace.box.dim.p3(ptr addrspace(3) %addr, i32 immarg %ord, i32 %new_value)
+```
-Syntax:
-"""""""
+##### Overview:
-.. code-block:: llvm
+The '`@llvm.nvvm.tensormap.replace.box.dim.*`' intrinsics replace the
+`%ord`-th element of the `box_dim` field of the tensor-map object with
+`%new_value`. `%ord` must be in the range [0, 4].
- declare void @llvm.nvvm.tensormap.replace.box.dim.p1(ptr addrspace(1) %addr, i32 immarg %ord, i32 %new_value)
- declare void @llvm.nvvm.tensormap.replace.box.dim.p3(ptr addrspace(3) %addr, i32 immarg %ord, i32 %new_value)
+#### '`llvm.nvvm.tensormap.replace.elemtype`'
-Overview:
-"""""""""
+##### Syntax:
-The '``@llvm.nvvm.tensormap.replace.box.dim.*``' intrinsics replace the
-``%ord``-th element of the ``box_dim`` field of the tensor-map object with
-``%new_value``. ``%ord`` must be in the range [0, 4].
+```llvm
+declare void @llvm.nvvm.tensormap.replace.elemtype.p1(ptr addrspace(1) %addr, i32 immarg %new_value)
+declare void @llvm.nvvm.tensormap.replace.elemtype.p3(ptr addrspace(3) %addr, i32 immarg %new_value)
+```
-'``llvm.nvvm.tensormap.replace.elemtype``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+##### Overview:
-Syntax:
-"""""""
+The '`@llvm.nvvm.tensormap.replace.elemtype.*`' intrinsics replace the
+`elemtype` field of the tensor-map object with the type specified by
+`%new_value`.
-.. code-block:: llvm
+##### Semantics:
- declare void @llvm.nvvm.tensormap.replace.elemtype.p1(ptr addrspace(1) %addr, i32 immarg %new_value)
- declare void @llvm.nvvm.tensormap.replace.elemtype.p3(ptr addrspace(3) %addr, i32 immarg %new_value)
-
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.tensormap.replace.elemtype.*``' intrinsics replace the
-``elemtype`` field of the tensor-map object with the type specified by
-``%new_value``.
-
-Semantics:
-""""""""""
-
-The following table shows the mapping of ``%new_value`` to the actual element
+The following table shows the mapping of `%new_value` to the actual element
type:
- ============================ =====
- Element Type Value
- ============================ =====
- ``u8`` 0
- ``u16`` 1
- ``u32`` 2
- ``s32`` 3
- ``u64`` 4
- ``s64`` 5
- ``f16`` 6
- ``f32`` 7
- ``f32.ftz`` 8
- ``f64`` 9
- ``bf16`` 10
- ``tf32`` 11
- ``tf32.ftz`` 12
- ``b4x16`` 13
- ``b4x16_p64`` 14
- ``b6x16_p32`` or ``b6p2x16`` 15
- ============================ =====
-
-'``llvm.nvvm.tensormap.replace.interleave.layout``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
+| Element Type | Value |
+| ------------------------ | ----- |
+| `u8` | 0 |
+| `u16` | 1 |
+| `u32` | 2 |
+| `s32` | 3 |
+| `u64` | 4 |
+| `s64` | 5 |
+| `f16` | 6 |
+| `f32` | 7 |
+| `f32.ftz` | 8 |
+| `f64` | 9 |
+| `bf16` | 10 |
+| `tf32` | 11 |
+| `tf32.ftz` | 12 |
+| `b4x16` | 13 |
+| `b4x16_p64` | 14 |
+| `b6x16_p32` or `b6p2x16` | 15 |
-.. code-block:: llvm
+#### '`llvm.nvvm.tensormap.replace.interleave.layout`'
- declare void @llvm.nvvm.tensormap.replace.interleave.layout.p1(ptr addrspace(1) %addr, i32 immarg %new_value)
- declare void @llvm.nvvm.tensormap.replace.interleave.layout.p3(ptr addrspace(3) %addr, i32 immarg %new_value)
+##### Syntax:
-Overview:
-"""""""""
+```llvm
+declare void @llvm.nvvm.tensormap.replace.interleave.layout.p1(ptr addrspace(1) %addr, i32 immarg %new_value)
+declare void @llvm.nvvm.tensormap.replace.interleave.layout.p3(ptr addrspace(3) %addr, i32 immarg %new_value)
+```
-The '``@llvm.nvvm.tensormap.replace.interleave.layout.*``' intrinsics replace
-the ``interleave_layout`` field of the tensor-map object with the layout
-specified by ``%new_value``.
+##### Overview:
-Semantics:
-""""""""""
+The '`@llvm.nvvm.tensormap.replace.interleave.layout.*`' intrinsics replace
+the `interleave_layout` field of the tensor-map object with the layout
+specified by `%new_value`.
-The following table shows the mapping of ``%new_value`` to the actual layout:
+##### Semantics:
- ================== =====
- Interleave Layout Value
- ================== =====
- ``No interleave`` 0
- ``16B interleave`` 1
- ``32B interleave`` 2
- ================== =====
+The following table shows the mapping of `%new_value` to the actual layout:
-'``llvm.nvvm.tensormap.replace.swizzle_mode``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+| Interleave Layout | Value |
+| ----------------- | ----- |
+| `No interleave` | 0 |
+| `16B interleave` | 1 |
+| `32B interleave` | 2 |
-Syntax:
-"""""""
+#### '`llvm.nvvm.tensormap.replace.swizzle_mode`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.tensormap.replace.swizzle.mode.p1(ptr addrspace(1) %addr, i32 immarg %new_value)
- declare void @llvm.nvvm.tensormap.replace.swizzle.mode.p3(ptr addrspace(3) %addr, i32 immarg %new_value)
+```llvm
+declare void @llvm.nvvm.tensormap.replace.swizzle.mode.p1(ptr addrspace(1) %addr, i32 immarg %new_value)
+declare void @llvm.nvvm.tensormap.replace.swizzle.mode.p3(ptr addrspace(3) %addr, i32 immarg %new_value)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.tensormap.replace.swizzle.mode.*``' intrinsics replace the
-``swizzle_mode`` field of the tensor-map object with the swizzle mode specified
-by ``%new_value``.
+The '`@llvm.nvvm.tensormap.replace.swizzle.mode.*`' intrinsics replace the
+`swizzle_mode` field of the tensor-map object with the swizzle mode specified
+by `%new_value`.
-Semantics:
-""""""""""
+##### Semantics:
-The following table shows the mapping of ``%new_value`` to the actual swizzle
+The following table shows the mapping of `%new_value` to the actual swizzle
mode:
- ================ =====
- Swizzle Mode Value
- ================ =====
- ``No swizzle`` 0
- ``32B swizzle`` 1
- ``64B swizzle`` 2
- ``128B swizzle`` 3
- ``96B swizzle`` 4
- ================ =====
-
-'``llvm.nvvm.tensormap.replace.swizzle_atomicity``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+| Swizzle Mode | Value |
+| -------------- | ----- |
+| `No swizzle` | 0 |
+| `32B swizzle` | 1 |
+| `64B swizzle` | 2 |
+| `128B swizzle` | 3 |
+| `96B swizzle` | 4 |
-Syntax:
-"""""""
+#### '`llvm.nvvm.tensormap.replace.swizzle_atomicity`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.tensormap.replace.swizzle.atomicity.p1(ptr addrspace(1) %addr, i32 immarg %new_value)
- declare void @llvm.nvvm.tensormap.replace.swizzle.atomicity.p3(ptr addrspace(3) %addr, i32 immarg %new_value)
+```llvm
+declare void @llvm.nvvm.tensormap.replace.swizzle.atomicity.p1(ptr addrspace(1) %addr, i32 immarg %new_value)
+declare void @llvm.nvvm.tensormap.replace.swizzle.atomicity.p3(ptr addrspace(3) %addr, i32 immarg %new_value)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.tensormap.replace.swizzle.atomicity.*``' intrinsics replace
-the ``swizzle_atomicity`` field of the tensor-map object with the swizzle
-atomicity specified by ``%new_value``.
+The '`@llvm.nvvm.tensormap.replace.swizzle.atomicity.*`' intrinsics replace
+the `swizzle_atomicity` field of the tensor-map object with the swizzle
+atomicity specified by `%new_value`.
-Semantics:
-""""""""""
+##### Semantics:
-The following table shows the mapping of ``%new_value`` to the actual swizzle
+The following table shows the mapping of `%new_value` to the actual swizzle
atomicity:
- ================= =====
- Swizzle Atomicity Value
- ================= =====
- ``16B`` 0
- ``32B`` 1
- ``32B + 8B flip`` 2
- ``64B`` 3
- ================= =====
+| Swizzle Atomicity | Value |
+| ----------------- | ----- |
+| `16B` | 0 |
+| `32B` | 1 |
+| `32B + 8B flip` | 2 |
+| `64B` | 3 |
-'``llvm.nvvm.tensormap.replace.fill_mode``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.tensormap.replace.fill_mode`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.tensormap.replace.fill.mode.p1(ptr addrspace(1) %addr, i32 immarg %new_value)
+declare void @llvm.nvvm.tensormap.replace.fill.mode.p3(ptr addrspace(3) %addr, i32 immarg %new_value)
+```
- declare void @llvm.nvvm.tensormap.replace.fill.mode.p1(ptr addrspace(1) %addr, i32 immarg %new_value)
- declare void @llvm.nvvm.tensormap.replace.fill.mode.p3(ptr addrspace(3) %addr, i32 immarg %new_value)
+##### Overview:
-Overview:
-"""""""""
+The '`@llvm.nvvm.tensormap.replace.fill.mode.*`' intrinsics replace the
+`fill_mode` field of the tensor-map object with the fill mode specified by
+`%new_value`.
-The '``@llvm.nvvm.tensormap.replace.fill.mode.*``' intrinsics replace the
-``fill_mode`` field of the tensor-map object with the fill mode specified by
-``%new_value``.
+##### Semantics:
-Semantics:
-""""""""""
+The following table shows the mapping of `%new_value` to the actual fill mode:
-The following table shows the mapping of ``%new_value`` to the actual fill mode:
+| Fill Mode | Value |
+| -------------- | ----- |
+| `Zero fill` | 0 |
+| `OOB-NaN fill` | 1 |
- ================ =====
- Fill Mode Value
- ================ =====
- ``Zero fill`` 0
- ``OOB-NaN fill`` 1
- ================ =====
+### TCGEN05 family of Intrinsics
-TCGEN05 family of Intrinsics
-----------------------------
+The llvm.nvvm.tcgen05.\* intrinsics model the TCGEN05 family of instructions
+exposed by PTX. These intrinsics use 'Tensor Memory' (henceforth `tmem`).
+NVPTX represents this memory using `addrspace(6)` and is always 32-bits.
-The llvm.nvvm.tcgen05.* intrinsics model the TCGEN05 family of instructions
-exposed by PTX. These intrinsics use 'Tensor Memory' (henceforth ``tmem``).
-NVPTX represents this memory using ``addrspace(6)`` and is always 32-bits.
-
-For more information, refer to the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-memory>`__.
+For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-memory).
The tensor-memory pointers may only be used with the tcgen05 intrinsics.
There are specialized load/store instructions provided (tcgen05.ld/st) to
work with tensor-memory.
-For more information on tensor-memory load/store instructions, refer to `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-memory-and-register-load-store-instructions>`__.
-
-'``llvm.nvvm.tcgen05.alloc``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information on tensor-memory load/store instructions, refer to [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-memory-and-register-load-store-instructions).
-Syntax:
-"""""""
+#### '`llvm.nvvm.tcgen05.alloc`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.tcgen05.alloc.cg1(ptr %dst, i32 %ncols)
- declare void @llvm.nvvm.tcgen05.alloc.cg2(ptr %dst, i32 %ncols)
- declare void @llvm.nvvm.tcgen05.alloc.shared.cg1(ptr addrspace(3) %dst, i32 %ncols)
- declare void @llvm.nvvm.tcgen05.alloc.shared.cg2(ptr addrspace(3) %dst, i32 %ncols)
+```llvm
+declare void @llvm.nvvm.tcgen05.alloc.cg1(ptr %dst, i32 %ncols)
+declare void @llvm.nvvm.tcgen05.alloc.cg2(ptr %dst, i32 %ncols)
+declare void @llvm.nvvm.tcgen05.alloc.shared.cg1(ptr addrspace(3) %dst, i32 %ncols)
+declare void @llvm.nvvm.tcgen05.alloc.shared.cg2(ptr addrspace(3) %dst, i32 %ncols)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.tcgen05.alloc.*``' intrinsics correspond to the
-``tcgen05.alloc.cta_group*.sync.aligned.b32`` family of PTX instructions.
-The ``tcgen05.alloc`` is a potentially blocking instruction which dynamically
+The '`@llvm.nvvm.tcgen05.alloc.*`' intrinsics correspond to the
+`tcgen05.alloc.cta_group*.sync.aligned.b32` family of PTX instructions.
+The `tcgen05.alloc` is a potentially blocking instruction which dynamically
allocates the specified number of columns in the Tensor Memory and writes the
address of the allocated Tensor Memory into shared memory at the location
-specified by ``%dst``. The 32-bit operand ``%ncols`` specifies the number of
-columns to be allocated and it must be a power-of-two. The ``.shared`` variant
-explicitly uses shared memory address space for the ``%dst`` operand. The
-``.cg1`` and ``.cg2`` variants generate ``cta_group::1`` and ``cta_group::2``
+specified by `%dst`. The 32-bit operand `%ncols` specifies the number of
+columns to be allocated and it must be a power-of-two. The `.shared` variant
+explicitly uses shared memory address space for the `%dst` operand. The
+`.cg1` and `.cg2` variants generate `cta_group::1` and `cta_group::2`
variants of the instruction respectively.
-For more information, refer to the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-memory-allocation-and-management-instructions>`__.
+For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-memory-allocation-and-management-instructions).
-'``llvm.nvvm.tcgen05.dealloc``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.tcgen05.dealloc`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.tcgen05.dealloc.cg1(ptr addrspace(6) %tmem_addr, i32 %ncols)
+declare void @llvm.nvvm.tcgen05.dealloc.cg2(ptr addrspace(6) %tmem_addr, i32 %ncols)
+```
- declare void @llvm.nvvm.tcgen05.dealloc.cg1(ptr addrspace(6) %tmem_addr, i32 %ncols)
- declare void @llvm.nvvm.tcgen05.dealloc.cg2(ptr addrspace(6) %tmem_addr, i32 %ncols)
+##### Overview:
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.tcgen05.dealloc.*``' intrinsics correspond to the
-``tcgen05.dealloc.*`` set of PTX instructions. The ``tcgen05.dealloc``
+The '`@llvm.nvvm.tcgen05.dealloc.*`' intrinsics correspond to the
+`tcgen05.dealloc.*` set of PTX instructions. The `tcgen05.dealloc`
instructions deallocates the Tensor Memory specified by the Tensor Memory
-address ``%tmem_addr``. The operand ``%tmem_addr`` must point to a previous
-Tensor Memory allocation. The 32-bit operand ``%ncols`` specifies the number
-of columns to be de-allocated. The ``.cg1`` and ``.cg2`` variants generate
-``cta_group::1`` and ``cta_group::2`` variants of the instruction respectively.
-
-For more information, refer to the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-memory-allocation-and-management-instructions>`__.
+address `%tmem_addr`. The operand `%tmem_addr` must point to a previous
+Tensor Memory allocation. The 32-bit operand `%ncols` specifies the number
+of columns to be de-allocated. The `.cg1` and `.cg2` variants generate
+`cta_group::1` and `cta_group::2` variants of the instruction respectively.
-'``llvm.nvvm.tcgen05.relinq.alloc.permit``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-memory-allocation-and-management-instructions).
-Syntax:
-"""""""
+#### '`llvm.nvvm.tcgen05.relinq.alloc.permit`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.tcgen05.relinq.alloc.permit.cg1()
- declare void @llvm.nvvm.tcgen05.relinq.alloc.permit.cg2()
+```llvm
+declare void @llvm.nvvm.tcgen05.relinq.alloc.permit.cg1()
+declare void @llvm.nvvm.tcgen05.relinq.alloc.permit.cg2()
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.tcgen05.relinq.alloc.permit.*``' intrinsics correspond to the
-``tcgen05.relinquish_alloc_permit.*`` set of PTX instructions. This instruction
+The '`@llvm.nvvm.tcgen05.relinq.alloc.permit.*`' intrinsics correspond to the
+`tcgen05.relinquish_alloc_permit.*` set of PTX instructions. This instruction
specifies that the CTA of the executing thread is relinquishing the right to
-allocate Tensor Memory. So, it is illegal for a CTA to perform ``tcgen05.alloc``
+allocate Tensor Memory. So, it is illegal for a CTA to perform `tcgen05.alloc`
after any of its constituent threads execute
-``tcgen05.relinquish_alloc_permit``. The ``.cg1`` and ``.cg2`` variants generate
-``cta_group::1`` and ``cta_group::2`` flavors of the instruction respectively.
+`tcgen05.relinquish_alloc_permit`. The `.cg1` and `.cg2` variants generate
+`cta_group::1` and `cta_group::2` flavors of the instruction respectively.
-For more information, refer to the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-memory-allocation-and-management-instructions>`__.
+For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tensor-memory-allocation-and-management-instructions).
-'``llvm.nvvm.tcgen05.commit``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.tcgen05.commit`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.tcgen05.commit.{cg1,cg2}(ptr %mbar)
+declare void @llvm.nvvm.tcgen05.commit.shared.{cg1,cg2}(ptr addrspace(3) %mbar)
+declare void @llvm.nvvm.tcgen05.commit.mc.{cg1,cg2}(ptr %mbar, i16 %mc)
+declare void @llvm.nvvm.tcgen05.commit.mc.shared.{cg1,cg2}(ptr addrspace(3) %mbar, i16 %mc)
+```
- declare void @llvm.nvvm.tcgen05.commit.{cg1,cg2}(ptr %mbar)
- declare void @llvm.nvvm.tcgen05.commit.shared.{cg1,cg2}(ptr addrspace(3) %mbar)
- declare void @llvm.nvvm.tcgen05.commit.mc.{cg1,cg2}(ptr %mbar, i16 %mc)
- declare void @llvm.nvvm.tcgen05.commit.mc.shared.{cg1,cg2}(ptr addrspace(3) %mbar, i16 %mc)
+##### Overview:
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.tcgen05.commit.*``' intrinsics correspond to the
-``tcgen05.commit.{cg1/cg2}.mbarrier::arrive::one.*`` set of PTX instructions.
-The ``tcgen05.commit`` is an asynchronous instruction which makes the mbarrier
-object (``%mbar``) track the completion of all prior asynchronous tcgen05
-operations. The ``.mc`` variants allow signaling on the mbarrier objects of
-multiple CTAs (specified by ``%mc``) in the cluster. The ``.cg1`` and ``.cg2``
-variants generate ```cta_group::1`` and ``cta_group::2`` flavors of the
+The '`@llvm.nvvm.tcgen05.commit.*`' intrinsics correspond to the
+`tcgen05.commit.{cg1/cg2}.mbarrier::arrive::one.*` set of PTX instructions.
+The `tcgen05.commit` is an asynchronous instruction which makes the mbarrier
+object (`%mbar`) track the completion of all prior asynchronous tcgen05
+operations. The `.mc` variants allow signaling on the mbarrier objects of
+multiple CTAs (specified by `%mc`) in the cluster. The `.cg1` and `.cg2`
+variants generate `cta_group::1` and `cta_group::2` flavors of the
instruction respectively.
-For more information, refer to the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen-async-sync-operations-commit>`__.
-
-'``llvm.nvvm.tcgen05.wait``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
+For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen-async-sync-operations-commit).
-.. code-block:: llvm
+#### '`llvm.nvvm.tcgen05.wait`'
- declare void @llvm.nvvm.tcgen05.wait.ld()
- declare void @llvm.nvvm.tcgen05.wait.st()
+##### Syntax:
-Overview:
-"""""""""
+```llvm
+declare void @llvm.nvvm.tcgen05.wait.ld()
+declare void @llvm.nvvm.tcgen05.wait.st()
+```
-The '``@llvm.nvvm.tcgen05.wait.ld/st``' intrinsics correspond to the
-``tcgen05.wait::{ld/st}.sync.aligned`` pair of PTX instructions. The
-``tcgen05.wait::ld`` causes the executing thread to block until all prior
-``tcgen05.ld`` operations issued by the executing thread have completed. The
-``tcgen05.wait::st`` causes the executing thread to block until all prior
-``tcgen05.st`` operations issued by the executing thread have completed.
+##### Overview:
-For more information, refer to the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instructions-tcgen05-wait>`__.
+The '`@llvm.nvvm.tcgen05.wait.ld/st`' intrinsics correspond to the
+`tcgen05.wait::{ld/st}.sync.aligned` pair of PTX instructions. The
+`tcgen05.wait::ld` causes the executing thread to block until all prior
+`tcgen05.ld` operations issued by the executing thread have completed. The
+`tcgen05.wait::st` causes the executing thread to block until all prior
+`tcgen05.st` operations issued by the executing thread have completed.
-'``llvm.nvvm.tcgen05.fence``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instructions-tcgen05-wait).
-Syntax:
-"""""""
+#### '`llvm.nvvm.tcgen05.fence`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.tcgen05.fence.before.thread.sync()
- declare void @llvm.nvvm.tcgen05.fence.after.thread.sync()
+```llvm
+declare void @llvm.nvvm.tcgen05.fence.before.thread.sync()
+declare void @llvm.nvvm.tcgen05.fence.after.thread.sync()
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.tcgen05.fence.*``' intrinsics correspond to the
-``tcgen05.fence::{before/after}_thread_sync`` pair of PTX instructions. These
+The '`@llvm.nvvm.tcgen05.fence.*`' intrinsics correspond to the
+`tcgen05.fence::{before/after}_thread_sync` pair of PTX instructions. These
instructions act as code motion fences for asynchronous tcgen05 operations.
-For more information, refer to the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tensorcore-5th-generation-instructions-tcgen05-fence>`__.
+For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tensorcore-5th-generation-instructions-tcgen05-fence).
-'``llvm.nvvm.tcgen05.shift``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.tcgen05.shift`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.tcgen05.shift.down.cg1(ptr addrspace(6) %tmem_addr)
+declare void @llvm.nvvm.tcgen05.shift.down.cg2(ptr addrspace(6) %tmem_addr)
+```
- declare void @llvm.nvvm.tcgen05.shift.down.cg1(ptr addrspace(6) %tmem_addr)
- declare void @llvm.nvvm.tcgen05.shift.down.cg2(ptr addrspace(6) %tmem_addr)
+##### Overview:
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.tcgen05.shift.{cg1/cg2}``' intrinsics correspond to the
-``tcgen05.shift.{cg1/cg2}`` PTX instructions. The ``tcgen05.shift`` is an
+The '`@llvm.nvvm.tcgen05.shift.{cg1/cg2}`' intrinsics correspond to the
+`tcgen05.shift.{cg1/cg2}` PTX instructions. The `tcgen05.shift` is an
asynchronous instruction which initiates the shifting of 32-byte elements
downwards across all the rows, except the last, by one row. The address operand
-``%tmem_addr`` specifies the base address of the matrix in the Tensor Memory
+`%tmem_addr` specifies the base address of the matrix in the Tensor Memory
whose rows must be down shifted.
-For more information, refer to the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instructions-tcgen05-shift>`__.
-
-'``llvm.nvvm.tcgen05.cp``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
-
-.. code-block:: llvm
-
- declare void @llvm.nvvm.tcgen05.cp.4x256b.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
- declare void @llvm.nvvm.tcgen05.cp.128x256b.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
- declare void @llvm.nvvm.tcgen05.cp.128x128b.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
- declare void @llvm.nvvm.tcgen05.cp.32x128b_warpx4.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
- declare void @llvm.nvvm.tcgen05.cp.64x128b_warpx2_02_13.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
- declare void @llvm.nvvm.tcgen05.cp.64x128b_warpx2_01_23.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
-
- declare void @llvm.nvvm.tcgen05.cp.4x256b.b6x16_p32.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
- declare void @llvm.nvvm.tcgen05.cp.128x256b.b6x16_p32.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
- declare void @llvm.nvvm.tcgen05.cp.128x128b.b6x16_p32.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
- declare void @llvm.nvvm.tcgen05.cp.32x128b_warpx4.b6x16_p32.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
- declare void @llvm.nvvm.tcgen05.cp.64x128b_warpx2_02_13.b6x16_p32.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
- declare void @llvm.nvvm.tcgen05.cp.64x128b_warpx2_01_23.b6x16_p32.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
-
- declare void @llvm.nvvm.tcgen05.cp.4x256b.b4x16_p64.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
- declare void @llvm.nvvm.tcgen05.cp.128x256b.b4x16_p64.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
- declare void @llvm.nvvm.tcgen05.cp.128x128b.b4x16_p64.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
- declare void @llvm.nvvm.tcgen05.cp.32x128b_warpx4.b4x16_p64.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
- declare void @llvm.nvvm.tcgen05.cp.64x128b_warpx2_02_13.b4x16_p64.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
- declare void @llvm.nvvm.tcgen05.cp.64x128b_warpx2_01_23.b4x16_p64.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
-
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.tcgen05.cp.{shape}.{src_fmt}.{cg1/cg2}``' intrinsics
-correspond to the ``tcgen05.cp.*`` family of PTX instructions. The
-``tcgen05.cp`` instruction initiates an asynchronous copy operation from shared
-memory to the location specified by ``%tmem_addr`` in Tensor Memory. The 64-bit
-register operand ``%sdesc`` is the matrix descriptor representing the source
+For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instructions-tcgen05-shift).
+
+#### '`llvm.nvvm.tcgen05.cp`'
+
+##### Syntax:
+
+```llvm
+declare void @llvm.nvvm.tcgen05.cp.4x256b.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+declare void @llvm.nvvm.tcgen05.cp.128x256b.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+declare void @llvm.nvvm.tcgen05.cp.128x128b.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+declare void @llvm.nvvm.tcgen05.cp.32x128b_warpx4.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+declare void @llvm.nvvm.tcgen05.cp.64x128b_warpx2_02_13.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+declare void @llvm.nvvm.tcgen05.cp.64x128b_warpx2_01_23.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+
+declare void @llvm.nvvm.tcgen05.cp.4x256b.b6x16_p32.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+declare void @llvm.nvvm.tcgen05.cp.128x256b.b6x16_p32.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+declare void @llvm.nvvm.tcgen05.cp.128x128b.b6x16_p32.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+declare void @llvm.nvvm.tcgen05.cp.32x128b_warpx4.b6x16_p32.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+declare void @llvm.nvvm.tcgen05.cp.64x128b_warpx2_02_13.b6x16_p32.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+declare void @llvm.nvvm.tcgen05.cp.64x128b_warpx2_01_23.b6x16_p32.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+
+declare void @llvm.nvvm.tcgen05.cp.4x256b.b4x16_p64.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+declare void @llvm.nvvm.tcgen05.cp.128x256b.b4x16_p64.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+declare void @llvm.nvvm.tcgen05.cp.128x128b.b4x16_p64.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+declare void @llvm.nvvm.tcgen05.cp.32x128b_warpx4.b4x16_p64.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+declare void @llvm.nvvm.tcgen05.cp.64x128b_warpx2_02_13.b4x16_p64.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+declare void @llvm.nvvm.tcgen05.cp.64x128b_warpx2_01_23.b4x16_p64.{cg1,cg2}(ptr addrspace(6) %tmem_addr, i64 %sdesc)
+```
+
+##### Overview:
+
+The '`@llvm.nvvm.tcgen05.cp.{shape}.{src_fmt}.{cg1/cg2}`' intrinsics
+correspond to the `tcgen05.cp.*` family of PTX instructions. The
+`tcgen05.cp` instruction initiates an asynchronous copy operation from shared
+memory to the location specified by `%tmem_addr` in Tensor Memory. The 64-bit
+register operand `%sdesc` is the matrix descriptor representing the source
matrix in shared memory that needs to be copied.
The valid shapes for the copy operation are:
{128x256b, 4x256b, 128x128b, 64x128b_warpx2_02_13, 64x128b_warpx2_01_23,
32x128b_warpx4}.
-Shapes ``64x128b`` and ``32x128b`` require dedicated multicast qualifiers,
+Shapes `64x128b` and `32x128b` require dedicated multicast qualifiers,
which are appended to the corresponding intrinsic names.
Optionally, the data can be decompressed from the source format in the shared
memory to the destination format in Tensor Memory during the copy operation.
-Currently, only ``.b8x16`` is supported as destination format. The valid source
-formats are ``.b6x16_p32`` and ``.b4x16_p64``.
-
-When the source format is ``.b6x16_p32``, a contiguous set of 16 elements of
-6-bits each followed by four bytes of padding (``_p32``) in shared memory is
-decompressed into 16 elements of 8-bits (``.b8x16``) each in the Tensor Memory.
+Currently, only `.b8x16` is supported as destination format. The valid source
+formats are `.b6x16_p32` and `.b4x16_p64`.
-When the source format is ``.b4x16_p64``, a contiguous set of 16 elements of
-4-bits each followed by eight bytes of padding (``_p64``) in shared memory is
-decompressed into 16 elements of 8-bits (``.b8x16``) each in the Tensor Memory.
+When the source format is `.b6x16_p32`, a contiguous set of 16 elements of
+6-bits each followed by four bytes of padding (`_p32`) in shared memory is
+decompressed into 16 elements of 8-bits (`.b8x16`) each in the Tensor Memory.
-For more information on the decompression schemes, refer to the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#optional-decompression>`__.
+When the source format is `.b4x16_p64`, a contiguous set of 16 elements of
+4-bits each followed by eight bytes of padding (`_p64`) in shared memory is
+decompressed into 16 elements of 8-bits (`.b8x16`) each in the Tensor Memory.
-For more information on the tcgen05.cp instruction, refer to the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instructions-tcgen05-cp>`__.
+For more information on the decompression schemes, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#optional-decompression).
-'``llvm.nvvm.tcgen05.ld.*``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information on the tcgen05.cp instruction, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instructions-tcgen05-cp).
-Syntax:
-"""""""
+#### '`llvm.nvvm.tcgen05.ld.*`'
-.. code-block:: llvm
+##### Syntax:
- declare <n x i32> @llvm.nvvm.tcgen05.ld.<shape>.<num>(ptr addrspace(6) %tmem_addr, i1 %pack)
+```llvm
+declare <n x i32> @llvm.nvvm.tcgen05.ld.<shape>.<num>(ptr addrspace(6) %tmem_addr, i1 %pack)
- declare <n x i32> @llvm.nvvm.tcgen05.ld.red.32x32b.<num>.i32(ptr addrspace(6) %tmem_addr, i32 %redOp)
+declare <n x i32> @llvm.nvvm.tcgen05.ld.red.32x32b.<num>.i32(ptr addrspace(6) %tmem_addr, i32 %redOp)
- declare <n x i32> @llvm.nvvm.tcgen05.ld.red.32x32b.<num>.f32(ptr addrspace(6) %tmem_addr, i32 %redOp, i1 %abs, i1 %nan)
+declare <n x i32> @llvm.nvvm.tcgen05.ld.red.32x32b.<num>.f32(ptr addrspace(6) %tmem_addr, i32 %redOp, i1 %abs, i1 %nan)
- declare <n x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.<num>(ptr addrspace(6) %tmem_addr, i64 %offset, i1 %pack)
+declare <n x i32> @llvm.nvvm.tcgen05.ld.16x32bx2.<num>(ptr addrspace(6) %tmem_addr, i64 %offset, i1 %pack)
- declare <n x i32> @llvm.nvvm.tcgen05.ld.red.16x32bx2.<num>.i32(ptr addrspace(6) %tmem_addr, i64 %offset, i32 %redOp)
+declare <n x i32> @llvm.nvvm.tcgen05.ld.red.16x32bx2.<num>.i32(ptr addrspace(6) %tmem_addr, i64 %offset, i32 %redOp)
- declare <n x i32> @llvm.nvvm.tcgen05.ld.red.16x32bx2.<num>.f32(ptr addrspace(6) %tmem_addr, i64 %offset, i32 %redOp, i1 %abs, i1 %nan)
+declare <n x i32> @llvm.nvvm.tcgen05.ld.red.16x32bx2.<num>.f32(ptr addrspace(6) %tmem_addr, i64 %offset, i32 %redOp, i1 %abs, i1 %nan)
+```
-Overview:
-"""""""""
+##### Overview:
This group of intrinsics asynchronously load data from the Tensor Memory at the
ocation specified by the 32-bit address operand `tmem_addr` into the destination
@@ -3189,18 +2813,16 @@ Allowed value for the 'shape' in the second intrinsic is `16x32bx2`.
The result of the intrinsic is a vector consisting of one or more 32-bit
registers derived from `shape` and `num` as shown below.
-=========== ========================= ========== ==========
- num/shape 16x32bx2/16x64b/32x32b 16x128b 16x256b
-=========== ========================= ========== ==========
- x1 1 2 4
- x2 2 4 8
- x4 4 8 16
- x8 8 16 32
- x16 16 32 64
- x32 32 64 128
- x64 64 128 NA
- x128 128 NA NA
-=========== ========================= ========== ==========
+| num/shape | 16x32bx2/16x64b/32x32b | 16x128b | 16x256b |
+| --------- | ---------------------- | ------- | ------- |
+| x1 | 1 | 2 | 4 |
+| x2 | 2 | 4 | 8 |
+| x4 | 4 | 8 | 16 |
+| x8 | 8 | 16 | 32 |
+| x16 | 16 | 32 | 64 |
+| x32 | 32 | 64 | 128 |
+| x64 | 64 | 128 | NA |
+| x128 | 128 | NA | NA |
The last argument `i1 %pack` is a compile-time constant which when set,
indicates that the adjacent columns are packed into a single 32-bit element
@@ -3212,35 +2834,28 @@ respectively
`%redOp` flag:
-=========== =============
- value operation
-=========== =============
- 0 min
- 1 max
-=========== =============
-
-For more information, refer to the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instructions-tcgen05-ld>`__.
-
+| value | operation |
+| ----- | --------- |
+| 0 | min |
+| 1 | max |
-'``llvm.nvvm.tcgen05.st.*``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instructions-tcgen05-ld).
-Syntax:
-"""""""
+#### '`llvm.nvvm.tcgen05.st.*`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.tcgen05.st.<shape>.<num>(ptr addrspace(6) %tmem_addr, <n x i32> %args, i1 %unpack)
+```llvm
+declare void @llvm.nvvm.tcgen05.st.<shape>.<num>(ptr addrspace(6) %tmem_addr, <n x i32> %args, i1 %unpack)
- declare void @llvm.nvvm.tcgen05.st.16x32bx2.<num>(ptr addrspace(6) %tmem_addr, <n x i32> %args, i64 %offset, i1 %unpack)
+declare void @llvm.nvvm.tcgen05.st.16x32bx2.<num>(ptr addrspace(6) %tmem_addr, <n x i32> %args, i64 %offset, i1 %unpack)
+```
-Overview:
-"""""""""
+##### Overview:
This group of intrinsics asynchronously store data from the source vector into
the Tensor Memory at the location specified by the 32-bit address operand
-'tmem_addr` collectively across all threads of the warps.
+`tmem_addr` collectively across all threads of the warps.
All the threads in the warp must specify the same value of `tmem_addr`, which
must be the base address of the collective load operation. Otherwise, the
@@ -3272,21 +2887,18 @@ The last argument `i1 %unpack` is a compile-time constant which when set,
indicates that a 32-bit element in the register to be unpacked into two 16-bit
elements and store them in adjacent columns.
-For more information, refer to the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instructions-tcgen05-st>`__.
+For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instructions-tcgen05-st).
-tcgen05.mma Intrinsics
-----------------------
+### tcgen05.mma Intrinsics
-Overview
-^^^^^^^^
+#### Overview
`tcgen05.mma` operation of shape `M x N x K` perform matrix multiplication and
accumulation of the form: `D = A * B + D` where:
- - the `A` matrix has shape `M x K`, in either `Tensor Memory` or `Shared Memory`
- - the `B` matrix has shape `K x N`, in `Shared Memory` of the current CTA and, optionally in peer CTA
- - the `D` matrix is of the shape `M x N`, in `Tensor Memory`
+> - the `A` matrix has shape `M x K`, in either `Tensor Memory` or `Shared Memory`
+> - the `B` matrix has shape `K x N`, in `Shared Memory` of the current CTA and, optionally in peer CTA
+> - the `D` matrix is of the shape `M x N`, in `Tensor Memory`
Optionally an input predicate can be used to disable the input (`%enable_inp_d`)
from the accumulator matrix and the following operation can be performed as
@@ -3297,24 +2909,15 @@ various kinds based on input types and the throughput of the multiplication
operation. The following table shows the
diff erent kinds of MMA operations
that are supported:
-+------------+--------------------------------------------+
-| .kind | Supported Input Types |
-+============+============================================+
-| f16 | F16 and BF16 |
-+------------+--------------------------------------------+
-| tf32 | TF32 |
-+------------+--------------------------------------------+
-| f8f6f4 | All combinations of F8, F6, and F4 |
-+------------+--------------------------------------------+
-| i8 | Signed and Unsigned 8-bit Integers |
-+------------+--------------------------------------------+
-| mxf8f6f4 | MX-floating point formats |
-+------------+--------------------------------------------+
-| mxf4 | MX-floating point formats (FP4) |
-+------------+--------------------------------------------+
-| mxf4nvf4 | MXF4 + custom NVIDIA 4-bit floating point |
-| | (with common scaling factor) |
-+------------+--------------------------------------------+
+| .kind | Supported Input Types |
+| -------- | ---------------------------------------------------------------------- |
+| f16 | F16 and BF16 |
+| tf32 | TF32 |
+| f8f6f4 | All combinations of F8, F6, and F4 |
+| i8 | Signed and Unsigned 8-bit Integers |
+| mxf8f6f4 | MX-floating point formats |
+| mxf4 | MX-floating point formats (FP4) |
+| mxf4nvf4 | MXF4 + custom NVIDIA 4-bit floating point (with common scaling factor) |
`tcgen05.mma.sp` supports sparse variant of `A` with shape `M x K` stored in
packed form as `M X (K / 2)` in memory. The `%spmetadata` specifies the mapping
@@ -3327,7 +2930,7 @@ memory to form the matrix `A` and matrix `B` before performing the MMA
operation. Scale factors for `A` and `B` matrices need to be duplicated to all
32 lane partitions of tensor memory. The shape of `%scale_a` and `%scale_b`
matrices depend on the `.scale_vectorsize` described in
-`here <https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-mma-scale-valid-comb>`__
+[here](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-mma-scale-valid-comb)
The sparsity metadata (`%spmetadata`) as well as the block-scale inputs for
`A / B` matrices (`%scale_a` and `%scale_b`) reside in Tensor Memory.
@@ -3341,24 +2944,30 @@ intrinsic specifies the nature of the re-use
There are three kinds of matrix descriptors used by the tcgen05 family of
instructions:
-+----------------------------+-----------------------------------------------------------------------------------------------------------+-------------+
-| Descriptor | Description | Size (bits) |
-+============================+===========================================================================================================+=============+
-| Shared Memory Descriptor | Describes properties of multiplicand matrix | |
-| | in shared memory, including its location | |
-| | within the CTA's shared memory. | 64 |
-| | `PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-shared-memory-descriptor>`__ | |
-+----------------------------+-----------------------------------------------+-------------+---------------------------------------------+-------------+
-| Instruction Descriptor | Describes shapes, types, and details of | |
-| | all matrices and the MMA operation. | 32 |
-| | `PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-zero-column-mask-descriptor>`__ | |
-+----------------------------+-----------------------------------------------+-------------+---------------------------------------------+-------------+
-| Zero-Column Mask Descriptor| Generates a mask specifying which columns of | |
-| | B matrix are zeroed in the MMA operation, | |
-| | regardless of values in shared memory. | 64 |
-| | Total mask size = N bits | |
-| | `PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instruction-descriptor>`__ | |
-+----------------------------+-----------------------------------------------+-------------+---------------------------------------------+-------------+
+```{list-table}
+:widths: 28 60 12
+:header-rows: 1
+
+ * - Descriptor
+ - Description
+ - Size (bits)
+ * - Shared Memory Descriptor
+ - Describes properties of multiplicand matrix in shared memory, including
+ its location within the CTA's shared memory.
+ [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-shared-memory-descriptor)
+ - 64
+ * - Instruction Descriptor
+ - Describes shapes, types, and details of all matrices and the MMA
+ operation.
+ [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-zero-column-mask-descriptor)
+ - 32
+ * - Zero-Column Mask Descriptor
+ - Generates a mask specifying which columns of B matrix are zeroed in the
+ MMA operation, regardless of values in shared memory. Total mask size =
+ N bits
+ [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instruction-descriptor)
+ - 64
+```
`tcgen05.mma` can be used for general matrix multiplication or for convolution
operations. In case of convolutions, the `activations` can be stored in either
@@ -3387,72 +2996,54 @@ of the vector (leftmost in syntax) corresponding to the lane 0 of the Tensor
Memory. If a bit in the mask is 1, then the corresponding lane in the Tensor
Memory for the resultant matrix D will not be updated
-Intrinsic Design:
-^^^^^^^^^^^^^^^^^
+#### Intrinsic Design:
Given the broad feature set of `tcgen05.mma` instruction modeling these
through intrinsics is highly complex, and the following table outlines the large
number of intrinsics required to fully support the `tcgen05.mma` instruction
set.
-+------------------------------------+---------------------------------------------------------------------------------------------------+----------------+
| variant | Configuration | Total Variants |
-+====================================+===================================================================================================+================+
+| ---------------------------------- | ------------------------------------------------------------------------------------------------- | -------------- |
| tcgen05.mma.shared | 2 (space) x 2 (sp) x 4 (kind) x 2 (cta_group) x 4 (collector_usage) | 128 |
-+------------------------------------+---------------------------------------------------------------------------------------------------+----------------+
| tcgen05.mma.tensor.ashift | 2 (sp) x 4 (kind) x 2 (cta_group) x 2 (collector_usage) | 32 |
-+------------------------------------+---------------------------------------------------------------------------------------------------+----------------+
| tcgen05.mma.scale_d | 2 (space) x 2 (sp) x 2 (kind) x 2 (cta_group) x 4 (collector_usage) | 128 |
-+------------------------------------+---------------------------------------------------------------------------------------------------+----------------+
| tcgen05.mma.scale_d.tensor.ashift | 2 (sp) x 2 (kind) x 2 (cta_group) x 2 (collector_usage) | 16 |
-+------------------------------------+---------------------------------------------------------------------------------------------------+----------------+
| tcgen05.mma.disable_output_lane | 2 (space) x 2 (sp) x 4 (kind) x 2 (cta_group) x 4 (collector_usage) | 128 |
-+------------------------------------+---------------------------------------------------------------------------------------------------+----------------+
| tcgen05.mma.disable_output_lane... | 2 (sp) x 4 (kind) x 2 (cta_group) x 2 (collector_usage) | 32 |
-+------------------------------------+---------------------------------------------------------------------------------------------------+----------------+
| tcgen05.mma.block_scale | 2 (space) x 1 (mxf4nvf4) x 2 (cta_group) x 2 (scale_vec_size) x 4 (collector_usage) | 32 |
-+------------------------------------+---------------------------------------------------------------------------------------------------+----------------+
| tcgen05.mma.block_scale | 2 (space) x 1 (mxf4) x 2 (cta_group) x 2 (scale_vec_size) x 4 (collector_usage) | 32 |
-+------------------------------------+---------------------------------------------------------------------------------------------------+----------------+
| tcgen05.mma.block_scale | 2 (space) x 1 (mxf8f6f4) x 2 (cta_group) x 2 (scale_vec_size) x 4 (collector_usage) | 32 |
-+------------------------------------+---------------------------------------------------------------------------------------------------+----------------+
| tcgen05.mma.ws | 2 (space) x 2 (sp) x 4 (kind) x 2 (zero_col_mask) x 4 (collector_usage_op) x 4 (collector_buffer) | 256 |
-+------------------------------------+---------------------------------------------------------------------------------------------------+----------------+
| Total | | 816 |
-+------------------------------------+---------------------------------------------------------------------------------------------------+----------------+
-
To reduce the number of possible intrinsic variations, we've modeled the
`tcgen05.mma` instructions using flag operands. We've added range checks to
these flags to prevent invalid values. We also expanded some flags back into
intrinsic modifiers to avoid supporting invalid combinations of features.
+#### '`llvm.nvvm.tcgen05.mma.*`'
-'``llvm.nvvm.tcgen05.mma.*``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
-
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i32 %kind_flag, i32 %cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.tensor<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i32 %kind_flag, i32 %cta_group_flag, i32 %collector_usage_a_op_flag)
+```llvm
+declare void @llvm.nvvm.tcgen05.mma.shared(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i32 %kind_flag, i32 %cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.tensor<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i32 %kind_flag, i32 %cta_group_flag, i32 %collector_usage_a_op_flag)
- ; .sp variants
- declare void @llvm.nvvm.tcgen05.mma.sp.shared(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, i32 %kind_flag, i32 %cta_group_flag, i32 %collector_usage_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.tensor<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, i32 %kind_flag, i32 %cta_group_flag, i32 %collector_usage_a_op_flag)
+; .sp variants
+declare void @llvm.nvvm.tcgen05.mma.sp.shared(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, i32 %kind_flag, i32 %cta_group_flag, i32 %collector_usage_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.tensor<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, i32 %kind_flag, i32 %cta_group_flag, i32 %collector_usage_a_op_flag)
- ; .scale_d variants
- declare void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %scale_d_imm, i32 %cta_group_flag, i32 %kind_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.tensor.scale_d<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %scale_d_imm, i32 %cta_group_flag, i32 %kind_flag, i32 %collector_usage_a_op_flag)
+; .scale_d variants
+declare void @llvm.nvvm.tcgen05.mma.shared.scale_d(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %scale_d_imm, i32 %cta_group_flag, i32 %kind_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.tensor.scale_d<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %scale_d_imm, i32 %cta_group_flag, i32 %kind_flag, i32 %collector_usage_a_op_flag)
- ; sp.scale_d variants
- declare void @llvm.nvvm.tcgen05.mma.sp.shared.scale_d(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, i64 %scale_d_imm, i32 %cta_group_flag, i32 %collector_usage_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.tensor.scale_d<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, i64 %scale_d_imm, i32 %cta_group, i32 %collector_usage_a_op_flag)
+; sp.scale_d variants
+declare void @llvm.nvvm.tcgen05.mma.sp.shared.scale_d(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, i64 %scale_d_imm, i32 %cta_group_flag, i32 %collector_usage_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.tensor.scale_d<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, i64 %scale_d_imm, i32 %cta_group, i32 %collector_usage_a_op_flag)
+```
-Overview:
-"""""""""
+##### Overview:
`nvvm.tcgen05.mma` is an asynchronous intrinsic which initiates an `M x N x K`
matrix multiply and accumulate operation, `D = A * B + D` where the `A` matrix
@@ -3463,7 +3054,7 @@ specified to scale the input matrix `D` as follows:
`D = A * B + D * (2 ^ - %scale_d_imm)`. The valid range of values for argument
`%scale_d_imm` is `[0, 15]`. The 32-bit register operand idesc is the
instruction descriptor as described in
-`Instruction descriptor <https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instruction-descriptor>`__
+[Instruction descriptor](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instruction-descriptor)
`nvvm.tcgen05.mma` has single thread semantics, unlike the collective
instructions `nvvm.mma.sync` or the PTX `wgmma.mma_async` instruction. So, a
@@ -3481,81 +3072,73 @@ for matrix `A`. It is illegal to specify either of `USE` or `FILL` for
`%collector_usage_a_op_flag` along with `.ashift`
For more information, refer to the
-`PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-mma-instructions-mma>`__
+[PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-mma-instructions-mma)
The following tables describe the possible values of the flag arguments
`%kind_flag` flag:
-============= ==========
- `kind_flag` value
-============= ==========
- F16 0
- TF32 1
- F8F6F4 2
- I8 3
-============= ==========
+| `kind_flag` | value |
+| ----------- | ----- |
+| F16 | 0 |
+| TF32 | 1 |
+| F8F6F4 | 2 |
+| I8 | 3 |
`%cta_group_flag` flag:
-================= ==========
- `cta_group_flag` value
-================= ==========
- CG1 1
- CG2 2
-================= ==========
+| `cta_group_flag` | value |
+| ---------------- | ----- |
+| CG1 | 1 |
+| CG2 | 2 |
`%collector_usage_a_op_flag` flag:
-============================= ==========
- `collector_usage_a_op_flag` value
-============================= ==========
- DISCARD 0
- LASTUSE 1
- USE 2
- FILL 3
-============================= ==========
-
-'``llvm.nvvm.tcgen05.mma.block_scale*``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
-
-.. code-block:: llvm
-
- ; mxf8f6f4
- declare void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.tensor.mxf8f6f4.block_scale(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.tensor.mxf8f6f4.block_scale.block32(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.shared.mxf8f6f4.block_scale(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.tensor.mxf8f6f4.block_scale(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.tensor.mxf8f6f4.block_scale.block32(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
-
- ; mxf4
- declare void @llvm.nvvm.tcgen05.mma.shared.mxf4.block_scale(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.tensor.mxf4.block_scale(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.shared.mxf4.block_scale.block32(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.tensor.mxf4.block_scale.block32(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.shared.mxf4.block_scale(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.tensor.mxf4.block_scale(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.shared.mxf4.block_scale.block32(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.tensor.mxf4.block_scale.block32(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
-
- ; mxf4nvf4
- declare void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.tensor.mxf4nvf4.block_scale.block16(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block32(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.tensor.mxf4nvf4.block_scale.block32(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.tensor.mxf4nvf4.block_scale.block16(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.shared.mxf4nvf4.block_scale.block32(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.tensor.mxf4nvf4.block_scale.block32(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
-
-Overview:
-"""""""""
+| `collector_usage_a_op_flag` | value |
+| --------------------------- | ----- |
+| DISCARD | 0 |
+| LASTUSE | 1 |
+| USE | 2 |
+| FILL | 3 |
+
+#### '`llvm.nvvm.tcgen05.mma.block_scale*`'
+
+##### Syntax:
+
+```llvm
+; mxf8f6f4
+declare void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.tensor.mxf8f6f4.block_scale(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.tensor.mxf8f6f4.block_scale.block32(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.shared.mxf8f6f4.block_scale(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.tensor.mxf8f6f4.block_scale(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.shared.mxf8f6f4.block_scale.block32(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.tensor.mxf8f6f4.block_scale.block32(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+
+; mxf4
+declare void @llvm.nvvm.tcgen05.mma.shared.mxf4.block_scale(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.tensor.mxf4.block_scale(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.shared.mxf4.block_scale.block32(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.tensor.mxf4.block_scale.block32(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.shared.mxf4.block_scale(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.tensor.mxf4.block_scale(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.shared.mxf4.block_scale.block32(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.tensor.mxf4.block_scale.block32(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+
+; mxf4nvf4
+declare void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.tensor.mxf4nvf4.block_scale.block16(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.shared.mxf4nvf4.block_scale.block32(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.tensor.mxf4nvf4.block_scale.block32(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.shared.mxf4nvf4.block_scale.block16(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.tensor.mxf4nvf4.block_scale.block16(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.shared.mxf4nvf4.block_scale.block32(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.tensor.mxf4nvf4.block_scale.block32(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, ptr addrspace(6) %scale_a, ptr addrspace(6) %scale_b, i32 cta_group_flag, i32 %collector_usage_a_op_flag)
+```
+
+##### Overview:
+
`nvvm.tcgen05.mma.block_scale` is an asynchronous intrinsic which initiates
an `M x N x K` matrix multiply and accumulate operation
`D = (A * scale_a) * (B * scale_b) + D` where the `A` matrix is `M x K`, the
@@ -3564,7 +3147,7 @@ are scaled with `%scale_A` and `%scale_B` matrices respectively before
performing the matrix multiply and accumulate operation. The operation of the
form `D = A*B` is issued when the input predicate argument `%enable_inp_d` is
false. The 32-bit register operand idesc is the instruction descriptor as
-described in `Instruction descriptor <https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instruction-descriptor>`__
+described in [Instruction descriptor](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instruction-descriptor)
`nvvm.tcgen05.mma.block_scale` has single thread semantics, unlike the
collective instructions `nvvm.mma.sync` or the PTX `wgmma.mma_async`
@@ -3578,63 +3161,56 @@ The `%collector_usage_a_op_flag` flag specifies the usage of collector buffer
for matrix `A`
For more information, refer to the
-`PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-mma-instructions-mma>`__
+[PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-mma-instructions-mma)
The following tables describe the possible values of the flag arguments
`%cta_group`:
-============= ==========
- `cta_group` value
-============= ==========
- CG1 1
- CG2 2
-============= ==========
+| `cta_group` | value |
+| ----------- | ----- |
+| CG1 | 1 |
+| CG2 | 2 |
`%collector_usage_a_op_flag`:
-============================= ==========
- `collector_usage_a_op_flag` value
-============================= ==========
- DISCARD 0
- LASTUSE 1
- USE 2
- FILL 3
-============================= ==========
-
-'``llvm.nvvm.tcgen05.mma.disable_output_lane*``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
-
-.. code-block:: llvm
-
- declare void @llvm.nvvm.tcgen05.mma.shared.disable_output_lane.cg1(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lane_v4, i32 %kind_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.shared.disable_output_lane.cg2(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, <8 x i32> %disable_output_lane_v8, i32 %kind_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.tensor.disable_output_lane.cg1<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lane_v4, i32 %kind_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.tensor.disable_output_lane.cg2<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, <8 x i32> %disable_output_lane_v8, i32 %kind_flag, i32 %collector_usage_a_op_flag)
-
- ; .sp variants
- declare void @llvm.nvvm.tcgen05.mma.sp.shared.disable_output_lane.cg1(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, <4 x i32> %disable_output_lane_v4, i32 %kind_flag, i32 %collector_usage_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.shared.disable_output_lane.cg2(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, <8 x i32> %disable_output_lane_v8, i32 %kind_flag, i32 %collector_usage_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.tensor.disable_output_lane.cg1<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, <4 x i32> %disable_output_lane_v4, i32 %kind_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.tensor.disable_output_lane.cg2<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, <8 x i32> %disable_output_lane_v8, i32 %kind_flag, i32 %collector_usage_a_op_flag)
-
- ; .scale_d variants
- declare void @llvm.nvvm.tcgen05.mma.shared.scale_d.disable_output_lane.cg1(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %scale_d_imm, <4 x i32> %disable_output_lane_v4, i32 %kind_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.shared.scale_d.disable_output_lane.cg2(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %scale_d_imm, <8 x i32> %disable_output_lane_v8, i32 %kind_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %scale_d_imm, <4 x i32> %disable_output_lane_v4, i32 %kind_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg2<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %scale_d_imm, <8 x i32> %disable_output_lane_v8, i32 %kind_flag, i32 %collector_usage_a_op_flag)
-
- ; .sp.scale_d variants
- declare void @llvm.nvvm.tcgen05.mma.sp.shared.scale_d.disable_output_lane.cg1(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, i64 %scale_d_imm, <4 x i32> %disable_output_lane_v4, i32 %kind_flag, i32 %collector_usage_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.shared.scale_d.disable_output_lane.cg2(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, i64 %scale_d_imm, <8 x i32> %disable_output_lane_v8, i32 %kind_flag, i32 %collector_usage_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.tensor.scale_d.disable_output_lane.cg1<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, i64 %scale_d_imm, <4 x i32> %disable_output_lane_v4, i32 %kind_flag, i32 %collector_usage_a_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.sp.tensor.scale_d.disable_output_lane.cg2<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, i64 %scale_d_imm, <8 x i32> %disable_output_lane_v8, i32 %kind_flag, i32 %collector_usage_a_op_flag)
-
-Overview:
-"""""""""
+| `collector_usage_a_op_flag` | value |
+| --------------------------- | ----- |
+| DISCARD | 0 |
+| LASTUSE | 1 |
+| USE | 2 |
+| FILL | 3 |
+
+#### '`llvm.nvvm.tcgen05.mma.disable_output_lane*`'
+
+##### Syntax:
+
+```llvm
+declare void @llvm.nvvm.tcgen05.mma.shared.disable_output_lane.cg1(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lane_v4, i32 %kind_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.shared.disable_output_lane.cg2(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, <8 x i32> %disable_output_lane_v8, i32 %kind_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.tensor.disable_output_lane.cg1<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, <4 x i32> %disable_output_lane_v4, i32 %kind_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.tensor.disable_output_lane.cg2<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, <8 x i32> %disable_output_lane_v8, i32 %kind_flag, i32 %collector_usage_a_op_flag)
+
+; .sp variants
+declare void @llvm.nvvm.tcgen05.mma.sp.shared.disable_output_lane.cg1(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, <4 x i32> %disable_output_lane_v4, i32 %kind_flag, i32 %collector_usage_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.shared.disable_output_lane.cg2(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, <8 x i32> %disable_output_lane_v8, i32 %kind_flag, i32 %collector_usage_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.tensor.disable_output_lane.cg1<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, <4 x i32> %disable_output_lane_v4, i32 %kind_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.tensor.disable_output_lane.cg2<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, <8 x i32> %disable_output_lane_v8, i32 %kind_flag, i32 %collector_usage_a_op_flag)
+
+; .scale_d variants
+declare void @llvm.nvvm.tcgen05.mma.shared.scale_d.disable_output_lane.cg1(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %scale_d_imm, <4 x i32> %disable_output_lane_v4, i32 %kind_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.shared.scale_d.disable_output_lane.cg2(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %scale_d_imm, <8 x i32> %disable_output_lane_v8, i32 %kind_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg1<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %scale_d_imm, <4 x i32> %disable_output_lane_v4, i32 %kind_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.tensor.scale_d.disable_output_lane.cg2<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %scale_d_imm, <8 x i32> %disable_output_lane_v8, i32 %kind_flag, i32 %collector_usage_a_op_flag)
+
+; .sp.scale_d variants
+declare void @llvm.nvvm.tcgen05.mma.sp.shared.scale_d.disable_output_lane.cg1(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, i64 %scale_d_imm, <4 x i32> %disable_output_lane_v4, i32 %kind_flag, i32 %collector_usage_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.shared.scale_d.disable_output_lane.cg2(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, i64 %scale_d_imm, <8 x i32> %disable_output_lane_v8, i32 %kind_flag, i32 %collector_usage_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.tensor.scale_d.disable_output_lane.cg1<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, i64 %scale_d_imm, <4 x i32> %disable_output_lane_v4, i32 %kind_flag, i32 %collector_usage_a_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.sp.tensor.scale_d.disable_output_lane.cg2<.ashift>(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, ptr addrspace(6) %spmetadata, i1 %enable_inp_d, i64 %scale_d_imm, <8 x i32> %disable_output_lane_v8, i32 %kind_flag, i32 %collector_usage_a_op_flag)
+```
+
+##### Overview:
`nvvm.tcgen05.mma.disable_output_lane` is an asynchronous intrinsic which
initiates an `M x N x K` matrix multiply and accumulate operation
@@ -3645,7 +3221,7 @@ argument `%scale_d_imm` can be specified to scale the input matrix `D` as
follows: `D = A*B+D * (2 ^ - %scale_d_imm)`. The valid range of values for
argument `%scale_d_imm` is `[0, 15]`. The 32-bit register operand idesc is the
instruction descriptor as described in
-`Instruction descriptor <https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instruction-descriptor>`__
+[Instruction descriptor](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instruction-descriptor)
The vector operand `%disable_output_lane` specifies the lane(s) in the Tensor
Memory that should be not be updated with the resultant matrix `D`. Elements of
@@ -3664,72 +3240,62 @@ whole matrix multiply and accumulate operation
When `.sp` is specifed, the dimension of A matrix is `M x (K / 2)` and requires
specifiying an additional `%spmetadata` argument.
- `.ashift` shifts the rows of the A matrix down by one row, except for the last
- row in the Tensor Memory. `.ashift` is only allowed with M = 128 or M = 256.
+> `.ashift` shifts the rows of the A matrix down by one row, except for the last
+> row in the Tensor Memory. `.ashift` is only allowed with M = 128 or M = 256.
The `%collector_usage_a_op_flag` flag specifies the usage of collector buffer
for matrix `A`. It is illegal to specify either of `USE` or `FILL` for
`%collector_usage_a_op_flag` along with `.ashift`
-For more information, refer to the `PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-mma-instructions-mma>`__
+For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-mma-instructions-mma)
The following tables describes the possible values of the flag arguments
`%kind_flag`:
-============= ==========
- `kind_flag` value
-============= ==========
- F16 0
- TF32 1
- F8F6F4 2
- I8 3
-============= ==========
+| `kind_flag` | value |
+| ----------- | ----- |
+| F16 | 0 |
+| TF32 | 1 |
+| F8F6F4 | 2 |
+| I8 | 3 |
`%cta_group_flag`:
-================= ==========
- `cta_group_flag` value
-================= ==========
- CG1 1
- CG2 2
-================= ==========
+| `cta_group_flag` | value |
+| ---------------- | ----- |
+| CG1 | 1 |
+| CG2 | 2 |
`%collector_usage_a_op_flag`:
-============================= ==========
- `collector_usage_a_op_flag` value
-============================= ==========
- DISCARD 0
- LASTUSE 1
- USE 2
- FILL 3
-============================= ==========
-
+| `collector_usage_a_op_flag` | value |
+| --------------------------- | ----- |
+| DISCARD | 0 |
+| LASTUSE | 1 |
+| USE | 2 |
+| FILL | 3 |
-'``llvm.nvvm.tcgen05.mma.ws*``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.tcgen05.mma.ws*`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+// tcgen05.mma.ws
+declare void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.ws.tensor(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.ws.shared.zero_col_mask(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %zero_col_mask, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.ws.shared.zero_col_mask(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %zero_col_mask, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.ws.tensor.zero_col_mask(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %zero_col_mask, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
- // tcgen05.mma.ws
- declare void @llvm.nvvm.tcgen05.mma.ws.shared(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.ws.tensor(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.ws.shared.zero_col_mask(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %zero_col_mask, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.ws.shared.zero_col_mask(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %zero_col_mask, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.ws.tensor.zero_col_mask(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, i64 %zero_col_mask, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
+; .sp variants
+declare void @llvm.nvvm.tcgen05.mma.ws.sp.shared(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.ws.sp.tensor(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.ws.sp.shared.zero_col_mask(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, i64 %zero_col_mask, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
+declare void @llvm.nvvm.tcgen05.mma.ws.sp.tensor.zero_col_mask(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, i64 %zero_col_mask, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
+```
- ; .sp variants
- declare void @llvm.nvvm.tcgen05.mma.ws.sp.shared(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.ws.sp.tensor(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.ws.sp.shared.zero_col_mask(ptr addrspace(6) %d, i64 %adesc, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, i64 %zero_col_mask, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
- declare void @llvm.nvvm.tcgen05.mma.ws.sp.tensor.zero_col_mask(ptr addrspace(6) %d, ptr addrspace(6) %atensor, i64 %bdesc, i32 %idesc, i1 %enable_inp_d, ptr addrspace(6) %spmetadata, i64 %zero_col_mask, i32 %kind_flag, i32 %collector_usage_b_buffer_flag, i32 %collector_usage_b_op_flag)
-
-Overview:
-"""""""""
+##### Overview:
`nvvm.tcgen05.mma.ws` is an asynchronous intrinsic which initiates an
`M x N x K` weight stationary convolution matrix multiply and accumulate
@@ -3740,7 +3306,7 @@ immediate argument `%scale_d_imm` can be specified to scale the input matrix `D`
as follows: `D = A*B+D * (2 ^ - %scale_d_imm)`. The valid range of values for
argument `%scale_d_imm` is `[0, 15]`. The 32-bit register operand idesc is the
instruction descriptor as described in
-`Instruction descriptor <https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instruction-descriptor>`__
+[Instruction descriptor](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-instruction-descriptor)
`nvvm.tcgen05.mma` has single thread semantics, unlike the collective
instructions `nvvm.mma.sync` or the PTX `wgmma.mma_async` instruction. So, a
@@ -3751,7 +3317,7 @@ When `.sp` is specifed, the dimension of A matrix is `M x (K / 2)` and requires
specifiying an additional `%spmetadata` argument
The operand `%zero_col_mask` is a 64-bit register which specifies the
-`Zero-Column Mask Descriptor <https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-zero-column-mask-descriptor>`__.
+[Zero-Column Mask Descriptor](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-zero-column-mask-descriptor).
The zero-column mask descriptor is used to generate a mask that specifies which
columns of `B` matrix will have zero value for the matrix multiply and
accumulate operation regardless of the values present in the shared memory.
@@ -3760,445 +3326,395 @@ The `%collector_usage_b_buffer_flag` and `%collector_usage_b_op_flag` together
flag specifies the usage of collector buffer for Matrix `B`.
For more information, refer to the
-`PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-mma-instructions-mma-ws>`__
+[PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#tcgen05-mma-instructions-mma-ws)
The following tables describes the possible values of the flag arguments
`%kind_flag`:
-============= ==========
- `kind_flag` value
-============= ==========
- F16 0
- TF32 1
- F8F6F4 2
- I8 3
-============= ==========
+| `kind_flag` | value |
+| ----------- | ----- |
+| F16 | 0 |
+| TF32 | 1 |
+| F8F6F4 | 2 |
+| I8 | 3 |
`%collector_usage_b_buffer_flag`:
-================================ ==========
- `collector_usage_b_buffer_flag` value
-================================ ==========
- B0 0
- B1 1
- B2 2
- B3 3
-================================ ==========
+| `collector_usage_b_buffer_flag` | value |
+| ------------------------------- | ----- |
+| B0 | 0 |
+| B1 | 1 |
+| B2 | 2 |
+| B3 | 3 |
`%collector_usage_b_op_flag`:
-============================= ==========
- `collector_usage_b_op_flag` value
-============================= ==========
- DISCARD 0
- LASTUSE 1
- USE 2
- FILL 3
-============================= ==========
-
-Store Intrinsics
-----------------
+| `collector_usage_b_op_flag` | value |
+| --------------------------- | ----- |
+| DISCARD | 0 |
+| LASTUSE | 1 |
+| USE | 2 |
+| FILL | 3 |
-'``llvm.nvvm.st.bulk.*``'
-^^^^^^^^^^^^^^^^^^^^^^^^^
+### Store Intrinsics
-Syntax:
-"""""""
+#### '`llvm.nvvm.st.bulk.*`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.st.bulk(ptr addrspace(1) %dst, i64 %size, i64 immarg %initval)
- declare void @llvm.nvvm.st.bulk.shared.cta(ptr addrspace(3) %dst, i64 %size, i64 immarg %initval)
+```llvm
+declare void @llvm.nvvm.st.bulk(ptr addrspace(1) %dst, i64 %size, i64 immarg %initval)
+declare void @llvm.nvvm.st.bulk.shared.cta(ptr addrspace(3) %dst, i64 %size, i64 immarg %initval)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``@llvm.nvvm.st.bulk.*``' intrinsics initialize a region of shared memory
+The '`@llvm.nvvm.st.bulk.*`' intrinsics initialize a region of shared memory
starting from the location specified by the destination address operand `%dst`.
-The integer operand `%size` specifies the amount of memory to be initialized in
-terms of number of bytes and must be a multiple of 8. Otherwise, the behavior
+The integer operand `%size` specifies the amount of memory to be initialized in
+terms of number of bytes and must be a multiple of 8. Otherwise, the behavior
is undefined.
-The integer immediate operand `%initval` specifies the initialization value for
+The integer immediate operand `%initval` specifies the initialization value for
the memory locations. The only numeric value allowed is 0.
-The ``@llvm.nvvm.st.bulk.shared.cta`` and ``@llvm.nvvm.st.bulk`` intrinsics are
-similar but the latter uses generic addressing (see `Generic Addressing
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#generic-addressing>`__).
+The `@llvm.nvvm.st.bulk.shared.cta` and `@llvm.nvvm.st.bulk` intrinsics are
+similar but the latter uses generic addressing (see [Generic Addressing](https://docs.nvidia.com/cuda/parallel-thread-execution/#generic-addressing)).
-For more information, refer `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-st-bulk>`__.
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-st-bulk).
-'``llvm.nvvm.st.async``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.st.async`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.st.async.i32(ptr addrspace(7) %dest_addr, i32 %value, ptr addrspace(7) %mbarrier_addr)
+declare void @llvm.nvvm.st.async.i64(ptr addrspace(7) %dest_addr, i64 %value, ptr addrspace(7) %mbarrier_addr)
+declare void @llvm.nvvm.st.async.i128(ptr addrspace(7) %dest_addr, i128 %value, ptr addrspace(7) %mbarrier_addr)
+```
- declare void @llvm.nvvm.st.async.i32(ptr addrspace(7) %dest_addr, i32 %value, ptr addrspace(7) %mbarrier_addr)
- declare void @llvm.nvvm.st.async.i64(ptr addrspace(7) %dest_addr, i64 %value, ptr addrspace(7) %mbarrier_addr)
- declare void @llvm.nvvm.st.async.i128(ptr addrspace(7) %dest_addr, i128 %value, ptr addrspace(7) %mbarrier_addr)
-
-Overview:
-"""""""""
+##### Overview:
-The '``llvm.nvvm.st.async``' intrinsic initiates a weak
-asynchronous store operation to shared memory that stores the value specified
-by the `%value` operand to the destination address specified by the
-`%dest_addr` operand. The `%value` operand must be an ``i32``, ``i64`` or
-``i128``, lowering to the ``.b32``, ``.b64`` and ``.b128`` variants
-of the ``st.async`` PTX instruction respectively.
+The '`llvm.nvvm.st.async`' intrinsic initiates a weak
+asynchronous store operation to shared memory that stores the value specified
+by the `%value` operand to the destination address specified by the
+`%dest_addr` operand. The `%value` operand must be an `i32`, `i64` or
+`i128`, lowering to the `.b32`, `.b64` and `.b128` variants
+of the `st.async` PTX instruction respectively.
-The store operation is treated as a weak memory operation. The effects of this
-operation become visible to other threads only when synchronization is
+The store operation is treated as a weak memory operation. The effects of this
+operation become visible to other threads only when synchronization is
established by other means.
-The operation is performed asynchronously and the completion is signalled using
-the mbarrier object specified by the `%mbarrier_addr` operand. Upon completion,
-a `complete-tx <https://docs.nvidia.com/cuda/parallel-thread-execution/#parallel-synchronization-and-communication-instructions-mbarrier-complete-tx-operation>`__ operation is performed on the mbarrier object, with the
+The operation is performed asynchronously and the completion is signalled using
+the mbarrier object specified by the `%mbarrier_addr` operand. Upon completion,
+a [complete-tx](https://docs.nvidia.com/cuda/parallel-thread-execution/#parallel-synchronization-and-communication-instructions-mbarrier-complete-tx-operation) operation is performed on the mbarrier object, with the
`completeCount` argument equal to the amount of data stored in bytes.
-For more information, refer `PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-st-async>`__.
-
-'``llvm.nvvm.st.async.{sys,gpu}``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Syntax:
-"""""""
-
-.. code-block:: llvm
-
- ; sys scope
- declare void @llvm.nvvm.st.async.sys.i8(ptr addrspace(1) %dest_addr, i8 %value, i1 immarg %is_multimem)
- declare void @llvm.nvvm.st.async.sys.i16(ptr addrspace(1) %dest_addr, i16 %value, i1 immarg %is_multimem)
- declare void @llvm.nvvm.st.async.sys.i32(ptr addrspace(1) %dest_addr, i32 %value, i1 immarg %is_multimem)
- declare void @llvm.nvvm.st.async.sys.i64(ptr addrspace(1) %dest_addr, i64 %value, i1 immarg %is_multimem)
-
- ; gpu scope
- declare void @llvm.nvvm.st.async.gpu.i8(ptr addrspace(1) %dest_addr, i8 %value, i1 immarg %is_multimem)
- declare void @llvm.nvvm.st.async.gpu.i16(ptr addrspace(1) %dest_addr, i16 %value, i1 immarg %is_multimem)
- declare void @llvm.nvvm.st.async.gpu.i32(ptr addrspace(1) %dest_addr, i32 %value, i1 immarg %is_multimem)
- declare void @llvm.nvvm.st.async.gpu.i64(ptr addrspace(1) %dest_addr, i64 %value, i1 immarg %is_multimem)
-
-Overview:
-"""""""""
-
-The '``llvm.nvvm.st.async.sys``' and
-'``llvm.nvvm.st.async.gpu``' intrinsics initiate an
-asynchronous release store to global memory that stores the value specified by
-the `%value` operand to the destination address specified by the `%dest_addr`
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-st-async).
+
+#### '`llvm.nvvm.st.async.{sys,gpu}`'
+
+##### Syntax:
+
+```llvm
+; sys scope
+declare void @llvm.nvvm.st.async.sys.i8(ptr addrspace(1) %dest_addr, i8 %value, i1 immarg %is_multimem)
+declare void @llvm.nvvm.st.async.sys.i16(ptr addrspace(1) %dest_addr, i16 %value, i1 immarg %is_multimem)
+declare void @llvm.nvvm.st.async.sys.i32(ptr addrspace(1) %dest_addr, i32 %value, i1 immarg %is_multimem)
+declare void @llvm.nvvm.st.async.sys.i64(ptr addrspace(1) %dest_addr, i64 %value, i1 immarg %is_multimem)
+
+; gpu scope
+declare void @llvm.nvvm.st.async.gpu.i8(ptr addrspace(1) %dest_addr, i8 %value, i1 immarg %is_multimem)
+declare void @llvm.nvvm.st.async.gpu.i16(ptr addrspace(1) %dest_addr, i16 %value, i1 immarg %is_multimem)
+declare void @llvm.nvvm.st.async.gpu.i32(ptr addrspace(1) %dest_addr, i32 %value, i1 immarg %is_multimem)
+declare void @llvm.nvvm.st.async.gpu.i64(ptr addrspace(1) %dest_addr, i64 %value, i1 immarg %is_multimem)
+```
+
+##### Overview:
+
+The '`llvm.nvvm.st.async.sys`' and
+'`llvm.nvvm.st.async.gpu`' intrinsics initiate an
+asynchronous release store to global memory that stores the value specified by
+the `%value` operand to the destination address specified by the `%dest_addr`
operand.
-The `%is_multimem` immediate argument selects the variant of the store. When it
-is `0`, a regular ``st.async`` is emitted. When it is `1`, a
-``multimem.st.async`` is emitted and the `%dest_addr` operand must be a
+The `%is_multimem` immediate argument selects the variant of the store. When it
+is `0`, a regular `st.async` is emitted. When it is `1`, a
+`multimem.st.async` is emitted and the `%dest_addr` operand must be a
multimem address.
-The store carries ``.release`` semantics — prior stores from the current thread
+The store carries `.release` semantics — prior stores from the current thread
are made visible to other threads in the scope.
-The scope of this operation can be ``sys`` or ``gpu``.
+The scope of this operation can be `sys` or `gpu`.
These intrinsics lower to the `.b8`, `.b16`, `.b32`, and `.b64` variants of the
-``st.async`` instruction.
-
-For more information, refer `PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-st-async>`__.
+`st.async` instruction.
-'``llvm.nvvm.st.async.mmio.sys``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-st-async).
-Syntax:
-"""""""
+#### '`llvm.nvvm.st.async.mmio.sys`'
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.st.async.mmio.sys.i8(ptr addrspace(1) %dest_addr, i8 %value)
- declare void @llvm.nvvm.st.async.mmio.sys.i16(ptr addrspace(1) %dest_addr, i16 %value)
- declare void @llvm.nvvm.st.async.mmio.sys.i32(ptr addrspace(1) %dest_addr, i32 %value)
- declare void @llvm.nvvm.st.async.mmio.sys.i64(ptr addrspace(1) %dest_addr, i64 %value)
+```llvm
+declare void @llvm.nvvm.st.async.mmio.sys.i8(ptr addrspace(1) %dest_addr, i8 %value)
+declare void @llvm.nvvm.st.async.mmio.sys.i16(ptr addrspace(1) %dest_addr, i16 %value)
+declare void @llvm.nvvm.st.async.mmio.sys.i32(ptr addrspace(1) %dest_addr, i32 %value)
+declare void @llvm.nvvm.st.async.mmio.sys.i64(ptr addrspace(1) %dest_addr, i64 %value)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``llvm.nvvm.st.async.mmio.sys``' intrinsic performs an `MMIO <https://docs.nvidia.com/cuda/parallel-thread-execution/#mmio-operation>`__
-store to global memory with ``.release`` semantics at the ``sys`` scope.
+The '`llvm.nvvm.st.async.mmio.sys`' intrinsic performs an [MMIO](https://docs.nvidia.com/cuda/parallel-thread-execution/#mmio-operation)
+store to global memory with `.release` semantics at the `sys` scope.
-For more information, refer `PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-st-async>`__.
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/#data-movement-and-conversion-instructions-st-async).
-clusterlaunchcontrol Intrinsics
--------------------------------
+### clusterlaunchcontrol Intrinsics
-'``llvm.nvvm.clusterlaunchcontrol.try_cancel*``' Intrinsics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.clusterlaunchcontrol.try_cancel*`' Intrinsics
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare void @llvm.nvvm.clusterlaunchcontrol.try_cancel.async.shared(ptr addrspace(3) %addr, ptr addrspace(3) %mbar)
+declare void @llvm.nvvm.clusterlaunchcontrol.try_cancel.async.multicast.shared(ptr addrspace(3) %addr, ptr addrspace(3) %mbar)
+```
- declare void @llvm.nvvm.clusterlaunchcontrol.try_cancel.async.shared(ptr addrspace(3) %addr, ptr addrspace(3) %mbar)
- declare void @llvm.nvvm.clusterlaunchcontrol.try_cancel.async.multicast.shared(ptr addrspace(3) %addr, ptr addrspace(3) %mbar)
+##### Overview:
-Overview:
-"""""""""
-
-The ``clusterlaunchcontrol.try_cancel`` intrinsics requests atomically cancelling
+The `clusterlaunchcontrol.try_cancel` intrinsics requests atomically cancelling
the launch of a cluster that has not started running yet. It asynchronously
non-atomically writes a 16-byte opaque response to shared memory, pointed to by
-16-byte-aligned ``addr`` indicating whether the operation succeeded or failed.
-``addr`` and 8-byte-aligned ``mbar`` must refer to ``shared::cta`` otherwise the
+16-byte-aligned `addr` indicating whether the operation succeeded or failed.
+`addr` and 8-byte-aligned `mbar` must refer to `shared::cta` otherwise the
behavior is undefined. The completion of the asynchronous operation is tracked
-using the mbarrier completion mechanism at ``.cluster`` scope referenced by the
-shared memory pointer, ``mbar``. On success, the opaque response contains the
+using the mbarrier completion mechanism at `.cluster` scope referenced by the
+shared memory pointer, `mbar`. On success, the opaque response contains the
CTA id of the first CTA of the canceled cluster; no other successful response
-from other ``clusterlaunchcontrol.try_cancel`` operations from the same grid
+from other `clusterlaunchcontrol.try_cancel` operations from the same grid
will contain that id.
-The ``multicast`` variant specifies that the response is asynchronously
+The `multicast` variant specifies that the response is asynchronously
non-atomically written to the corresponding shared memory location of each CTA
in the requesting cluster. The completion of the write of each local response is
tracked by independent mbarriers at the corresponding shared memory location of
each CTA in theccluster.
-For more information, refer `PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/?a#parallel-synchronization-and-communication-instructions-clusterlaunchcontrol-try-cancel>`__.
-
-'``llvm.nvvm.clusterlaunchcontrol.query_cancel.is_canceled``' Intrinsic
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/?a#parallel-synchronization-and-communication-instructions-clusterlaunchcontrol-try-cancel).
-Syntax:
-"""""""
+#### '`llvm.nvvm.clusterlaunchcontrol.query_cancel.is_canceled`' Intrinsic
-.. code-block:: llvm
+##### Syntax:
- declare i1 @llvm.nvvm.clusterlaunchcontrol.query_cancel.is_canceled(i128 %try_cancel_response)
+```llvm
+declare i1 @llvm.nvvm.clusterlaunchcontrol.query_cancel.is_canceled(i128 %try_cancel_response)
+```
-Overview:
-"""""""""
+##### Overview:
-The ``llvm.nvvm.clusterlaunchcontrol.query_cancel.is_canceled`` intrinsic
+The `llvm.nvvm.clusterlaunchcontrol.query_cancel.is_canceled` intrinsic
decodes the opaque response written by the
-``llvm.nvvm.clusterlaunchcontrol.try_cancel`` operation.
+`llvm.nvvm.clusterlaunchcontrol.try_cancel` operation.
-The intrinsic returns ``0`` (false) if the request failed. If the request
-succeeded, it returns ``1`` (true). A true result indicates that:
+The intrinsic returns `0` (false) if the request failed. If the request
+succeeded, it returns `1` (true). A true result indicates that:
- the thread block cluster whose first CTA id matches that of the response
handle will not run, and
-- no other successful response of another ``try_cancel`` request in the grid
+- no other successful response of another `try_cancel` request in the grid
will contain the first CTA id of that cluster
-For more information, refer `PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/?a#parallel-synchronization-and-communication-instructions-clusterlaunchcontrol-query-cancel>`__.
-
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/?a#parallel-synchronization-and-communication-instructions-clusterlaunchcontrol-query-cancel).
-'``llvm.nvvm.clusterlaunchcontrol.query_cancel.get_first_ctaid.*``' Intrinsics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.clusterlaunchcontrol.query_cancel.get_first_ctaid.*`' Intrinsics
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare i32 @llvm.nvvm.clusterlaunchcontrol.query_cancel.get_first_ctaid.x(i128 %try_cancel_response)
+declare i32 @llvm.nvvm.clusterlaunchcontrol.query_cancel.get_first_ctaid.y(i128 %try_cancel_response)
+declare i32 @llvm.nvvm.clusterlaunchcontrol.query_cancel.get_first_ctaid.z(i128 %try_cancel_response)
+```
- declare i32 @llvm.nvvm.clusterlaunchcontrol.query_cancel.get_first_ctaid.x(i128 %try_cancel_response)
- declare i32 @llvm.nvvm.clusterlaunchcontrol.query_cancel.get_first_ctaid.y(i128 %try_cancel_response)
- declare i32 @llvm.nvvm.clusterlaunchcontrol.query_cancel.get_first_ctaid.z(i128 %try_cancel_response)
+##### Overview:
-Overview:
-"""""""""
-
-The ``clusterlaunchcontrol.query_cancel.get_first_ctaid.*`` intrinsic can be
+The `clusterlaunchcontrol.query_cancel.get_first_ctaid.*` intrinsic can be
used to decode the successful opaque response written by the
-``llvm.nvvm.clusterlaunchcontrol.try_cancel`` operation.
+`llvm.nvvm.clusterlaunchcontrol.try_cancel` operation.
If the request succeeded:
-- ``llvm.nvvm.clusterlaunchcontrol.query_cancel.get_first_ctaid.{x,y,z}``
+- `llvm.nvvm.clusterlaunchcontrol.query_cancel.get_first_ctaid.{x,y,z}`
returns the coordinate of the first CTA in the canceled cluster, either x, y,
or z.
If the request failed, the behavior of these intrinsics is undefined.
-For more information, refer `PTX ISA <https://docs.nvidia.com/cuda/parallel-thread-execution/?a#parallel-synchronization-and-communication-instructions-clusterlaunchcontrol-query-cancel>`__.
-
-Perf Monitor Event Intrinsics
------------------------------
+For more information, refer [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/?a#parallel-synchronization-and-communication-instructions-clusterlaunchcontrol-query-cancel).
-'``llvm.nvvm.pm.event.mask``' Intrinsic
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+### Perf Monitor Event Intrinsics
-Syntax:
-"""""""
+#### '`llvm.nvvm.pm.event.mask`' Intrinsic
-.. code-block:: llvm
+##### Syntax:
- declare void @llvm.nvvm.pm.event.mask(i16 immarg %mask_val)
+```llvm
+declare void @llvm.nvvm.pm.event.mask(i16 immarg %mask_val)
+```
-Overview:
-"""""""""
+##### Overview:
-The '``llvm.nvvm.pm.event.mask``' intrinsic triggers one or more performance
-monitor events. Each bit in the 16-bit immediate operand `%mask_val`` controls
+The '`llvm.nvvm.pm.event.mask`' intrinsic triggers one or more performance
+monitor events. Each bit in the 16-bit immediate operand `%mask_val` controls
an event.
-For more information on the pmevent instructions, refer to the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#miscellaneous-instructions-pmevent>`__.
+For more information on the pmevent instructions, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#miscellaneous-instructions-pmevent).
-Warp-level Matrix Transpose Intrinsics
----------------------------------------
+### Warp-level Matrix Transpose Intrinsics
-'``llvm.nvvm.movmatrix.sync.aligned.m8n8.trans.b16``'
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#### '`llvm.nvvm.movmatrix.sync.aligned.m8n8.trans.b16`'
-Syntax:
-"""""""
+##### Syntax:
-.. code-block:: llvm
+```llvm
+declare i32 @llvm.nvvm.movmatrix.sync.aligned.m8n8.trans.b16(i32 %src)
+```
- declare i32 @llvm.nvvm.movmatrix.sync.aligned.m8n8.trans.b16(i32 %src)
+##### Overview:
-Overview:
-"""""""""
-
-The '``@llvm.nvvm.movmatrix.sync.aligned.m8n8.trans.b16``' intrinsic
+The '`@llvm.nvvm.movmatrix.sync.aligned.m8n8.trans.b16`' intrinsic
transposes an 8x8 matrix of 16-bit elements distributed across all 32
threads of a warp. Each thread provides a 32-bit register containing two
-packed ``.b16`` elements, and receives back two packed ``.b16`` elements
+packed `.b16` elements, and receives back two packed `.b16` elements
from the transposed matrix in the same format.
-The mandatory ``.sync`` qualifier indicates that ``movmatrix`` causes the
+The mandatory `.sync` qualifier indicates that `movmatrix` causes the
executing thread to wait until all threads in the warp execute the same
-``movmatrix`` intrinsic before resuming execution.
+`movmatrix` intrinsic before resuming execution.
-The mandatory ``.aligned`` qualifier indicates that all threads in the warp
-must execute the same ``movmatrix`` intrinsic. In conditionally executed
-code, a ``movmatrix`` intrinsic should only be used if it is known that
+The mandatory `.aligned` qualifier indicates that all threads in the warp
+must execute the same `movmatrix` intrinsic. In conditionally executed
+code, a `movmatrix` intrinsic should only be used if it is known that
all threads in the warp evaluate the condition identically, otherwise the
behavior is undefined.
-For more information, refer to the `PTX ISA
-<https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-instructions-movmatrix>`__.
-
+For more information, refer to the [PTX ISA](https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#warp-level-matrix-instructions-movmatrix).
-Other Intrinsics
-----------------
+### Other Intrinsics
For the full set of NVPTX intrinsics, please see the
-``include/llvm/IR/IntrinsicsNVVM.td`` file in the LLVM source tree.
+`include/llvm/IR/IntrinsicsNVVM.td` file in the LLVM source tree.
+(libdevice)=
-.. _libdevice:
+## Linking with Libdevice
-Linking with Libdevice
-======================
-
-The CUDA Toolkit comes with an LLVM bitcode library called ``libdevice`` that
+The CUDA Toolkit comes with an LLVM bitcode library called `libdevice` that
implements many common mathematical functions. This library can be used as a
high-performance math library for any compilers using the LLVM NVPTX target.
-The library can be found under ``nvvm/libdevice/`` in the CUDA Toolkit and
+The library can be found under `nvvm/libdevice/` in the CUDA Toolkit and
there is a separate version for each compute architecture.
For a list of all math functions implemented in libdevice, see
-`libdevice Users Guide <http://docs.nvidia.com/cuda/libdevice-users-guide/index.html>`__.
+[libdevice Users Guide](http://docs.nvidia.com/cuda/libdevice-users-guide/index.html).
To accommodate various math-related compiler flags that can affect code
generation of libdevice code, the library code depends on a special LLVM IR
-pass (``NVVMReflect``) to handle conditional compilation within LLVM IR. This
-pass looks for calls to the ``@__nvvm_reflect`` function and replaces them
+pass (`NVVMReflect`) to handle conditional compilation within LLVM IR. This
+pass looks for calls to the `@__nvvm_reflect` function and replaces them
with constants based on the defined reflection parameters. Such conditional
code often follows a pattern:
-.. code-block:: c++
-
- float my_function(float a) {
- if (__nvvm_reflect("FASTMATH"))
- return my_function_fast(a);
- else
- return my_function_precise(a);
- }
+```c++
+float my_function(float a) {
+ if (__nvvm_reflect("FASTMATH"))
+ return my_function_fast(a);
+ else
+ return my_function_precise(a);
+}
+```
The default value for all unspecified reflection parameters is zero.
-The ``NVVMReflect`` pass should be executed early in the optimization
-pipeline, immediately after the link stage. The ``internalize`` pass is also
+The `NVVMReflect` pass should be executed early in the optimization
+pipeline, immediately after the link stage. The `internalize` pass is also
recommended to remove unused math functions from the resulting PTX. For an
-input IR module ``module.bc``, the following compilation flow is recommended:
+input IR module `module.bc`, the following compilation flow is recommended:
-The ``NVVMReflect`` pass will attempt to remove dead code even without
+The `NVVMReflect` pass will attempt to remove dead code even without
optimizations. This allows potentially incompatible instructions to be avoided
-at all optimizations levels by using the ``__CUDA_ARCH`` argument.
+at all optimizations levels by using the `__CUDA_ARCH` argument.
-1. Save list of external functions in ``module.bc``
-2. Link ``module.bc`` with ``libdevice.compute_XX.YY.bc``
+1. Save list of external functions in `module.bc`
+2. Link `module.bc` with `libdevice.compute_XX.YY.bc`
3. Internalize all functions not in list from (1)
4. Eliminate all unused internal functions
-5. Run ``NVVMReflect`` pass
+5. Run `NVVMReflect` pass
6. Run standard optimization pipeline
-.. note::
+:::{note}
+`linkonce` and `linkonce_odr` linkage types are not suitable for the
+libdevice functions. It is possible to link two IR modules that have been
+linked against libdevice using
diff erent reflection variables.
+:::
- ``linkonce`` and ``linkonce_odr`` linkage types are not suitable for the
- libdevice functions. It is possible to link two IR modules that have been
- linked against libdevice using
diff erent reflection variables.
-
-Since the ``NVVMReflect`` pass replaces conditionals with constants, it will
+Since the `NVVMReflect` pass replaces conditionals with constants, it will
often leave behind dead code of the form:
-.. code-block:: llvm
-
- entry:
- ..
- br i1 true, label %foo, label %bar
- foo:
- ..
- bar:
- ; Dead code
- ..
-
-Therefore, it is recommended that ``NVVMReflect`` is executed early in the
+```llvm
+entry:
+ ..
+ br i1 true, label %foo, label %bar
+foo:
+ ..
+bar:
+ ; Dead code
+ ..
+```
+
+Therefore, it is recommended that `NVVMReflect` is executed early in the
optimization pipeline before dead-code elimination.
-The NVPTX TargetMachine knows how to schedule ``NVVMReflect`` at the beginning
+The NVPTX TargetMachine knows how to schedule `NVVMReflect` at the beginning
of your pass manager; just use the following code when setting up your pass
-manager and the PassBuilder will use ``registerPassBuilderCallbacks`` to let
+manager and the PassBuilder will use `registerPassBuilderCallbacks` to let
NVPTXTargetMachine::registerPassBuilderCallbacks add the pass to the
pass manager:
-.. code-block:: c++
-
- std::unique_ptr<TargetMachine> TM = ...;
- PassBuilder PB(TM);
- ModulePassManager MPM;
- PB.parsePassPipeline(MPM, ...);
+```c++
+std::unique_ptr<TargetMachine> TM = ...;
+PassBuilder PB(TM);
+ModulePassManager MPM;
+PB.parsePassPipeline(MPM, ...);
+```
-Reflection Parameters
----------------------
+### Reflection Parameters
The libdevice library currently uses the following reflection parameters to
control code generation:
-==================== ======================================================
-Flag Description
-==================== ======================================================
-``__CUDA_FTZ=[0,1]`` Use optimized code paths that flush subnormals to zero
-==================== ======================================================
+| Flag | Description |
+| ------------------ | ------------------------------------------------------ |
+| `__CUDA_FTZ=[0,1]` | Use optimized code paths that flush subnormals to zero |
The value of this flag is determined by the "nvvm-reflect-ftz" module flag.
The following sets the ftz flag to 1.
-.. code-block:: llvm
+```llvm
+!llvm.module.flags = !{!0}
+!0 = !{i32 4, !"nvvm-reflect-ftz", i32 1}
+```
- !llvm.module.flags = !{!0}
- !0 = !{i32 4, !"nvvm-reflect-ftz", i32 1}
-
-(``i32 4`` indicates that the value set here overrides the value in another
-module we link with. See the `LangRef <LangRef.html#module-flags-metadata>`
+(`i32 4` indicates that the value set here overrides the value in another
+module we link with. See the [LangRef](project:LangRef.md#module-flags-metadata)
for details.)
-Executing PTX
-=============
+## Executing PTX
The most common way to execute PTX assembly on a GPU device is to use the CUDA
Driver API. This API is a low-level interface to the GPU driver and allows for
@@ -4206,195 +3722,174 @@ JIT compilation of PTX code to native GPU machine code.
Initializing the Driver API:
-.. code-block:: c++
-
- CUdevice device;
- CUcontext context;
+```c++
+CUdevice device;
+CUcontext context;
- // Initialize the driver API
- cuInit(0);
- // Get a handle to the first compute device
- cuDeviceGet(&device, 0);
- // Create a compute device context
- cuCtxCreate(&context, 0, device);
+// Initialize the driver API
+cuInit(0);
+// Get a handle to the first compute device
+cuDeviceGet(&device, 0);
+// Create a compute device context
+cuCtxCreate(&context, 0, device);
+```
JIT compiling a PTX string to a device binary:
-.. code-block:: c++
-
- CUmodule module;
- CUfunction function;
+```c++
+CUmodule module;
+CUfunction function;
- // JIT compile a null-terminated PTX string
- cuModuleLoadData(&module, (void*)PTXString);
+// JIT compile a null-terminated PTX string
+cuModuleLoadData(&module, (void*)PTXString);
- // Get a handle to the "myfunction" kernel function
- cuModuleGetFunction(&function, module, "myfunction");
+// Get a handle to the "myfunction" kernel function
+cuModuleGetFunction(&function, module, "myfunction");
+```
-For full examples of executing PTX assembly, please see the `CUDA Samples
-<https://developer.nvidia.com/cuda-downloads>`_ distribution.
+For full examples of executing PTX assembly, please see the [CUDA Samples](https://developer.nvidia.com/cuda-downloads) distribution.
+## Common Issues
-Common Issues
-=============
+### ptxas complains of undefined function: \_\_nvvm_reflect
-ptxas complains of undefined function: __nvvm_reflect
------------------------------------------------------
+When linking with libdevice, the `NVVMReflect` pass must be used. See
+{ref}`libdevice` for more information.
-When linking with libdevice, the ``NVVMReflect`` pass must be used. See
-:ref:`libdevice` for more information.
-
-
-Tutorial: A Simple Compute Kernel
-=================================
+## Tutorial: A Simple Compute Kernel
To start, let us take a look at a simple compute kernel written directly in
LLVM IR. The kernel implements vector addition, where each thread computes one
-element of the output vector C from the input vectors A and B. To make this
+element of the output vector C from the input vectors A and B. To make this
easier, we also assume that only a single CTA (thread block) will be launched,
and that it will be one dimensional.
-
-The Kernel
-----------
-
-.. code-block:: llvm
-
- target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v32:32:32-v64:64:64-v128:128:128-n16:32:64"
- target triple = "nvptx64-nvidia-cuda"
-
- ; Intrinsic to read X component of thread ID
- declare i32 @llvm.nvvm.read.ptx.sreg.tid.x() readnone nounwind
-
- define ptx_kernel void @kernel(ptr addrspace(1) %A,
- ptr addrspace(1) %B,
- ptr addrspace(1) %C) {
- entry:
- ; What is my ID?
- %id = tail call i32 @llvm.nvvm.read.ptx.sreg.tid.x() readnone nounwind
-
- ; Compute pointers into A, B, and C
- %ptrA = getelementptr float, ptr addrspace(1) %A, i32 %id
- %ptrB = getelementptr float, ptr addrspace(1) %B, i32 %id
- %ptrC = getelementptr float, ptr addrspace(1) %C, i32 %id
-
- ; Read A, B
- %valA = load float, ptr addrspace(1) %ptrA, align 4
- %valB = load float, ptr addrspace(1) %ptrB, align 4
-
- ; Compute C = A + B
- %valC = fadd float %valA, %valB
-
- ; Store back to C
- store float %valC, ptr addrspace(1) %ptrC, align 4
-
- ret void
- }
-
-
-We can use the LLVM ``llc`` tool to directly run the NVPTX code generator:
-
-.. code-block:: text
-
- # llc -mcpu=sm_20 kernel.ll -o kernel.ptx
-
-
-.. note::
-
- If you want to generate 32-bit code, change ``p:64:64:64`` to ``p:32:32:32``
- in the module data layout string and use ``nvptx-nvidia-cuda`` as the
- target triple.
-
-
-The output we get from ``llc`` (as of LLVM 3.4):
-
-.. code-block:: text
-
- //
- // Generated by LLVM NVPTX Back-End
- //
-
- .version 3.1
- .target sm_20
- .address_size 64
-
- // .globl kernel
- // @kernel
- .visible .entry kernel(
- .param .u64 kernel_param_0,
- .param .u64 kernel_param_1,
- .param .u64 kernel_param_2
- )
- {
- .reg .f32 %f<4>;
- .reg .s32 %r<2>;
- .reg .s64 %rl<8>;
-
- // %bb.0: // %entry
- ld.param.u64 %rl1, [kernel_param_0];
- mov.u32 %r1, %tid.x;
- mul.wide.s32 %rl2, %r1, 4;
- add.s64 %rl3, %rl1, %rl2;
- ld.param.u64 %rl4, [kernel_param_1];
- add.s64 %rl5, %rl4, %rl2;
- ld.param.u64 %rl6, [kernel_param_2];
- add.s64 %rl7, %rl6, %rl2;
- ld.global.f32 %f1, [%rl3];
- ld.global.f32 %f2, [%rl5];
- add.f32 %f3, %f1, %f2;
- st.global.f32 [%rl7], %f3;
- ret;
- }
-
-
-Dissecting the Kernel
----------------------
+### The Kernel
+
+```llvm
+target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v32:32:32-v64:64:64-v128:128:128-n16:32:64"
+target triple = "nvptx64-nvidia-cuda"
+
+; Intrinsic to read X component of thread ID
+declare i32 @llvm.nvvm.read.ptx.sreg.tid.x() readnone nounwind
+
+define ptx_kernel void @kernel(ptr addrspace(1) %A,
+ ptr addrspace(1) %B,
+ ptr addrspace(1) %C) {
+entry:
+ ; What is my ID?
+ %id = tail call i32 @llvm.nvvm.read.ptx.sreg.tid.x() readnone nounwind
+
+ ; Compute pointers into A, B, and C
+ %ptrA = getelementptr float, ptr addrspace(1) %A, i32 %id
+ %ptrB = getelementptr float, ptr addrspace(1) %B, i32 %id
+ %ptrC = getelementptr float, ptr addrspace(1) %C, i32 %id
+
+ ; Read A, B
+ %valA = load float, ptr addrspace(1) %ptrA, align 4
+ %valB = load float, ptr addrspace(1) %ptrB, align 4
+
+ ; Compute C = A + B
+ %valC = fadd float %valA, %valB
+
+ ; Store back to C
+ store float %valC, ptr addrspace(1) %ptrC, align 4
+
+ ret void
+}
+```
+
+We can use the LLVM `llc` tool to directly run the NVPTX code generator:
+
+```text
+# llc -mcpu=sm_20 kernel.ll -o kernel.ptx
+```
+
+:::{note}
+If you want to generate 32-bit code, change `p:64:64:64` to `p:32:32:32`
+in the module data layout string and use `nvptx-nvidia-cuda` as the
+target triple.
+:::
+
+The output we get from `llc` (as of LLVM 3.4):
+
+```text
+//
+// Generated by LLVM NVPTX Back-End
+//
+
+.version 3.1
+.target sm_20
+.address_size 64
+
+ // .globl kernel
+ // @kernel
+.visible .entry kernel(
+ .param .u64 kernel_param_0,
+ .param .u64 kernel_param_1,
+ .param .u64 kernel_param_2
+)
+{
+ .reg .f32 %f<4>;
+ .reg .s32 %r<2>;
+ .reg .s64 %rl<8>;
+
+// %bb.0: // %entry
+ ld.param.u64 %rl1, [kernel_param_0];
+ mov.u32 %r1, %tid.x;
+ mul.wide.s32 %rl2, %r1, 4;
+ add.s64 %rl3, %rl1, %rl2;
+ ld.param.u64 %rl4, [kernel_param_1];
+ add.s64 %rl5, %rl4, %rl2;
+ ld.param.u64 %rl6, [kernel_param_2];
+ add.s64 %rl7, %rl6, %rl2;
+ ld.global.f32 %f1, [%rl3];
+ ld.global.f32 %f2, [%rl5];
+ add.f32 %f3, %f1, %f2;
+ st.global.f32 [%rl7], %f3;
+ ret;
+}
+```
+
+### Dissecting the Kernel
Now let us dissect the LLVM IR that makes up this kernel.
-Data Layout
-^^^^^^^^^^^
+#### Data Layout
The data layout string determines the size in bits of common data types, their
-ABI alignment, and their storage size. For NVPTX, you should use one of the
+ABI alignment, and their storage size. For NVPTX, you should use one of the
following:
32-bit PTX:
-.. code-block:: llvm
-
- target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v32:32:32-v64:64:64-v128:128:128-n16:32:64"
+```llvm
+target datalayout = "e-p:32:32:32-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v32:32:32-v64:64:64-v128:128:128-n16:32:64"
+```
64-bit PTX:
-.. code-block:: llvm
-
- target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v32:32:32-v64:64:64-v128:128:128-n16:32:64"
-
+```llvm
+target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v32:32:32-v64:64:64-v128:128:128-n16:32:64"
+```
-Target Intrinsics
-^^^^^^^^^^^^^^^^^
+#### Target Intrinsics
-In this example, we use the ``@llvm.nvvm.read.ptx.sreg.tid.x`` intrinsic to
+In this example, we use the `@llvm.nvvm.read.ptx.sreg.tid.x` intrinsic to
read the X component of the current thread's ID, which corresponds to a read
-of register ``%tid.x`` in PTX. The NVPTX back-end supports a large set of
-intrinsics. A short list is shown below; please see
-``include/llvm/IR/IntrinsicsNVVM.td`` for the full list.
+of register `%tid.x` in PTX. The NVPTX back-end supports a large set of
+intrinsics. A short list is shown below; please see
+`include/llvm/IR/IntrinsicsNVVM.td` for the full list.
+| Intrinsic | CUDA Equivalent |
+| --------------------------------------------- | ----------------- |
+| `i32 @llvm.nvvm.read.ptx.sreg.tid.{x,y,z}` | threadIdx.{x,y,z} |
+| `i32 @llvm.nvvm.read.ptx.sreg.ctaid.{x,y,z}` | blockIdx.{x,y,z} |
+| `i32 @llvm.nvvm.read.ptx.sreg.ntid.{x,y,z}` | blockDim.{x,y,z} |
+| `i32 @llvm.nvvm.read.ptx.sreg.nctaid.{x,y,z}` | gridDim.{x,y,z} |
+| `void @llvm.nvvm.barrier0()` | \_\_syncthreads() |
-================================================ ====================
-Intrinsic CUDA Equivalent
-================================================ ====================
-``i32 @llvm.nvvm.read.ptx.sreg.tid.{x,y,z}`` threadIdx.{x,y,z}
-``i32 @llvm.nvvm.read.ptx.sreg.ctaid.{x,y,z}`` blockIdx.{x,y,z}
-``i32 @llvm.nvvm.read.ptx.sreg.ntid.{x,y,z}`` blockDim.{x,y,z}
-``i32 @llvm.nvvm.read.ptx.sreg.nctaid.{x,y,z}`` gridDim.{x,y,z}
-``void @llvm.nvvm.barrier0()`` __syncthreads()
-================================================ ====================
-
-
-Address Spaces
-^^^^^^^^^^^^^^
+#### Address Spaces
You may have noticed that all of the pointer types in the LLVM IR example had
an explicit address space specifier. What is address space 1? NVIDIA GPU
@@ -4407,361 +3902,352 @@ devices (generally) have four types of memory:
These
diff erent types of memory are represented in LLVM IR as address spaces.
There is also a fifth address space used by the NVPTX code generator that
-corresponds to the "generic" address space. This address space can represent
-addresses in any other address space (with a few exceptions). This allows
+corresponds to the "generic" address space. This address space can represent
+addresses in any other address space (with a few exceptions). This allows
users to write IR functions that can load/store memory using the same
instructions. Intrinsics are provided to convert pointers between the generic
and non-generic address spaces.
-See :ref:`address_spaces` and :ref:`nvptx_intrinsics` for more information.
+See {ref}`address-spaces` and {ref}`nvptx-intrinsics` for more information.
-
-Running the Kernel
-------------------
+### Running the Kernel
Generating PTX from LLVM IR is all well and good, but how do we execute it on
a real GPU device? The CUDA Driver API provides a convenient mechanism for
loading and JIT compiling PTX to a native GPU device, and launching a kernel.
-The API is similar to OpenCL. A simple example showing how to load and
+The API is similar to OpenCL. A simple example showing how to load and
execute our vector addition code is shown below. Note that for brevity this
code does not perform much error checking!
-.. note::
+:::{note}
+You can also use the `ptxas` tool provided by the CUDA Toolkit to offline
+compile PTX to machine code (SASS) for a specific GPU architecture. Such
+binaries can be loaded by the CUDA Driver API in the same way as PTX. This
+can be useful for reducing startup time by precompiling the PTX kernels.
+:::
+
+```c++
+#include <iostream>
+#include <fstream>
+#include <cassert>
+#include "cuda.h"
+
+
+void checkCudaErrors(CUresult err) {
+ assert(err == CUDA_SUCCESS);
+}
+
+/// main - Program entry point
+int main(int argc, char **argv) {
+ CUdevice device;
+ CUmodule cudaModule;
+ CUcontext context;
+ CUfunction function;
+ CUlinkState linker;
+ int devCount;
+
+ // CUDA initialization
+ checkCudaErrors(cuInit(0));
+ checkCudaErrors(cuDeviceGetCount(&devCount));
+ checkCudaErrors(cuDeviceGet(&device, 0));
+
+ char name[128];
+ checkCudaErrors(cuDeviceGetName(name, 128, device));
+ std::cout << "Using CUDA Device [0]: " << name << "\n";
+
+ int devMajor, devMinor;
+ checkCudaErrors(cuDeviceComputeCapability(&devMajor, &devMinor, device));
+ std::cout << "Device Compute Capability: "
+ << devMajor << "." << devMinor << "\n";
+ if (devMajor < 2) {
+ std::cerr << "ERROR: Device 0 is not SM 2.0 or greater\n";
+ return 1;
+ }
+
+ std::ifstream t("kernel.ptx");
+ if (!t.is_open()) {
+ std::cerr << "kernel.ptx not found\n";
+ return 1;
+ }
+ std::string str((std::istreambuf_iterator<char>(t)),
+ std::istreambuf_iterator<char>());
+
+ // Create driver context
+ checkCudaErrors(cuCtxCreate(&context, 0, device));
- You can also use the ``ptxas`` tool provided by the CUDA Toolkit to offline
- compile PTX to machine code (SASS) for a specific GPU architecture. Such
- binaries can be loaded by the CUDA Driver API in the same way as PTX. This
- can be useful for reducing startup time by precompiling the PTX kernels.
+ // Create module for object
+ checkCudaErrors(cuModuleLoadDataEx(&cudaModule, str.c_str(), 0, 0, 0));
+ // Get kernel function
+ checkCudaErrors(cuModuleGetFunction(&function, cudaModule, "kernel"));
-.. code-block:: c++
+ // Device data
+ CUdeviceptr devBufferA;
+ CUdeviceptr devBufferB;
+ CUdeviceptr devBufferC;
- #include <iostream>
- #include <fstream>
- #include <cassert>
- #include "cuda.h"
+ checkCudaErrors(cuMemAlloc(&devBufferA, sizeof(float)*16));
+ checkCudaErrors(cuMemAlloc(&devBufferB, sizeof(float)*16));
+ checkCudaErrors(cuMemAlloc(&devBufferC, sizeof(float)*16));
+ float* hostA = new float[16];
+ float* hostB = new float[16];
+ float* hostC = new float[16];
- void checkCudaErrors(CUresult err) {
- assert(err == CUDA_SUCCESS);
+ // Populate input
+ for (unsigned i = 0; i != 16; ++i) {
+ hostA[i] = (float)i;
+ hostB[i] = (float)(2*i);
+ hostC[i] = 0.0f;
}
- /// main - Program entry point
- int main(int argc, char **argv) {
- CUdevice device;
- CUmodule cudaModule;
- CUcontext context;
- CUfunction function;
- CUlinkState linker;
- int devCount;
-
- // CUDA initialization
- checkCudaErrors(cuInit(0));
- checkCudaErrors(cuDeviceGetCount(&devCount));
- checkCudaErrors(cuDeviceGet(&device, 0));
-
- char name[128];
- checkCudaErrors(cuDeviceGetName(name, 128, device));
- std::cout << "Using CUDA Device [0]: " << name << "\n";
-
- int devMajor, devMinor;
- checkCudaErrors(cuDeviceComputeCapability(&devMajor, &devMinor, device));
- std::cout << "Device Compute Capability: "
- << devMajor << "." << devMinor << "\n";
- if (devMajor < 2) {
- std::cerr << "ERROR: Device 0 is not SM 2.0 or greater\n";
- return 1;
- }
-
- std::ifstream t("kernel.ptx");
- if (!t.is_open()) {
- std::cerr << "kernel.ptx not found\n";
- return 1;
- }
- std::string str((std::istreambuf_iterator<char>(t)),
- std::istreambuf_iterator<char>());
-
- // Create driver context
- checkCudaErrors(cuCtxCreate(&context, 0, device));
-
- // Create module for object
- checkCudaErrors(cuModuleLoadDataEx(&cudaModule, str.c_str(), 0, 0, 0));
-
- // Get kernel function
- checkCudaErrors(cuModuleGetFunction(&function, cudaModule, "kernel"));
-
- // Device data
- CUdeviceptr devBufferA;
- CUdeviceptr devBufferB;
- CUdeviceptr devBufferC;
-
- checkCudaErrors(cuMemAlloc(&devBufferA, sizeof(float)*16));
- checkCudaErrors(cuMemAlloc(&devBufferB, sizeof(float)*16));
- checkCudaErrors(cuMemAlloc(&devBufferC, sizeof(float)*16));
-
- float* hostA = new float[16];
- float* hostB = new float[16];
- float* hostC = new float[16];
-
- // Populate input
- for (unsigned i = 0; i != 16; ++i) {
- hostA[i] = (float)i;
- hostB[i] = (float)(2*i);
- hostC[i] = 0.0f;
- }
-
- checkCudaErrors(cuMemcpyHtoD(devBufferA, &hostA[0], sizeof(float)*16));
- checkCudaErrors(cuMemcpyHtoD(devBufferB, &hostB[0], sizeof(float)*16));
-
-
- unsigned blockSizeX = 16;
- unsigned blockSizeY = 1;
- unsigned blockSizeZ = 1;
- unsigned gridSizeX = 1;
- unsigned gridSizeY = 1;
- unsigned gridSizeZ = 1;
-
- // Kernel parameters
- void *KernelParams[] = { &devBufferA, &devBufferB, &devBufferC };
-
- std::cout << "Launching kernel\n";
-
- // Kernel launch
- checkCudaErrors(cuLaunchKernel(function, gridSizeX, gridSizeY, gridSizeZ,
- blockSizeX, blockSizeY, blockSizeZ,
- 0, NULL, KernelParams, NULL));
-
- // Retrieve device data
- checkCudaErrors(cuMemcpyDtoH(&hostC[0], devBufferC, sizeof(float)*16));
-
-
- std::cout << "Results:\n";
- for (unsigned i = 0; i != 16; ++i) {
- std::cout << hostA[i] << " + " << hostB[i] << " = " << hostC[i] << "\n";
- }
-
-
- // Clean up after ourselves
- delete [] hostA;
- delete [] hostB;
- delete [] hostC;
-
- // Clean-up
- checkCudaErrors(cuMemFree(devBufferA));
- checkCudaErrors(cuMemFree(devBufferB));
- checkCudaErrors(cuMemFree(devBufferC));
- checkCudaErrors(cuModuleUnload(cudaModule));
- checkCudaErrors(cuCtxDestroy(context));
-
- return 0;
+ checkCudaErrors(cuMemcpyHtoD(devBufferA, &hostA[0], sizeof(float)*16));
+ checkCudaErrors(cuMemcpyHtoD(devBufferB, &hostB[0], sizeof(float)*16));
+
+
+ unsigned blockSizeX = 16;
+ unsigned blockSizeY = 1;
+ unsigned blockSizeZ = 1;
+ unsigned gridSizeX = 1;
+ unsigned gridSizeY = 1;
+ unsigned gridSizeZ = 1;
+
+ // Kernel parameters
+ void *KernelParams[] = { &devBufferA, &devBufferB, &devBufferC };
+
+ std::cout << "Launching kernel\n";
+
+ // Kernel launch
+ checkCudaErrors(cuLaunchKernel(function, gridSizeX, gridSizeY, gridSizeZ,
+ blockSizeX, blockSizeY, blockSizeZ,
+ 0, NULL, KernelParams, NULL));
+
+ // Retrieve device data
+ checkCudaErrors(cuMemcpyDtoH(&hostC[0], devBufferC, sizeof(float)*16));
+
+
+ std::cout << "Results:\n";
+ for (unsigned i = 0; i != 16; ++i) {
+ std::cout << hostA[i] << " + " << hostB[i] << " = " << hostC[i] << "\n";
}
-You will need to link with the CUDA driver and specify the path to cuda.h.
+ // Clean up after ourselves
+ delete [] hostA;
+ delete [] hostB;
+ delete [] hostC;
-.. code-block:: text
+ // Clean-up
+ checkCudaErrors(cuMemFree(devBufferA));
+ checkCudaErrors(cuMemFree(devBufferB));
+ checkCudaErrors(cuMemFree(devBufferC));
+ checkCudaErrors(cuModuleUnload(cudaModule));
+ checkCudaErrors(cuCtxDestroy(context));
- # clang++ sample.cpp -o sample -O2 -g -I/usr/local/cuda-5.5/include -lcuda
+ return 0;
+}
+```
+
+You will need to link with the CUDA driver and specify the path to cuda.h.
-We don't need to specify a path to ``libcuda.so`` since this is installed in a
+```text
+# clang++ sample.cpp -o sample -O2 -g -I/usr/local/cuda-5.5/include -lcuda
+```
+
+We don't need to specify a path to `libcuda.so` since this is installed in a
system location by the driver, not the CUDA toolkit.
If everything goes as planned, you should see the following output when
running the compiled program:
-.. code-block:: text
-
- Using CUDA Device [0]: GeForce GTX 680
- Device Compute Capability: 3.0
- Launching kernel
- Results:
- 0 + 0 = 0
- 1 + 2 = 3
- 2 + 4 = 6
- 3 + 6 = 9
- 4 + 8 = 12
- 5 + 10 = 15
- 6 + 12 = 18
- 7 + 14 = 21
- 8 + 16 = 24
- 9 + 18 = 27
- 10 + 20 = 30
- 11 + 22 = 33
- 12 + 24 = 36
- 13 + 26 = 39
- 14 + 28 = 42
- 15 + 30 = 45
-
-.. note::
-
- You will likely see a
diff erent device identifier based on your hardware
-
-
-Tutorial: Linking with Libdevice
-================================
+```text
+Using CUDA Device [0]: GeForce GTX 680
+Device Compute Capability: 3.0
+Launching kernel
+Results:
+0 + 0 = 0
+1 + 2 = 3
+2 + 4 = 6
+3 + 6 = 9
+4 + 8 = 12
+5 + 10 = 15
+6 + 12 = 18
+7 + 14 = 21
+8 + 16 = 24
+9 + 18 = 27
+10 + 20 = 30
+11 + 22 = 33
+12 + 24 = 36
+13 + 26 = 39
+14 + 28 = 42
+15 + 30 = 45
+```
+
+:::{note}
+You will likely see a
diff erent device identifier based on your hardware
+:::
+
+## Tutorial: Linking with Libdevice
In this tutorial, we show a simple example of linking LLVM IR with the
libdevice library. We will use the same kernel as the previous tutorial,
-except that we will compute ``C = pow(A, B)`` instead of ``C = A + B``.
-Libdevice provides an ``__nv_powf`` function that we will use.
-
-.. code-block:: llvm
+except that we will compute `C = pow(A, B)` instead of `C = A + B`.
+Libdevice provides an `__nv_powf` function that we will use.
- target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v32:32:32-v64:64:64-v128:128:128-n16:32:64"
- target triple = "nvptx64-nvidia-cuda"
+```llvm
+target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v16:16:16-v32:32:32-v64:64:64-v128:128:128-n16:32:64"
+target triple = "nvptx64-nvidia-cuda"
- ; Intrinsic to read X component of thread ID
- declare i32 @llvm.nvvm.read.ptx.sreg.tid.x() readnone nounwind
- ; libdevice function
- declare float @__nv_powf(float, float)
+; Intrinsic to read X component of thread ID
+declare i32 @llvm.nvvm.read.ptx.sreg.tid.x() readnone nounwind
+; libdevice function
+declare float @__nv_powf(float, float)
- define ptx_kernel void @kernel(ptr addrspace(1) %A,
- ptr addrspace(1) %B,
- ptr addrspace(1) %C) {
- entry:
- ; What is my ID?
- %id = tail call i32 @llvm.nvvm.read.ptx.sreg.tid.x() readnone nounwind
+define ptx_kernel void @kernel(ptr addrspace(1) %A,
+ ptr addrspace(1) %B,
+ ptr addrspace(1) %C) {
+entry:
+ ; What is my ID?
+ %id = tail call i32 @llvm.nvvm.read.ptx.sreg.tid.x() readnone nounwind
- ; Compute pointers into A, B, and C
- %ptrA = getelementptr float, ptr addrspace(1) %A, i32 %id
- %ptrB = getelementptr float, ptr addrspace(1) %B, i32 %id
- %ptrC = getelementptr float, ptr addrspace(1) %C, i32 %id
+ ; Compute pointers into A, B, and C
+ %ptrA = getelementptr float, ptr addrspace(1) %A, i32 %id
+ %ptrB = getelementptr float, ptr addrspace(1) %B, i32 %id
+ %ptrC = getelementptr float, ptr addrspace(1) %C, i32 %id
- ; Read A, B
- %valA = load float, ptr addrspace(1) %ptrA, align 4
- %valB = load float, ptr addrspace(1) %ptrB, align 4
+ ; Read A, B
+ %valA = load float, ptr addrspace(1) %ptrA, align 4
+ %valB = load float, ptr addrspace(1) %ptrB, align 4
- ; Compute C = pow(A, B)
- %valC = call float @__nv_powf(float %valA, float %valB)
+ ; Compute C = pow(A, B)
+ %valC = call float @__nv_powf(float %valA, float %valB)
- ; Store back to C
- store float %valC, ptr addrspace(1) %ptrC, align 4
-
- ret void
- }
+ ; Store back to C
+ store float %valC, ptr addrspace(1) %ptrC, align 4
+ ret void
+}
+```
To compile this kernel, we perform the following steps:
1. Link with libdevice
2. Internalize all but the public kernel function
-3. Run ``NVVMReflect`` and set ``__CUDA_FTZ`` to 0
+3. Run `NVVMReflect` and set `__CUDA_FTZ` to 0
4. Optimize the linked module
5. Codegen the module
-
-These steps can be performed by the LLVM ``llvm-link``, ``opt``, and ``llc``
+These steps can be performed by the LLVM `llvm-link`, `opt`, and `llc`
tools. In a complete compiler, these steps can also be performed entirely
programmatically by setting up an appropriate pass configuration (see
-:ref:`libdevice`).
-
-.. code-block:: text
+{ref}`libdevice`).
- # llvm-link t2.bc libdevice.compute_20.10.bc -o t2.linked.bc
- # opt -internalize -internalize-public-api-list=kernel -nvvm-reflect-list=__CUDA_FTZ=0 -nvvm-reflect -O3 t2.linked.bc -o t2.opt.bc
- # llc -mcpu=sm_20 t2.opt.bc -o t2.ptx
-
-.. note::
-
- The ``-nvvm-reflect-list=_CUDA_FTZ=0`` is not strictly required, as any
- undefined variables will default to zero. It is shown here for evaluation
- purposes.
+```text
+# llvm-link t2.bc libdevice.compute_20.10.bc -o t2.linked.bc
+# opt -internalize -internalize-public-api-list=kernel -nvvm-reflect-list=__CUDA_FTZ=0 -nvvm-reflect -O3 t2.linked.bc -o t2.opt.bc
+# llc -mcpu=sm_20 t2.opt.bc -o t2.ptx
+```
+:::{note}
+The `-nvvm-reflect-list=_CUDA_FTZ=0` is not strictly required, as any
+undefined variables will default to zero. It is shown here for evaluation
+purposes.
+:::
This gives us the following PTX (excerpt):
-.. code-block:: text
-
- //
- // Generated by LLVM NVPTX Back-End
- //
-
- .version 3.1
- .target sm_20
- .address_size 64
-
- // .globl kernel
- // @kernel
- .visible .entry kernel(
- .param .u64 kernel_param_0,
- .param .u64 kernel_param_1,
- .param .u64 kernel_param_2
- )
- {
- .reg .pred %p<30>;
- .reg .f32 %f<111>;
- .reg .s32 %r<21>;
- .reg .s64 %rl<8>;
-
- // %bb.0: // %entry
- ld.param.u64 %rl2, [kernel_param_0];
- mov.u32 %r3, %tid.x;
- ld.param.u64 %rl3, [kernel_param_1];
- mul.wide.s32 %rl4, %r3, 4;
- add.s64 %rl5, %rl2, %rl4;
- ld.param.u64 %rl6, [kernel_param_2];
- add.s64 %rl7, %rl3, %rl4;
- add.s64 %rl1, %rl6, %rl4;
- ld.global.f32 %f1, [%rl5];
- ld.global.f32 %f2, [%rl7];
- setp.eq.f32 %p1, %f1, 0f3F800000;
- setp.eq.f32 %p2, %f2, 0f00000000;
- or.pred %p3, %p1, %p2;
- @%p3 bra BB0_1;
- bra.uni BB0_2;
- BB0_1:
- mov.f32 %f110, 0f3F800000;
- st.global.f32 [%rl1], %f110;
- ret;
- BB0_2: // %__nv_isnanf.exit.i
- abs.f32 %f4, %f1;
- setp.gtu.f32 %p4, %f4, 0f7F800000;
- @%p4 bra BB0_4;
- // %bb.3: // %__nv_isnanf.exit5.i
- abs.f32 %f5, %f2;
- setp.le.f32 %p5, %f5, 0f7F800000;
- @%p5 bra BB0_5;
- BB0_4: // %.critedge1.i
- add.f32 %f110, %f1, %f2;
- st.global.f32 [%rl1], %f110;
- ret;
- BB0_5: // %__nv_isinff.exit.i
-
- ...
-
- BB0_26: // %__nv_truncf.exit.i.i.i.i.i
- mul.f32 %f90, %f107, 0f3FB8AA3B;
- cvt.rzi.f32.f32 %f91, %f90;
- mov.f32 %f92, 0fBF317200;
- fma.rn.f32 %f93, %f91, %f92, %f107;
- mov.f32 %f94, 0fB5BFBE8E;
- fma.rn.f32 %f95, %f91, %f94, %f93;
- mul.f32 %f89, %f95, 0f3FB8AA3B;
- // inline asm
- ex2.approx.ftz.f32 %f88,%f89;
- // inline asm
- add.f32 %f96, %f91, 0f00000000;
- ex2.approx.f32 %f97, %f96;
- mul.f32 %f98, %f88, %f97;
- setp.lt.f32 %p15, %f107, 0fC2D20000;
- selp.f32 %f99, 0f00000000, %f98, %p15;
- setp.gt.f32 %p16, %f107, 0f42D20000;
- selp.f32 %f110, 0f7F800000, %f99, %p16;
- setp.eq.f32 %p17, %f110, 0f7F800000;
- @%p17 bra BB0_28;
- // %bb.27:
- fma.rn.f32 %f110, %f110, %f108, %f110;
- BB0_28: // %__internal_accurate_powf.exit.i
- setp.lt.f32 %p18, %f1, 0f00000000;
- setp.eq.f32 %p19, %f3, 0f3F800000;
- and.pred %p20, %p18, %p19;
- @!%p20 bra BB0_30;
- bra.uni BB0_29;
- BB0_29:
- mov.b32 %r9, %f110;
- xor.b32 %r10, %r9, -2147483648;
- mov.b32 %f110, %r10;
- BB0_30: // %__nv_powf.exit
- st.global.f32 [%rl1], %f110;
- ret;
- }
+```text
+//
+// Generated by LLVM NVPTX Back-End
+//
+
+.version 3.1
+.target sm_20
+.address_size 64
+
+ // .globl kernel
+ // @kernel
+.visible .entry kernel(
+ .param .u64 kernel_param_0,
+ .param .u64 kernel_param_1,
+ .param .u64 kernel_param_2
+)
+{
+ .reg .pred %p<30>;
+ .reg .f32 %f<111>;
+ .reg .s32 %r<21>;
+ .reg .s64 %rl<8>;
+
+// %bb.0: // %entry
+ ld.param.u64 %rl2, [kernel_param_0];
+ mov.u32 %r3, %tid.x;
+ ld.param.u64 %rl3, [kernel_param_1];
+ mul.wide.s32 %rl4, %r3, 4;
+ add.s64 %rl5, %rl2, %rl4;
+ ld.param.u64 %rl6, [kernel_param_2];
+ add.s64 %rl7, %rl3, %rl4;
+ add.s64 %rl1, %rl6, %rl4;
+ ld.global.f32 %f1, [%rl5];
+ ld.global.f32 %f2, [%rl7];
+ setp.eq.f32 %p1, %f1, 0f3F800000;
+ setp.eq.f32 %p2, %f2, 0f00000000;
+ or.pred %p3, %p1, %p2;
+ @%p3 bra BB0_1;
+ bra.uni BB0_2;
+BB0_1:
+ mov.f32 %f110, 0f3F800000;
+ st.global.f32 [%rl1], %f110;
+ ret;
+BB0_2: // %__nv_isnanf.exit.i
+ abs.f32 %f4, %f1;
+ setp.gtu.f32 %p4, %f4, 0f7F800000;
+ @%p4 bra BB0_4;
+// %bb.3: // %__nv_isnanf.exit5.i
+ abs.f32 %f5, %f2;
+ setp.le.f32 %p5, %f5, 0f7F800000;
+ @%p5 bra BB0_5;
+BB0_4: // %.critedge1.i
+ add.f32 %f110, %f1, %f2;
+ st.global.f32 [%rl1], %f110;
+ ret;
+BB0_5: // %__nv_isinff.exit.i
+
+ ...
+
+BB0_26: // %__nv_truncf.exit.i.i.i.i.i
+ mul.f32 %f90, %f107, 0f3FB8AA3B;
+ cvt.rzi.f32.f32 %f91, %f90;
+ mov.f32 %f92, 0fBF317200;
+ fma.rn.f32 %f93, %f91, %f92, %f107;
+ mov.f32 %f94, 0fB5BFBE8E;
+ fma.rn.f32 %f95, %f91, %f94, %f93;
+ mul.f32 %f89, %f95, 0f3FB8AA3B;
+ // inline asm
+ ex2.approx.ftz.f32 %f88,%f89;
+ // inline asm
+ add.f32 %f96, %f91, 0f00000000;
+ ex2.approx.f32 %f97, %f96;
+ mul.f32 %f98, %f88, %f97;
+ setp.lt.f32 %p15, %f107, 0fC2D20000;
+ selp.f32 %f99, 0f00000000, %f98, %p15;
+ setp.gt.f32 %p16, %f107, 0f42D20000;
+ selp.f32 %f110, 0f7F800000, %f99, %p16;
+ setp.eq.f32 %p17, %f110, 0f7F800000;
+ @%p17 bra BB0_28;
+// %bb.27:
+ fma.rn.f32 %f110, %f110, %f108, %f110;
+BB0_28: // %__internal_accurate_powf.exit.i
+ setp.lt.f32 %p18, %f1, 0f00000000;
+ setp.eq.f32 %p19, %f3, 0f3F800000;
+ and.pred %p20, %p18, %p19;
+ @!%p20 bra BB0_30;
+ bra.uni BB0_29;
+BB0_29:
+ mov.b32 %r9, %f110;
+ xor.b32 %r10, %r9, -2147483648;
+ mov.b32 %f110, %r10;
+BB0_30: // %__nv_powf.exit
+ st.global.f32 [%rl1], %f110;
+ ret;
+}
+```
diff --git a/llvm/docs/ORCv2.md b/llvm/docs/ORCv2.md
index 333977a0aaa66..100fef4b12081 100644
--- a/llvm/docs/ORCv2.md
+++ b/llvm/docs/ORCv2.md
@@ -1,20 +1,17 @@
-===============================
-ORC Design and Implementation
-===============================
+# ORC Design and Implementation
-.. contents::
- :local:
+```{contents}
+:local: true
+```
-Introduction
-============
+## Introduction
This document aims to provide a high-level overview of the design and
implementation of the ORC JIT APIs. Except where otherwise stated all discussion
refers to the modern ORCv2 APIs (available since LLVM 7). Clients wishing to
-transition from OrcV1 should see Section :ref:`transitioning_orcv1_to_orcv2`.
+transition from OrcV1 should see Section {ref}`transitioning_orcv1_to_orcv2`.
-Use-cases
-=========
+## Use-cases
ORC provides a modular API for building JIT compilers. There are a number
of use cases for such an API. For example:
@@ -35,38 +32,37 @@ optimizations within an existing JIT infrastructure.
By adopting a modular, library-based design we aim to make ORC useful in as many
of these contexts as possible.
-Features
-========
+## Features
ORC provides the following features:
**JIT-linking**
- ORC provides APIs to link relocatable object files (COFF, ELF, MachO) [1]_
+: ORC provides APIs to link relocatable object files (COFF, ELF, MachO) [^1]
into a target process at runtime. The target process may be the same process
that contains the JIT session object and jit-linker, or may be another process
(even one running on a
diff erent machine or architecture) that communicates
with the JIT via RPC.
**LLVM IR compilation**
- ORC provides off the shelf components (IRCompileLayer, SimpleCompiler,
+: ORC provides off the shelf components (IRCompileLayer, SimpleCompiler,
ConcurrentIRCompiler) that make it easy to add LLVM IR to a JIT'd process.
**Eager and lazy compilation**
- By default, ORC will compile symbols as soon as they are looked up in the JIT
- session object (``ExecutionSession``). Compiling eagerly by default makes it
+: By default, ORC will compile symbols as soon as they are looked up in the JIT
+ session object (`ExecutionSession`). Compiling eagerly by default makes it
easy to use ORC as an in-memory compiler for an existing JIT (similar to how
MCJIT is commonly used). However ORC also provides built-in support for lazy
- compilation via lazy-reexports (see :ref:`Laziness`).
+ compilation via lazy-reexports (see {ref}`Laziness`).
**Support for Custom Compilers and Program Representations**
- Clients can supply custom compilers for each symbol that they define in their
+: Clients can supply custom compilers for each symbol that they define in their
JIT session. ORC will run the user-supplied compiler when the a definition of
a symbol is needed. ORC is actually fully language agnostic: LLVM IR is not
treated specially, and is supported via the same wrapper mechanism (the
- ``MaterializationUnit`` class) that is used for custom compilers.
+ `MaterializationUnit` class) that is used for custom compilers.
**Concurrent JIT'd code** and **Concurrent Compilation**
- JIT'd code may be executed in multiple threads, may spawn new threads, and may
+: JIT'd code may be executed in multiple threads, may spawn new threads, and may
re-enter the ORC (e.g. to request lazy compilation) concurrently from multiple
threads. Compilers launched my ORC can run concurrently (provided the client
sets up an appropriate dispatcher). Built-in dependency tracking ensures that
@@ -74,15 +70,14 @@ ORC provides the following features:
have also been JIT'd and they are safe to call or use.
**Removable Code**
- Resources for JIT'd program representations
+: Resources for JIT'd program representations
**Orthogonality** and **Composability**
- Each of the features above can be used independently. It is possible to put
+: Each of the features above can be used independently. It is possible to put
ORC components together to make a non-lazy, in-process, single threaded JIT
or a lazy, out-of-process, concurrent JIT, or anything in between.
-LLJIT and LLLazyJIT
-===================
+## LLJIT and LLLazyJIT
ORC provides two basic JIT classes off-the-shelf. These are useful both as
examples of how to assemble ORC components to make a JIT, and as replacements
@@ -103,110 +98,109 @@ JIT API.
LLJIT and LLLazyJIT instances can be created using their respective builder
classes: LLJITBuilder and LLazyJITBuilder. For example, assuming you have a
-module ``M`` loaded on a ThreadSafeContext ``Ctx``:
+module `M` loaded on a ThreadSafeContext `Ctx`:
-.. code-block:: c++
+```c++
+// Try to detect the host arch and construct an LLJIT instance.
+auto JIT = LLJITBuilder().create();
- // Try to detect the host arch and construct an LLJIT instance.
- auto JIT = LLJITBuilder().create();
+// If we could not construct an instance, return an error.
+if (!JIT)
+ return JIT.takeError();
- // If we could not construct an instance, return an error.
- if (!JIT)
- return JIT.takeError();
+// Add the module.
+if (auto Err = JIT->addIRModule(TheadSafeModule(std::move(M), Ctx)))
+ return Err;
- // Add the module.
- if (auto Err = JIT->addIRModule(TheadSafeModule(std::move(M), Ctx)))
- return Err;
+// Look up the JIT'd code entry point.
+auto EntrySym = JIT->lookup("entry");
+if (!EntrySym)
+ return EntrySym.takeError();
- // Look up the JIT'd code entry point.
- auto EntrySym = JIT->lookup("entry");
- if (!EntrySym)
- return EntrySym.takeError();
+// Cast the entry point address to a function pointer.
+auto *Entry = EntrySym.getAddress().toPtr<void(*)()>();
- // Cast the entry point address to a function pointer.
- auto *Entry = EntrySym.getAddress().toPtr<void(*)()>();
-
- // Call into JIT'd code.
- Entry();
+// Call into JIT'd code.
+Entry();
+```
The builder classes provide a number of configuration options that can be
specified before the JIT instance is constructed. For example:
-.. code-block:: c++
-
- // Build an LLLazyJIT instance that uses four worker threads for compilation,
- // and jumps to a specific error handler (rather than null) on lazy compile
- // failures.
+```c++
+// Build an LLLazyJIT instance that uses four worker threads for compilation,
+// and jumps to a specific error handler (rather than null) on lazy compile
+// failures.
- void handleLazyCompileFailure() {
- // JIT'd code will jump here if lazy compilation fails, giving us an
- // opportunity to exit or throw an exception into JIT'd code.
- throw JITFailed();
- }
+void handleLazyCompileFailure() {
+ // JIT'd code will jump here if lazy compilation fails, giving us an
+ // opportunity to exit or throw an exception into JIT'd code.
+ throw JITFailed();
+}
- auto JIT = LLLazyJITBuilder()
- .setNumCompileThreads(4)
- .setLazyCompileFailureAddr(
- ExecutorAddr::fromPtr(&handleLazyCompileFailure))
- .create();
+auto JIT = LLLazyJITBuilder()
+ .setNumCompileThreads(4)
+ .setLazyCompileFailureAddr(
+ ExecutorAddr::fromPtr(&handleLazyCompileFailure))
+ .create();
- // ...
+// ...
+```
For users wanting to get started with LLJIT a minimal example program can be
-found at ``llvm/examples/HowToUseLLJIT``.
+found at `llvm/examples/HowToUseLLJIT`.
-Design Overview
-===============
+## Design Overview
ORC's JIT program model aims to emulate the linking and symbol resolution
rules used by the static and dynamic linkers. This allows ORC to JIT
arbitrary LLVM IR, including IR produced by an ordinary static compiler (e.g.
-clang) that uses constructs like symbol linkage and visibility, and weak [3]_
+clang) that uses constructs like symbol linkage and visibility, and weak [^3]
and common symbol definitions.
-To see how this works, imagine a program ``foo`` which links against a pair
-of dynamic libraries: ``libA`` and ``libB``. On the command line, building this
+To see how this works, imagine a program `foo` which links against a pair
+of dynamic libraries: `libA` and `libB`. On the command line, building this
program might look like:
-.. code-block:: bash
-
- $ clang++ -shared -o libA.dylib a1.cpp a2.cpp
- $ clang++ -shared -o libB.dylib b1.cpp b2.cpp
- $ clang++ -o myapp myapp.cpp -L. -lA -lB
- $ ./myapp
+```bash
+$ clang++ -shared -o libA.dylib a1.cpp a2.cpp
+$ clang++ -shared -o libB.dylib b1.cpp b2.cpp
+$ clang++ -o myapp myapp.cpp -L. -lA -lB
+$ ./myapp
+```
In ORC, this would translate into API calls on a hypothetical CXXCompilingLayer
(with error checking omitted for brevity) as:
-.. code-block:: c++
+```c++
+ExecutionSession ES;
+RTDyldObjectLinkingLayer ObjLinkingLayer(
+ ES, []() { return std::make_unique<SectionMemoryManager>(); });
+CXXCompileLayer CXXLayer(ES, ObjLinkingLayer);
- ExecutionSession ES;
- RTDyldObjectLinkingLayer ObjLinkingLayer(
- ES, []() { return std::make_unique<SectionMemoryManager>(); });
- CXXCompileLayer CXXLayer(ES, ObjLinkingLayer);
+// Create JITDylib "A" and add code to it using the CXX layer.
+auto &LibA = ES.createJITDylib("A");
+CXXLayer.add(LibA, MemoryBuffer::getFile("a1.cpp"));
+CXXLayer.add(LibA, MemoryBuffer::getFile("a2.cpp"));
- // Create JITDylib "A" and add code to it using the CXX layer.
- auto &LibA = ES.createJITDylib("A");
- CXXLayer.add(LibA, MemoryBuffer::getFile("a1.cpp"));
- CXXLayer.add(LibA, MemoryBuffer::getFile("a2.cpp"));
+// Create JITDylib "B" and add code to it using the CXX layer.
+auto &LibB = ES.createJITDylib("B");
+CXXLayer.add(LibB, MemoryBuffer::getFile("b1.cpp"));
+CXXLayer.add(LibB, MemoryBuffer::getFile("b2.cpp"));
- // Create JITDylib "B" and add code to it using the CXX layer.
- auto &LibB = ES.createJITDylib("B");
- CXXLayer.add(LibB, MemoryBuffer::getFile("b1.cpp"));
- CXXLayer.add(LibB, MemoryBuffer::getFile("b2.cpp"));
+// Create and specify the search order for the main JITDylib. This is
+// equivalent to a "links against" relationship in a command-line link.
+auto &MainJD = ES.createJITDylib("main");
+MainJD.addToLinkOrder(&LibA);
+MainJD.addToLinkOrder(&LibB);
+CXXLayer.add(MainJD, MemoryBuffer::getFile("main.cpp"));
- // Create and specify the search order for the main JITDylib. This is
- // equivalent to a "links against" relationship in a command-line link.
- auto &MainJD = ES.createJITDylib("main");
- MainJD.addToLinkOrder(&LibA);
- MainJD.addToLinkOrder(&LibB);
- CXXLayer.add(MainJD, MemoryBuffer::getFile("main.cpp"));
+// Look up the JIT'd main, cast it to a function pointer, then call it.
+auto MainSym = ExitOnErr(ES.lookup({&MainJD}, "main"));
+auto *Main = MainSym.getAddress().toPtr<int(*)(int, char *[])>();
- // Look up the JIT'd main, cast it to a function pointer, then call it.
- auto MainSym = ExitOnErr(ES.lookup({&MainJD}, "main"));
- auto *Main = MainSym.getAddress().toPtr<int(*)(int, char *[])>();
-
- int Result = Main(...);
+int Result = Main(...);
+```
This example tells us nothing about *how* or *when* compilation will happen.
That will depend on the implementation of the hypothetical CXXCompilingLayer.
@@ -231,13 +225,13 @@ addresses for symbols: (1) It triggers compilation of the symbol(s) searched for
synchronization mechanism for concurrent compilation. The pseudo-code for the
lookup process is:
-.. code-block:: none
-
- construct a query object from a query set and query handler
- lock the session
- lodge query against requested symbols, collect required materializers (if any)
- unlock the session
- dispatch materializers (if any)
+```none
+construct a query object from a query set and query handler
+lock the session
+lodge query against requested symbols, collect required materializers (if any)
+unlock the session
+dispatch materializers (if any)
+```
In this context a materializer is something that provides a working definition
of a symbol upon request. Usually materializers are just wrappers for compilers,
@@ -262,21 +256,17 @@ materializer is run on the calling thread. Clients are free to create new
threads to run materializers, or to send the work to a work queue for a thread
pool (this is what LLJIT/LLLazyJIT do).
-Top Level APIs
-==============
+## Top Level APIs
Many of ORC's top-level APIs are visible in the example above:
- *ExecutionSession* represents the JIT'd program and provides context for the
JIT: It contains the JITDylibs, error reporting mechanisms, and dispatches the
materializers.
-
- *JITDylibs* provide the symbol tables.
-
- *Layers* (ObjLinkingLayer and CXXLayer) are wrappers around compilers and
allow clients to add uncompiled program representations supported by those
compilers to JITDylibs.
-
- *ResourceTrackers* allow you to remove code.
Several other important APIs are used explicitly. JIT clients need not be aware
@@ -291,33 +281,30 @@ of them, but Layer authors will use them:
ownership of the program representation will be passed back on the stack,
rather than having to be fished out of a Layer member, which would require
synchronization).
-
- *MaterializationResponsibility* - When a MaterializationUnit hands a program
representation back to the layer it comes with an associated
MaterializationResponsibility object. This object tracks the definitions
that must be materialized and provides a way to notify the JITDylib once they
are either successfully materialized or a failure occurs.
-Absolute Symbols, Aliases, and Reexports
-========================================
+## Absolute Symbols, Aliases, and Reexports
ORC makes it easy to define symbols with absolute addresses, or symbols that
are simply aliases of other symbols:
-Absolute Symbols
-----------------
+### Absolute Symbols
Absolute symbols are symbols that map directly to addresses without requiring
further materialization, for example: "foo" = 0x1234. One use case for
absolute symbols is allowing resolution of process symbols. E.g.
-.. code-block:: c++
-
- JD.define(absoluteSymbols(SymbolMap({
- { Mangle("printf"),
- { ExecutorAddr::fromPtr(&printf),
- JITSymbolFlags::Callable } }
- });
+```c++
+JD.define(absoluteSymbols(SymbolMap({
+ { Mangle("printf"),
+ { ExecutorAddr::fromPtr(&printf),
+ JITSymbolFlags::Callable } }
+ });
+```
With this mapping established code added to the JIT can refer to printf
symbolically rather than requiring the address of printf to be "baked in".
@@ -326,95 +313,92 @@ to be re-used across JIT sessions as the JIT'd code no longer changes, only the
absolute symbol definition does.
For process and library symbols the DynamicLibrarySearchGenerator utility (See
-:ref:`How to Add Process and Library Symbols to JITDylibs
-<ProcessAndLibrarySymbols>`) can be used to automatically build absolute
+{ref}`How to Add Process and Library Symbols to JITDylibs <ProcessAndLibrarySymbols>`) can be used to automatically build absolute
symbol mappings for you. However the absoluteSymbols function is still useful
for making non-global objects in your JIT visible to JIT'd code. For example,
imagine that your JIT standard library needs access to your JIT object to make
some calls. We could bake the address of your object into the library, but then
it would need to be recompiled for each session:
-.. code-block:: c++
-
- // From standard library for JIT'd code:
+```c++
+// From standard library for JIT'd code:
- class MyJIT {
- public:
- void log(const char *Msg);
- };
+class MyJIT {
+public:
+ void log(const char *Msg);
+};
- void log(const char *Msg) { ((MyJIT*)0x1234)->log(Msg); }
+void log(const char *Msg) { ((MyJIT*)0x1234)->log(Msg); }
+```
We can turn this into a symbolic reference in the JIT standard library:
-.. code-block:: c++
+```c++
+extern MyJIT *__MyJITInstance;
- extern MyJIT *__MyJITInstance;
-
- void log(const char *Msg) { __MyJITInstance->log(Msg); }
+void log(const char *Msg) { __MyJITInstance->log(Msg); }
+```
And then make our JIT object visible to the JIT standard library with an
absolute symbol definition when the JIT is started:
-.. code-block:: c++
-
- MyJIT J = ...;
+```c++
+MyJIT J = ...;
- auto &JITStdLibJD = ... ;
+auto &JITStdLibJD = ... ;
- JITStdLibJD.define(absoluteSymbols(SymbolMap({
- { Mangle("__MyJITInstance"),
- { ExecutorAddr::fromPtr(&J), JITSymbolFlags() } }
- });
+JITStdLibJD.define(absoluteSymbols(SymbolMap({
+ { Mangle("__MyJITInstance"),
+ { ExecutorAddr::fromPtr(&J), JITSymbolFlags() } }
+ });
+```
-Aliases and Reexports
----------------------
+### Aliases and Reexports
Aliases and reexports allow you to define new symbols that map to existing
symbols. This can be useful for changing linkage relationships between symbols
across sessions without having to recompile code. For example, imagine that
-JIT'd code has access to a log function, ``void log(const char*)`` for which
-there are two implementations in the JIT standard library: ``log_fast`` and
-``log_detailed``. Your JIT can choose which one of these definitions will be
-used when the ``log`` symbol is referenced by setting up an alias at JIT startup
+JIT'd code has access to a log function, `void log(const char*)` for which
+there are two implementations in the JIT standard library: `log_fast` and
+`log_detailed`. Your JIT can choose which one of these definitions will be
+used when the `log` symbol is referenced by setting up an alias at JIT startup
time:
-.. code-block:: c++
-
- auto &JITStdLibJD = ... ;
+```c++
+auto &JITStdLibJD = ... ;
- auto LogImplementationSymbol =
- Verbose ? Mangle("log_detailed") : Mangle("log_fast");
+auto LogImplementationSymbol =
+ Verbose ? Mangle("log_detailed") : Mangle("log_fast");
- JITStdLibJD.define(
- symbolAliases(SymbolAliasMap({
- { Mangle("log"),
- { LogImplementationSymbol
- JITSymbolFlags::Exported | JITSymbolFlags::Callable } }
- });
+JITStdLibJD.define(
+ symbolAliases(SymbolAliasMap({
+ { Mangle("log"),
+ { LogImplementationSymbol
+ JITSymbolFlags::Exported | JITSymbolFlags::Callable } }
+ });
+```
-The ``symbolAliases`` function allows you to define aliases within a single
-JITDylib. The ``reexports`` function provides the same functionality, but
+The `symbolAliases` function allows you to define aliases within a single
+JITDylib. The `reexports` function provides the same functionality, but
operates across JITDylib boundaries. E.g.
-.. code-block:: c++
-
- auto &JD1 = ... ;
- auto &JD2 = ... ;
+```c++
+auto &JD1 = ... ;
+auto &JD2 = ... ;
- // Make 'bar' in JD2 an alias for 'foo' from JD1.
- JD2.define(
- reexports(JD1, SymbolAliasMap({
- { Mangle("bar"), { Mangle("foo"), JITSymbolFlags::Exported } }
- });
+// Make 'bar' in JD2 an alias for 'foo' from JD1.
+JD2.define(
+ reexports(JD1, SymbolAliasMap({
+ { Mangle("bar"), { Mangle("foo"), JITSymbolFlags::Exported } }
+ });
+```
The reexports utility can be handy for composing a single JITDylib interface by
re-exporting symbols from several other JITDylibs.
-.. _Laziness:
+(Laziness)=
-Laziness
-========
+## Laziness
Laziness in ORC is provided by a utility called "lazy reexports". A lazy
reexport is similar to a regular reexport or alias: It provides a new name for
@@ -443,32 +427,30 @@ allow the address to be taken without forcing materialization of the reexport.
Usage example:
-If JITDylib ``JD`` contains definitions for symbols ``foo_body`` and
-``bar_body``, we can create lazy entry points ``Foo`` and ``Bar`` in JITDylib
-``JD2`` by calling:
+If JITDylib `JD` contains definitions for symbols `foo_body` and
+`bar_body`, we can create lazy entry points `Foo` and `Bar` in JITDylib
+`JD2` by calling:
-.. code-block:: c++
-
- auto ReexportFlags = JITSymbolFlags::Exported | JITSymbolFlags::Callable;
- JD2.define(
- lazyReexports(CallThroughMgr, StubsMgr, JD,
- SymbolAliasMap({
- { Mangle("foo"), { Mangle("foo_body"), ReexportedFlags } },
- { Mangle("bar"), { Mangle("bar_body"), ReexportedFlags } }
- }));
+```c++
+auto ReexportFlags = JITSymbolFlags::Exported | JITSymbolFlags::Callable;
+JD2.define(
+ lazyReexports(CallThroughMgr, StubsMgr, JD,
+ SymbolAliasMap({
+ { Mangle("foo"), { Mangle("foo_body"), ReexportedFlags } },
+ { Mangle("bar"), { Mangle("bar_body"), ReexportedFlags } }
+ }));
+```
A full example of how to use lazyReexports with the LLJIT class can be found at
-``llvm/examples/OrcV2Examples/LLJITWithLazyReexports``.
+`llvm/examples/OrcV2Examples/LLJITWithLazyReexports`.
-Supporting Custom Compilers
-===========================
+## Supporting Custom Compilers
TBD.
-.. _transitioning_orcv1_to_orcv2:
+(transitioning_orcv1_to_orcv2)=
-Transitioning from ORCv1 to ORCv2
-=================================
+## Transitioning from ORCv1 to ORCv2
Since LLVM 7.0, new ORC development work has focused on adding support for
concurrent JIT compilation. The new APIs (including new layer interfaces and
@@ -481,226 +463,220 @@ prefix in LLVM 8.0, and have deprecation warnings attached in LLVM 9.0. In LLVM
12.0 ORCv1 will be removed entirely.
Transitioning from ORCv1 to ORCv2 should be easy for most clients. Most of the
-ORCv1 layers and utilities have ORCv2 counterparts [2]_ that can be directly
+ORCv1 layers and utilities have ORCv2 counterparts [^2] that can be directly
substituted. However there are some design
diff erences between ORCv1 and ORCv2
to be aware of:
- 1. ORCv2 fully adopts the JIT-as-linker model that began with MCJIT. Modules
- (and other program representations, e.g. Object Files) are no longer added
- directly to JIT classes or layers. Instead, they are added to ``JITDylib``
- instances *by* layers. The ``JITDylib`` determines *where* the definitions
- reside, the layers determine *how* the definitions will be compiled.
- Linkage relationships between ``JITDylibs`` determine how inter-module
- references are resolved, and symbol resolvers are no longer used. See the
- section `Design Overview`_ for more details.
-
- Unless multiple JITDylibs are needed to model linkage relationships, ORCv1
- clients should place all code in a single JITDylib.
- MCJIT clients should use LLJIT (see `LLJIT and LLLazyJIT`_), and can place
- code in LLJIT's default created main JITDylib (See
- ``LLJIT::getMainJITDylib()``).
-
- 2. All JIT stacks now need an ``ExecutionSession`` instance. ExecutionSession
- manages the string pool, error reporting, synchronization, and symbol
- lookup.
-
- 3. ORCv2 uses uniqued strings (``SymbolStringPtr`` instances) rather than
- string values in order to reduce memory overhead and improve lookup
- performance. See the subsection `How to manage symbol strings`_.
-
- 4. IR layers require ThreadSafeModule instances, rather than
- std::unique_ptr<Module>s. ThreadSafeModule is a wrapper that ensures that
- Modules that use the same LLVMContext are not accessed concurrently.
- See `How to use ThreadSafeModule and ThreadSafeContext`_.
-
- 5. Symbol lookup is no longer handled by layers. Instead, there is a
- ``lookup`` method on JITDylib that takes a list of JITDylibs to scan.
-
- .. code-block:: c++
-
- ExecutionSession ES;
- JITDylib &JD1 = ...;
- JITDylib &JD2 = ...;
-
- auto Sym = ES.lookup({&JD1, &JD2}, ES.intern("_main"));
-
- 6. The removeModule/removeObject methods are replaced by
- ``ResourceTracker::remove``.
- See the subsection `How to remove code`_.
+1. ORCv2 fully adopts the JIT-as-linker model that began with MCJIT. Modules
+ (and other program representations, e.g. Object Files) are no longer added
+ directly to JIT classes or layers. Instead, they are added to `JITDylib`
+ instances *by* layers. The `JITDylib` determines *where* the definitions
+ reside, the layers determine *how* the definitions will be compiled.
+ Linkage relationships between `JITDylibs` determine how inter-module
+ references are resolved, and symbol resolvers are no longer used. See the
+ section [Design Overview](#design-overview) for more details.
+
+ Unless multiple JITDylibs are needed to model linkage relationships, ORCv1
+ clients should place all code in a single JITDylib.
+ MCJIT clients should use LLJIT (see [LLJIT and LLLazyJIT](#lljit-and-lllazyjit)), and can place
+ code in LLJIT's default created main JITDylib (See
+ `LLJIT::getMainJITDylib()`).
+
+2. All JIT stacks now need an `ExecutionSession` instance. ExecutionSession
+ manages the string pool, error reporting, synchronization, and symbol
+ lookup.
+
+3. ORCv2 uses uniqued strings (`SymbolStringPtr` instances) rather than
+ string values in order to reduce memory overhead and improve lookup
+ performance. See the subsection [How to manage symbol strings](#how-to-manage-symbol-strings).
+
+4. IR layers require ThreadSafeModule instances, rather than
+ `std::unique_ptr<Module>`s. ThreadSafeModule is a wrapper that ensures that
+ Modules that use the same LLVMContext are not accessed concurrently.
+ See [How to use ThreadSafeModule and ThreadSafeContext](#how-to-use-threadsafemodule-and-threadsafecontext).
+
+5. Symbol lookup is no longer handled by layers. Instead, there is a
+ `lookup` method on JITDylib that takes a list of JITDylibs to scan.
+
+ ```c++
+ ExecutionSession ES;
+ JITDylib &JD1 = ...;
+ JITDylib &JD2 = ...;
+
+ auto Sym = ES.lookup({&JD1, &JD2}, ES.intern("_main"));
+ ```
+
+6. The removeModule/removeObject methods are replaced by
+ `ResourceTracker::remove`.
+ See the subsection [How to remove code](#how-to-remove-code).
For code examples and suggestions of how to use the ORCv2 APIs, please see
-the section `How-tos`_.
+the section [How-tos](#how-tos).
-How-tos
-=======
+## How-tos
-How to manage symbol strings
-----------------------------
+### How to manage symbol strings
Symbol strings in ORC are uniqued to improve lookup performance, reduce memory
overhead, and allow symbol names to function as efficient keys. To get the
-unique ``SymbolStringPtr`` for a string value, call the
-``ExecutionSession::intern`` method:
+unique `SymbolStringPtr` for a string value, call the
+`ExecutionSession::intern` method:
- .. code-block:: c++
-
- ExecutionSession ES;
- /// ...
- auto MainSymbolName = ES.intern("main");
+```c++
+ExecutionSession ES;
+/// ...
+auto MainSymbolName = ES.intern("main");
+```
If you wish to perform lookup using the C/IR name of a symbol you will also
need to apply the platform linker-mangling before interning the string. On
Linux this mangling is a no-op, but on other platforms it usually involves
-adding a prefix to the string (e.g. '_' on Darwin). The mangling scheme is
+adding a prefix to the string (e.g. `_` on Darwin). The mangling scheme is
based on the DataLayout for the target. Given a DataLayout and an
ExecutionSession, you can create a MangleAndInterner function object that
will perform both jobs for you:
- .. code-block:: c++
-
- ExecutionSession ES;
- const DataLayout &DL = ...;
- MangleAndInterner Mangle(ES, DL);
+```c++
+ExecutionSession ES;
+const DataLayout &DL = ...;
+MangleAndInterner Mangle(ES, DL);
- // ...
+// ...
- // Portable IR-symbol-name lookup:
- auto Sym = ES.lookup({&MainJD}, Mangle("main"));
+// Portable IR-symbol-name lookup:
+auto Sym = ES.lookup({&MainJD}, Mangle("main"));
+```
-How to create JITDylibs and set up linkage relationships
---------------------------------------------------------
+### How to create JITDylibs and set up linkage relationships
In ORC, all symbol definitions reside in JITDylibs. JITDylibs are created by
-calling the ``ExecutionSession::createJITDylib`` method with a unique name:
-
- .. code-block:: c++
+calling the `ExecutionSession::createJITDylib` method with a unique name:
- ExecutionSession ES;
- auto &JD = ES.createJITDylib("libFoo.dylib");
+```c++
+ExecutionSession ES;
+auto &JD = ES.createJITDylib("libFoo.dylib");
+```
-The JITDylib is owned by the ``ExecutionEngine`` instance and will be freed
+The JITDylib is owned by the `ExecutionEngine` instance and will be freed
when it is destroyed.
-How to remove code
-------------------
+### How to remove code
To remove an individual module from a JITDylib it must first be added using an
-explicit ``ResourceTracker``. The module can then be removed by calling
-``ResourceTracker::remove``:
+explicit `ResourceTracker`. The module can then be removed by calling
+`ResourceTracker::remove`:
- .. code-block:: c++
+```c++
+auto &JD = ... ;
+auto M = ... ;
- auto &JD = ... ;
- auto M = ... ;
+auto RT = JD.createResourceTracker();
+Layer.add(RT, std::move(M)); // Add M to JD, tracking resources with RT
- auto RT = JD.createResourceTracker();
- Layer.add(RT, std::move(M)); // Add M to JD, tracking resources with RT
-
- RT.remove(); // Remove M from JD.
+RT.remove(); // Remove M from JD.
+```
Modules added directly to a JITDylib will be tracked by that JITDylib's default
resource tracker.
-All code can be removed from a JITDylib by calling ``JITDylib::clear``. This
+All code can be removed from a JITDylib by calling `JITDylib::clear`. This
leaves the cleared JITDylib in an empty but usable state.
-JITDylibs can be removed by calling ``ExecutionSession::removeJITDylib``. This
+JITDylibs can be removed by calling `ExecutionSession::removeJITDylib`. This
clears the JITDylib and then puts it into a defunct state. No further operations
can be performed on the JITDylib, and it will be destroyed as soon as the last
handle to it is released.
An example of how to use the resource management APIs can be found at
-``llvm/examples/OrcV2Examples/LLJITRemovableCode``.
+`llvm/examples/OrcV2Examples/LLJITRemovableCode`.
+### How to add the support for custom program representation
-How to add the support for custom program representation
---------------------------------------------------------
-In order to add the support for a custom program representation, a custom ``MaterializationUnit``
-for the program representation, and a custom ``Layer`` are needed. The Layer will have two
-operations: ``add`` and ``emit``. The ``add`` operation takes an instance of your program
-representation, builds one of your custom ``MaterializationUnits`` to hold it, then adds it
-to a ``JITDylib``. The emit operation takes a ``MaterializationResponsibility`` object and an
+In order to add the support for a custom program representation, a custom `MaterializationUnit`
+for the program representation, and a custom `Layer` are needed. The Layer will have two
+operations: `add` and `emit`. The `add` operation takes an instance of your program
+representation, builds one of your custom `MaterializationUnits` to hold it, then adds it
+to a `JITDylib`. The emit operation takes a `MaterializationResponsibility` object and an
instance of your program representation and materializes it, usually by compiling it and handing
-the resulting object off to an ``ObjectLinkingLayer``.
+the resulting object off to an `ObjectLinkingLayer`.
-Your custom ``MaterializationUnit`` will have two operations: ``materialize`` and ``discard``. The
-``materialize`` function will be called for you when any symbol provided by the unit is looked up,
-and it should just call the ``emit`` function on your layer, passing in the given
-``MaterializationResponsibility`` and the wrapped program representation. The ``discard`` function
+Your custom `MaterializationUnit` will have two operations: `materialize` and `discard`. The
+`materialize` function will be called for you when any symbol provided by the unit is looked up,
+and it should just call the `emit` function on your layer, passing in the given
+`MaterializationResponsibility` and the wrapped program representation. The `discard` function
will be called if some weak symbol provided by your unit is not needed (because the JIT found an
overriding definition). You can use this to drop your definition early, or just ignore it and let
the linker drops the definition later.
Here is an example of an ASTLayer:
- .. code-block:: c++
-
- // ... In you JIT class
- AstLayer astLayer;
- // ...
+```c++
+// ... In you JIT class
+AstLayer astLayer;
+// ...
- class AstMaterializationUnit : public orc::MaterializationUnit {
- public:
- AstMaterializationUnit(AstLayer &l, Ast &ast)
- : llvm::orc::MaterializationUnit(l.getInterface(ast)), astLayer(l),
- ast(ast) {};
+class AstMaterializationUnit : public orc::MaterializationUnit {
+public:
+ AstMaterializationUnit(AstLayer &l, Ast &ast)
+ : llvm::orc::MaterializationUnit(l.getInterface(ast)), astLayer(l),
+ ast(ast) {};
- llvm::StringRef getName() const override {
- return "AstMaterializationUnit";
- }
+ llvm::StringRef getName() const override {
+ return "AstMaterializationUnit";
+ }
- void materialize(std::unique_ptr<orc::MaterializationResponsibility> r) override {
- astLayer.emit(std::move(r), ast);
- };
+ void materialize(std::unique_ptr<orc::MaterializationResponsibility> r) override {
+ astLayer.emit(std::move(r), ast);
+ };
- private:
- void discard(const llvm::orc::JITDylib &jd, const llvm::orc::SymbolStringPtr &sym) override {
- llvm_unreachable("functions are not overridable");
- }
+private:
+ void discard(const llvm::orc::JITDylib &jd, const llvm::orc::SymbolStringPtr &sym) override {
+ llvm_unreachable("functions are not overridable");
+ }
- AstLayer &astLayer;
- Ast *
- };
+ AstLayer &astLayer;
+ Ast *
+};
- class AstLayer {
- llvhm::orc::IRLayer &baseLayer;
- llvhm::orc::MangleAndInterner &mangler;
+class AstLayer {
+ llvhm::orc::IRLayer &baseLayer;
+ llvhm::orc::MangleAndInterner &mangler;
- public:
- AstLayer(llvm::orc::IRLayer &baseLayer, llvm::orc::MangleAndInterner &mangler)
- : baseLayer(baseLayer), mangler(mangler){};
+public:
+ AstLayer(llvm::orc::IRLayer &baseLayer, llvm::orc::MangleAndInterner &mangler)
+ : baseLayer(baseLayer), mangler(mangler){};
- llvm::Error add(llvm::orc::ResourceTrackerSP &rt, Ast &ast) {
- return rt->getJITDylib().define(std::make_unique<AstMaterializationUnit>(*this, ast), rt);
- }
+ llvm::Error add(llvm::orc::ResourceTrackerSP &rt, Ast &ast) {
+ return rt->getJITDylib().define(std::make_unique<AstMaterializationUnit>(*this, ast), rt);
+ }
- void emit(std::unique_ptr<orc::MaterializationResponsibility> mr, Ast &ast) {
- // compileAst is just function that compiles the given AST and returns
- // a `llvm::orc::ThreadSafeModule`
- baseLayer.emit(std::move(mr), compileAst(ast));
- }
+ void emit(std::unique_ptr<orc::MaterializationResponsibility> mr, Ast &ast) {
+ // compileAst is just function that compiles the given AST and returns
+ // a `llvm::orc::ThreadSafeModule`
+ baseLayer.emit(std::move(mr), compileAst(ast));
+ }
- llvm::orc::MaterializationUnit::Interface getInterface(Ast &ast) {
- SymbolFlagsMap Symbols;
- // Find all the symbols in the AST and for each of them
- // add it to the Symbols map.
- Symbols[mangler(someNameFromAST)] =
- JITSymbolFlags(JITSymbolFlags::Exported | JITSymbolFlags::Callable);
- return MaterializationUnit::Interface(std::move(Symbols), nullptr);
- }
- };
+ llvm::orc::MaterializationUnit::Interface getInterface(Ast &ast) {
+ SymbolFlagsMap Symbols;
+ // Find all the symbols in the AST and for each of them
+ // add it to the Symbols map.
+ Symbols[mangler(someNameFromAST)] =
+ JITSymbolFlags(JITSymbolFlags::Exported | JITSymbolFlags::Callable);
+ return MaterializationUnit::Interface(std::move(Symbols), nullptr);
+ }
+};
+```
-Take look at the source code of `Building A JIT's Chapter 4 <tutorial/BuildingAJIT4.html>`_ for a complete example.
+Take look at the source code of {doc}`Building A JIT's Chapter 4 <tutorial/BuildingAJIT4>` for a complete example.
-How to use ThreadSafeModule and ThreadSafeContext
--------------------------------------------------
+### How to use ThreadSafeModule and ThreadSafeContext
ThreadSafeModule and ThreadSafeContext are wrappers around Modules and
LLVMContexts respectively. A ThreadSafeModule is a pair of a
-std::unique_ptr<Module> and a (possibly shared) ThreadSafeContext value. A
-ThreadSafeContext is a pair of a std::unique_ptr<LLVMContext> and a lock.
+`std::unique_ptr<Module>` and a (possibly shared) ThreadSafeContext value. A
+ThreadSafeContext is a pair of a `std::unique_ptr<LLVMContext>` and a lock.
This design serves two purposes: providing a locking scheme and lifetime
management for LLVMContexts. The ThreadSafeContext may be locked to prevent
accidental concurrent access by two Modules that use the same LLVMContext.
@@ -709,97 +685,96 @@ to it are destroyed, allowing the context memory to be reclaimed as soon as
the Modules referring to it are destroyed.
ThreadSafeContexts can be explicitly constructed from a
-std::unique_ptr<LLVMContext>:
-
- .. code-block:: c++
+`std::unique_ptr<LLVMContext>`:
- ThreadSafeContext TSCtx(std::make_unique<LLVMContext>());
+```c++
+ThreadSafeContext TSCtx(std::make_unique<LLVMContext>());
+```
-ThreadSafeModules can be constructed from a pair of a std::unique_ptr<Module>
+ThreadSafeModules can be constructed from a pair of a `std::unique_ptr<Module>`
and a ThreadSafeContext value. ThreadSafeContext values may be shared between
multiple ThreadSafeModules:
- .. code-block:: c++
+```c++
+ThreadSafeModule TSM1(
+ std::make_unique<Module>("M1", *TSCtx.getContext()), TSCtx);
- ThreadSafeModule TSM1(
- std::make_unique<Module>("M1", *TSCtx.getContext()), TSCtx);
-
- ThreadSafeModule TSM2(
- std::make_unique<Module>("M2", *TSCtx.getContext()), TSCtx);
+ThreadSafeModule TSM2(
+ std::make_unique<Module>("M2", *TSCtx.getContext()), TSCtx);
+```
Before using a ThreadSafeContext, clients should ensure that either the context
is only accessible on the current thread, or that the context is locked. In the
example above (where the context is never locked) we rely on the fact that both
-``TSM1`` and ``TSM2``, and TSCtx are all created on one thread. If a context is
+`TSM1` and `TSM2`, and TSCtx are all created on one thread. If a context is
going to be shared between threads then it must be locked before any accessing
or creating any Modules attached to it. E.g.
- .. code-block:: c++
-
- ThreadSafeContext TSCtx(std::make_unique<LLVMContext>());
+```c++
+ThreadSafeContext TSCtx(std::make_unique<LLVMContext>());
- DefaultThreadPool TP(NumThreads);
- JITStack J;
+DefaultThreadPool TP(NumThreads);
+JITStack J;
- for (auto &ModulePath : ModulePaths) {
- TP.async(
- [&]() {
- auto Lock = TSCtx.getLock();
- auto M = loadModuleOnContext(ModulePath, TSCtx.getContext());
- J.addModule(ThreadSafeModule(std::move(M), TSCtx));
- });
- }
+for (auto &ModulePath : ModulePaths) {
+ TP.async(
+ [&]() {
+ auto Lock = TSCtx.getLock();
+ auto M = loadModuleOnContext(ModulePath, TSCtx.getContext());
+ J.addModule(ThreadSafeModule(std::move(M), TSCtx));
+ });
+}
- TP.wait();
+TP.wait();
+```
To make exclusive access to Modules easier to manage the ThreadSafeModule class
-provides a convenience function, ``withModuleDo``, that implicitly (1) locks the
+provides a convenience function, `withModuleDo`, that implicitly (1) locks the
associated context, (2) runs a given function object, (3) unlocks the context,
and (3) returns the result generated by the function object. E.g.
- .. code-block:: c++
-
- ThreadSafeModule TSM = getModule(...);
+```c++
+ThreadSafeModule TSM = getModule(...);
- // Dump the module:
- size_t NumFunctionsInModule =
- TSM.withModuleDo(
- [](Module &M) { // <- Context locked before entering lambda.
- return M.size();
- } // <- Context unlocked after leaving.
- );
+// Dump the module:
+size_t NumFunctionsInModule =
+ TSM.withModuleDo(
+ [](Module &M) { // <- Context locked before entering lambda.
+ return M.size();
+ } // <- Context unlocked after leaving.
+ );
+```
Clients wishing to maximize possibilities for concurrent compilation will want
to create every new ThreadSafeModule on a new ThreadSafeContext. For this
reason a convenience constructor for ThreadSafeModule is provided that implicitly
-constructs a new ThreadSafeContext value from a std::unique_ptr<LLVMContext>:
-
- .. code-block:: c++
-
- // Maximize concurrency opportunities by loading every module on a
- // separate context.
- for (const auto &IRPath : IRPaths) {
- auto Ctx = std::make_unique<LLVMContext>();
- auto M = std::make_unique<Module>("M", *Ctx);
- CompileLayer.add(MainJD, ThreadSafeModule(std::move(M), std::move(Ctx)));
- }
+constructs a new ThreadSafeContext value from a `std::unique_ptr<LLVMContext>`:
+
+```c++
+// Maximize concurrency opportunities by loading every module on a
+// separate context.
+for (const auto &IRPath : IRPaths) {
+ auto Ctx = std::make_unique<LLVMContext>();
+ auto M = std::make_unique<Module>("M", *Ctx);
+ CompileLayer.add(MainJD, ThreadSafeModule(std::move(M), std::move(Ctx)));
+}
+```
Clients who plan to run single-threaded may choose to save memory by loading
all modules on the same context:
- .. code-block:: c++
-
- // Save memory by using one context for all Modules:
- ThreadSafeContext TSCtx(std::make_unique<LLVMContext>());
- for (const auto &IRPath : IRPaths) {
- ThreadSafeModule TSM(parsePath(IRPath, *TSCtx.getContext()), TSCtx);
- CompileLayer.add(MainJD, ThreadSafeModule(std::move(TSM));
- }
+```c++
+// Save memory by using one context for all Modules:
+ThreadSafeContext TSCtx(std::make_unique<LLVMContext>());
+for (const auto &IRPath : IRPaths) {
+ ThreadSafeModule TSM(parsePath(IRPath, *TSCtx.getContext()), TSCtx);
+ CompileLayer.add(MainJD, ThreadSafeModule(std::move(TSM));
+}
+```
-.. _ProcessAndLibrarySymbols:
+(ProcessAndLibrarySymbols)=
-How to Add Process and Library Symbols to JITDylibs
-===================================================
+## How to Add Process and Library Symbols to JITDylibs
JIT'd code may need to access symbols in the host program or in supporting
libraries. The best way to enable this is to reflect these symbols into your
@@ -810,18 +785,18 @@ so visible to the JIT linker during linking).
One way to reflect external symbols is to add them manually using the
absoluteSymbols function:
- .. code-block:: c++
-
- const DataLayout &DL = getDataLayout();
- MangleAndInterner Mangle(ES, DL);
+```c++
+const DataLayout &DL = getDataLayout();
+MangleAndInterner Mangle(ES, DL);
- auto &JD = ES.createJITDylib("main");
+auto &JD = ES.createJITDylib("main");
- JD.define(
- absoluteSymbols({
- { Mangle("puts"), ExecutorAddr::fromPtr(&puts)},
- { Mangle("gets"), ExecutorAddr::fromPtr(&getS)}
- }));
+JD.define(
+ absoluteSymbols({
+ { Mangle("puts"), ExecutorAddr::fromPtr(&puts)},
+ { Mangle("gets"), ExecutorAddr::fromPtr(&getS)}
+ }));
+```
Using absoluteSymbols is reasonable if the set of symbols to be reflected is
small and fixed. On the other hand, if the set of symbols is large or variable
@@ -831,51 +806,51 @@ to a JITDylib, receiving a callback whenever a lookup within that JITDylib fails
to find one or more symbols. The definition generator is given a chance to
produce a definition of the missing symbol(s) before the lookup proceeds.
-ORC provides the ``DynamicLibrarySearchGenerator`` utility for reflecting symbols
+ORC provides the `DynamicLibrarySearchGenerator` utility for reflecting symbols
from the process (or a specific dynamic library) for you. For example, to reflect
the whole interface of a runtime library:
- .. code-block:: c++
+```c++
+const DataLayout &DL = getDataLayout();
+auto &JD = ES.createJITDylib("main");
- const DataLayout &DL = getDataLayout();
- auto &JD = ES.createJITDylib("main");
+if (auto DLSGOrErr =
+ DynamicLibrarySearchGenerator::Load("/path/to/lib"
+ DL.getGlobalPrefix()))
+ JD.addGenerator(std::move(*DLSGOrErr);
+else
+ return DLSGOrErr.takeError();
- if (auto DLSGOrErr =
- DynamicLibrarySearchGenerator::Load("/path/to/lib"
- DL.getGlobalPrefix()))
- JD.addGenerator(std::move(*DLSGOrErr);
- else
- return DLSGOrErr.takeError();
+// IR added to JD can now link against all symbols exported by the library
+// at '/path/to/lib'.
+CompileLayer.add(JD, loadModule(...));
+```
- // IR added to JD can now link against all symbols exported by the library
- // at '/path/to/lib'.
- CompileLayer.add(JD, loadModule(...));
-
-The ``DynamicLibrarySearchGenerator`` utility can also be constructed with a
+The `DynamicLibrarySearchGenerator` utility can also be constructed with a
filter function to restrict the set of symbols that may be reflected. For
example, to expose an allowed set of symbols from the main process:
- .. code-block:: c++
-
- const DataLayout &DL = getDataLayout();
- MangleAndInterner Mangle(ES, DL);
+```c++
+const DataLayout &DL = getDataLayout();
+MangleAndInterner Mangle(ES, DL);
- auto &JD = ES.createJITDylib("main");
+auto &JD = ES.createJITDylib("main");
- DenseSet<SymbolStringPtr> AllowList({
- Mangle("puts"),
- Mangle("gets")
- });
+DenseSet<SymbolStringPtr> AllowList({
+ Mangle("puts"),
+ Mangle("gets")
+ });
- // Use GetForCurrentProcess with a predicate function that checks the
- // allowed list.
- JD.addGenerator(cantFail(DynamicLibrarySearchGenerator::GetForCurrentProcess(
- DL.getGlobalPrefix(),
- [&](const SymbolStringPtr &S) { return AllowList.count(S); })));
+// Use GetForCurrentProcess with a predicate function that checks the
+// allowed list.
+JD.addGenerator(cantFail(DynamicLibrarySearchGenerator::GetForCurrentProcess(
+ DL.getGlobalPrefix(),
+ [&](const SymbolStringPtr &S) { return AllowList.count(S); })));
- // IR added to JD can now link against any symbols exported by the process
- // and contained in the list.
- CompileLayer.add(JD, loadModule(...));
+// IR added to JD can now link against any symbols exported by the process
+// and contained in the list.
+CompileLayer.add(JD, loadModule(...));
+```
References to process or library symbols could also be hardcoded into your IR
or object files using the symbols' raw addresses, however symbolic resolution
@@ -883,29 +858,26 @@ using the JIT symbol tables should be preferred: it keeps the IR and objects
readable and reusable in subsequent JIT sessions. Hardcoded addresses are
diff icult to read, and usually only good for one session.
-Roadmap
-=======
+## Roadmap
ORC is still undergoing active development. Some current and future works are
listed below.
-Current Work
-------------
+### Current Work
1. **TargetProcessControl: Improvements to in-tree support for out-of-process
execution**
- The ``TargetProcessControl`` API provides various operations on the JIT
+ The `TargetProcessControl` API provides various operations on the JIT
target process (the one which will execute the JIT'd code), including
memory allocation, memory writes, function execution, and process queries
(e.g. for the target triple). By targeting this API new components can be
developed which will work equally well for in-process and out-of-process
JITing.
-
2. **ORC RPC based TargetProcessControl implementation**
- An ORC RPC based implementation of the ``TargetProcessControl`` API is
+ An ORC RPC based implementation of the `TargetProcessControl` API is
currently under development to enable easy out-of-process JITing via
file descriptors / sockets.
@@ -918,8 +890,7 @@ Current Work
ExecutionSession and leaving the JITDylib instance in a defunct state until
all references to it have been released).
-Near Future Work
-----------------
+### Near Future Work
1. **ORC JIT Runtime Libraries**
@@ -941,8 +912,7 @@ Near Future Work
as the underlying JIT linker. We will need a new solution for JITLink based
platforms.
-Further Future Work
--------------------
+### Further Future Work
1. **Speculative Compilation**
@@ -951,24 +921,23 @@ Further Future Work
but which we have reason to believe will be needed in the future. This can be
used to hide compile latency and improve JIT throughput. A proof-of-concept
example of speculative compilation with ORC has already been developed (see
- ``llvm/examples/SpeculativeJIT``). Future work on this is likely to focus on
+ `llvm/examples/SpeculativeJIT`). Future work on this is likely to focus on
re-using and improving existing profiling support (currently used by PGO) to
feed speculation decisions, as well as built-in tools to simplify use of
speculative compilation.
-.. [1] Formats/architectures vary in terms of supported features. MachO and
- ELF tend to have better support than COFF. Patches very welcome!
-
-.. [2] The ``LazyEmittingLayer``, ``RemoteObjectClientLayer`` and
- ``RemoteObjectServerLayer`` do not have counterparts in the new
- system. In the case of ``LazyEmittingLayer`` it was simply no longer
- needed: in ORCv2, deferring compilation until symbols are looked up is
- the default. The removal of ``RemoteObjectClientLayer`` and
- ``RemoteObjectServerLayer`` means that JIT stacks can no longer be split
- across processes, however this functionality appears not to have been
- used.
-
-.. [3] Weak definitions are currently handled correctly within dylibs, but if
- multiple dylibs provide a weak definition of a symbol then each will end
- up with its own definition (similar to how weak definitions are handled
- in Windows DLLs). This will be fixed in the future.
+[^1]: Formats/architectures vary in terms of supported features. MachO and
+ ELF tend to have better support than COFF. Patches very welcome!
+
+[^2]: The `LazyEmittingLayer`, `RemoteObjectClientLayer` and
+ `RemoteObjectServerLayer` do not have counterparts in the new system. In the
+ case of `LazyEmittingLayer` it was simply no longer needed: in ORCv2,
+ deferring compilation until symbols are looked up is the default. The
+ removal of `RemoteObjectClientLayer` and `RemoteObjectServerLayer` means
+ that JIT stacks can no longer be split across processes, however this
+ functionality appears not to have been used.
+
+[^3]: Weak definitions are currently handled correctly within dylibs, but if
+ multiple dylibs provide a weak definition of a symbol then each will end up
+ with its own definition (similar to how weak definitions are handled in
+ Windows DLLs). This will be fixed in the future.
diff --git a/llvm/docs/Remarks.md b/llvm/docs/Remarks.md
index a22e8c30dfa30..148776dbf5c7f 100644
--- a/llvm/docs/Remarks.md
+++ b/llvm/docs/Remarks.md
@@ -1,12 +1,10 @@
-=======
-Remarks
-=======
+# Remarks
-.. contents::
- :local:
+```{contents}
+:local: true
+```
-Introduction to the LLVM remark diagnostics
-===========================================
+## Introduction to the LLVM remark diagnostics
LLVM is able to emit diagnostics from passes describing whether an optimization
has been performed or missed for a particular reason, which should give more
@@ -14,78 +12,72 @@ insight to users about what the compiler did during the compilation pipeline.
There are three main remark types:
-``Passed``
+`Passed`
+: Remarks that describe a successful optimization performed by the compiler.
- Remarks that describe a successful optimization performed by the compiler.
+ **Example:**
- :Example:
+ ```
+ foo inlined into bar with (cost=always): always inline attribute
+ ```
- ::
+`Missed`
+: Remarks that describe an attempt to an optimization by the compiler that
+ could not be performed.
- foo inlined into bar with (cost=always): always inline attribute
+ **Example:**
-``Missed``
+ ```
+ foo not inlined into bar because it should never be inlined
+ (cost=never): noinline function attribute
+ ```
- Remarks that describe an attempt to an optimization by the compiler that
- could not be performed.
+`Analysis`
+: Remarks that describe the result of an analysis, that can bring more
+ information to the user regarding the generated code.
- :Example:
+ **Example:**
- ::
+ ```
+ 16 stack bytes in function
+ ```
- foo not inlined into bar because it should never be inlined
- (cost=never): noinline function attribute
+ ```
+ 10 instructions in function
+ ```
-``Analysis``
-
- Remarks that describe the result of an analysis, that can bring more
- information to the user regarding the generated code.
-
- :Example:
-
- ::
-
- 16 stack bytes in function
-
- ::
-
- 10 instructions in function
-
-Enabling optimization remarks
-=============================
+## Enabling optimization remarks
There are two modes that are supported for enabling optimization remarks in
LLVM: through remark diagnostics, or through serialized remarks.
See also the clang flags
-`-Rpass <https://clang.llvm.org/docs/UsersManual.html#options-to-emit-optimization-reports>`_
+[-Rpass](https://clang.llvm.org/docs/UsersManual.html#options-to-emit-optimization-reports)
and
-`-fsave-optimization-record <http://clang.llvm.org/docs/UsersManual.html#cmdoption-f-no-save-optimization-record>`_.
+[-fsave-optimization-record](http://clang.llvm.org/docs/UsersManual.html#cmdoption-f-no-save-optimization-record).
-Remark diagnostics
-------------------
+### Remark diagnostics
Optimization remarks can be emitted as diagnostics. These diagnostics will be
-propagated to front-ends if desired, or emitted by tools like :doc:`llc
-<CommandGuide/llc>` or :doc:`opt <CommandGuide/opt>`.
-
-.. option:: -pass-remarks=<regex>
-
- Enables optimization remarks from passes whose name match the given (POSIX)
- regular expression.
+propagated to front-ends if desired, or emitted by tools like {doc}`llc
+<CommandGuide/llc>` or {doc}`opt <CommandGuide/opt>`.
-.. option:: -pass-remarks-missed=<regex>
+```{option} -pass-remarks=<regex>
+Enables optimization remarks from passes whose name match the given (POSIX)
+regular expression.
+```
- Enables missed optimization remarks from passes whose name match the given
- (POSIX) regular expression.
+```{option} -pass-remarks-missed=<regex>
+Enables missed optimization remarks from passes whose name match the given
+(POSIX) regular expression.
+```
-.. option:: -pass-remarks-analysis=<regex>
+```{option} -pass-remarks-analysis=<regex>
+Enables optimization analysis remarks from passes whose name match the given
+(POSIX) regular expression.
+```
- Enables optimization analysis remarks from passes whose name match the given
- (POSIX) regular expression.
-
-Serialized remarks
-------------------
+### Serialized remarks
While diagnostics are useful during development, it is often more useful to
refer to optimization remarks post-compilation, typically during performance
@@ -94,125 +86,138 @@ analysis.
For that, LLVM can serialize the remarks produced for each compilation unit to
a file that can be consumed later.
-By default, the format of the serialized remarks is :ref:`YAML
-<yamlremarks>`, and it can be accompanied by a :ref:`section <remarkssection>`
+By default, the format of the serialized remarks is {ref}`YAML
+<yamlremarks>`, and it can be accompanied by a {ref}`section <remarkssection>`
in the object files to easily retrieve it.
-:doc:`llc <CommandGuide/llc>` and :doc:`opt <CommandGuide/opt>` support the
+{doc}`llc <CommandGuide/llc>` and {doc}`opt <CommandGuide/opt>` support the
following options:
+`Basic options`
-``Basic options``
+```{option} -pass-remarks-output=<filename>
+Enables the serialization of remarks to a file specified in `<filename>`.
- .. option:: -pass-remarks-output=<filename>
+By default, the output is serialized to {ref}`YAML <yamlremarks>`.
+```
- Enables the serialization of remarks to a file specified in <filename>.
+```{option} -pass-remarks-format=<format>
+Specifies the output format of the serialized remarks.
- By default, the output is serialized to :ref:`YAML <yamlremarks>`.
+Supported formats:
- .. option:: -pass-remarks-format=<format>
+- {ref}`yaml <yamlremarks>` (default)
+- {ref}`bitstream <bitstreamremarks>`
+```
- Specifies the output format of the serialized remarks.
+`Content configuration`
- Supported formats:
+```{option} -pass-remarks-filter=<regex>
+Only passes whose name match the given (POSIX) regular expression will be
+serialized to the final output.
+```
- * :ref:`yaml <yamlremarks>` (default)
- * :ref:`bitstream <bitstreamremarks>`
+```{option} -pass-remarks-with-hotness
+With PGO, include profile count in optimization remarks.
+```
-``Content configuration``
+```{option} -pass-remarks-hotness-threshold
+The minimum profile count required for an optimization remark to be
+emitted.
+```
- .. option:: -pass-remarks-filter=<regex>
+Other tools that support remarks:
- Only passes whose name match the given (POSIX) regular expression will be
- serialized to the final output.
+{program}`llvm-lto`
- .. option:: -pass-remarks-with-hotness
+```{option} -lto-pass-remarks-output=<filename>
+```
- With PGO, include profile count in optimization remarks.
+```{option} -lto-pass-remarks-filter=<regex>
+```
- .. option:: -pass-remarks-hotness-threshold
+```{option} -lto-pass-remarks-format=<format>
+```
- The minimum profile count required for an optimization remark to be
- emitted.
+```{option} -lto-pass-remarks-with-hotness
+```
-Other tools that support remarks:
+```{option} -lto-pass-remarks-hotness-threshold
+```
-:program:`llvm-lto`
+{program}`gold-plugin` and {program}`lld`
- .. option:: -lto-pass-remarks-output=<filename>
- .. option:: -lto-pass-remarks-filter=<regex>
- .. option:: -lto-pass-remarks-format=<format>
- .. option:: -lto-pass-remarks-with-hotness
- .. option:: -lto-pass-remarks-hotness-threshold
+```{option} -opt-remarks-filename=<filename>
+```
-:program:`gold-plugin` and :program:`lld`
+```{option} -opt-remarks-filter=<regex>
+```
- .. option:: -opt-remarks-filename=<filename>
- .. option:: -opt-remarks-filter=<regex>
- .. option:: -opt-remarks-format=<format>
- .. option:: -opt-remarks-with-hotness
+```{option} -opt-remarks-format=<format>
+```
-.. _yamlremarks:
+```{option} -opt-remarks-with-hotness
+```
-YAML remarks
-============
+(yamlremarks)=
-A typical remark serialized to YAML looks like this:
+## YAML remarks
-.. code-block:: yaml
+A typical remark serialized to YAML looks like this:
- --- !<TYPE>
- Pass: <pass>
- Name: <name>
- DebugLoc: { File: <file>, Line: <line>, Column: <column> }
- Function: <function>
- Hotness: <hotness>
- Args:
- - <key>: <value>
- DebugLoc: { File: <arg-file>, Line: <arg-line>, Column: <arg-column> }
+```yaml
+--- !<TYPE>
+Pass: <pass>
+Name: <name>
+DebugLoc: { File: <file>, Line: <line>, Column: <column> }
+Function: <function>
+Hotness: <hotness>
+Args:
+ - <key>: <value>
+ DebugLoc: { File: <arg-file>, Line: <arg-line>, Column: <arg-column> }
+```
The following entries are mandatory:
-* ``<TYPE>``: can be ``Passed``, ``Missed``, ``Analysis``,
- ``AnalysisFPCommute``, ``AnalysisAliasing``, ``Failure``.
-* ``<pass>``: the name of the pass that emitted this remark.
-* ``<name>``: the name of the remark coming from ``<pass>``.
-* ``<function>``: the mangled name of the function.
+- `<TYPE>`: can be `Passed`, `Missed`, `Analysis`,
+ `AnalysisFPCommute`, `AnalysisAliasing`, `Failure`.
+- `<pass>`: the name of the pass that emitted this remark.
+- `<name>`: the name of the remark coming from `<pass>`.
+- `<function>`: the mangled name of the function.
-If a ``DebugLoc`` entry is specified, the following fields are required:
+If a `DebugLoc` entry is specified, the following fields are required:
-* ``<file>``
-* ``<line>``
-* ``<column>``
+- `<file>`
+- `<line>`
+- `<column>`
-If an ``arg`` entry is specified, the following fields are required:
+If an `arg` entry is specified, the following fields are required:
-* ``<key>``
-* ``<value>``
+- `<key>`
+- `<value>`
-If a ``DebugLoc`` entry is specified within an ``arg`` entry, the following
+If a `DebugLoc` entry is specified within an `arg` entry, the following
fields are required:
-* ``<arg-file>``
-* ``<arg-line>``
-* ``<arg-column>``
+- `<arg-file>`
+- `<arg-line>`
+- `<arg-column>`
-.. _optviewer:
+(optviewer)=
-YAML metadata
--------------
+### YAML metadata
The metadata used together with the YAML format is:
-* a magic number: "REMARKS\\0"
-* the version number: a little-endian uint64_t
-* 8 zero bytes. This space was previously used to encode the size of a string
+- a magic number: "REMARKS\\0"
+- the version number: a little-endian uint64_t
+- 8 zero bytes. This space was previously used to encode the size of a string
table. String table support for YAML remarks has been removed, use the
bitstream format instead.
Optional:
-* the absolute file path to the serialized remark diagnostics: a
+- the absolute file path to the serialized remark diagnostics: a
null-terminated string.
When the metadata is serialized separately from the remarks, the file path
@@ -221,23 +226,21 @@ should be present and point to the file where the remarks are serialized to.
In case the metadata only acts as a header to the remarks, the file path can be
omitted.
-.. _bitstreamremarks:
+(bitstreamremarks)=
-LLVM bitstream remarks
-======================
+## LLVM bitstream remarks
-This format is using :doc:`LLVM bitstream <BitCodeFormat>` to serialize remarks
+This format is using {doc}`LLVM bitstream <BitCodeFormat>` to serialize remarks
and their associated metadata.
-A bitstream remark stream can be identified by the magic number ``"RMRK"`` that
+A bitstream remark stream can be identified by the magic number `"RMRK"` that
is placed at the very beginning.
The format for serializing remarks is composed of two
diff erent block types:
-.. _bitstreamremarksmetablock:
+(bitstreamremarksmetablock)=
-META_BLOCK
-----------
+### META_BLOCK
The block providing information about the rest of the content in the stream.
@@ -245,298 +248,297 @@ Exactly one block is expected. Having multiple metadata blocks is an error.
This block can contain the following records:
-.. _bitstreamremarksrecordmetacontainerinfo:
+(bitstreamremarksrecordmetacontainerinfo)=
-``RECORD_META_CONTAINER_INFO``
+`RECORD_META_CONTAINER_INFO`
- The container version and type.
+The container version and type.
- Version: u32
+Version: u32
- Type: u2
+Type: u2
-.. _bitstreamremarksrecordmetaremarkversion:
+(bitstreamremarksrecordmetaremarkversion)=
-``RECORD_META_REMARK_VERSION``
+`RECORD_META_REMARK_VERSION`
- The version of the remark entries. This can change independently from the
- container version.
+The version of the remark entries. This can change independently from the
+container version.
- Version: u32
+Version: u32
-.. _bitstreamremarksrecordmetastrtab:
+(bitstreamremarksrecordmetastrtab)=
-``RECORD_META_STRTAB``
+`RECORD_META_STRTAB`
- The string table used by the remark entries. The format of the string table
- is a sequence of strings separated by ``\0``.
+The string table used by the remark entries. The format of the string table
+is a sequence of strings separated by `\0`.
-.. _bitstreamremarksrecordmetaexternalfile:
+(bitstreamremarksrecordmetaexternalfile)=
-``RECORD_META_EXTERNAL_FILE``
+`RECORD_META_EXTERNAL_FILE`
- The external remark file path that contains the remark blocks associated
- with this metadata. This is an absolute path.
+The external remark file path that contains the remark blocks associated
+with this metadata. This is an absolute path.
-.. _bitstreamremarksremarkblock:
+(bitstreamremarksremarkblock)=
-REMARK_BLOCK
-------------
+### REMARK_BLOCK
The block describing a remark entry.
0 or more blocks per file are allowed. Each block will depend on the
-:ref:`META_BLOCK <bitstreamremarksmetablock>` in order to be parsed correctly.
+{ref}`META_BLOCK <bitstreamremarksmetablock>` in order to be parsed correctly.
This block can contain the following records:
-``RECORD_REMARK_HEADER``
+`RECORD_REMARK_HEADER`
- The header of the remark. This contains all the mandatory information about
- a remark.
+The header of the remark. This contains all the mandatory information about
+a remark.
- +---------------+---------------------------+
- | Type | u3 |
- +---------------+---------------------------+
- | Remark name | VBR6 (string table index) |
- +---------------+---------------------------+
- | Pass name | VBR6 (string table index) |
- +---------------+---------------------------+
- | Function name | VBR6 (string table index) |
- +---------------+---------------------------+
+```{list-table}
+* - Type
+ - u3
+* - Remark name
+ - VBR6 (string table index)
+* - Pass name
+ - VBR6 (string table index)
+* - Function name
+ - VBR6 (string table index)
+```
-``RECORD_REMARK_DEBUG_LOC``
+`RECORD_REMARK_DEBUG_LOC`
- The source location for the corresponding remark. This record is optional.
+The source location for the corresponding remark. This record is optional.
- +--------+---------------------------+
- | File | VBR7 (string table index) |
- +--------+---------------------------+
- | Line | u32 |
- +--------+---------------------------+
- | Column | u32 |
- +--------+---------------------------+
+```{list-table}
+* - File
+ - VBR7 (string table index)
+* - Line
+ - u32
+* - Column
+ - u32
+```
-``RECORD_REMARK_HOTNESS``
+`RECORD_REMARK_HOTNESS`
- The hotness of the remark. This record is optional.
+The hotness of the remark. This record is optional.
- +---------------+---------------------+
- | Hotness | VBR8 (string table index) |
- +---------------+---------------------+
+```{list-table}
+* - Hotness
+ - VBR8 (string table index)
+```
-``RECORD_REMARK_ARG_WITH_DEBUGLOC``
+`RECORD_REMARK_ARG_WITH_DEBUGLOC`
- A remark argument with an associated debug location.
+A remark argument with an associated debug location.
- +--------+---------------------------+
- | Key | VBR7 (string table index) |
- +--------+---------------------------+
- | Value | VBR7 (string table index) |
- +--------+---------------------------+
- | File | VBR7 (string table index) |
- +--------+---------------------------+
- | Line | u32 |
- +--------+---------------------------+
- | Column | u32 |
- +--------+---------------------------+
+```{list-table}
+* - Key
+ - VBR7 (string table index)
+* - Value
+ - VBR7 (string table index)
+* - File
+ - VBR7 (string table index)
+* - Line
+ - u32
+* - Column
+ - u32
+```
-``RECORD_REMARK_ARG_WITHOUT_DEBUGLOC``
+`RECORD_REMARK_ARG_WITHOUT_DEBUGLOC`
- A remark argument with an associated debug location.
+A remark argument with an associated debug location.
- +--------+---------------------------+
- | Key | VBR7 (string table index) |
- +--------+---------------------------+
- | Value | VBR7 (string table index) |
- +--------+---------------------------+
+```{list-table}
+* - Key
+ - VBR7 (string table index)
+* - Value
+ - VBR7 (string table index)
+```
-The remark container
---------------------
+### The remark container
The bitstream remark container supports multiple types:
-.. _bitstreamremarksfileexternal:
+(bitstreamremarksfileexternal)=
+
+`RemarksFileExternal: a link to an external remarks file`
-``RemarksFileExternal: a link to an external remarks file``
+This container type expects only a {ref}`META_BLOCK <bitstreamremarksmetablock>` containing only:
- This container type expects only a :ref:`META_BLOCK <bitstreamremarksmetablock>` containing only:
+- {ref}`RECORD_META_CONTAINER_INFO <bitstreamremarksrecordmetacontainerinfo>`
+- {ref}`RECORD_META_STRTAB <bitstreamremarksrecordmetastrtab>`
+- {ref}`RECORD_META_EXTERNAL_FILE <bitstreamremarksrecordmetaexternalfile>`
- * :ref:`RECORD_META_CONTAINER_INFO <bitstreamremarksrecordmetacontainerinfo>`
- * :ref:`RECORD_META_STRTAB <bitstreamremarksrecordmetastrtab>`
- * :ref:`RECORD_META_EXTERNAL_FILE <bitstreamremarksrecordmetaexternalfile>`
+Typically, this is emitted in a section in the object files, allowing
+clients to retrieve remarks and their associated metadata directly from
+intermediate products.
- Typically, this is emitted in a section in the object files, allowing
- clients to retrieve remarks and their associated metadata directly from
- intermediate products.
+The container versions of the external separate container should match in order to
+have a well-formed file.
- The container versions of the external separate container should match in order to
- have a well-formed file.
+(bitstreamremarksfile)=
-.. _bitstreamremarksfile:
+`RemarksFile: a standalone remarks file`
-``RemarksFile: a standalone remarks file``
+This container type expects a {ref}`META_BLOCK <bitstreamremarksmetablock>` containing only:
- This container type expects a :ref:`META_BLOCK <bitstreamremarksmetablock>` containing only:
+- {ref}`RECORD_META_CONTAINER_INFO <bitstreamremarksrecordmetacontainerinfo>`
+- {ref}`RECORD_META_REMARK_VERSION <bitstreamremarksrecordmetaremarkversion>`
- * :ref:`RECORD_META_CONTAINER_INFO <bitstreamremarksrecordmetacontainerinfo>`
- * :ref:`RECORD_META_REMARK_VERSION <bitstreamremarksrecordmetaremarkversion>`
+Then, this container type expects 1 or more {ref}`REMARK_BLOCK <bitstreamremarksremarkblock>`.
+If no remarks are emitted, the meta blocks are also not emitted, so the file is empty.
- Then, this container type expects 1 or more :ref:`REMARK_BLOCK <bitstreamremarksremarkblock>`.
- If no remarks are emitted, the meta blocks are also not emitted, so the file is empty.
+After the remark blocks, another {ref}`META_BLOCK <bitstreamremarksmetablock>` is emitted, containing:
- After the remark blocks, another :ref:`META_BLOCK <bitstreamremarksmetablock>` is emitted, containing:
- * :ref:`RECORD_META_STRTAB <bitstreamremarksrecordmetastrtab>`
+- {ref}`RECORD_META_STRTAB <bitstreamremarksrecordmetastrtab>`
- When the parser reads this container type, it jumps to the end of the file
- to read the string table before parsing the individual remarks.
+When the parser reads this container type, it jumps to the end of the file
+to read the string table before parsing the individual remarks.
- Standalone remarks files can be referenced by the
- :ref:`RECORD_META_EXTERNAL_FILE <bitstreamremarksrecordmetaexternalfile>`
- entry in the :ref:`RemarksFileExternal
- <bitstreamremarksfileexternal>` container.
+Standalone remarks files can be referenced by the
+{ref}`RECORD_META_EXTERNAL_FILE <bitstreamremarksrecordmetaexternalfile>`
+entry in the {ref}`RemarksFileExternal
+<bitstreamremarksfileexternal>` container.
-.. FIXME: Add complete output of :program:`llvm-bcanalyzer` on the
diff erent container types (once format changes are completed)
+% FIXME: Add complete output of llvm-bcanalyzer on the
diff erent container types (once format changes are completed)
-opt-viewer
-==========
+## opt-viewer
-The ``opt-viewer`` directory contains a collection of tools that visualize and
+The `opt-viewer` directory contains a collection of tools that visualize and
summarize serialized remarks.
-The tools only support the ``yaml`` format.
+The tools only support the `yaml` format.
-.. _optviewerpy:
+(optviewerpy)=
-opt-viewer.py
--------------
+### opt-viewer.py
Output a HTML page which gives visual feedback on compiler interactions with
your program.
- :Examples:
-
- ::
-
- $ opt-viewer.py my_yaml_file.opt.yaml
-
- ::
+**Examples:**
- $ opt-viewer.py my_build_dir/
+```
+$ opt-viewer.py my_yaml_file.opt.yaml
+```
+```
+$ opt-viewer.py my_build_dir/
+```
-opt-stats.py
-------------
+### opt-stats.py
Output statistics about the optimization remarks in the input set.
- :Example:
+**Example:**
- ::
+```
+$ opt-stats.py my_yaml_file.opt.yaml
- $ opt-stats.py my_yaml_file.opt.yaml
+Total number of remarks 3
- Total number of remarks 3
+Top 10 remarks by pass:
+ inline 33%
+ asm-printer 33%
+ prolog-epilog 33%
- Top 10 remarks by pass:
- inline 33%
- asm-printer 33%
- prolog-epilog 33%
+Top 10 remarks:
+ asm-printer/InstructionCount 33%
+ inline/NoDefinition 33%
+ prolog-epilog/StackSize 33%
+```
- Top 10 remarks:
- asm-printer/InstructionCount 33%
- inline/NoDefinition 33%
- prolog-epilog/StackSize 33%
-
-opt-
diff .py
------------
+### opt-
diff .py
Produce a new YAML file which contains all of the changes in optimizations
between two YAML files.
Typically, this tool should be used to do
diff s between:
-* new compiler + fixed source vs old compiler + fixed source
-* fixed compiler + new source vs fixed compiler + old source
-
-This
diff file can be displayed using :ref:`opt-viewer.py <optviewerpy>`.
+- new compiler + fixed source vs old compiler + fixed source
+- fixed compiler + new source vs fixed compiler + old source
- :Example:
+This
diff file can be displayed using {ref}`opt-viewer.py <optviewerpy>`.
- ::
+**Example:**
- $ opt-
diff .py my_opt_yaml1.opt.yaml my_opt_yaml2.opt.yaml -o my_opt_
diff .opt.yaml
- $ opt-viewer.py my_opt_
diff .opt.yaml
+```
+$ opt-
diff .py my_opt_yaml1.opt.yaml my_opt_yaml2.opt.yaml -o my_opt_
diff .opt.yaml
+$ opt-viewer.py my_opt_
diff .opt.yaml
+```
-.. _remarkssection:
+(remarkssection)=
-Emitting remark diagnostics in the object file
-==============================================
+## Emitting remark diagnostics in the object file
A section containing metadata on remark diagnostics will be emitted for the
following formats:
-* ``bitstream``
+- `bitstream`
-This can be overridden by using the flag ``-remarks-section=<bool>``.
+This can be overridden by using the flag `-remarks-section=<bool>`.
The section is named:
-* ``__LLVM,__remarks`` (MachO)
+- `__LLVM,__remarks` (MachO)
-C API
-=====
+## C API
LLVM provides a library that can be used to parse remarks through a shared
-library named ``libRemarks``.
+library named `libRemarks`.
The typical usage through the C API is like the following:
-.. code-block:: c
-
- LLVMRemarkParserRef Parser = LLVMRemarkParserCreateYAML(Buf, Size);
- LLVMRemarkEntryRef Remark = NULL;
- while ((Remark = LLVMRemarkParserGetNext(Parser))) {
- // use Remark
- LLVMRemarkEntryDispose(Remark); // Release memory.
- }
- bool HasError = LLVMRemarkParserHasError(Parser);
- LLVMRemarkParserDispose(Parser);
+```c
+LLVMRemarkParserRef Parser = LLVMRemarkParserCreateYAML(Buf, Size);
+LLVMRemarkEntryRef Remark = NULL;
+while ((Remark = LLVMRemarkParserGetNext(Parser))) {
+ // use Remark
+ LLVMRemarkEntryDispose(Remark); // Release memory.
+}
+bool HasError = LLVMRemarkParserHasError(Parser);
+LLVMRemarkParserDispose(Parser);
+```
-Remark streamers
-================
+## Remark streamers
-The ``RemarkStreamer`` interface is used to unify the serialization
+The `RemarkStreamer` interface is used to unify the serialization
capabilities of remarks across all the components that can generate remarks.
All remark serialization should go through the main remark streamer, the
-``llvm::remarks::RemarkStreamer`` set up in the ``LLVMContext``. The interface
-takes remark objects converted to ``llvm::remarks::Remark``, and takes care of
+`llvm::remarks::RemarkStreamer` set up in the `LLVMContext`. The interface
+takes remark objects converted to `llvm::remarks::Remark`, and takes care of
serializing it to the requested format, using the requested type of metadata,
etc.
Typically, a specialized remark streamer will hold a reference to the one set
-up in the ``LLVMContext``, and will operate on its own type of diagnostics.
+up in the `LLVMContext`, and will operate on its own type of diagnostics.
-For example, LLVM IR passes will emit ``llvm::DiagnosticInfoOptimization*``
-that get converted to ``llvm::remarks::Remark`` objects. Then, clang could set
-up its own specialized remark streamer that takes ``clang::Diagnostic``
+For example, LLVM IR passes will emit `llvm::DiagnosticInfoOptimization*`
+that get converted to `llvm::remarks::Remark` objects. Then, clang could set
+up its own specialized remark streamer that takes `clang::Diagnostic`
objects. This can allow various components of the frontend to emit remarks
using the same techniques as the LLVM remarks.
This gives us the following advantages:
-* Composition: during the compilation pipeline, multiple components can set up
+- Composition: during the compilation pipeline, multiple components can set up
their specialized remark streamers that all emit remarks through the same
main streamer.
-* Re-using the remark infrastructure in ``lib/Remarks``.
-* Using the same file and format for the remark emitters created throughout the
+- Re-using the remark infrastructure in `lib/Remarks`.
+- Using the same file and format for the remark emitters created throughout the
compilation.
at the cost of an extra layer of abstraction.
-.. FIXME: add documentation for llvm-opt-report.
-.. FIXME: add documentation for Passes supporting optimization remarks
-.. FIXME: add documentation for IR Passes
-.. FIXME: add documentation for CodeGen Passes
+% FIXME: add documentation for llvm-opt-report.
+
+% FIXME: add documentation for Passes supporting optimization remarks
+
+% FIXME: add documentation for IR Passes
+
+% FIXME: add documentation for CodeGen Passes
diff --git a/llvm/docs/SPIRVUsage.md b/llvm/docs/SPIRVUsage.md
index a820e2e6ff6e4..3ef099be78a61 100644
--- a/llvm/docs/SPIRVUsage.md
+++ b/llvm/docs/SPIRVUsage.md
@@ -1,29 +1,26 @@
-=============================
-User Guide for SPIR-V Target
-=============================
+# User Guide for SPIR-V Target
-.. contents::
- :local:
+```{contents}
+:local: true
+```
-.. toctree::
- :hidden:
+```{toctree}
+:hidden:
+```
-Introduction
-============
+## Introduction
The SPIR-V target provides code generation for the SPIR-V binary format described
-in `the official SPIR-V specification <https://www.khronos.org/registry/SPIR-V/>`_.
+in [the official SPIR-V specification](https://www.khronos.org/registry/SPIR-V/).
-Usage
-=====
+## Usage
The SPIR-V backend can be invoked either from LLVM's Static Compiler (llc) or Clang,
allowing developers to compile LLVM intermediate language (IL) files or OpenCL kernel
sources directly to SPIR-V. This section outlines the usage of various commands to
leverage the SPIR-V backend for
diff erent purposes.
-Static Compiler Commands
-------------------------
+### Static Compiler Commands
1. **Basic SPIR-V Compilation**
Command: `llc -mtriple=spirv32-unknown-unknown input.ll -o output.spvt`
@@ -35,125 +32,109 @@ Static Compiler Commands
3. **Compilation with NonSemantic.Shader.DebugInfo support**
Command: `llc -g --spirv-ext=+SPV_KHR_non_semantic_info input.ll -o output.spvt`
- Description: Compiles an LLVM IL file to SPIR-V with NonSemantic.Shader.DebugInfo.100 instructions. The ``-g`` flag causes the backend to emit NSDI instructions when the module contains debug metadata. The required SPV_KHR_non_semantic_info extension must be enabled explicitly.
+ Description: Compiles an LLVM IL file to SPIR-V with NonSemantic.Shader.DebugInfo.100 instructions. The `-g` flag causes the backend to emit NSDI instructions when the module contains debug metadata. The required SPV_KHR_non_semantic_info extension must be enabled explicitly.
- Note: ``--spv-emit-nonsemantic-debug-info`` is a deprecated synonym for ``-g`` and will be removed in a future release.
+ Note: `--spv-emit-nonsemantic-debug-info` is a deprecated synonym for `-g` and will be removed in a future release.
4. **SPIR-V Binary Generation**
Command: `llc -O0 -mtriple=spirv64-unknown-unknown -filetype=obj input.ll -o output.spvt`
Description: Generates a SPIR-V object file (`output.spvt`) from an LLVM module, targeting a 64-bit SPIR-V architecture with no optimizations.
-Clang Commands
---------------
+### Clang Commands
1. **SPIR-V Generation**
Command: `clang –target=spirv64 input.cl`
Description: Generates a SPIR-V file directly from an OpenCL kernel source file (`input.cl`).
-Compiler Options
-================
+## Compiler Options
-.. _spirv-target-triples:
+(spirv-target-triples)=
-Target Triples
---------------
+### Target Triples
For cross-compilation into SPIR-V use option
-``-target <Architecture><Subarchitecture>-<Vendor>-<OS>-<Environment>``
+`-target <Architecture><Subarchitecture>-<Vendor>-<OS>-<Environment>`
to specify the target triple:
- .. table:: SPIR-V Architectures
-
- ============ ==============================================================
- Architecture Description
- ============ ==============================================================
- ``spirv32`` SPIR-V with 32-bit pointer width.
- ``spirv64`` SPIR-V with 64-bit pointer width.
- ``spirv`` SPIR-V with logical memory layout.
- ============ ==============================================================
-
- .. table:: SPIR-V Subarchitectures
-
- =============== ==============================================================
- Subarchitecture Description
- =============== ==============================================================
- *<empty>* SPIR-V version deduced by backend based on the input.
- ``v1.0`` SPIR-V version 1.0.
- ``v1.1`` SPIR-V version 1.1.
- ``v1.2`` SPIR-V version 1.2.
- ``v1.3`` SPIR-V version 1.3.
- ``v1.4`` SPIR-V version 1.4.
- ``v1.5`` SPIR-V version 1.5.
- ``v1.6`` SPIR-V version 1.6.
- =============== ==============================================================
-
- .. table:: SPIR-V Vendors
-
- ===================== ==============================================================
- Vendor Description
- ===================== ==============================================================
- *<empty>*/``unknown`` Generic SPIR-V target without any vendor-specific settings.
- ``amd`` AMDGCN SPIR-V target, with support for target specific
- builtins and ASM, meant to be consumed by AMDGCN toolchains.
- ===================== ==============================================================
-
- .. table:: Operating Systems
-
- ===================== ==============================================================
- OS Description
- ===================== ==============================================================
- *<empty>*/``unknown`` Defaults to the OpenCL runtime.
- ``vulkan`` Vulkan shader runtime.
- ``vulkan1.2`` Vulkan 1.2 runtime, corresponding to SPIR-V 1.5.
- ``vulkan1.3`` Vulkan 1.3 runtime, corresponding to SPIR-V 1.6.
- ``amdhsa`` AMDHSA runtime, meant to be used on HSA compatible runtimes,
- corresponding to SPIR-V 1.6.
- ===================== ==============================================================
-
- .. table:: SPIR-V Environments
-
- ===================== ==============================================================
- Environment Description
- ===================== ==============================================================
- *<empty>*/``unknown`` OpenCL environment or deduced by backend based on the input.
- ===================== ==============================================================
+**SPIR-V Architectures**
+
+| Architecture | Description |
+| --- | --- |
+| `spirv32` | SPIR-V with 32-bit pointer width. |
+| `spirv64` | SPIR-V with 64-bit pointer width. |
+| `spirv` | SPIR-V with logical memory layout. |
+
+**SPIR-V Subarchitectures**
+
+| Subarchitecture | Description |
+| --- | --- |
+| `<empty>` | SPIR-V version deduced by backend based on the input. |
+| `v1.0` | SPIR-V version 1.0. |
+| `v1.1` | SPIR-V version 1.1. |
+| `v1.2` | SPIR-V version 1.2. |
+| `v1.3` | SPIR-V version 1.3. |
+| `v1.4` | SPIR-V version 1.4. |
+| `v1.5` | SPIR-V version 1.5. |
+| `v1.6` | SPIR-V version 1.6. |
+
+**SPIR-V Vendors**
+
+| Vendor | Description |
+| --- | --- |
+| `<empty>` / `unknown` | Generic SPIR-V target without any vendor-specific settings. |
+| `amd` | AMDGCN SPIR-V target, with support for target specific builtins and ASM, meant to be consumed by AMDGCN toolchains. |
+
+**Operating Systems**
+
+| OS | Description |
+| --- | --- |
+| `<empty>` / `unknown` | Defaults to the OpenCL runtime. |
+| `vulkan` | Vulkan shader runtime. |
+| `vulkan1.2` | Vulkan 1.2 runtime, corresponding to SPIR-V 1.5. |
+| `vulkan1.3` | Vulkan 1.3 runtime, corresponding to SPIR-V 1.6. |
+| `amdhsa` | AMDHSA runtime, meant to be used on HSA compatible runtimes, corresponding to SPIR-V 1.6. |
+
+**SPIR-V Environments**
+
+| Environment | Description |
+| --- | --- |
+| `<empty>` / `unknown` | OpenCL environment or deduced by backend based on the input. |
Example:
-``-target spirv64v1.0`` can be used to compile for SPIR-V version 1.0 with 64-bit pointer width.
+`-target spirv64v1.0` can be used to compile for SPIR-V version 1.0 with 64-bit pointer width.
-``-target spirv64-amd-amdhsa`` can be used to compile for AMDGCN flavoured SPIR-V with 64-bit pointer width.
+`-target spirv64-amd-amdhsa` can be used to compile for AMDGCN flavoured SPIR-V with 64-bit pointer width.
-.. _spirv-extensions:
+(spirv-extensions)=
-Extensions
-----------
+### Extensions
-The SPIR-V backend supports a variety of `extensions <https://github.com/KhronosGroup/SPIRV-Registry/tree/main/extensions>`_
+The SPIR-V backend supports a variety of [extensions](https://github.com/KhronosGroup/SPIRV-Registry/tree/main/extensions)
that enable or enhance features beyond the core SPIR-V specification.
-The enabled extensions can be controlled using the ``-spirv-ext`` option followed by a list of
-extensions to enable or disable, each prefixed with ``+`` or ``-``, respectively.
+The enabled extensions can be controlled using the `-spirv-ext` option followed by a list of
+extensions to enable or disable, each prefixed with `+` or `-`, respectively.
To enable multiple extensions, list them separated by comma. For example, to enable support for atomic operations on floating-point numbers and arbitrary precision integers, use:
-``-spirv-ext=+SPV_EXT_shader_atomic_float_add,+SPV_ALTERA_arbitrary_precision_integers``
+`-spirv-ext=+SPV_EXT_shader_atomic_float_add,+SPV_ALTERA_arbitrary_precision_integers`
To enable all extensions, use the following option:
-``-spirv-ext=all``
+`-spirv-ext=all`
To enable all KHR extensions, use the following option:
-``-spirv-ext=khr``
+`-spirv-ext=khr`
-To enable all extensions except specified, specify ``all`` followed by a list of disallowed extensions. For example:
-``-spirv-ext=all,-SPV_ALTERA_arbitrary_precision_integers``
+To enable all extensions except specified, specify `all` followed by a list of disallowed extensions. For example:
+`-spirv-ext=all,-SPV_ALTERA_arbitrary_precision_integers`
Below is a list of supported SPIR-V extensions, sorted alphabetically by their extension names:
-.. list-table:: Supported SPIR-V Extensions
- :widths: 50 150
- :header-rows: 1
+```{list-table} Supported SPIR-V Extensions
+:widths: 50 150
+:header-rows: 1
* - Extension Name
- Description
@@ -284,116 +265,129 @@ Below is a list of supported SPIR-V extensions, sorted alphabetically by their e
* - ``SPV_KHR_poison_freeze``
- Adds instructions to represent a poison value and freeze. Also adds an execution mode to control poison behavior.
+```
-SPIR-V representation in LLVM IR
-================================
+## SPIR-V representation in LLVM IR
SPIR-V is intentionally designed for seamless integration with various Intermediate
Representations (IRs), including LLVM IR, facilitating straightforward mappings for
most of its entities. The development of the SPIR-V backend has been guided by a
-principle of compatibility with the `Khronos Group SPIR-V LLVM Translator <https://github.com/KhronosGroup/SPIRV-LLVM-Translator>`_.
+principle of compatibility with the [Khronos Group SPIR-V LLVM Translator](https://github.com/KhronosGroup/SPIRV-LLVM-Translator).
Consequently, the input representation accepted by the SPIR-V backend aligns closely
-with that detailed in `the SPIR-V Representation in LLVM document <https://github.com/KhronosGroup/SPIRV-LLVM-Translator/blob/main/docs/SPIRVRepresentationInLLVM.rst>`_.
+with that detailed in [the SPIR-V Representation in LLVM document](https://github.com/KhronosGroup/SPIRV-LLVM-Translator/blob/main/docs/SPIRVRepresentationInLLVM.rst).
This document, along with the sections that follow, delineate the main points and focus
on any
diff erences between the LLVM IR that this backend processes and the conventions
used by other tools.
-.. _spirv-special-types:
+(spirv-special-types)=
-Special types
--------------
+### Special types
SPIR-V specifies several kinds of opaque types. These types are represented
using target extension types and are represented as follows:
- .. table:: SPIR-V Opaque Types
-
- ================== ======================= ===========================================================================================
- SPIR-V Type LLVM type name LLVM type arguments
- ================== ======================= ===========================================================================================
- OpTypeImage ``spirv.Image`` sampled type, dimensionality, depth, arrayed, MS, sampled, image format, [access qualifier]
- OpTypeImage ``spirv.SignedImage`` sampled type, dimensionality, depth, arrayed, MS, sampled, image format, [access qualifier]
- OpTypeSampler ``spirv.Sampler`` (none)
- OpTypeSampledImage ``spirv.SampledImage`` sampled type, dimensionality, depth, arrayed, MS, sampled, image format, [access qualifier]
- OpTypeEvent ``spirv.Event`` (none)
- OpTypeDeviceEvent ``spirv.DeviceEvent`` (none)
- OpTypeReserveId ``spirv.ReserveId`` (none)
- OpTypeQueue ``spirv.Queue`` (none)
- OpTypePipe ``spirv.Pipe`` access qualifier
- OpTypePipeStorage ``spirv.PipeStorage`` (none)
- NA ``spirv.VulkanBuffer`` ElementType, StorageClass, IsWriteable
- ================== ======================= ===========================================================================================
-
-All integer arguments take the same value as they do in their `corresponding
-SPIR-V instruction <https://registry.khronos.org/SPIR-V/specs/unified1/SPIRV.html#_type_declaration_instructions>`_.
-For example, the OpenCL type ``image2d_depth_ro_t`` would be represented in
-SPIR-V IR as ``target("spirv.Image", void, 1, 1, 0, 0, 0, 0, 0)``, with its
-dimensionality parameter as ``1`` meaning 2D. Sampled image types include the
+```{list-table} SPIR-V Opaque Types
+:widths: 20 25 55
+:header-rows: 1
+
+ * - SPIR-V Type
+ - LLVM type name
+ - LLVM type arguments
+ * - OpTypeImage
+ - `spirv.Image`
+ - sampled type, dimensionality, depth, arrayed, MS, sampled, image format, [access qualifier]
+ * - OpTypeImage
+ - `spirv.SignedImage`
+ - sampled type, dimensionality, depth, arrayed, MS, sampled, image format, [access qualifier]
+ * - OpTypeSampler
+ - `spirv.Sampler`
+ - (none)
+ * - OpTypeSampledImage
+ - `spirv.SampledImage`
+ - sampled type, dimensionality, depth, arrayed, MS, sampled, image format, [access qualifier]
+ * - OpTypeEvent
+ - `spirv.Event`
+ - (none)
+ * - OpTypeDeviceEvent
+ - `spirv.DeviceEvent`
+ - (none)
+ * - OpTypeReserveId
+ - `spirv.ReserveId`
+ - (none)
+ * - OpTypeQueue
+ - `spirv.Queue`
+ - (none)
+ * - OpTypePipe
+ - `spirv.Pipe`
+ - access qualifier
+ * - OpTypePipeStorage
+ - `spirv.PipeStorage`
+ - (none)
+ * - NA
+ - `spirv.VulkanBuffer`
+ - ElementType, StorageClass, IsWriteable
+```
+
+All integer arguments take the same value as they do in their [corresponding
+SPIR-V instruction](https://registry.khronos.org/SPIR-V/specs/unified1/SPIRV.html#_type_declaration_instructions).
+For example, the OpenCL type `image2d_depth_ro_t` would be represented in
+SPIR-V IR as `target("spirv.Image", void, 1, 1, 0, 0, 0, 0, 0)`, with its
+dimensionality parameter as `1` meaning 2D. Sampled image types include the
parameters of its underlying image type, so that a sampled image for the
previous type has the representation
-``target("spirv.SampledImage, void, 1, 1, 0, 0, 0, 0, 0)``.
+`target("spirv.SampledImage, void, 1, 1, 0, 0, 0, 0, 0)`.
-The
diff erences between ``spirv.Image`` and ``spirv.SignedImage`` is that the
+The
diff erences between `spirv.Image` and `spirv.SignedImage` is that the
backend will generate code assuming that the format of the image is a signed
integer instead of unsigned. This is required because llvm-ir will create the
same sampled type for signed and unsigned integers. If the image format is
unknown, the backend cannot distinguish the two case.
-See `wg-hlsl proposal 0018 <https://github.com/llvm/wg-hlsl/blob/main/proposals/0018-spirv-resource-representation.md>`_
-for details on ``spirv.VulkanBuffer``.
+See [wg-hlsl proposal 0018](https://github.com/llvm/wg-hlsl/blob/main/proposals/0018-spirv-resource-representation.md)
+for details on `spirv.VulkanBuffer`.
-.. _inline-spirv-types:
+(inline-spirv-types)=
-Inline SPIR-V Types
--------------------
+### Inline SPIR-V Types
-HLSL allows users to create types representing specific SPIR-V types, using ``vk::SpirvType`` and
-``vk::SpirvOpaqueType``. These are specified in the `Inline SPIR-V`_ proposal. They may be
+HLSL allows users to create types representing specific SPIR-V types, using `vk::SpirvType` and
+`vk::SpirvOpaqueType`. These are specified in the [Inline SPIR-V] proposal. They may be
represented using target extension types:
-.. _Inline SPIR-V: https://microsoft.github.io/hlsl-specs/proposals/0011-inline-spirv.html#types
+| LLVM type name | LLVM type arguments | LLVM integer arguments |
+| --- | --- | --- |
+| `spirv.Type` | SPIR-V operands | opcode, size, alignment |
+| `spirv.IntegralConstant` | integral type | value |
+| `spirv.Literal` | (none) | value |
- .. table:: Inline SPIR-V Types
-
- ========================== =================== =========================
- LLVM type name LLVM type arguments LLVM integer arguments
- ========================== =================== =========================
- ``spirv.Type`` SPIR-V operands opcode, size, alignment
- ``spirv.IntegralConstant`` integral type value
- ``spirv.Literal`` (none) value
- ========================== =================== =========================
-
-The operand arguments to ``spirv.Type`` may be either a ``spirv.IntegralConstant`` type,
-representing an ``OpConstant`` id operand, a ``spirv.Literal`` type, representing an immediate
+The operand arguments to `spirv.Type` may be either a `spirv.IntegralConstant` type,
+representing an `OpConstant` id operand, a `spirv.Literal` type, representing an immediate
literal operand, or any other type, representing the id of that type as an operand.
-``spirv.IntegralConstant`` and ``spirv.Literal`` may not be used outside of this context.
+`spirv.IntegralConstant` and `spirv.Literal` may not be used outside of this context.
-For example, ``OpTypeArray`` (opcode 28) takes an id for the element type and an id for the element
+For example, `OpTypeArray` (opcode 28) takes an id for the element type and an id for the element
length, so an array of 16 integers could be declared as:
-``target("spirv.Type", i32, target("spirv.IntegralConstant", i32, 16), 28, 64, 32)``
+`target("spirv.Type", i32, target("spirv.IntegralConstant", i32, 16), 28, 64, 32)`
This will be lowered to:
-``OpTypeArray %int %int_16``
+`OpTypeArray %int %int_16`
-``OpTypeVector`` takes an id for the component type and a literal for the component count, so a
+`OpTypeVector` takes an id for the component type and a literal for the component count, so a
4-integer vector could be declared as:
-``target("spirv.Type", i32, target("spirv.Literal", 4), 23, 16, 32)``
+`target("spirv.Type", i32, target("spirv.Literal", 4), 23, 16, 32)`
This will be lowered to:
-``OpTypeVector %int 4``
+`OpTypeVector %int 4`
-See `Target Extension Types for Inline SPIR-V and Decorated Types`_ for further details.
+See [Target Extension Types for Inline SPIR-V and Decorated Types] for further details.
-.. _Target Extension Types for Inline SPIR-V and Decorated Types: https://github.com/llvm/wg-hlsl/blob/main/proposals/0017-inline-spirv-and-decorated-types.md
+(spirv-intrinsics)=
-.. _spirv-intrinsics:
-
-Target Intrinsics
------------------
+### Target Intrinsics
The SPIR-V backend employs several LLVM IR intrinsics that facilitate various low-level
operations essential for generating correct and efficient SPIR-V code. These intrinsics
@@ -401,9 +395,9 @@ cover a range of functionalities from type assignment and memory management to c
flow and atomic operations. Below is a detailed table of selected intrinsics used in the
SPIR-V backend, along with their descriptions and argument details.
-.. list-table:: LLVM IR Intrinsics for SPIR-V
- :widths: 25 15 20 40
- :header-rows: 1
+```{list-table} LLVM IR Intrinsics for SPIR-V
+:widths: 25 15 20 40
+:header-rows: 1
* - Intrinsic ID
- Return Type
@@ -540,140 +534,140 @@ SPIR-V backend, along with their descriptions and argument details.
* - `int_spv_resource_handlefrombinding`
- spirv.Image
- `[32-bit Integer set, 32-bit Integer binding, 32-bit Integer arraySize, 32-bit Integer index, bool isUniformIndex]`
- - Returns the handle for the resource at the given set and binding.\
- If `arraySize > 1`, then the binding represents an array of resources\
- of the given size, and the handle for the resource at the given index is returned.\
+ - Returns the handle for the resource at the given set and binding.
+ If `arraySize > 1`, then the binding represents an array of resources
+ of the given size, and the handle for the resource at the given index is returned.
If the index is possibly non-uniform, then `isUniformIndex` must get set to true.
* - `int_spv_typeBufferLoad`
- Scalar or vector
- `[spirv.Image ImageBuffer, 32-bit Integer coordinate]`
- - Loads a value from a Vulkan image buffer at the given coordinate. The \
- image buffer data is assumed to be stored as a 4-element vector. If the \
- return type is a scalar, then the first element of the vector is \
- returned. If the return type is an n-element vector, then the first \
+ - Loads a value from a Vulkan image buffer at the given coordinate. The
+ image buffer data is assumed to be stored as a 4-element vector. If the
+ return type is a scalar, then the first element of the vector is
+ returned. If the return type is an n-element vector, then the first
n-elements of the 4-element vector are returned.
* - `int_spv_resource_store_typedbuffer`
- void
- `[spirv.Image Image, 32-bit Integer coordinate, vec4 data]`
- - Stores the data to the image buffer at the given coordinate. The \
+ - Stores the data to the image buffer at the given coordinate. The
data must be a 4-element vector.
+```
-.. _spirv-builtin-functions:
+(spirv-builtin-functions)=
-Builtin Functions
------------------
+### Builtin Functions
The following section highlights the representation of SPIR-V builtins in LLVM IR,
emphasizing builtins that do not have direct counterparts in LLVM.
-Instructions as Function Calls
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#### Instructions as Function Calls
SPIR-V builtins without direct LLVM counterparts are represented as LLVM function calls.
These functions, termed SPIR-V builtin functions, follow an IA64 mangling scheme with
SPIR-V-specific extensions. Parsing non-mangled calls to builtins is supported in some cases,
but not tested extensively. The general format is:
-.. code-block:: c
-
- __spirv_{OpCodeName}{_OptionalPostfixes}
+```c
+__spirv_{OpCodeName}{_OptionalPostfixes}
+```
Where `{OpCodeName}` is the SPIR-V opcode name sans the "Op" prefix, and
`{OptionalPostfixes}` are decoration-specific postfixes, if any. The mangling and
postfixes allow for the representation of SPIR-V's rich instruction set within LLVM's
framework.
-Extended Instruction Sets
-~~~~~~~~~~~~~~~~~~~~~~~~~
+#### Extended Instruction Sets
SPIR-V defines several extended instruction sets for additional functionalities, such as
OpenCL-specific operations. In LLVM IR, these are represented by function calls to
mangled builtins and selected based on the environment. For example:
-.. code-block:: c
-
- acos_f32
+```c
+acos_f32
+```
represents the `acos` function from the OpenCL extended instruction set for a float32
input.
-Builtin Variables
-~~~~~~~~~~~~~~~~~
+#### Builtin Variables
SPIR-V builtin variables, which provide access to special hardware or execution model
properties, are mapped to either LLVM function calls or LLVM global variables. The
representation follows the naming convention:
-.. code-block:: c
-
- __spirv_BuiltIn{VariableName}
+```c
+__spirv_BuiltIn{VariableName}
+```
For instance, the SPIR-V builtin `GlobalInvocationId` is accessible in LLVM IR as
`__spirv_BuiltInGlobalInvocationId`.
-Vector Load and Store Builtins
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#### Vector Load and Store Builtins
SPIR-V's capabilities for loading and storing vectors are represented in LLVM IR using
functions that mimic the SPIR-V instructions. These builtins handle cases that LLVM's
native instructions do not directly support, enabling fine-grained control over memory
operations.
-Atomic Operations
-~~~~~~~~~~~~~~~~~
+#### Atomic Operations
SPIR-V's atomic operations, especially those operating on floating-point data, are
represented in LLVM IR with corresponding function calls. These builtins ensure
atomicity in operations where LLVM might not have direct support, essential for parallel
execution and synchronization.
-Image Operations
-~~~~~~~~~~~~~~~~
+#### Image Operations
SPIR-V provides extensive support for image and sampler operations, which LLVM
represents through function calls to builtins. These include image reads, writes, and
queries, allowing detailed manipulation of image data and parameters.
-Group and Subgroup Operations
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#### Group and Subgroup Operations
For workgroup and subgroup operations, LLVM uses function calls to represent SPIR-V's
group-based instructions. These builtins facilitate group synchronization, data sharing,
and collective operations essential for efficient parallel computation.
-SPIR-V Instructions Mapped to LLVM Metadata
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+#### SPIR-V Instructions Mapped to LLVM Metadata
+
Some SPIR-V instructions don't have a direct equivalent in the LLVM IR language. To
address this, the SPIR-V Target uses
diff erent specific LLVM named metadata to convey
the necessary information. The SPIR-V specification allows multiple module-scope
instructions, where as LLVM named metadata must be unique. Therefore, the encoding of
such instructions has the following format:
-.. code-block:: llvm
-
- !spirv.<OpCodeName> = !{!<InstructionMetadata1>, !<InstructionMetadata2>, ..}
- !<InstructionMetadata1> = !{<Operand1>, <Operand2>, ..}
- !<InstructionMetadata2> = !{<Operand1>, <Operand2>, ..}
+```llvm
+!spirv.<OpCodeName> = !{!<InstructionMetadata1>, !<InstructionMetadata2>, ..}
+!<InstructionMetadata1> = !{<Operand1>, <Operand2>, ..}
+!<InstructionMetadata2> = !{<Operand1>, <Operand2>, ..}
+```
Below, you will find the mappings between SPIR-V instruction and their corresponding
LLVM IR representations.
-+--------------------+---------------------------------------------------------+
-| SPIR-V instruction | LLVM IR |
-+====================+=========================================================+
-| OpMemoryModel | .. code-block:: llvm |
-| | |
-| | !spirv.MemoryModel = !{!0} |
-| | !0 = !{i32 0, i32 1} |
-| | ; Set addressing model to Logical (0) and memory |
-| | ; model to GLSL450 (1). Valid memory models: |
-| | ; Simple (0), GLSL450 (1), OpenCL (2), |
-| | ; VulkanKHR (3). |
-+--------------------+---------------------------------------------------------+
-| OpExecutionMode | .. code-block:: llvm |
-| | |
-| | !spirv.ExecutionMode = !{!0} |
-| | !0 = !{void @worker, i32 30, i32 262149} |
-| | ; Set execution mode with id 30 (VecTypeHint) and |
-| | ; literal `262149` operand. |
-+--------------------+---------------------------------------------------------+
+````{list-table}
+:widths: 25 75
+:header-rows: 1
+
+ * - SPIR-V instruction
+ - LLVM IR
+ * - OpMemoryModel
+ - ```llvm
+ !spirv.MemoryModel = !{!0}
+ !0 = !{i32 0, i32 1}
+ ; Set addressing model to Logical (0) and memory
+ ; model to GLSL450 (1). Valid memory models:
+ ; Simple (0), GLSL450 (1), OpenCL (2),
+ ; VulkanKHR (3).
+ ```
+ * - OpExecutionMode
+ - ```llvm
+ !spirv.ExecutionMode = !{!0}
+ !0 = !{void @worker, i32 30, i32 262149}
+ ; Set execution mode with id 30 (VecTypeHint) and
+ ; literal `262149` operand.
+ ```
+````
+
+[inline spir-v]: https://microsoft.github.io/hlsl-specs/proposals/0011-inline-spirv.html#types
+[target extension types for inline spir-v and decorated types]: https://github.com/llvm/wg-hlsl/blob/main/proposals/0017-inline-spirv-and-decorated-types.md
More information about the llvm-commits
mailing list