<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<p style="font-family:Arial;font-size:10pt;color:#0078D7;margin:15pt;" align="Left">
[AMD Official Use Only - Internal Distribution Only]<br>
</p>
<br>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Your "@kernel" function isn't a kernel, it's the default C calling convention. You need to use the amdgpu_kernel calling convention</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
-Matt<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> llvm-dev <llvm-dev-bounces@lists.llvm.org> on behalf of Frank Winter via llvm-dev <llvm-dev@lists.llvm.org><br>
<b>Sent:</b> Wednesday, April 22, 2020 1:57 PM<br>
<b>To:</b> LLVM Dev <llvm-dev@lists.llvm.org><br>
<b>Subject:</b> [llvm-dev] ROCm module from LLVM AMDGPU backend</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">[CAUTION: External Email]<br>
<br>
Hi,<br>
<br>
I'm trying to launch a GPU kernel which was compiled by the LLVM<br>
AMDGPU backend. Currently I'm having no success with it and I was<br>
hoping someone tuned in on here might have an idea.<br>
<br>
It seems that tensorflow is doing a similar thing. So I was reading<br>
the tensorflow code on github and I believe the following setup is<br>
pretty close in the vital parts:<br>
<br>
<br>
1) Compile an LLVM IR module (see below) with AMDGPU backend to a<br>
'module.o' file. Using this triple/CPU:<br>
<br>
llvm::Triple TheTriple;<br>
TheTriple.setArch (llvm::Triple::ArchType::amdgcn);<br>
TheTriple.setVendor (llvm::Triple::VendorType::AMD);<br>
TheTriple.setOS (llvm::Triple::OSType::AMDHSA);<br>
<br>
std::string CPUStr("gfx906");<br>
<br>
LLVM IR passes that I use:<br>
<br>
TargetLibraryInfoWrapperPass<br>
TargetMachine->addPassesToEmitFile with CGFT_ObjectFile<br>
<br>
<br>
2) LLVM linker generates a shared lib using 'system()' call<br>
<br>
ld.lld -shared module.o -o module.so<br>
<br>
<br>
3) Reading this shared module back into a 'vector<uint8> shared'<br>
<br>
<br>
4) Using HIP to load this module:<br>
<br>
hipModule_t module;<br>
ret = hipModuleLoadData( &module , shared.data() );<br>
<br>
(this returns hipSuccess)<br>
<br>
5) Trying to get a HIP function:<br>
<br>
hipFunction_t kernel;<br>
ret = hipModuleGetFunction(&kernel, module, "kernel" );<br>
<br>
.. and this fails with HIP error code 500 !?<br>
<br>
<br>
I believe the vital steps here concerning ROCm are similar<br>
(identical?) to what's in tensorflow but I don't get it to work.<br>
<br>
I have to admit that I did not build tensorflow to see if the AMD GPU<br>
bits actually work. I read the comments and some are saying that it<br>
comes with some performance overhead. Performance isn't the point at<br>
the moment - I'm working on a proof-of-concept.<br>
<br>
My test machine has an 'AMD gfx906' card installed.<br>
<br>
Digging deeper, the hipModule_t is a pointer to ihipModule_t and<br>
printing out the values after loading the module gives<br>
<br>
ihip->fileName =<br>
ihip->hash = 3943538976062281088<br>
ihip->kernargs.size() = 0<br>
ihip->executable.handle = 42041072<br>
<br>
It's not telling me much. 'Not sure what to do with the handle for the<br>
executable.<br>
<br>
Any ideas what could be tried next?<br>
<br>
<br>
Frank<br>
<br>
<br>
--------------------------------------------------------------<br>
<br>
LLVM IR module<br>
<br>
target datalayout =<br>
"e-p:64:64-p1:64:64-p2:32:32-p3:32:32-p4:64:64-p5:32:32-p6:32:32-i64:64-v16:16-v24:32-v32:32-v48:64-v96:128-v192:256-v256:256-v512:512-v1024:1024-v2048:2048-n32:64-S32-A5-ni:7"<br>
<br>
define void @kernel(i1 %arg0, i32 %arg1, i32 %arg2, i32 %arg3, i1 %arg4,<br>
i32* %arg5, i1* %arg6, float* %arg7, float* %arg8, float* %arg9) {<br>
entrypoint:<br>
%0 = sext i1 %arg4 to i32<br>
%1 = xor i32 -1, %0<br>
%2 = call i32 @llvm.amdgcn.workitem.id.x()<br>
%3 = icmp sge i32 %2, %arg1<br>
br i1 %3, label %L0, label %L1<br>
<br>
L0: ; preds = %entrypoint<br>
ret void<br>
<br>
L1: ; preds = %entrypoint<br>
%4 = trunc i32 %1 to i1<br>
br i1 %4, label %L3, label %L4<br>
<br>
L2: ; preds = %L6, %L5, %L4<br>
%5 = phi i32 [ %7, %L4 ], [ %8, %L5 ], [ %2, %L6 ]<br>
br i1 %arg0, label %L7, label %L8<br>
<br>
L3: ; preds = %L1<br>
br i1 %arg0, label %L5, label %L6<br>
<br>
L4: ; preds = %L1<br>
%6 = getelementptr i32, i32* %arg5, i32 %2<br>
%7 = load i32, i32* %6<br>
br label %L2<br>
<br>
L5: ; preds = %L3<br>
%8 = add nsw i32 %2, %arg2<br>
br label %L2<br>
<br>
L6: ; preds = %L3<br>
br label %L2<br>
<br>
L7: ; preds = %L2<br>
%9 = icmp sgt i32 %5, %arg3<br>
br i1 %9, label %L12, label %L13<br>
<br>
L8: ; preds = %L2<br>
%10 = getelementptr i1, i1* %arg6, i32 %5<br>
%11 = load i1, i1* %10<br>
%12 = sext i1 %11 to i32<br>
%13 = xor i32 -1, %12<br>
%14 = trunc i32 %13 to i1<br>
br i1 %14, label %L10, label %L11<br>
<br>
L9: ; preds = %L15, %L11<br>
%15 = add nsw i32 0, %5<br>
%16 = add nsw i32 0, %5<br>
%17 = getelementptr float, float* %arg8, i32 %16<br>
%18 = load float, float* %17<br>
%19 = add nsw i32 0, %5<br>
%20 = getelementptr float, float* %arg9, i32 %19<br>
%21 = load float, float* %20<br>
%22 = fmul float %18, %21<br>
%23 = getelementptr float, float* %arg7, i32 %15<br>
store float %22, float* %23<br>
ret void<br>
<br>
L10: ; preds = %L8<br>
ret void<br>
<br>
L11: ; preds = %L8<br>
br label %L9<br>
<br>
L12: ; preds = %L7<br>
ret void<br>
<br>
L13: ; preds = %L7<br>
%24 = icmp slt i32 %5, %arg2<br>
br i1 %24, label %L14, label %L15<br>
<br>
L14: ; preds = %L13<br>
ret void<br>
<br>
L15: ; preds = %L13<br>
br label %L9<br>
}<br>
<br>
; Function Attrs: nounwind readnone speculatable<br>
declare i32 @llvm.amdgcn.workitem.id.x() #0<br>
<br>
attributes #0 = { nounwind readnone speculatable }<br>
<br>
<br>
<br>
------------------------------------------------------------------------------<br>
<br>
<br>
<br>
The following is the assembly output the AMDGPU backend generates:<br>
<br>
<br>
output: .text<br>
.amdgcn_target "amdgcn-amd-amdhsa--gfx906"<br>
.globl kernel<br>
.p2align 2<br>
.type kernel,@function<br>
kernel:<br>
s_waitcnt vmcnt(0) expcnt(0) lgkmcnt(0)<br>
v_and_b32_e32 v4, 1, v4<br>
v_cmp_eq_u32_e64 s[4:5], 1, v4<br>
v_and_b32_e32 v0, 1, v0<br>
v_and_b32_e32 v4, 0x3ff, v15<br>
v_cmp_eq_u32_e32 vcc, 1, v0<br>
v_cmp_lt_i32_e64 s[6:7], v4, v1<br>
s_and_saveexec_b64 s[8:9], s[6:7]<br>
s_cbranch_execz BB0_16<br>
BB0_1:<br>
s_and_saveexec_b64 s[6:7], s[4:5]<br>
s_xor_b64 s[6:7], exec, s[6:7]<br>
s_cbranch_execz BB0_3<br>
BB0_2:<br>
v_lshlrev_b32_e32 v0, 2, v4<br>
v_add_co_u32_e64 v0, s[4:5], v5, v0<br>
v_addc_co_u32_e64 v1, s[4:5], 0, v6, s[4:5]<br>
flat_load_dword v0, v[0:1]<br>
BB0_3:<br>
s_or_saveexec_b64 s[4:5], s[6:7]<br>
s_xor_b64 exec, exec, s[4:5]<br>
s_cbranch_execz BB0_7<br>
BB0_4:<br>
s_xor_b64 s[6:7], vcc, -1<br>
s_waitcnt vmcnt(0) lgkmcnt(0)<br>
v_add_u32_e32 v0, v4, v2<br>
s_and_saveexec_b64 s[10:11], s[6:7]<br>
s_xor_b64 s[6:7], exec, s[10:11]<br>
BB0_5:<br>
v_mov_b32_e32 v0, v4<br>
BB0_6:<br>
s_or_b64 exec, exec, s[6:7]<br>
BB0_7:<br>
s_or_b64 exec, exec, s[4:5]<br>
s_xor_b64 s[6:7], vcc, -1<br>
s_mov_b64 s[4:5], 0<br>
s_and_saveexec_b64 s[10:11], s[6:7]<br>
s_xor_b64 s[6:7], exec, s[10:11]<br>
s_cbranch_execz BB0_9<br>
BB0_8:<br>
s_waitcnt vmcnt(0) lgkmcnt(0)<br>
v_ashrrev_i32_e32 v1, 31, v0<br>
v_add_co_u32_e32 v4, vcc, v7, v0<br>
v_addc_co_u32_e32 v5, vcc, v8, v1, vcc<br>
flat_load_ubyte v1, v[4:5]<br>
s_waitcnt vmcnt(0) lgkmcnt(0)<br>
v_and_b32_e32 v1, 1, v1<br>
v_cmp_eq_u32_e32 vcc, 1, v1<br>
s_and_b64 s[4:5], vcc, exec<br>
BB0_9:<br>
s_or_saveexec_b64 s[6:7], s[6:7]<br>
s_xor_b64 exec, exec, s[6:7]<br>
s_cbranch_execz BB0_13<br>
BB0_10:<br>
s_waitcnt vmcnt(0) lgkmcnt(0)<br>
v_cmp_le_i32_e32 vcc, v0, v3<br>
s_mov_b64 s[12:13], s[4:5]<br>
s_and_saveexec_b64 s[10:11], vcc<br>
BB0_11:<br>
v_cmp_ge_i32_e32 vcc, v0, v2<br>
s_andn2_b64 s[12:13], s[4:5], exec<br>
s_and_b64 s[14:15], vcc, exec<br>
s_or_b64 s[12:13], s[12:13], s[14:15]<br>
BB0_12:<br>
s_or_b64 exec, exec, s[10:11]<br>
s_andn2_b64 s[4:5], s[4:5], exec<br>
s_and_b64 s[10:11], s[12:13], exec<br>
s_or_b64 s[4:5], s[4:5], s[10:11]<br>
BB0_13:<br>
s_or_b64 exec, exec, s[6:7]<br>
s_and_saveexec_b64 s[6:7], s[4:5]<br>
s_cbranch_execz BB0_15<br>
BB0_14:<br>
s_waitcnt vmcnt(0) lgkmcnt(0)<br>
v_ashrrev_i32_e32 v1, 31, v0<br>
v_lshlrev_b64 v[0:1], 2, v[0:1]<br>
v_add_co_u32_e32 v2, vcc, v11, v0<br>
v_addc_co_u32_e32 v3, vcc, v12, v1, vcc<br>
flat_load_dword v4, v[2:3]<br>
v_add_co_u32_e32 v2, vcc, v13, v0<br>
v_addc_co_u32_e32 v3, vcc, v14, v1, vcc<br>
flat_load_dword v2, v[2:3]<br>
v_add_co_u32_e32 v0, vcc, v9, v0<br>
v_addc_co_u32_e32 v1, vcc, v10, v1, vcc<br>
s_waitcnt vmcnt(0) lgkmcnt(0)<br>
v_mul_f32_e32 v2, v4, v2<br>
flat_store_dword v[0:1], v2<br>
BB0_15:<br>
s_or_b64 exec, exec, s[6:7]<br>
BB0_16:<br>
s_or_b64 exec, exec, s[8:9]<br>
s_waitcnt vmcnt(0) lgkmcnt(0)<br>
s_setpc_b64 s[30:31]<br>
.Lfunc_end0:<br>
.size kernel, .Lfunc_end0-kernel<br>
<br>
.section ".note.GNU-stack"<br>
.amdgpu_metadata<br>
---<br>
amdhsa.kernels: []<br>
amdhsa.version:<br>
- 1<br>
- 0<br>
...<br>
<br>
.end_amdgpu_metadata<br>
<br>
<br>
<br>
-----------------------------------------------------------------------<br>
<br>
<br>
rocminfo output:<br>
<br>
<br>
Agent 1 and 2 are the host's Intel CPUs, then Agent 3 - 6 look like:<br>
<br>
*******<br>
Agent 3<br>
*******<br>
Name: gfx906<br>
Marketing Name: Vega 20<br>
Vendor Name: AMD<br>
Feature: KERNEL_DISPATCH<br>
Profile: BASE_PROFILE<br>
Float Round Mode: NEAR<br>
Max Queue Number: 128(0x80)<br>
Queue Min Size: 4096(0x1000)<br>
Queue Max Size: 131072(0x20000)<br>
Queue Type: MULTI<br>
Node: 2<br>
Device Type: GPU<br>
Cache Info:<br>
L1: 16(0x10) KB<br>
Chip ID: 26273(0x66a1)<br>
Cacheline Size: 64(0x40)<br>
Max Clock Freq. (MHz): 1725<br>
BDFID: 35328<br>
Internal Node ID: 2<br>
Compute Unit: 60<br>
SIMDs per CU: 4<br>
Shader Engines: 4<br>
Shader Arrs. per Eng.: 1<br>
WatchPts on Addr. Ranges:4<br>
Features: KERNEL_DISPATCH<br>
Fast F16 Operation: FALSE<br>
Wavefront Size: 64(0x40)<br>
Workgroup Max Size: 1024(0x400)<br>
Workgroup Max Size per Dimension:<br>
x 1024(0x400)<br>
y 1024(0x400)<br>
z 1024(0x400)<br>
Max Waves Per CU: 40(0x28)<br>
Max Work-item Per CU: 2560(0xa00)<br>
Grid Max Size: 4294967295(0xffffffff)<br>
Grid Max Size per Dimension:<br>
x 4294967295(0xffffffff)<br>
y 4294967295(0xffffffff)<br>
z 4294967295(0xffffffff)<br>
Max fbarriers/Workgrp: 32<br>
Pool Info:<br>
Pool 1<br>
Segment: GLOBAL; FLAGS: COARSE GRAINED<br>
Size: 33538048(0x1ffc000) KB<br>
Allocatable: TRUE<br>
Alloc Granule: 4KB<br>
Alloc Alignment: 4KB<br>
Acessible by all: FALSE<br>
Pool 2<br>
Segment: GLOBAL; FLAGS: FINE GRAINED<br>
Size: 33538048(0x1ffc000) KB<br>
Allocatable: TRUE<br>
Alloc Granule: 4KB<br>
Alloc Alignment: 4KB<br>
Acessible by all: FALSE<br>
Pool 3<br>
Segment: GROUP<br>
Size: 64(0x40) KB<br>
Allocatable: FALSE<br>
Alloc Granule: 0KB<br>
Alloc Alignment: 0KB<br>
Acessible by all: FALSE<br>
ISA Info:<br>
ISA 1<br>
Name: amdgcn-amd-amdhsa--gfx906<br>
Machine Models: HSA_MACHINE_MODEL_LARGE<br>
Profiles: HSA_PROFILE_BASE<br>
Default Rounding Mode: NEAR<br>
Default Rounding Mode: NEAR<br>
Fast f16: TRUE<br>
Workgroup Max Size: 1024(0x400)<br>
Workgroup Max Size per Dimension:<br>
x 1024(0x400)<br>
y 1024(0x400)<br>
z 1024(0x400)<br>
Grid Max Size: 4294967295(0xffffffff)<br>
Grid Max Size per Dimension:<br>
x 4294967295(0xffffffff)<br>
y 4294967295(0xffffffff)<br>
z 4294967295(0xffffffff)<br>
FBarrier Max Size: 32<br>
<br>
<br>
_______________________________________________<br>
LLVM Developers mailing list<br>
llvm-dev@lists.llvm.org<br>
<a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fllvm-dev&data=02%7C01%7CMatthew.Arsenault%40amd.com%7C89e2121592ab48c84c1908d7e6ffce69%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637231858665874178&sdata=DcmMDUa7TNCPu1lPd3ZP5jcuiuCsBrQtrTfKUKxX%2FGw%3D&reserved=0">https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.llvm.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fllvm-dev&data=02%7C01%7CMatthew.Arsenault%40amd.com%7C89e2121592ab48c84c1908d7e6ffce69%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637231858665874178&sdata=DcmMDUa7TNCPu1lPd3ZP5jcuiuCsBrQtrTfKUKxX%2FGw%3D&reserved=0</a><br>
</div>
</span></font></div>
</div>
</body>
</html>