<div dir="ltr">Hi Ronan,<div><br></div><div>Thanks for the feedback!</div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">2016-03-03 5:50 GMT-05:00 Ronan KERYELL via cfe-dev <span dir="ltr"><<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">>>>>> On Wed, 24 Feb 2016 19:01:31 -0500, Samuel F Antao via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>> said:<br>
<br>
Samuel> Hi all,<br>
<br>
Hi Samuel!<br>
<br>
Samuel> I’d like to propose a change in the Driver implementation<br>
Samuel> to support programming models that require offloading with a<br>
Samuel> unified infrastructure. The goal is to have a design that<br>
Samuel> is general enough to cover different programming models with<br>
Samuel> as little as possible customization that is<br>
Samuel> programming-model specific. Some of this discussion already<br>
Samuel> took place in <a href="http://reviews.llvm.org/D9888" rel="noreferrer" target="_blank">http://reviews.llvm.org/D9888</a> but would like<br>
Samuel> to continue that here in he mailing list and try to collect<br>
Samuel> as much feedback as possible.<br>
<br>
Samuel> Currently, there are two programming models supported by<br>
Samuel> clang that require offloading - CUDA and OpenMP. Examples of<br>
Samuel> other offloading models that can could benefit of a unified<br>
Samuel> driver design as they become supported in clang are also<br>
Samuel> SYCL (<a href="https://www.khronos.org/sycl" rel="noreferrer" target="_blank">https://www.khronos.org/sycl</a>) and OpenACC<br>
Samuel> (<a href="http://www.openacc.org/" rel="noreferrer" target="_blank">http://www.openacc.org/</a>).<br>
<br>
Great proposal!<br>
<br>
Very à propos since I am just thinking about implementing it with Clang<br>
in my SYCL implementation (see<br>
<a href="https://github.com/amd/triSYCL#possible-futures" rel="noreferrer" target="_blank">https://github.com/amd/triSYCL#possible-futures</a> for possible way I am<br>
thinking of).<br></blockquote><div><br></div><div>That's great what I am proposing is aligned with your project needs!</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Samuel> OpenMP (Host IR has to be read by the device to determine<br>
Samuel> which declarations have to be emitted and the device binary<br>
Samuel> is embedded in the host binary at link phase through a<br>
Samuel> proper linker script):<br>
<br>
Samuel> Src -> Host PP -> A<br>
<br>
Samuel> A -> HostCompile -> B<br>
<br>
Samuel> A,B -> DeviceCompile -> C<br>
<br>
Samuel> C -> DeviceAssembler -> D<br>
<br>
Samuel> E -> DeviceLinker -> F<br>
<br>
Samuel> B -> HostAssembler -> G<br>
<br>
Samuel> G,F -> HostLinker -> Out<br>
<br>
In SYCL it would be pretty close. Something like:<br>
<span class=""><br>
Src -> Host PP -> A<br>
<br>
A -> HostCompile -> B<br>
<br>
</span>B -> HostAssembler -> C<br>
<br>
Src -> Device PP -> D<br>
<br>
D -> DeviceCompile -> E<br>
<br>
E -> DeviceAssembler -> F<br>
<br>
F -> DeviceLinker -> G<br>
<br>
C,G -> HostLinker -> Out<br></blockquote><div><br></div><div>The idea of the driver design is to allow you to use the DAG that is more appropriate for your programming model without having to touch parts of the infrastructure that do not necessarily relate with offloading. Your DAG should be easily supported by the design.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Samuel> As an hypothetical example, lets assume we wanted to compile<br>
Samuel> code that uses both CUDA for a nvptx64 device, OpenMP for an<br>
Samuel> x86_64 device, and a powerpc64le host, one could invoke the<br>
Samuel> driver as:<br>
<br>
Samuel> clang -target powerpc64le-ibm-linux-gnu <more host options><br>
<br>
Samuel> -target-offload=nvptx64-nvidia-cuda -fcuda -mcpu sm_35 <more<br>
Samuel> options for the nvptx toolchain><br>
<br>
Samuel> -target-offload=x86_64-pc-linux-gnu -fopenmp <more options<br>
Samuel> for the x86_64 toolchain><br>
<br>
Just to be sure to understand: you are thinking about being able to<br>
outline several "languages" at once, such as CUDA *and* OpenMP, right ?<br></blockquote><div><br></div><div>That's correct. All the toolchains foreach programming model would be extracted from the input arguments before getting into the commands generation.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I think it is required for serious applications. For example, in the HPC<br>
world, it is common to have hybrid multi-node heterogeneous applications<br>
that use MPI+OpenMP+OpenCL for example. Since MPI and OpenCL are just<br>
libraries, there is only OpenMP to off-load here. But if we move to<br>
OpenCL SYCL instead with MPI+OpenMP+SYCL then both OpenMP and SYCL have<br>
to be managed by the Clang off-loading infrastructure at the same time<br>
and be sure they combine gracefully...<br></blockquote><div><br></div><div>Yes SYCL could coexist with OpenMP, and OpenMP should coexist with CUDA. I assume you are talking about the clang-driver infrastructure, because for the codegen infrastructure, each programming model will have its own specifics. </div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I think your second proposal about (un)bundling can already manage this.<br>
<br>
Otherwise, what about the code outlining itself used in the off-loading<br>
process? The code generation itself requires to outline the kernel code<br>
to some external functions to be compiled by the kernel compiler. Do you<br>
think it is up to the programmer to re-use the recipes used by OpenMP<br>
and CUDA for example or it would be interesting to have a third proposal<br>
to abstract more the outliner to be configurable to handle globally<br>
OpenMP, CUDA, SYCL...?<br></blockquote><div><br></div><div>The code generation has to be implemented/tuned for the programming model. The driver itself can only help in the outlining by forwarding the right files to the right place. </div><div><br></div><div>In codegen there is some sort of filtering going on. The AST for host an device is similar. In the device frontend, for CUDA (and OpenCL I believe) the declarations that don't have the device attribute are just ignored - basically the outlining is already done, only have to device what is device/host. In OpenMP is more complicated because the model allows offloading to fail, so the host and device codegen have to match- that's why the host IR is passed to the device frontend so that it makes sure it matches the host implementation in terms of outlined kernels (the host produces metadata to ease this process).</div><div><br></div><div>Having said that, I think that it would be hard to have something very different from what we have already in clang, so I suspect different programming models will fall in one of these two categories: explicit outlining (CUDA OpenCL) and implicit outlining (OpenMP). For both categories, the way device attributes are specified in the frontend or the metadata used to pass information from host to device can be adapted to better suit multiple programing models. However, this is a separate discussion from the driver support.</div><div> </div><div>Thanks again,</div><div>Samuel</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Thanks a lot,<br>
<span class="HOEnZb"><font color="#888888">--<br>
Ronan KERYELL<br>
Xilinx Research Labs, Dublin, Ireland<br>
</font></span><div class="HOEnZb"><div class="h5">_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
</div></div></blockquote></div><br></div></div>