<div dir="ltr"><div>Let me provide a more concrete example of what I had in mind (sorry for the long email...):<br></div><div><br></div><div>Lets assume I have two source files <a href="http://a.cu">a.cu</a> and <a href="http://b.cu">b.cu</a>, both with OpenMP and CUDA kernels and several device functions. Device functions defined in <a href="http://a.cu">a.cu</a> are used in <a href="http://b.cu">b.cu</a> and vice-versa. </div><div><br></div><div>I have a system that consists of a ppc64le host, an nvptx64 device and a ppc64le accelerator. The exact details of this system are not relevant, I am just using this example because it is something supported by the prototype we have in github and exemplifies that the same target can be both host and device (in general we can have AMD GPUs, DSPs, FPGAs, each with its own binary formats). </div><div><br></div><div>###</div><div>Case A: We pass all the source files and compile in a single run of clang.</div><div><br></div><div>A1) clang invocation (the openmp options are just an example, *.cu extension tells the driver we should compile for CUDA as well):</div><div> clang -target ppc64le -fopenmp -fomptargets=nvptx64,ppc64le <a href="http://a.cu">a.cu</a> <a href="http://b.cu">b.cu</a></div><div><br></div><div>A2) driver commands:</div><div> // Obtain cuda obj for <a href="http://a.cu">a.cu</a></div><div> clang -cc1 -fcuda-is-device -target nvptx64 -S <a href="http://a.cu">a.cu</a> -o a-device-cuda-nvptx64.s</div><div> ptxas -c a-device-cuda-nvptx64.s -o a-device-cuda-nvptx64.o</div><div><br></div><div><div> // Obtain openmp nvptx64 obj for <a href="http://a.cu">a.cu</a></div><div> clang -cc1 -fopenmp-is-device -target nvptx64 -S <a href="http://a.cu">a.cu</a> -o a-device-openmp-nvptx64.s</div><div> ptxas -c a-device-openmp-nvptx64.s -o a-device-openmp-nvptx64.o</div></div><div><br></div><div><div> // Obtain openmp ppc64le device obj for <a href="http://a.cu">a.cu</a></div><div> clang -cc1 -fopenmp-is-device -target ppc64le -c <a href="http://a.cu">a.cu</a> -o a-device-openmp-ppc64le.o</div></div><div><br></div><div><div> // Obtain openmp ppc64le host obj for <a href="http://a.cu">a.cu</a></div><div> clang -cc1 -fopenmp -target ppc64le -c <a href="http://a.cu">a.cu</a> -o a-host-cuda-openmp.o</div></div><div><br></div><div> // Repeat everything for <a href="http://b.cu">b.cu</a></div><div><br></div><div> // Obtain cuda device image:</div><div> nvlink a-device-cuda-nvptx64.o b-device-cuda-nvptx64.o -o a.out-device-cuda-nvptx64</div><div><br></div><div> // Obtain openmp nvptx device image:</div><div> nvlink a-device-openmp-nvptx64.o b-device-openmp-nvptx64.o -o a.out-device-openmp-nvptx64</div><div><br></div><div> // Obtain openmp ppc64le device image (making it a shared library is just easier to load, but not required):</div><div> ld -shared a-device-openmp-ppc64le.o b-device-openmp-ppc64le.o -o a.out-device-openmp-ppc64le</div><div><br></div><div> // Create a host compatible object for each device (this is done at link time only, the user is not expected to deal with this tool - the name of the tool and commands are just an example).</div><div> // I am calling this "embed-mode" but is not really embedding, is generate an host object that solely contains device code...</div><div> clang-bundler -embed -host ppc64le -models=cuda,openmp,openmp -targets=nvptx64,nvptx64,ppc64le -inputs=a.out-device-cuda-nvptx64,a.out-device-openmp-nvptx64,a.out-device-openmp-ppc64le -outputs=a.out-device-cuda-nvptx64.o,a.out-device-openmp-nvptx64.o,a.out-device-openmp-ppc64le.o</div><div><br></div><div> // Generate the host binary</div><div> ld a-host-cuda-openmp.o a.out-device-cuda-nvptx64.o a.out-device-openmp-nvptx64.o a.out-device-openmp-ppc64le.o -o a.out</div><div><br></div><div><div>###</div><div>Case B: We do separate compilation.</div></div><div><div>B1) clang invocation (the openmp options are just an example, *.cu extension tells the driver we should compile for CUDA as well):</div><div> clang -c -target ppc64le -fopenmp -fomptargets=nvptx64,ppc64le <a href="http://a.cu">a.cu</a> <a href="http://b.cu">b.cu</a></div></div><div> clang -target ppc64le -fopenmp -fomptargets=nvptx64,ppc64le a.o b.o</div><div><br></div><div>B2) driver commands - 1st run:</div><div><br></div><div> // Same thing as in A2) up to the nvlink command</div><div><br></div><div> // Create the bundles a.o and b.o so that the user sees a single file.</div><div> clang-bundler -bundle -models=host,cuda,openmp,openmp -targets=nvptx64,nvptx64,ppc64le -inputs=a-host-cuda-openmp.o,a-device-cuda-nvptx64.o,a-device-openmp-nvptx64.o,a-device-openmp-ppc64le.o -outputs=a.o</div><div> clang-bundler -bundle -models=host,cuda,openmp,openmp -targets=nvptx64,nvptx64,ppc64le -inputs=b-host-cuda-openmp.o,b-device-cuda-nvptx64.o,b-device-openmp-nvptx64.o,b-device-openmp-ppc64le.o -outputs=b.o<br></div><div><br></div><div>B2) driver commands - 2nd run:</div><div> // Attempt to unbundle the inputs because the user is asking for offloading support and they are not source files. If the bundler understands the file is a bundle it creates the individual files, otherwise create empty files for the devices.</div><div> clang-bundler -unbundle -models=host,cuda,openmp,openmp -targets=nvptx64,nvptx64,ppc64le -outputs =a-host-cuda-openmp.o,a-device-cuda-nvptx64.o,a-device-openmp-nvptx64.o,a-device-openmp-ppc64le.o -inputs=a.o</div><div> clang-bundler -unbundle -models=host,cuda,openmp,openmp -targets=nvptx64,nvptx64,ppc64le -outputs =b-host-cuda-openmp.o,b-device-cuda-nvptx64.o,b-device-openmp-nvptx64.o,b-device-openmp-ppc64le.o -inputs=b.o</div><div><br></div><div> // Same thing in A2 starting from the nvlink commands.</div><div><br></div><div>###</div><div>A few more comments/observations:</div><div><br></div><div>- I don't have any other tool that would be compatible with a bundle of all four object files, doesn't matter how I do it, this is just not supported by any other compiler and I'm not doing anything wrong as per the programming models specification. But, I have tools compatible with each object. That's is way I agree we should give the user the ability to extract objects. I agree with Justin that extracting stuff from tar is more convenient that doing it from ELF (I don't think most users will now how to do it from ELF, and it is very likely they don't even know what ELF is, for them is just a binary, so not exposing these details is desirable). I am not trying to push for a specific format, I just think it should be target agnostic.</div><div><br></div><div>- Creating linkable host objects shouldn't be that hard (clang-bundler -embed). For CUDA I can create an IR file that defines "__cuda_fatbin_wrapper" with the bytes coming from the fat binary (that global would have to be made external though). For OpenMP it would do something similar but for ".omp_offloading.img_start/end.<device>". The resulting IR could then be converted into an object by calling the host backend. The tool itself wouldn't care about obj formats.</div><div><br></div><div>- If the bundler uses a target agnostic format, my example above would just work regardless of the compiler phases -- -S, -emit-ast, -emit-llvm, -emit-ast would work just fine. </div><div><br></div><div>- Extending the bundler to enable compatibility modes can then be discussed on a per-case basis if there are a lot of users asking for that feature. If that is to be supported, someone with experience with that format could contribute it and there is nothing in the design that prevents it. Expecting to have something that works for all formats obj formats and device combinations from the very beginning seems not reasonable. My plan of action was to contribute something that works and can scale to as many use cases as possible, I can't however contribute specific code for every possible device, in particular those that are not supported in clang yet. </div><div><br></div><div>- The user does not have to deal with the bundling tool at any time if we adopt a format that makes it easy to extract objects. I don't know if one should avoid completely having a tool (again the user doesn't have to use it, but it can look it up in the install tree of clang). An alternative is supporting something like clang -ccbundle. Maybe other people are aware of what are the precedents in clang?</div><div><br></div><div>About shared libraries: I am not sure what is the concern with shared libraries. They would be created as binaries would, and would contain bundles on them. Nothing that is proposed here precludes that. Supporting device-side shared libraries depends on whether the runtime library implements a loader or not. It is very possible I am missing something here...</div><div><br></div><div>Thanks!</div><div>Samuel</div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-03-15 14:36 GMT-04:00 Eric Christopher via cfe-dev <span dir="ltr"><<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><br><div class="gmail_quote"><span class=""><div dir="ltr">On Tue, Mar 15, 2016 at 12:25 AM C Bergström <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Afaik - The way that each vendor *embeds* is a little different and<br>
non-conflicting. (AMD vs NV vs Intel) Embedding for each vendor in a<br>
single object may work. Wouldn't this achieve the desired goal without<br>
the bundling? Bundling is a non-solution - it's just delays dealing<br>
with the offload target code, at which point you have to extract and<br>
jump around to make sure it all magically works (fragile - This side<br>
up)<br>
<br></blockquote><div><br></div></span><div>I'm somewhat in agreement here. An alternate idea is to just "do what we've always done" and have compatibility modes where we try to do what's expected for the particular end use that we're trying to copy. That would get us maximum flexibility (at the cost of a number of implementations, but abstraction hopefully ftw?).</div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
With embedding approach we have all the pieces to at least read/embed<br>
- it's just dealing with elf/macho/pxe objects and data sections.<br>
<br></blockquote><div><br></div></span><div>Agreed.</div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Also there's the question of how to deal with shared libraries - Even<br>
if initially it's not a concern, eventually someone will request it.<br>
If this isn't ever going to be supported or a concern it's not valid<br>
of course..<br>
<br></blockquote><div><br></div></span><div>I'm sure someone is going to want it eventually. </div><div><br></div><div>-eric</div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Tue, Mar 15, 2016 at 2:54 PM, Justin Lebar via cfe-dev<br>
<<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<br>
>> I agree with Justin when he says that supporting all possible combinations of host-device bundling, each with a custom format adds a lot of complexity and does not necessarily helps the user.<br>
><br>
> Maybe it would be helpful to speak concretely about this. What is a<br>
> specific case you're concerned about?<br>
><br>
> Frankly if we try to build a thing that does everything -- both embeds<br>
> and bundles -- I'm concerned that will be much more complex than<br>
> either approach by itself. But maybe there's something about bundling<br>
> code for many different devices that's qualitatively different than<br>
> bundling code for one or just a few devices? For example, are there<br>
> extant formats that we would like to be compatible with that are<br>
> incompatible with each other? Again, I think it would be helpful to<br>
> be concrete.<br>
><br>
>> Using a header compatible with tar is not something hard to do, having the bundler tool as part of clang ensures that it works across operating systems too.<br>
><br>
> This is a minor point, but -- without getting myself back into the<br>
> minefield of discussing whether using the tar format is or is not good<br>
> -- if we do use said format, given what I understand at the moment,<br>
> I'm not in favor of building our own user-facing tool, if that's<br>
> possible. If you're on Windows and want to inspect a tar file you<br>
> can, download 7z. It's no different than downloading objdump. But we<br>
> can postpone this discussion until later; I suspect it won't be<br>
> necessary, given the pushback to using tar in the first place.<br>
><br>
>> this leaves room to have custom-format bundling activated with options for the cases (would have to be evaluated one by one) that would greatly benefit of interoperability.<br>
><br>
> This is, as I read it, exactly what everyone else has been arguing so<br>
> strenuously against (and what I conceded to).<br>
><br>
> On Mon, Mar 14, 2016 at 7:25 PM, Samuel F Antao <<a href="mailto:sfantao@us.ibm.com" target="_blank">sfantao@us.ibm.com</a>> wrote:<br>
>> Hi all,<br>
>><br>
>> I decided a take shot on a possible implementation for the part of this<br>
>> proposal that I think is more consensual (i.e. the part that does not relate<br>
>> with the bundling). I posted three patches (<a href="http://reviews.llvm.org/D18170" rel="noreferrer" target="_blank">http://reviews.llvm.org/D18170</a>,<br>
>> <a href="http://reviews.llvm.org/D18171" rel="noreferrer" target="_blank">http://reviews.llvm.org/D18171</a>, <a href="http://reviews.llvm.org/D18172" rel="noreferrer" target="_blank">http://reviews.llvm.org/D18172</a>) with a<br>
>> possible implementation, so that we have something more concrete to discuss.<br>
>> Let me know your thoughts.<br>
>><br>
>> Going back to the bundling discussion:<br>
>><br>
>> I agree with Justin when he says that supporting all possible combinations<br>
>> of host-device bundling, each with a custom format adds a lot of complexity<br>
>> and does not necessarily helps the user. Therefore, I think reasonable to<br>
>> have intermediate files bundled in some already existent format (say tar)<br>
>> that is agnostic of the programming model. Actually, that was my motivation<br>
>> when I proposed the custom format in the bundler.<br>
>><br>
>> When I look at all the different concerns I think that a possible solution<br>
>> is to have a bundler with three operation modes:<br>
>> i) "Embed": it generates a host object that contains the device image and<br>
>> properly defines the symbols a programming model requires. Therefore it can<br>
>> be linked with host objects successfully. This file is never exposed to the<br>
>> user unless save-temps is used.<br>
>> ii) "Bundle": Combines host and device object using, by default, a format<br>
>> easy to interact with that is agnostic of the programming model.<br>
>> iii) "Unbundle": The inverse of ii), it assumes the input uses that default<br>
>> format.<br>
>><br>
>> Using a header compatible with tar is not something hard to do, having the<br>
>> bundler tool as part of clang ensures that it works across operating systems<br>
>> too. At the same time, this leaves room to have custom-format bundling<br>
>> activated with options for the cases (would have to be evaluated one by one)<br>
>> that would greatly benefit of interoperability.<br>
>><br>
>> Does this sound reasonable?<br>
>><br>
>> Thanks!<br>
>> Samuel<br>
>><br>
>> 2016-03-10 13:59 GMT-05:00 Justin Lebar via cfe-dev<br>
>> <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>>:<br>
>>><br>
>>> > Justin, is this convincing enough?<br>
>>><br>
>>> Okay, okay. Uncle.<br>
>>><br>
>>> There are two things here that I find convincing.<br>
>>><br>
>>> 1) Although we're not going to be compatible with the details of some,<br>
>>> if not all, other compilers' formats, we can at least be compatible<br>
>>> with the spirit by using object files as opposed to tar.<br>
>>><br>
>>> 2) The postscript in Andrey's e-mail:<br>
>>><br>
>>> > Re: objdump doesn't understand ELF format with code for multiple<br>
>>> > targets. The same is true for fat executable files as well, isn't? So if we<br>
>>> > need to teach objdump how to recognize fat files, we already have this<br>
>>> > problem.<br>
>>><br>
>>> It's probably much more important that objdump work on executables<br>
>>> than on object files, since if you have object files, you can probably<br>
>>> recompile with -save-temps, but if you only have an executable, you<br>
>>> don't necessarily have access to intermediate files, or even a<br>
>>> compiler for the relevant architecture, much less the specific<br>
>>> compiler which generated the executable.<br>
>>><br>
>>> Shoving device code into the host *executable* seems unavoidable. I'm<br>
>>> still not thrilled with doing the same for object files -- it still<br>
>>> feels like we're using ELF when we actually want an archive format --<br>
>>> but (1) makes it less bad.<br>
>>><br>
>>> -Justin<br>
>>><br>
>>> On Wed, Mar 9, 2016 at 5:59 AM, Andrey Bokhanko<br>
>>> <<a href="mailto:andreybokhanko@gmail.com" target="_blank">andreybokhanko@gmail.com</a>> wrote:<br>
>>> > All,<br>
>>> ><br>
>>> > I asked Intel GCC guys who implemented OpenMP offloading support in GCC,<br>
>>> > and<br>
>>> > as they told me, GCC also employs option #4 from Hal's list -- it puts<br>
>>> > both<br>
>>> > host and target code in a single ELF file. "Code" in GCC case is always<br>
>>> > GCC's IR (Gimple), though -- they require GCC invocation from linker in<br>
>>> > order to produce a multi-target executable. This makes GCC<br>
>>> > non-interoperable<br>
>>> > with any other offloading compiler and effectively produces its own<br>
>>> > standard.<br>
>>> ><br>
>>> > Thus, prior art from:<br>
>>> > * nvcc<br>
>>> > * Pathscale<br>
>>> > * GCC<br>
>>> > * ICC<br>
>>> ><br>
>>> > indicates only one direction -- compiler driver produces a single object<br>
>>> > file with target code embedded in data section.<br>
>>> ><br>
>>> > Justin, is this convincing enough? I don't see any good reasons why<br>
>>> > clang<br>
>>> > should go against what every other compiler on the planet does.<br>
>>> ><br>
>>> > Re: objdump doesn't understand ELF format with code for multiple<br>
>>> > targets.<br>
>>> > The same is true for fat executable files as well, isn't? So if we need<br>
>>> > to<br>
>>> > teach objdump how to recognize fat files, we already have this problem.<br>
>>> ><br>
>>> > Yours,<br>
>>> > Andrey<br>
>>> > =====<br>
>>> > Software Engineer<br>
>>> > Intel Compiler Team<br>
>>> ><br>
>>> ><br>
>>> _______________________________________________<br>
>>> cfe-dev mailing list<br>
>>> <a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
>>> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
>><br>
>><br>
> _______________________________________________<br>
> cfe-dev mailing list<br>
> <a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
</blockquote></div></div></div></div>
<br>_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
<br></blockquote></div><br></div>