[cfe-dev] [OpenMP] offload support for static libraries

Thu Aug 16 16:26:00 PDT 2018

Hi Doru,

Thank you for the detailed description of your changes.

> Regarding your proposal, from your slides I understand that you perform a partial linking step as the first action for all object files and/or static libraries given as input. So this clang invocation:
> clang++ -L. -labc test.cpp -o test
> would result in the same compilation steps as the current Clang version performs because the initial stage of partial linking would have no work to do (since there are no object files present to be partially linked).

No, the intent is to change offload link step to always operate with fat objects and libs, so the compilation part of the action graph for that command should produce temporary fat object which is then passed to the partial linking. I have not described that in slides, but I agree that it is probably not obvious and should have been mentioned.

> Another question I have is regarding the "ld -r" box in your slides.
> How does ld -r work with "bundled" objects? Your diagram seems to imply that ld -r does the concatenation of all device images out of the box. Is this accurate?

Actually the technique that is currently used by the clang-offload-bundler tool for creating bundled (or fat) objects is very similar to what you are going to do for bundling NVPTX and host objects. The difference is in the initial "wrapper" that is created for the device code - you are using C++ structure, while clang-offload-bundler uses LLVM bitcode file. The bundler tool creates a temporary LLVM IR which contains a global initialized array holding the device object, and this array is allocated in a section with predefined name (the name includes offload target triple). This "wrapper" bitcode file is then compiled for the host and then partially linked against the host object.

So technically fat/bundled object is just a host ELF object which has one or more additional ELF section containing device object as data (one extra section per each offloading target). Linker concatenates sections with the same name while linking multiple objects files, so the result of partial linking will have the same named ELF sections holding device code concatenated from all input (fat) objects.

Clang-offload-bundler would need to extract each device object individually while doing unbundling operation on the partially linked object. A possible way to enable this would be creating one more section holding the device object size in addition to existing section with device object in fat/bundled object. That would allow clang bundler to get sizes of device objects that were concatenated by partial linking.

Thanks,
Serguei

From: Gheorghe-Teod Bercea [mailto:Gheorghe-Teod.Bercea at ibm.com]
Sent: Thursday, August 16, 2018 6:15 AM
To: Dmitriev, Serguei N <serguei.n.dmitriev at intel.com>
Cc: 'cfe-dev at lists.llvm.org' <cfe-dev at lists.llvm.org>; Jonas Hahnfeld <hahnjo at hahnjo.de>
Subject: RE: [cfe-dev] [OpenMP] offload support for static libraries

Hi Serguei,

Thanks a lot for the proposal.

My proposal reworks a little bit the way the OpenMP-NVPTX toolchain creates device object files: the device specific part of the object is "wrapped" in an NVLINK-friendly C++ structure that is then compiled for the host. The result is a host object file with a device part which NVLINK can detect (D.o). The D.o object file is then partially linked against the host object file H.o and thus we obtain HD.o. This is required because compilation is required to produce a single output object file (when doing "-c -o" for example). HD.o can now be passed to NVLINK directly or put in a static library and then passed to NVLINK. Either way, NVLINK will be able to detect the device part (due to the special wrapping that we did previously) without the need to "unbundle" the object file (prior to passing it to NVLINK).

The reason why the clang-offload-bundler is not involved in this is because we are using the standard object format for the object file that the OpenMP-NVPTX toolchain outputs so there's no need for a custom format in this case. The partial linking step is required to put together the host and device object files and to ensure that only one object file is produced even if we actually invoked two toolchains (one for host and one for the device).

Regarding your proposal, from your slides I understand that you perform a partial linking step as the first action for all object files and/or static libraries given as input. So this clang invocation:
clang++ -L. -labc test.cpp -o test
would result in the same compilation steps as the current Clang version performs because the initial stage of partial linking would have no work to do (since there are no object files present to be partially linked).

Another question I have is regarding the "ld -r" box in your slides.
How does ld -r work with "bundled" objects? Your diagram seems to imply that ld -r does the concatenation of all device images out of the box. Is this accurate?

Thanks a lot,

--Doru

From:        "Dmitriev, Serguei N" <serguei.n.dmitriev at intel.com<mailto:serguei.n.dmitriev at intel.com>>
To:        Jonas Hahnfeld <hahnjo at hahnjo.de<mailto:hahnjo at hahnjo.de>>
Cc:        "'cfe-dev at lists.llvm.org'" <cfe-dev at lists.llvm.org<mailto:cfe-dev at lists.llvm.org>>, Doru Bercea <gheorghe-teod.bercea at ibm.com<mailto:gheorghe-teod.bercea at ibm.com>>
Date:        08/15/2018 04:27 PM
Subject:        RE: [cfe-dev] [OpenMP] offload support for static libraries
________________________________

Hi Jonas,

I guess this patch implements the proposal which Doru presented on the "OpenMP / HPC in Clang / LLVM Multi-company" meeting. As I remember he suggested to eliminate use of clang-offload-bundler tool when offload target is NTVPTX by replacing bundling operation with partial linking of host and device objects, and then relying of the NVPTX linker to perform the unbundling operation at link phase. Based on Doru's explanations NVPTX linker "knows" how to extract device parts from such objects, so the explicit unbundling operation in not required. Doru, please correct me if my understanding is not fully accurate. Doru's proposal definitely achieves the same goal for NVPTX offloading target (i.e. enables offload in static libraries), but it is NVPTX specific and cannot be extended to other offloading targets (at least that is how it looked like when Doru described it).

I propose slightly different solution which I think should work for any generic OpenMP offload target (it was also discussed on the OpenMP multi-company meeting). In general case we have to use clang-offload-bundler because we cannot assume that device object(s) can be bundled with the host object by performing partial linking of host and device objects. So bundling and unbundling operation will still be done by the clang-offload-bundler tool. The main part of my suggestion is adding partial linking of fat objects (created by offload bundler tool) and static libraries (which are composed of fat objects) and only after that do the unbundling operation on the partially linked object (followed by the appropriate link actions for all offloading devices and then for the host). This would guarantee that device parts of fat objects from static libraries will participate in the device link actions, and thus would enable offloading for static libraries.

Thanks,
Serguei

-----Original Message-----
From: Jonas Hahnfeld [mailto:hahnjo at hahnjo.de]
Sent: Tuesday, August 14, 2018 1:52 PM
To: Dmitriev, Serguei N <serguei.n.dmitriev at intel.com<mailto:serguei.n.dmitriev at intel.com>>
Cc: 'cfe-dev at lists.llvm.org' <cfe-dev at lists.llvm.org<mailto:cfe-dev at lists.llvm.org>>; Doru Bercea <gheorghe-teod.bercea at ibm.com<mailto:gheorghe-teod.bercea at ibm.com>>
Subject: Re: [cfe-dev] [OpenMP] offload support for static libraries

This proposal has already been proposed for NVPTX in https://reviews.llvm.org/D47394, adding Doru.

Cheers,
Jonas

On 2018-08-14 18:43, Dmitriev, Serguei N via cfe-dev wrote:
> PROBLEM OVERVIEW
>
> OpenMP offload functionality is currently not supported in static
> libraries. Because of that an attempt to use offloading in static
> libraries ends up with a fallback execution of target regions on the
> host. This limitation clearly has significant impact on OpenMP offload
> usability.
>
> An output object file that is created by the compiler for offload
> compilation is a fat object. Such object files besides the code for
> the host architecture also contains code for the offloading targets
> which is stored as data in ELF sections with predefined names. Thus, a
> static library that is created from object files produced by offload
> compilation would be an archive of fat objects.
>
> Clang driver currently never passes fat objects directly to any
> toolchain. Instead it performs an unbundling operation for each fat
> object which extract host and device parts from the object. These
> parts are then independently processed by the corresponding target
> toolchains. However, current implementation does not assume that
> static archives may also be composed from fat objects. No unbundling
> is done for static archives (they are passed to linker as is) and thus
> device parts of objects from such archives get ignored.
>
> SUGGESTED SOLUTION
>
> It seems feasible to resolve this problem by changing the offload link
> process - adding an extra step to the link flow which will do a
> partial linking (ld -r) of fat objects and static libraries as shown
> on this diagram
>
> [Fat objects] \                                 / [Target1 link] \
>
>                [Partial linking] - [Unbundling] - [TargetN link] -
> [Host link]
>
> [Static libs] /                                 \--- Host part --/
>
> (You can also look at the .pdf file on this link
> https://drive.google.com/file/d/1ZTNoB-Ghin1BTaiZ312FMSRS6rISDtlr/view
> ?usp=sharing [1] for illustrations for the suggested change)
>
> Linker will pull in all necessary dependencies from static libraries
> while performing partial linking, so the result of partial linking
> would be a fat object with concatenated device parts from input fat
> objects and required dependencies from static libraries. These
> concatenated device objects will be stored in the corresponding ELF
> sections of the partially linked object.
>
> Unbundling operation on the partially linked object will create one or
> more device objects for each offloading target, and these objects will
> be linked by corresponding target toolchains the same way as it is
> done now. Offload bundler tool would require enhancements to support
> unbundling of multiple concatenated device objects for each offloading
> target.
>
> Host link action can be changed to use host part of the partially
> linked object while linking the final image.
>
> Do you see any potential problems in the proposed change?
>
> Links:
> ------
> [1]
> https://drive.google.com/file/d/1ZTNoB-Ghin1BTaiZ312FMSRS6rISDtlr/view
> ?usp=sharing _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org<mailto:cfe-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20180816/ccb21103/attachment.html>