[cfe-dev] [OpenMP] offload support for static libraries

[Dmitriev, Serguei N via cfe-dev]

Hi Jonas,

I guess this patch implements the proposal which Doru presented on the "OpenMP / HPC in Clang / LLVM Multi-company" meeting. As I remember he suggested to eliminate use of clang-offload-bundler tool when offload target is NTVPTX by replacing bundling operation with partial linking of host and device objects, and then relying of the NVPTX linker to perform the unbundling operation at link phase. Based on Doru's explanations NVPTX linker "knows" how to extract device parts from such objects, so the explicit unbundling operation in not required. Doru, please correct me if my understanding is not fully accurate. Doru's proposal definitely achieves the same goal for NVPTX offloading target (i.e. enables offload in static libraries), but it is NVPTX specific and cannot be extended to other offloading targets (at least that is how it looked like when Doru described it).

I propose slightly different solution which I think should work for any generic OpenMP offload target (it was also discussed on the OpenMP multi-company meeting). In general case we have to use clang-offload-bundler because we cannot assume that device object(s) can be bundled with the host object by performing partial linking of host and device objects. So bundling and unbundling operation will still be done by the clang-offload-bundler tool. The main part of my suggestion is adding partial linking of fat objects (created by offload bundler tool) and static libraries (which are composed of fat objects) and only after that do the unbundling operation on the partially linked object (followed by the appropriate link actions for all offloading devices and then for the host). This would guarantee that device parts of fat objects from static libraries will participate in the device link actions, and thus would enable offloading for static libraries.

Thanks,
Serguei

-----Original Message-----
From: Jonas Hahnfeld [mailto:hahnjo at hahnjo.de] 
Sent: Tuesday, August 14, 2018 1:52 PM
To: Dmitriev, Serguei N <serguei.n.dmitriev at intel.com>
Cc: 'cfe-dev at lists.llvm.org' <cfe-dev at lists.llvm.org>; Doru Bercea <gheorghe-teod.bercea at ibm.com>
Subject: Re: [cfe-dev] [OpenMP] offload support for static libraries

This proposal has already been proposed for NVPTX in https://reviews.llvm.org/D47394, adding Doru.

Cheers,
Jonas

On 2018-08-14 18:43, Dmitriev, Serguei N via cfe-dev wrote:
> PROBLEM OVERVIEW
> 
> OpenMP offload functionality is currently not supported in static 
> libraries. Because of that an attempt to use offloading in static 
> libraries ends up with a fallback execution of target regions on the 
> host. This limitation clearly has significant impact on OpenMP offload 
> usability.
> 
> An output object file that is created by the compiler for offload 
> compilation is a fat object. Such object files besides the code for 
> the host architecture also contains code for the offloading targets 
> which is stored as data in ELF sections with predefined names. Thus, a 
> static library that is created from object files produced by offload 
> compilation would be an archive of fat objects.
> 
> Clang driver currently never passes fat objects directly to any 
> toolchain. Instead it performs an unbundling operation for each fat 
> object which extract host and device parts from the object. These 
> parts are then independently processed by the corresponding target 
> toolchains. However, current implementation does not assume that 
> static archives may also be composed from fat objects. No unbundling 
> is done for static archives (they are passed to linker as is) and thus 
> device parts of objects from such archives get ignored.
> 
> SUGGESTED SOLUTION
> 
> It seems feasible to resolve this problem by changing the offload link 
> process - adding an extra step to the link flow which will do a 
> partial linking (ld -r) of fat objects and static libraries as shown 
> on this diagram
> 
> [Fat objects] \                                 / [Target1 link] \
> 
>                [Partial linking] - [Unbundling] - [TargetN link] - 
> [Host link]
> 
> [Static libs] /                                 \--- Host part --/
> 
> (You can also look at the .pdf file on this link 
> https://drive.google.com/file/d/1ZTNoB-Ghin1BTaiZ312FMSRS6rISDtlr/view
> ?usp=sharing [1] for illustrations for the suggested change)
> 
> Linker will pull in all necessary dependencies from static libraries 
> while performing partial linking, so the result of partial linking 
> would be a fat object with concatenated device parts from input fat 
> objects and required dependencies from static libraries. These 
> concatenated device objects will be stored in the corresponding ELF 
> sections of the partially linked object.
> 
> Unbundling operation on the partially linked object will create one or 
> more device objects for each offloading target, and these objects will 
> be linked by corresponding target toolchains the same way as it is 
> done now. Offload bundler tool would require enhancements to support 
> unbundling of multiple concatenated device objects for each offloading 
> target.
> 
> Host link action can be changed to use host part of the partially 
> linked object while linking the final image.
> 
> Do you see any potential problems in the proposed change?
> 
> Links:
> ------
> [1]
> https://drive.google.com/file/d/1ZTNoB-Ghin1BTaiZ312FMSRS6rISDtlr/view
> ?usp=sharing _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev