[clang] [Clang][SYCL] Introduce clang-sycl-link-wrapper to link SYCL offloading device code (Part 1 of many) (PR #112245)

[Arvind Sudarsanam via cfe-commits]

asudarsa wrote:

Hi @llvm-beanz and @jhuber6 

Thanks so much for all the feedback and discussion here. I am very much grateful for both your time and insights here. I will try to give my answers in this comment.
An overview of the SYCL compilation flow (with a bit more emphasis on the device code linking phases) can be seen in the RFC that we submitted earlier. 
RFC: https://discourse.llvm.org/t/rfc-offloading-design-for-sycl-offload-kind-and-spir-targets/74088

As shown in the RFC, SYCL device code linking process involves the following steps:
1. Linking of all LLVM IR bitcode device objects found in user code. llvm-link is used here.
2. Linking of fully linked LLVM IR device bitcode found in user code (output of the previous step) with SYCL specific device library files. llvm-link is used here.
3. Post-link steps: Some steps are required to conform to SYCL specification. Some kernels have kernel metadata (known as SYCL properties) that need to be communicated to SYCL runtime. This requires processing of LLVM IR metadata and generating some information that can be embedded into the device image and then processed by SYCL runtime. Also, we support device code splitting which is mostly done to improve run-time performance and also ensure SYCL spec conformance in some cases.
4. For SPIR-V based JIT and AOT targets, SPIR-V IR code generation is then performed using an external tool (llvm-spirv)
5. For AOT targets (both SPIR-V and non SPIR-V), we then invoke backend compilation tools (For e.g. ocloc for Intel graphics).

Quite a few of these steps have been fashioned to work around the fact that we do not have a regular SPIR-V backend in LLVM repo. Once the SPIR-V backend is ready to be included in compilation flows, we can remove the dependence on llvm-link and external tools like llvm-spirv and use the backend for LTO based linking and SPIR-V code generation.

We initially envisioned this flow to be embedded into clang-linker-wrapper. Due to specific variations from the community flow, especially the need for SYCL-specific post-link steps, we decided to move the logic into a stand-alone tool to do SYCL specific linking.

A primary reason for making this a clang-based tool was that it is quite similar in its functionality to existing tools like clang-linker-wrapper and clang-nvlink-wrapper. As @jhuber6 mentioned, I think it can be easily moved to llvm/tools if that's a blocking issue.

As far as testing support goes, the test that has been added in this PR 'clang-sycl-linker-test.cpp' covers the testing of an very initial flow (linking of source llvm IR device bitcodes, linking of device library files, and spir-v code generation). This test is expected to grow as we add other pieces (post-link steps, AOT compilation support, etc).

I can provide pointers to some extended documentation if the RFC is not in sufficient detail.

Thanks again for great review comments and feedback here.
Sincerely

https://github.com/llvm/llvm-project/pull/112245