[all-commits] [llvm/llvm-project] 37d056: [Clang] Introduce 'clang-nvlink-wrapper' to work a...

[Joseph Huber via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 37d0568a6593adfe791c1327d99731050540e97a
      https://github.com/llvm/llvm-project/commit/37d0568a6593adfe791c1327d99731050540e97a
  Author: Joseph Huber <huberjn at outlook.com>
  Date:   2024-07-22 (Mon, 22 Jul 2024)

  Changed paths:
    A clang/docs/ClangNVLinkWrapper.rst
    M clang/docs/index.rst
    M clang/lib/Driver/ToolChains/Cuda.cpp
    M clang/lib/Driver/ToolChains/Cuda.h
    M clang/test/Driver/cuda-cross-compiling.c
    A clang/test/Driver/nvlink-wrapper.c
    M clang/test/lit.cfg.py
    M clang/tools/CMakeLists.txt
    A clang/tools/clang-nvlink-wrapper/CMakeLists.txt
    A clang/tools/clang-nvlink-wrapper/ClangNVLinkWrapper.cpp
    A clang/tools/clang-nvlink-wrapper/NVLinkOpts.td

  Log Message:
  -----------
  [Clang] Introduce 'clang-nvlink-wrapper' to work around 'nvlink' (#96561)

Summary:
The `clang-nvlink-wrapper` is a utility that I removed awhile back
during the transition to the new driver. This patch adds back in a new,
upgraded version that does LTO + archive linking. It's not an easy
choice to reintroduce something I happily deleted, but this is the only
way to move forward with improving GPU support in LLVM.

While NVIDIA provides a linker called 'nvlink', its main interface is
very difficult to work with. It does not provide LTO, or static linking,
requires all files to be named a non-standard `.cubin`, and rejects link
jobs that other linkers would be fine with (i.e empty). I have spent a
great deal of time hacking around this in the GPU `libc` implementation,
where I deliberately avoid LTO and static linking and have about 100
lines of hacky CMake dedicated to storing these files in a format that
the clang-linker-wrapper accepts to avoid this limitation.

The main reason I want to re-intorudce this tool is because I am
planning on creating a more standard C/C++ toolchain for GPUs to use.
This will install files like the following.
```
<install>/lib/nvptx64-nvidia-cuda/libc.a
<install>/lib/nvptx64-nvidia-cuda/libc++.a
<install>/lib/nvptx64-nvidia-cuda/libomp.a
<install>/lib/clang/19/lib/nvptx64-nvidia-cuda/libclang_rt.builtins.a
```
Linking in these libraries will then simply require passing `-lc` like
is already done for non-GPU toolchains. However, this doesn't work with
the currently deficient `nvlink` linker, so I consider this a blocking
issue to massively improving the state of building GPU libraries.

In the future we may be able to convince NVIDIA to port their linker to
`ld.lld`, but for now this is the only workable solution that allows us
to hack around the weird behavior of their closed-source software.
This also copies some amount of logic from the clang-linker-wrapper,
but not enough for it to be worthwhile to merge them I feel. In the
future it may be possible to delete that handling from there entirely.



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications