<table border="1" cellspacing="0" cellpadding="8">
<tr>
<th>Issue</th>
<td>
<a href=https://github.com/llvm/llvm-project/issues/72734>72734</a>
</td>
</tr>
<tr>
<th>Summary</th>
<td>
[clang17] -nostdinc++ and -x cuda break cuda_wrappers
</td>
</tr>
<tr>
<th>Labels</th>
<td>
new issue
</td>
</tr>
<tr>
<th>Assignees</th>
<td>
</td>
</tr>
<tr>
<th>Reporter</th>
<td>
AustinSchuh
</td>
</tr>
</table>
<pre>
I'm using `-nostdinc++`, `--sysroot` and `-isystem` to compile CUDA code against a sysroot with clang. When I do this, I get an error about `__noinline__`.
```
In file included from frc971/orin/cuda.cc:3:
In file included from external/com_github_nvidia_cuco/include/cuco/pair.cuh:19:
In file included from external/com_github_nvidia_cuco/include/cuco/detail/traits.hpp:18:
In file included from external/com_github_nvidia_cccl/thrust/thrust/device_reference.h:33:
In file included from external/com_github_nvidia_cccl/thrust/thrust/device_ptr.h:33:
In file included from external/com_github_nvidia_cccl/thrust/thrust/memory.h:32:
In file included from external/com_github_nvidia_cccl/thrust/thrust/detail/type_traits/pointer_traits.h:31:
In file included from external/com_github_nvidia_cccl/thrust/thrust/iterator/iterator_traits.h:70:
In file included from external/com_github_nvidia_cccl/thrust/thrust/iterator/detail/device_system_tag.h:31:
In file included from external/com_github_nvidia_cccl/thrust/thrust/system/cuda/detail/execution_policy.h:43:
In file included from external/com_github_nvidia_cccl/thrust/thrust/system/cuda/config.h:50:
In file included from external/com_github_nvidia_cccl/cub/cub/util_debug.cuh:39:
In file included from external/com_github_nvidia_cccl/cub/cub/config.cuh:46:
In file included from external/com_github_nvidia_cccl/cub/cub/util_arch.cuh:47:
In file included from external/com_github_nvidia_cccl/cub/cub/util_macro.cuh:48:
In file included from external/com_github_nvidia_cccl/libcudacxx/include/cuda/std/utility:18:
In file included from external/com_github_nvidia_cccl/libcudacxx/include/cuda/std/detail/libcxx/include/utility:274:
In file included from external/com_github_nvidia_cccl/libcudacxx/include/cuda/std/detail/libcxx/include/__memory/construct_at.h:49:
In file included from external/amd64_debian_sysroot//usr/include/c++/12/memory:77:
In file included from external/amd64_debian_sysroot//usr/include/c++/12/bits/shared_ptr.h:53:
external/amd64_debian_sysroot//usr/include/c++/12/bits/shared_ptr_base.h:196:22: error:
expected expression
external/amd64_debian_sysroot//usr/include/crt/host_defines.h:83:33: note:
expanded from macro '__noinline__'
83 | __attribute__((noinline))
| ^
In file included from frc971/orin/cuda.cc:4:
In file included from external/com_github_nvidia_cuco/include/cuco/static_map.cuh:22:
external/com_github_nvidia_cuco/include/cuco/detail/static_map_kernels.cuh:56:5: error:
no matching 'operator new' function for non-allocating placement new expression; include <new>
56 | new (&slots[idx].first) atomic_key_type{k};
| ^~~
external/com_github_nvidia_cuco/include/cuco/detail/static_map.inl:49:11: note:
in instantiation of function template specialization 'cuco::detail::initialize<256UL,
cuda::atomic<unsigned long, cuda::std::__detail::thread_scope_device>, cuda::atomic<unsigned
long, cuda::std::__detail::thread_scope_device>, unsigned long, unsigned long,
cuco::pair<cuda::atomic<unsigned long, cuda::std::__detail::thread_scope_device>,
cuda::atomic<unsigned long, cuda::std::__detail::thread_scope_device>>>' requested here
49 | detail::initialize<block_size, atomic_key_type, atomic_mapped_type>
| ^
frc971/orin/cuda.cc:269:40: note: in instantiation of member function
'cuco::static_map<unsigned long, unsigned long>::static_map' requested here
269 | cuco::static_map<uint64_t, uint64_t> map(10000, empty_key_sentinel,
| ^
In file included from frc971/orin/cuda.cc:4:
In file included from external/com_github_nvidia_cuco/include/cuco/static_map.cuh:22:
external/com_github_nvidia_cuco/include/cuco/detail/static_map_kernels.cuh:57:5: error:
no matching 'operator new' function for non-allocating placement new expression; include <new>
57 | new (&slots[idx].second) atomic_mapped_type{v};
| ^~~
5 errors generated when compiling for sm_80.
```
>From reading online, this is because `cuda_wrappers` isn't wrapping the headers correctly. Digging in, `-v` shows that `cuda_wrappers` ends up really late in the system search path.
```
#include <...> search starts here:
bazel-out/k8-opt/bin/external/com_github_google_glog/_virtual_includes/glog
bazel-out/k8-opt/bin/external/com_github_gflags_gflags/_virtual_includes/gflags
external/amd64_debian_sysroot//usr/include/c++/12
external/amd64_debian_sysroot//usr/include/x86_64-linux-gnu/c++/12
external/amd64_debian_sysroot//usr/include/c++/12/backward
external/llvm_k8/lib/clang/17/include
external/amd64_debian_sysroot//usr/include/x86_64-linux-gnu
external/amd64_debian_sysroot//usr/include
external/com_github_nvidia_cccl/cub
external/com_github_nvidia_cccl/libcudacxx/include
external/com_github_nvidia_cccl/thrust
external/com_github_nvidia_cuco/include
external/llvm_k8/lib/clang/17/include/cuda_wrappers
external/amd64_debian_sysroot/usr/lib/cuda//include
```
I can fix this by managing `cuda_wrappers` myself in my compilation invocation, but this smells like a bug. `cuda_wrappers` should come first.
(full command line is:
```
clang '--target=x86_64-unknown-linux-gnu' -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=2' -ggdb3 -fstack-protector -fno-omit-frame-pointer -fcolor-diagnostics -Wall -Wthread-safety -Wself-assign -Bexternal/llvm_k8/bin/ -resource-dir external/llvm_k8/lib/clang/17 -D__STDC_FORMAT_MACROS -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS '-D_FILE_OFFSET_BITS=64' '-fmessage-length=100' '-fmacro-backtrace-limit=0' -ggdb3 -O2 -DNDEBUG -ffunction-sections -fdata-sections '-DAOS_DEBUG=0' -isystem external/llvm_k8/lib/clang/17/include/cuda_wrappers '-std=c++20' -nostdinc++ -isystem external/amd64_debian_sysroot//usr/include/c++/12 -isystem external/amd64_debian_sysroot//usr/include/x86_64-linux-gnu/c++/12 -isystem external/amd64_debian_sysroot//usr/include/c++/12/backward -nostdinc -isystem external/llvm_k8/lib/clang/17/include -isystem external/amd64_debian_sysroot//usr/local/include -isystem external/amd64_debian_sysroot//usr/x86_64-linux-gnu/include -isystem external/amd64_debian_sysroot//usr/include/x86_64-linux-gnu -isystem external/amd64_debian_sysroot//usr/include -isystem external/amd64_debian_sysroot//include -isystem external/amd64_debian_sysroot//usr/include '-std=gnu++20' -MD -MF bazel-out/k8-opt/bin/frc971/orin/_objs/cuda/cuda.pic.d '-frandom-seed=bazel-out/k8-opt/bin/frc971/orin/_objs/cuda/cuda.pic.o' -fPIC '-DGLOG_DEPRECATED=__attribute__((deprecated))' '-DGLOG_EXPORT=__attribute__((visibility("default")))' -D__CUDACC_RELAXED_CONSTEXPR__ -D__CUDACC_EXTENDED_LAMBDA__ '-DBAZEL_CURRENT_REPOSITORY=""' -iquote . -iquote bazel-out/k8-opt/bin -iquote external/com_github_google_glog -iquote bazel-out/k8-opt/bin/external/com_github_google_glog -iquote external/com_github_gflags_gflags -iquote bazel-out/k8-opt/bin/external/com_github_gflags_gflags -iquote external/com_github_nvidia_cuco -iquote bazel-out/k8-opt/bin/external/com_github_nvidia_cuco -iquote external/com_github_nvidia_cccl -iquote bazel-out/k8-opt/bin/external/com_github_nvidia_cccl -Ibazel-out/k8-opt/bin/external/com_github_google_glog/_virtual_includes/glog -Ibazel-out/k8-opt/bin/external/com_github_gflags_gflags/_virtual_includes/gflags -isystem external/com_github_nvidia_cuco/include -isystem bazel-out/k8-opt/bin/external/com_github_nvidia_cuco/include -isystem external/com_github_nvidia_cccl/cub -isystem bazel-out/k8-opt/bin/external/com_github_nvidia_cccl/cub -isystem external/com_github_nvidia_cccl/libcudacxx/include -isystem bazel-out/k8-opt/bin/external/com_github_nvidia_cccl/libcudacxx/include -isystem external/com_github_nvidia_cccl/thrust -isystem bazel-out/k8-opt/bin/external/com_github_nvidia_cccl/thrust '--cuda-gpu-arch=sm_80' '--cuda-gpu-arch=sm_86' '--cuda-gpu-arch=sm_87' -x cuda '--cuda-path=external/amd64_debian_sysroot/usr/lib/cuda/' '--ptxas-path=external/amd64_debian_sysroot/usr/bin/ptxas' '-D__CUDACC_VER_MAJOR__=11' '-D__CUDACC_VER_MINOR__=8' -Werror -Wextra -Wpointer-arith -Wstrict-aliasing -Wwrite-strings -Wtype-limits -Wsign-compare '-Wformat=2' '--sysroot=external/amd64_debian_sysroot' -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-fdebug-prefix-map=external/llvm_k8/=__bazel_toolchain_llvm_repo__/' -c frc971/orin/cuda.cc -o bazel-out/k8-opt/bin/frc971/orin/_objs/cuda/cuda.pic.o```
)
</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJzkWl9P6zgW_zThxXLVJm3TPvBQmjLq6nK5grLc2RfLdU4SD46dsR2g8zCffWUnLQXKn9uW1UiLEKS2z18f_845TqkxPJcAp8HgLBgkJ7S2hdKnk9pYLq9ZURcnS5WuTudBGJeoNlzmKBh2sVTGplyyIDxzv8NuEE79BDYro5WywbCLqEz9GDcrY6F0Q1YhpsqKC0DTm2SCmEoB0ZxyaSyiqCVGD9wWiAkq8w5CtwVINEepQrbgxgmaoxwsohKB1kojulS1dZIIkYpLwSUQEgy7naCbBN1J-3fYbX_9x7lEmdOCSybqFFKUaVWiTLNx3AvCc6W5DMJzVqe0w1gQTaIgmrxHCY8WtKTCEamS5NwW9ZLIe55ySljNVBCetxSerx-oKNcdVhdBNOmNv0JACpZyR2I15dZ0iqpyskYHyWLMcyx0bez2Qwr3nAHRkIEGyaDj7IoOc9wHsiqrv1JKCaXSq0ZC-EV2rDdoVQFpdskFhuLSgibrbXMK9L5EAW5BU6v01uO21Lj71VI3Hmi3tMEKYmn-lWa3iNSc8G0t4BFYbbmSpFKCs2bz-18TXi-VYEpmvDF7cAy_s3q5-VtbLkgKyzpvASc6DHBeS2i1b7j3h1-hP9WsWAuIv0JASZlWawnHQEnBl25z2ePjC3j2G25s2krmdnUsYP6MyE24u8UvFj6pE8b9f4A-hDQw3ISYsbpmllDbHMxfCGJapsO-OwKcSrKuUsJzZ7HRz1Vrq5rwvBdusoADw1-Iub2lLZsMYAqqId0kuMETAn2JELKkpknYvbE7uqFLd0155R4aycj_wGMFzELqHjQYw5U8RDHtJgplLEkh4xKavDOK1lkdSWVhhw5UbtzuTy0KwvhZ_RfGG4pRhIJ4igih1mq-rK2fHwXhaL0-CMfud0uEI_joJxjM9qsmDztXbxR7xlLLGSlp1SJYGO4Kmj2qxyfO5A60BGFaCQMXK4M3Q0UqVFLLCt81hLGqmqyPJDwEYYyyWjKXalHmxpTEVAjFqHXLK0EZlCCtW7wdatHZ2lcoiKaOUzTbyBwMN_vmyPweD41Q1gSDM54-BoOkk3Htsu8YUatKzsgdrIirvoL47C6IkyA6exkGwWD299_H9WOHS7HGr17vrTDnErnGiErLqXeUyp6cZqGsBLWATAWMU8H_atYEYezlRpMgmrSi_TOX3Pp1EETTcDC8-RaE021xHpL90sY1QTStpe8PUySUzF3r9bTGIbd_IGRbii000JQYpiogTU3ndugZ6Sv2rRZHEPJK4ZcDzw1e-8l1YkE0_VoP_K983fyGMdLwZw3GYXUBGjbi-2Mf1m_GxlIodkeM-xBOXx2Tp6GSVhWkzejmED7HzQ1Cvo2F4dAdgn53-xDsCvwSyiXoTfw_O6TbIf90yHb59PmA89QLmjfdFg7HrXFvyeLSDvvEejHr52iGPNdRr9vt-lsSKCu78u40IC2XIF6ERgs5_-eZJf6HZZb448xigCmZbqWW7QMSn93vzi7tMVlnmEFjsEE5SGcUpOihANlemzkDnE2mJKPN_dbzm63m77nbbIcOjkK1Nc7U36AhbtASGK0NoGDYdbFDHrRTVZtg2EXcyCCMLfJjjtwWgAqgKWiDmNIamBWrDkIJz3M370Kwuf27d_SmUA8G2YLanexBpgbVlVNOiBXyKYxLL6RpiJEB1-ihitrivSu8IIy29qvT6bjD1tIaS7U1zfldRypa0r9AYFW7ivNuhFVlfSksfdu_K4pzpXIBJBcqd43IPde2poK0Ul0F7af25Z4Jmpv231v8m8lGwuGV_wF8HkdDMuxjwWX9iHNZH5H1y-aEsrsHqtPXLIW4L8ndqOkTHaGgDtXPe_EWv2PauDevV4Tv3kF8fvXOZvrz5O3V02cIniP3flvRJKan0_9Zdza-bBk39wQvddkJenPEqMt6jw3OLVeopJLm7UuLV0hUrgyIzIFPuWrxtak2uLz32UJ5YFvWtuFnShDCIMHvAFG0rPMO2snXFKoWqeMIyHcbz1EsHGW1EG66pDJFDpoRNxuYemGZd6zLcRhbqnOwQZS0cVrLO6ke5PaZjBG-IeeXV4v5-e_k-vLmajrztMmLwSBKQr86z9NlhHBmLGV3uNLKAnOJFGdSYVVyizNNS8DtjTTCGVNCaZxymktlLGcG4VsqBMK3TTmKDc3ArhC-dd7FzRsmhM92Bk-DjwhrMKrWDHDK9efCDOGEkOtFMnWWXUwW5GIyvbq8fhqeXn6_Xky-v574Nr-Yb0Zb58y_zcjl-fn1bEHO5ovrIEqGfecfN52VYAzNAQuQuS2CKOm5Sm49SZlW2GGW1ZQBFrzkbo-62-69DBFOviezs5vfEM7W1Qk24P8bhLOUWvr02Ws1ubwmnmTDrX2Rdug59Ox9S5G00Bs2_J-_1dspbz9UP5DVB7nniIpupZ8nd-zt-L0Uc4WqOJDHDocdwu7tnTiE3S_SHsGArcD3TtkK_YsE4Yvz94q4lz0XUcs_zNbLG9eCVZx10gYXNJWpKrEBcOKOwFd5PbMf82kDD799u_yNJLMfV7PpZDFLgijZcb-ZQqWBuS6iveFsYauhnv38cXm12E15zw1fNm8C3OcwhYzWwgZhuObkmTlUnd4kk-mUXM2-TX7OkgZ4Zz9_XBGyPT37uZh9T2YJ-Ta5OEsmhDSKnE3-M_tGpjdXV7PvC3I1-3F5PV9cXv0eRImX1aQp_metLKDO5ukth24WfFjRf8jqM23BR-K2S_wDBO5k82H9tr_AXUw-rC8PF-eZzL-oTduL86dbtJ2w9FFZ_UR0yA59gI3vdR5HUGAHr30bmaNp8z7zzzdKR1OoZeereKcZzqsaU81cMemvcta4vHN2-O5s7AHy0d8Wb62qqC9V92y2WnmVfaTmV1k1bvGkm3SzyQL_nl2Ri8m_Lq8IcYV0740V8-_tipG37rb5ohe-hUerKcK3bTuCqea2cK2G1ZxZTAWn_jtq-PZBcwvYjUt3Om_tqmqLc_fJ9STYtXxUN1XBbaZ0Se26L_LWrw37hOm-eMaMSiU5owJXGjL-CE6UVHhZc2G5xE2voKF505muDU8mi5k3NghDDSllPl2HW65ZzC9m14vJxY-Pl72zIvPf_2h1w_7SOtlZ0fqKwIc8sUoJVlAuiZ_WUClXIfgQweyti2iE1TFKKfXqxm98kp5G6Tga0xM47cXdbhgPh-HgpDgdhaNxNBwM-oN-PBwNenF_OM563eFoycZpRsMTfhp2w6jX6426UTfqDzoxjHvjrB-mUZj1GURBvwsl5aLjLO0onZ9wY2o4jcM46p8IugRh_Pckw1DCA_KTzruD5ESfOhq8rHMT9LuCG2ueuFhuhf-CpW8RenEwSF61WlSmmzO81EDv0LOO7aTW4rSwtvLXBL6wbXCmw1TZbl37z_XwfwDz37JyGjq_egv-GwAA___o9tjr">