[libcxx-commits] [compiler-rt] [libc] [libcxx] [llvm] [libc] Add a dynamic loader and dlopen for x86_64 (draft, for discussion) (PR #236698)
Mike Lothian via libcxx-commits
libcxx-commits at lists.llvm.org
Thu Oct 29 02:08:23 PDT 2026
https://github.com/FireBurn created https://github.com/llvm/llvm-project/pull/236698
This is a draft to start a conversation, not something I expect to be merged as it stands. I would like to know which direction you want this to go before I put more work into any one part
It adds a way to build libc as a shared object and a dynamic loader that runs programs linked against it, with dlopen on top, for x86_64 Linux only. The ELF code (hashing, symbol lookup, segment mapping, relocation, TLS layout) is in libc/src/__support/elf and is shared by the startup loader and dlopen. The loader itself is libc/loader, built as ld-llvm-libc-x86_64.so.1 with no dependencies. The commits are in the order they build on each other, from LIBC_ENABLE_SHARED through dlopen, general dynamic TLS, $ORIGIN, dladdr, RTLD_DEFAULT and RTLD_NEXT, and dl_iterate_phdr over every loaded module
This is the first 46 commits of a longer series. All of the work is on https://github.com/FireBurn/llvm-project/tree/libc-shared-lib-support and goes well past this: lazy binding, TLS descriptors, DT_RELR, copy relocations, symbol versioning, LD_PRELOAD, indirect function resolvers, the r_debug interface for debuggers, a single early init entry point for the loader to call, and several hundred entry points that programs needed. I have kept those out of this PR so there is something small enough to read
The branch is being dogfooded on a real Gentoo system, with llvm-libc as the only C library. The loader starts a Plasma desktop, Firefox and Chromium there
What I ran on this 46 commit prefix on its own: it configures and builds libc.so, libm.so and the loader, and the ELF and link unit tests under libc/test/src pass. I have not run the whole libc test suite on it, and the desktop testing above was on the full branch, not on this prefix
Known gaps: x86_64 only, RTLD_LOCAL and RTLD_GLOBAL are not distinguished (everything is global), there is no ld.so command line options handling, and a resolver or initialiser that depends on another object being bound first is not supported. libc.so has no ABI stability and I think it should be labelled experimental
I introduced this work on Discourse in https://discourse.llvm.org/t/shared-llvm-libc-with-a-dynamic-loader-running-a-self-hosting-gentoo-system/91817 and the design discussion is in https://discourse.llvm.org/t/91826 (the requirements for one loader, where I have replied) and https://discourse.llvm.org/t/91825 (the hooks RFC). This differs from the RFC in that dlopen and the loader share one implementation in libc rather than going through weak stubs, and in how the loader hands libc what it loaded (a plain record of modules rather than a table of callbacks). I would like to know if that is the wrong trade
Questions for you:
- Is an in-tree loader wanted at all, or should this stay downstream
- If it is, would you rather see it as small PRs, and which piece should go first (the shared library build, the ELF support, or the loader)
- Should it use elfldltl instead of the ELF code here
- How should it fit with the RFC's interface for early init and TLS
- Which of aarch64 and riscv64 should come next
Assisted-by: Claude
I used Claude Code to help write and review this code and to run the builds and tests above
>From 690b7ca478e3e4fbdf4d4436cae5849f99b56340 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 03:42:42 +0100
Subject: [PATCH 01/46] [libc] Give public symbols default visibility in a
shared library
LIBC_NAMESPACE_DECL applies hidden visibility to every declaration,
which is right for internals but leaves the public C symbols hidden as
well. In a static archive that is harmless, but linking those objects
into a shared object makes every entrypoint local: a libc.so built from
them exported 37 symbols instead of 790.
LLVM_LIBC_FUNCTION_ATTR and LLVM_LIBC_VARIABLE_ATTR already exist to
control the public symbols independently of the namespace, so define
them to default visibility when LIBC_COPT_SHARED_LIBRARY is set. Both
use the GNU attribute spelling because they expand in positions where a
C++ attribute would appertain to the type rather than the declaration.
Errno and libc_errno need the same treatment: libm.so and libmvec.so are
separate objects that have to share libc.so's errno.
No change unless LIBC_COPT_SHARED_LIBRARY is defined.
Assisted-by: Claude
---
libc/src/__support/common.h | 14 ++++++++++++++
libc/src/__support/libc_errno.h | 14 ++++++++++++--
.../src/math/performance_testing/CMakeLists.txt | 2 +-
3 files changed, 27 insertions(+), 3 deletions(-)
diff --git a/libc/src/__support/common.h b/libc/src/__support/common.h
index 454dfb43b744e4..6b4d9663f04d59 100644
--- a/libc/src/__support/common.h
+++ b/libc/src/__support/common.h
@@ -25,13 +25,27 @@
#define LIBC_TARGET_USES_LEADING_UNDERSCORE
#endif
+// LIBC_NAMESPACE_DECL gives every declaration hidden visibility. The public C
+// symbols are controlled independently here, and must be given default
+// visibility to be exported from a shared object.
+// Both are spelled the GNU way rather than as C++ attributes: they expand in
+// positions where a `[[...]]` attribute would appertain to the type rather
+// than to the declaration.
#ifndef LLVM_LIBC_FUNCTION_ATTR
+#ifdef LIBC_COPT_SHARED_LIBRARY
+#define LLVM_LIBC_FUNCTION_ATTR __attribute__((visibility("default")))
+#else
#define LLVM_LIBC_FUNCTION_ATTR
#endif
+#endif
#ifndef LLVM_LIBC_VARIABLE_ATTR
+#ifdef LIBC_COPT_SHARED_LIBRARY
+#define LLVM_LIBC_VARIABLE_ATTR __attribute__((visibility("default")))
+#else
#define LLVM_LIBC_VARIABLE_ATTR
#endif
+#endif
// clang-format off
// Allow each function `func` to have extra attributes specified by defining:
diff --git a/libc/src/__support/libc_errno.h b/libc/src/__support/libc_errno.h
index 3720cdebd5d2ae..75e6841097aee6 100644
--- a/libc/src/__support/libc_errno.h
+++ b/libc/src/__support/libc_errno.h
@@ -83,12 +83,22 @@ namespace LIBC_NAMESPACE_DECL {
extern "C" int *__llvm_libc_errno() noexcept;
-struct Errno {
+// libm.so and libmvec.so must share libc.so's errno, so these cannot keep the
+// hidden visibility the surrounding namespace applies.
+#ifdef LIBC_COPT_SHARED_LIBRARY
+#define LIBC_ERRNO_VISIBILITY __attribute__((visibility("default")))
+#else
+#define LIBC_ERRNO_VISIBILITY
+#endif
+
+struct LIBC_ERRNO_VISIBILITY Errno {
void operator=(int);
operator int();
};
-extern Errno libc_errno;
+extern LIBC_ERRNO_VISIBILITY Errno libc_errno;
+
+#undef LIBC_ERRNO_VISIBILITY
} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/test/src/math/performance_testing/CMakeLists.txt b/libc/test/src/math/performance_testing/CMakeLists.txt
index 2053c68d5a4a41..fb90ff677a84c7 100644
--- a/libc/test/src/math/performance_testing/CMakeLists.txt
+++ b/libc/test/src/math/performance_testing/CMakeLists.txt
@@ -14,7 +14,7 @@ add_dependencies(
)
# A convenience target to build all performance tests.
-add_custom_target(libc-math-performance-tests)
+add_libc_testsuite(libc-math-performance-tests)
function(add_perf_binary target_name)
cmake_parse_arguments(
>From 7d0749e66c982e93a1d9e7fd0e2492040ead3967 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 03:42:52 +0100
Subject: [PATCH 02/46] [libc] Add LIBC_ENABLE_SHARED to build libc.so, libm.so
and libmvec.so
add_entrypoint_library only ever produced a static archive. Add a SHARED
option to it, and a LIBC_ENABLE_SHARED build option that uses it to
produce shared objects alongside the archives in a full build.
Building a working shared object needs five things beyond changing the
library type:
- The entrypoint objects and object libraries must be position
independent. They are shared between the archive and the shared
object, so the archive becomes position independent as well when the
option is on.
- Position independent code defaults to the general dynamic TLS model,
which needs __tls_get_addr. libc.so is loaded at startup rather than
dlopen'd, so initial-exec is both valid and cheaper.
- When scudo is the allocator, compiler-rt compiles it with flags of
its own, so it needs the same two changes there.
- Thread creation calls init_tls and cleanup_tls, which a static link
picks up from crt1.o. libc.so has to carry them itself.
- The link must exclude the driver's default libraries, which would
otherwise add a DT_NEEDED on the system libm, while still providing
the compiler runtime that the math and fixed-point entrypoints call
into.
The result is a self-contained libc.so.0 with no DT_NEEDED entries, and
libm.so.0 and libmvec.so.0 linked against it. SOVERSION comes from
LIBC_ABI_VERSION, which stays at 0 while there is no ABI stability
guarantee.
The option is off by default and requires LLVM_LIBC_FULL_BUILD. A
dynamically linked program still needs a loader, which is not part of
this change.
Assisted-by: Claude
---
.../lib/scudo/standalone/CMakeLists.txt | 8 ++++
libc/CMakeLists.txt | 25 +++++++++++
.../modules/LLVMLibCCompileOptionRules.cmake | 7 ++++
libc/cmake/modules/LLVMLibCLibraryRules.cmake | 41 +++++++++++++++++--
libc/cmake/modules/LLVMLibCObjectRules.cmake | 11 +++++
libc/docs/build_concepts.md | 21 ++++++++++
libc/lib/CMakeLists.txt | 38 +++++++++++++++++
7 files changed, 147 insertions(+), 4 deletions(-)
diff --git a/compiler-rt/lib/scudo/standalone/CMakeLists.txt b/compiler-rt/lib/scudo/standalone/CMakeLists.txt
index 20c59b46006136..97a585ef89e125 100644
--- a/compiler-rt/lib/scudo/standalone/CMakeLists.txt
+++ b/compiler-rt/lib/scudo/standalone/CMakeLists.txt
@@ -186,6 +186,14 @@ if(COMPILER_RT_BUILD_SCUDO_STANDALONE_WITH_LLVM_LIBC)
set(SCUDO_DEPS libc-headers)
list(APPEND SCUDO_CFLAGS "-ffreestanding")
+
+ # A shared libc links scudo into libc.so, so it is compiled the way the
+ # libc's own objects are: position independent, with initial-exec TLS.
+ if(LIBC_ENABLE_SHARED)
+ append_list_if(COMPILER_RT_HAS_FPIC_FLAG -fPIC SCUDO_CFLAGS)
+ append_list_if(COMPILER_RT_HAS_FTLS_MODEL_INITIAL_EXEC
+ -ftls-model=initial-exec SCUDO_CFLAGS)
+ endif()
endif()
append_list_if(COMPILER_RT_HAS_LIBPTHREAD -pthread SCUDO_LINK_FLAGS)
diff --git a/libc/CMakeLists.txt b/libc/CMakeLists.txt
index b6c6b0aa546e2d..4464a27817cdda 100644
--- a/libc/CMakeLists.txt
+++ b/libc/CMakeLists.txt
@@ -269,6 +269,31 @@ endif()
option(LLVM_LIBC_HERMETIC_TEST_USE_INTERNAL_STARTUP
"Link hermetic tests against LLVM libc's own startup object"
ON)
+option(LIBC_ENABLE_SHARED
+ "Additionally build the entrypoint libraries as shared objects" OFF)
+if(LIBC_ENABLE_SHARED AND NOT LLVM_LIBC_FULL_BUILD)
+ message(FATAL_ERROR
+ "LIBC_ENABLE_SHARED requires LLVM_LIBC_FULL_BUILD. The overlay build "
+ "produces a static archive that is linked ahead of the system libc.")
+endif()
+# Bumped when the shared object ABI changes incompatibly. LLVM-libc does not
+# offer ABI stability yet, so this stays at 0 until it does.
+set(LIBC_ABI_VERSION 0 CACHE STRING
+ "SOVERSION for the LLVM-libc shared objects")
+if(LIBC_ENABLE_SHARED)
+ execute_process(
+ COMMAND ${CMAKE_C_COMPILER} --rtlib=compiler-rt --print-libgcc-file-name
+ OUTPUT_VARIABLE LIBC_COMPILER_RT_BUILTINS
+ OUTPUT_STRIP_TRAILING_WHITESPACE
+ ERROR_QUIET)
+ if(NOT EXISTS "${LIBC_COMPILER_RT_BUILTINS}")
+ message(WARNING
+ "Could not locate the compiler-rt builtins archive. The shared objects "
+ "may fail to link.")
+ set(LIBC_COMPILER_RT_BUILTINS "")
+ endif()
+endif()
+
option(LLVM_LIBC_IMPLEMENTATION_DEFINED_TEST_BEHAVIOR "Build LLVM libc tests assuming our implementation-defined behavior" ON)
option(LLVM_LIBC_ENABLE_LINTING "Enables linting of libc source files" OFF)
option(LLVM_LIBC_ALL_HEADERS "Outputs all functions in header files, regardless of whether they are enabled on this target" OFF)
diff --git a/libc/cmake/modules/LLVMLibCCompileOptionRules.cmake b/libc/cmake/modules/LLVMLibCCompileOptionRules.cmake
index b26c1957880a5f..dc1634cd2d3a22 100644
--- a/libc/cmake/modules/LLVMLibCCompileOptionRules.cmake
+++ b/libc/cmake/modules/LLVMLibCCompileOptionRules.cmake
@@ -288,6 +288,13 @@ function(_get_common_compile_options_impl output_var)
list(APPEND compile_options "-fno-unwind-tables")
list(APPEND compile_options "-fno-asynchronous-unwind-tables")
list(APPEND compile_options "-fno-rtti")
+ if(LIBC_ENABLE_SHARED)
+ # Position independent code defaults to the general dynamic TLS model,
+ # which needs __tls_get_addr. libc.so is always loaded at startup rather
+ # than dlopen'd, so initial-exec is both valid and cheaper.
+ list(APPEND compile_options "-ftls-model=initial-exec")
+ list(APPEND compile_options "-DLIBC_COPT_SHARED_LIBRARY")
+ endif()
if(LIBC_CC_SUPPORTS_PATTERN_INIT)
list(APPEND compile_options "-ftrivial-auto-var-init=pattern")
endif()
diff --git a/libc/cmake/modules/LLVMLibCLibraryRules.cmake b/libc/cmake/modules/LLVMLibCLibraryRules.cmake
index cdff1b907214bb..46ba4cae98f19b 100644
--- a/libc/cmake/modules/LLVMLibCLibraryRules.cmake
+++ b/libc/cmake/modules/LLVMLibCLibraryRules.cmake
@@ -138,17 +138,24 @@ endfunction()
# A rule to build a library from a collection of entrypoint objects.
# Usage:
# add_entrypoint_library(
+# target_name
+# [SHARED]
+# [SOVERSION <version>]
# DEPENDS <list of add_entrypoint_object targets>
# )
#
+# Without SHARED a static archive is produced. SHARED builds a shared object
+# instead and requires LIBC_ENABLE_SHARED, which makes the entrypoint objects
+# position independent.
+#
# NOTE: If one wants an entrypoint to be available in a library, then they will
# have to list the entrypoint target explicitly in the DEPENDS list. Implicit
# entrypoint dependencies will not be added to the library.
function(add_entrypoint_library target_name)
cmake_parse_arguments(
"ENTRYPOINT_LIBRARY"
- "" # No optional arguments
- "" # No single value arguments
+ "SHARED" # Optional arguments
+ "SOVERSION" # Single value arguments
"DEPENDS" # Multi-value arguments
${ARGN}
)
@@ -156,6 +163,10 @@ function(add_entrypoint_library target_name)
message(FATAL_ERROR "'add_entrypoint_library' target requires a DEPENDS list "
"of 'add_entrypoint_object' targets.")
endif()
+ if(ENTRYPOINT_LIBRARY_SHARED AND NOT LIBC_ENABLE_SHARED)
+ message(FATAL_ERROR
+ "'add_entrypoint_library' with SHARED requires LIBC_ENABLE_SHARED.")
+ endif()
get_fq_deps_list(fq_deps_list ${ENTRYPOINT_LIBRARY_DEPENDS})
get_all_object_file_deps(all_deps "${fq_deps_list}")
@@ -165,12 +176,34 @@ function(add_entrypoint_library target_name)
list(APPEND objects $<$<STREQUAL:$<TARGET_NAME_IF_EXISTS:${dep}>,${dep}>:$<TARGET_OBJECTS:${dep}>>)
endforeach()
+ if(NOT ENTRYPOINT_LIBRARY_SHARED)
+ add_library(
+ ${target_name}
+ STATIC
+ ${objects}
+ )
+ set_target_properties(${target_name} PROPERTIES ARCHIVE_OUTPUT_DIRECTORY ${LIBC_LIBRARY_DIR})
+ return()
+ endif()
+
add_library(
${target_name}
- STATIC
+ SHARED
${objects}
)
- set_target_properties(${target_name} PROPERTIES ARCHIVE_OUTPUT_DIRECTORY ${LIBC_LIBRARY_DIR})
+ set_target_properties(${target_name}
+ PROPERTIES LIBRARY_OUTPUT_DIRECTORY ${LIBC_LIBRARY_DIR})
+ if(NOT ENTRYPOINT_LIBRARY_SOVERSION STREQUAL "")
+ set_target_properties(${target_name}
+ PROPERTIES SOVERSION ${ENTRYPOINT_LIBRARY_SOVERSION})
+ endif()
+ # Link nothing from the host toolchain except the compiler runtime, which the
+ # math and fixed-point entrypoints need for builtins such as __divti3. The
+ # driver defaults would otherwise add a DT_NEEDED on the system libm.
+ target_link_options(${target_name} PRIVATE -nostdlib)
+ if(LIBC_COMPILER_RT_BUILTINS)
+ target_link_libraries(${target_name} PRIVATE ${LIBC_COMPILER_RT_BUILTINS})
+ endif()
endfunction()
set(HDR_LIBRARY_TARGET_TYPE "HDR_LIBRARY")
diff --git a/libc/cmake/modules/LLVMLibCObjectRules.cmake b/libc/cmake/modules/LLVMLibCObjectRules.cmake
index ff701921c784b7..b5d8d108888be5 100644
--- a/libc/cmake/modules/LLVMLibCObjectRules.cmake
+++ b/libc/cmake/modules/LLVMLibCObjectRules.cmake
@@ -94,6 +94,10 @@ function(create_object_library fq_target_name)
target_include_directories(${fq_target_name} PRIVATE ${LIBC_SOURCE_DIR})
target_include_directories(${fq_target_name} PRIVATE ${LIBC_BUILD_DIR})
target_compile_options(${fq_target_name} PRIVATE ${compile_options})
+ if(LIBC_ENABLE_SHARED)
+ set_target_properties(${fq_target_name}
+ PROPERTIES POSITION_INDEPENDENT_CODE ON)
+ endif()
# loop through the deps, check if any have the TARGET_TYPE of ENTRYPOINT_OBJ_TARGET_TYPE, and print a warning if they do.
if(LIBC_CMAKE_VERBOSE_LOGGING)
@@ -324,6 +328,13 @@ function(create_entrypoint_object fq_target_name)
add_dependencies(${fq_target_name} ${full_deps_list})
target_link_libraries(${fq_target_name} ${full_deps_list})
+ # The same objects are packaged into both the static archive and, when
+ # enabled, the shared object, so they must be position independent.
+ if(LIBC_ENABLE_SHARED)
+ set_target_properties(${fq_target_name} ${internal_target_name}
+ PROPERTIES POSITION_INDEPENDENT_CODE ON)
+ endif()
+
# Builtin recognition causes issues when trying to implement the builtin
# functions themselves. The GPU backends do not use libcalls so we disable the
# known problematic ones on the entrypoints that implement them.
diff --git a/libc/docs/build_concepts.md b/libc/docs/build_concepts.md
index a6a93744978090..1046775149c4b6 100644
--- a/libc/docs/build_concepts.md
+++ b/libc/docs/build_concepts.md
@@ -40,6 +40,27 @@ cmake -S runtimes -B build -DLLVM_ENABLE_RUNTIMES="libc;compiler-rt" \
-DLLVM_LIBC_FULL_BUILD=ON ...
```
+Setting `LIBC_ENABLE_SHARED=ON` additionally builds `libc.so`, `libm.so` and
+`libmvec.so` alongside the archives. This makes the entrypoint objects position
+independent and gives the public symbols default visibility, so the archives
+built in this configuration are position independent too.
+
+```sh
+cmake -S runtimes -B build -DLLVM_ENABLE_RUNTIMES="libc;compiler-rt" \
+ -DLLVM_LIBC_FULL_BUILD=ON -DLIBC_ENABLE_SHARED=ON ...
+```
+
+Note that a dynamically linked program needs a dynamic loader, which LLVM-libc
+does not yet provide. Linking one against `libc.so` succeeds and reaches
+`main`, but faults on the way out: `libc.so` carries a `PT_TLS` segment of its
+own, while `crt1.o` sizes the thread local block from the main executable's
+`PT_TLS` alone and then installs its own thread pointer. Accesses to
+`libc.so`'s thread local storage land outside that block.
+
+Sizing thread local storage across every loaded module is the loader's job, so
+these shared objects are a prerequisite for adding one rather than something
+usable without one.
+
### 3. Bootstrap Build
A bootstrap build first builds the compiler (Clang) and other LLVM tools using
diff --git a/libc/lib/CMakeLists.txt b/libc/lib/CMakeLists.txt
index aa0b35e65a1a0b..09acbc20179fcf 100644
--- a/libc/lib/CMakeLists.txt
+++ b/libc/lib/CMakeLists.txt
@@ -35,6 +35,7 @@ else()
endif()
set(added_archive_targets "")
+set(added_shared_targets "")
set(added_bitcode_targets "")
foreach(archive IN ZIP_LISTS
libc_archive_names libc_archive_targets libc_archive_entrypoint_lists)
@@ -60,6 +61,33 @@ foreach(archive IN ZIP_LISTS
endif()
list(APPEND added_archive_targets ${archive_1})
+ if(LIBC_ENABLE_SHARED)
+ add_entrypoint_library(
+ ${archive_1}-shared
+ SHARED
+ SOVERSION ${LIBC_ABI_VERSION}
+ DEPENDS
+ ${${archive_2}}
+ )
+ set_target_properties(
+ ${archive_1}-shared
+ PROPERTIES
+ OUTPUT_NAME ${archive_0}
+ )
+ target_link_libraries(${archive_1}-shared PUBLIC libc-headers)
+ if(TARGET libc-startup)
+ add_dependencies(${archive_1}-shared libc-startup)
+ endif()
+ if(archive_1 STREQUAL "libc" AND TARGET ${tls_object})
+ target_sources(${archive_1}-shared
+ PRIVATE $<TARGET_OBJECTS:${tls_object}>)
+ else()
+ # libm and libmvec reference errno and other libc internals.
+ target_link_libraries(${archive_1}-shared PRIVATE libc-shared)
+ endif()
+ list(APPEND added_shared_targets ${archive_1}-shared)
+ endif()
+
# Add the offloading version of the library for offloading languages. These
# are installed in the standard search path separate from the other libraries.
if(LIBC_TARGET_OS_IS_GPU)
@@ -98,6 +126,14 @@ install(
COMPONENT libc
)
+if(added_shared_targets)
+ install(
+ TARGETS ${added_shared_targets}
+ LIBRARY DESTINATION ${LIBC_INSTALL_LIBRARY_DIR}
+ COMPONENT libc
+ )
+endif()
+
foreach(file ${added_bitcode_targets})
install(FILES $<TARGET_PROPERTY:${file},TARGET_FILE>
DESTINATION ${LIBC_INSTALL_LIBRARY_DIR}
@@ -143,6 +179,7 @@ endif()
add_custom_target(install-libc
DEPENDS ${added_archive_targets}
+ ${added_shared_targets}
${added_bitcode_targets}
${added_empty_archives}
${startup_target}
@@ -152,6 +189,7 @@ add_custom_target(install-libc
-P "${CMAKE_BINARY_DIR}/cmake_install.cmake")
add_custom_target(install-libc-stripped
DEPENDS ${added_archive_targets}
+ ${added_shared_targets}
${added_bitcode_targets}
${added_empty_archives}
${startup_target}
>From 47e4b78ead80acfcea700163e52f8befc82174ee Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 03:58:16 +0100
Subject: [PATCH 03/46] [libc] Add LIBC_SHARED_INTERNAL for symbols crossing a
shared object
LIBC_NAMESPACE_DECL gives every declaration hidden visibility, which
makes those symbols local when the objects are linked into libc.so. Some
internals have to stay visible across that edge: libm.so and libmvec.so
share libc.so's errno, and crt1.o binds to several globals that the
startup code initialises.
Replace the errno-specific macro added earlier with a single
LIBC_SHARED_INTERNAL, and apply it to environ, program_invocation_name,
program_invocation_short_name and _r_debug. It expands to nothing unless
LIBC_COPT_SHARED_LIBRARY is defined.
This is not yet the complete set. crt1.o also reaches for atexit and the
thread atexit callback manager, so linking a dynamic executable against
libc.so still fails. Settling the full startup-to-libc interface is a
follow-up rather than a symbol-at-a-time fix.
Assisted-by: Claude
---
libc/src/__support/libc_errno.h | 15 +++------------
libc/src/__support/macros/config.h | 9 +++++++++
libc/src/errno/program_invocation_name.h | 2 +-
libc/src/errno/program_invocation_short_name.h | 2 +-
libc/src/link/_r_debug.h | 2 +-
libc/src/unistd/environ.h | 2 +-
6 files changed, 16 insertions(+), 16 deletions(-)
diff --git a/libc/src/__support/libc_errno.h b/libc/src/__support/libc_errno.h
index 75e6841097aee6..a7ea9678d19c6e 100644
--- a/libc/src/__support/libc_errno.h
+++ b/libc/src/__support/libc_errno.h
@@ -83,22 +83,13 @@ namespace LIBC_NAMESPACE_DECL {
extern "C" int *__llvm_libc_errno() noexcept;
-// libm.so and libmvec.so must share libc.so's errno, so these cannot keep the
-// hidden visibility the surrounding namespace applies.
-#ifdef LIBC_COPT_SHARED_LIBRARY
-#define LIBC_ERRNO_VISIBILITY __attribute__((visibility("default")))
-#else
-#define LIBC_ERRNO_VISIBILITY
-#endif
-
-struct LIBC_ERRNO_VISIBILITY Errno {
+// libm.so and libmvec.so share libc.so's errno.
+struct LIBC_SHARED_INTERNAL Errno {
void operator=(int);
operator int();
};
-extern LIBC_ERRNO_VISIBILITY Errno libc_errno;
-
-#undef LIBC_ERRNO_VISIBILITY
+extern LIBC_SHARED_INTERNAL Errno libc_errno;
} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/src/__support/macros/config.h b/libc/src/__support/macros/config.h
index f256d5b9b99220..2ee4e3989d2578 100644
--- a/libc/src/__support/macros/config.h
+++ b/libc/src/__support/macros/config.h
@@ -74,6 +74,15 @@
#define LIBC_NAMESPACE_DECL LIBC_NAMESPACE
#endif
+// Marks an internal symbol that startup code or a sibling LLVM-libc shared
+// object has to bind to across a shared object boundary. LIBC_NAMESPACE_DECL
+// would otherwise give it hidden visibility, which makes it local to libc.so.
+#ifdef LIBC_COPT_SHARED_LIBRARY
+#define LIBC_SHARED_INTERNAL __attribute__((visibility("default")))
+#else
+#define LIBC_SHARED_INTERNAL
+#endif
+
// IMPORTANT (USE WITH CAUTION): This macro is intended to be used at the top of
// the file and set to 1. It alters the signatures of some functions to have
// constexpr qualifier and forces the use of constexpr-compatible
diff --git a/libc/src/errno/program_invocation_name.h b/libc/src/errno/program_invocation_name.h
index 9d5f6bec499ff8..00ec3802d0621f 100644
--- a/libc/src/errno/program_invocation_name.h
+++ b/libc/src/errno/program_invocation_name.h
@@ -18,7 +18,7 @@
namespace LIBC_NAMESPACE_DECL {
-extern char *program_invocation_name;
+extern LIBC_SHARED_INTERNAL char *program_invocation_name;
} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/src/errno/program_invocation_short_name.h b/libc/src/errno/program_invocation_short_name.h
index 60bfd9a7aac21d..162764f1a6ec5b 100644
--- a/libc/src/errno/program_invocation_short_name.h
+++ b/libc/src/errno/program_invocation_short_name.h
@@ -18,7 +18,7 @@
namespace LIBC_NAMESPACE_DECL {
-extern char *program_invocation_short_name;
+extern LIBC_SHARED_INTERNAL char *program_invocation_short_name;
} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/src/link/_r_debug.h b/libc/src/link/_r_debug.h
index a15023f6f88ee5..47a9f90d286281 100644
--- a/libc/src/link/_r_debug.h
+++ b/libc/src/link/_r_debug.h
@@ -19,7 +19,7 @@
namespace LIBC_NAMESPACE_DECL {
-extern struct r_debug _r_debug;
+extern LIBC_SHARED_INTERNAL struct r_debug _r_debug;
} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/src/unistd/environ.h b/libc/src/unistd/environ.h
index 3d1c8aed89e3b1..97b9f37a443e45 100644
--- a/libc/src/unistd/environ.h
+++ b/libc/src/unistd/environ.h
@@ -13,7 +13,7 @@
namespace LIBC_NAMESPACE_DECL {
-extern char **environ;
+extern LIBC_SHARED_INTERNAL char **environ;
} // namespace LIBC_NAMESPACE_DECL
>From e0dbd2e3ba6b4ce2b4feac22c99922e73f216a8f Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 04:00:29 +0100
Subject: [PATCH 04/46] [libc] Expose the remaining startup dependencies from
libc.so
crt1.o needs seventeen symbols. All but three are linker-provided
(_DYNAMIC, the init/fini/preinit array bounds, __rela_iplt_start and
end, and main). The three that come from libc are exit, atexit and
get_thread_atexit_callback_mgr, so mark them LIBC_SHARED_INTERNAL.
With these, a dynamically linked executable against libc.so.0 links and
reaches main:
clang -nostdlib -nostdlibinc -isystem <build>/include hello.c \
<build>/startup/linux/crt1.o -L<build>/lib -lc
It still faults on the way out, inside exit() called from do_start(),
which suggests the atexit state is not shared correctly between crt1.o
and libc.so. Fixing that needs the startup-to-libc interface defined
properly rather than more symbols exposed one at a time.
Assisted-by: Claude
---
libc/src/stdlib/atexit.h | 2 +-
libc/src/stdlib/exit.h | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/libc/src/stdlib/atexit.h b/libc/src/stdlib/atexit.h
index 39ce2dc52624a9..85610d71c3412b 100644
--- a/libc/src/stdlib/atexit.h
+++ b/libc/src/stdlib/atexit.h
@@ -14,7 +14,7 @@
namespace LIBC_NAMESPACE_DECL {
-int atexit(__atexithandler_t);
+LIBC_SHARED_INTERNAL int atexit(__atexithandler_t);
} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/src/stdlib/exit.h b/libc/src/stdlib/exit.h
index 3c9d37a342eb98..d55c180cacd97b 100644
--- a/libc/src/stdlib/exit.h
+++ b/libc/src/stdlib/exit.h
@@ -13,7 +13,7 @@
namespace LIBC_NAMESPACE_DECL {
-[[noreturn]] void exit(int status);
+[[noreturn]] LIBC_SHARED_INTERNAL void exit(int status);
} // namespace LIBC_NAMESPACE_DECL
>From 90c125a8caac8332545e7c8398a271e422be12e5 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 08:00:11 +0100
Subject: [PATCH 05/46] [libcxx] Link libunwind in the llvm-libc test
configuration
The link line pulled in libc++ and libc++abi but not the unwinder. This
is harmless for the configuration the cache file selects, which builds
with exceptions off, but an exceptions enabled build fails almost
everything with undefined references to _Unwind_Resume and
_Unwind_RaiseException.
Measured against llvm-libc with threads, exceptions, RTTI and filesystem
enabled, this takes the suite from 5128 failures to 128, and from 5099
tests passing to 10127 of 11578. The remaining failures are concentrated
in filesystem directory iteration and the time zone database.
Assisted-by: Claude
---
libcxx/test/configs/llvm-libc++-llvm-libc.cfg.in | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/libcxx/test/configs/llvm-libc++-llvm-libc.cfg.in b/libcxx/test/configs/llvm-libc++-llvm-libc.cfg.in
index 885715eb42dfc4..042514aa2d9069 100644
--- a/libcxx/test/configs/llvm-libc++-llvm-libc.cfg.in
+++ b/libcxx/test/configs/llvm-libc++-llvm-libc.cfg.in
@@ -8,7 +8,7 @@ config.substitutions.append(('%{compile_flags}',
'-nostdlibinc -I %{include-dir} -I %{target-include-dir} -isystem @CMAKE_BINARY_DIR@/libc/include -I %{libcxx-dir}/test/support -idirafter @LIBC_KERNEL_HEADERS@'
))
config.substitutions.append(('%{link_flags}',
- '-nodefaultlibs -nostartfiles -L %{lib-dir} -lc++ -lc++abi @CMAKE_BINARY_DIR@/libc/startup/linux/crt1.o -Wl,--start-group @CMAKE_BINARY_DIR@/libc/lib/libc.a @CMAKE_BINARY_DIR@/libc/lib/libm.a -static -fno-use-cxa-atexit @CMAKE_BINARY_DIR@/compiler-rt/lib/linux/libclang_rt.builtins-x86_64.a -Wl,--end-group -Wl,--defsym,__dso_handle=0'
+ '-nodefaultlibs -nostartfiles -L %{lib-dir} -lc++ -lc++abi -lunwind @CMAKE_BINARY_DIR@/libc/startup/linux/crt1.o -Wl,--start-group @CMAKE_BINARY_DIR@/libc/lib/libc.a @CMAKE_BINARY_DIR@/libc/lib/libm.a -static -fno-use-cxa-atexit @CMAKE_BINARY_DIR@/compiler-rt/lib/linux/libclang_rt.builtins-x86_64.a -Wl,--end-group -Wl,--defsym,__dso_handle=0'
))
config.substitutions.append(('%{exec}',
'%{executor} --execdir %{temp} -- '
>From 687d5d62eb316a29e58a12bd1a5c60df2435d0a2 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Sat, 5 Sep 2026 03:11:00 +0100
Subject: [PATCH 06/46] [libc] Export the errno accessor from the shared libc
<errno.h> defines errno as *__llvm_libc_errno(), so every program
touching errno binds to that symbol. It was declared inside
LIBC_NAMESPACE_DECL, which carries hidden visibility, so it did not
appear in libc.so and nothing setting errno could link against the
shared library.
It was the only symbol named by an installed header missing from
libc.so.
Assisted-by: Claude
---
libc/src/__support/libc_errno.h | 5 ++++-
libc/src/errno/libc_errno.cpp | 8 ++++++--
2 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/libc/src/__support/libc_errno.h b/libc/src/__support/libc_errno.h
index a7ea9678d19c6e..bbfdf136b0a606 100644
--- a/libc/src/__support/libc_errno.h
+++ b/libc/src/__support/libc_errno.h
@@ -81,7 +81,10 @@ LIBC_ERRNO_MODE_SYSTEM_INLINE.
namespace LIBC_NAMESPACE_DECL {
-extern "C" int *__llvm_libc_errno() noexcept;
+// This is what <errno.h> makes errno stand for, so a program which touches
+// errno binds to it. That crosses into libc.so, which the hidden visibility
+// LIBC_NAMESPACE_DECL carries would not allow.
+extern "C" LIBC_SHARED_INTERNAL int *__llvm_libc_errno() noexcept;
// libm.so and libmvec.so share libc.so's errno.
struct LIBC_SHARED_INTERNAL Errno {
diff --git a/libc/src/errno/libc_errno.cpp b/libc/src/errno/libc_errno.cpp
index e8960fcd89500f..68776e66ca4434 100644
--- a/libc/src/errno/libc_errno.cpp
+++ b/libc/src/errno/libc_errno.cpp
@@ -25,7 +25,9 @@ namespace {
LIBC_THREAD_LOCAL int thread_errno;
}
-extern "C" int *__llvm_libc_errno() noexcept { return &thread_errno; }
+extern "C" LIBC_SHARED_INTERNAL int *__llvm_libc_errno() noexcept {
+ return &thread_errno;
+}
void Errno::operator=(int a) { thread_errno = a; }
Errno::operator int() { return thread_errno; }
@@ -36,7 +38,9 @@ namespace {
int shared_errno;
}
-extern "C" int *__llvm_libc_errno() noexcept { return &shared_errno; }
+extern "C" LIBC_SHARED_INTERNAL int *__llvm_libc_errno() noexcept {
+ return &shared_errno;
+}
void Errno::operator=(int a) { shared_errno = a; }
Errno::operator int() { return shared_errno; }
>From 9e88ba7373cff68499bd18c981253d3fcab3daf6 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Sat, 5 Sep 2026 17:21:50 +0100
Subject: [PATCH 07/46] [libc] Leave empty archives where the split libraries
used to be
Everything is in libc, but a great deal of software still names the
separate libraries it was split across, and a link asking for one fails
when there is nothing of that name. An empty libpthread was already left
in place; libdl, libresolv, librt and libutil are now too. glibc keeps
them for the same reason, having moved most of what they held into libc.
Assisted-by: Claude
---
libc/lib/CMakeLists.txt | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/libc/lib/CMakeLists.txt b/libc/lib/CMakeLists.txt
index 09acbc20179fcf..bba7023b8c4e7d 100644
--- a/libc/lib/CMakeLists.txt
+++ b/libc/lib/CMakeLists.txt
@@ -99,6 +99,10 @@ foreach(archive IN ZIP_LISTS
list(APPEND added_bitcode_targets ${archive_1}bitcode)
endif()
endforeach()
+# Everything is in libc, but a great deal of software still asks for the
+# separate libraries it used to be split across, and a link that names one
+# fails if there is nothing of that name to find. Empty archives are what
+# glibc leaves in their place for the same reason.
if(libc_subarchive_entrypoints)
add_entrypoint_library(
@@ -141,11 +145,9 @@ foreach(file ${added_bitcode_targets})
)
endforeach()
-# Add empty archive libraries for compatibility with systems expecting certain
-# bits of libc to be located in separate files (e.g. libpthread).
set(added_empty_archives "")
if(LLVM_LIBC_FULL_BUILD AND ${LIBC_TARGET_OS} STREQUAL "linux")
- list(APPEND libc_empty_archive_names pthread dl rt)
+ list(APPEND libc_empty_archive_names pthread dl resolv rt util)
foreach(archive ${libc_empty_archive_names})
set(empty_archive_path ${LIBC_LIBRARY_DIR}/lib${archive}.a)
add_custom_command(
@@ -162,6 +164,8 @@ if(LLVM_LIBC_FULL_BUILD AND ${LIBC_TARGET_OS} STREQUAL "linux")
add_custom_target(empty-${archive} ALL DEPENDS ${empty_archive_path})
list(APPEND added_empty_archives empty-${archive})
endforeach()
+ # They are installed with libc, so building libc has to make them.
+ add_dependencies(libc ${added_empty_archives})
endif()
set(startup_target "")
>From f3e1e943a1b0be1549c4053a20c51d43008dacbd Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Wed, 14 Oct 2026 11:23:48 +0100
Subject: [PATCH 08/46] [libc] Export what a C++ runtime calls
A C++ runtime registers the destructor of an object with static storage
through __cxa_atexit and runs them through __cxa_finalize. Both were
defined but hidden with the rest of the library, so no C++ program with
an object that has a destructor could link against the shared libc.
Assisted-by: Claude
---
libc/src/stdlib/atexit.cpp | 8 ++++++--
libc/src/stdlib/exit.cpp | 6 +++++-
2 files changed, 11 insertions(+), 3 deletions(-)
diff --git a/libc/src/stdlib/atexit.cpp b/libc/src/stdlib/atexit.cpp
index 58e8a614517a4f..57672d6ffea907 100644
--- a/libc/src/stdlib/atexit.cpp
+++ b/libc/src/stdlib/atexit.cpp
@@ -19,13 +19,17 @@ LIBC_CONSTINIT ExitCallbackList atexit_callbacks;
Mutex handler_list_mtx(/*is_priority_inherit=*/false, /*is_recursive=*/false,
/*is_robust=*/false, /*is_pshared=*/false);
+// These two are what a C++ runtime calls to register the destructor of an
+// object with static storage and to run them, so they have to be visible from
+// outside the library rather than hidden with the rest of it.
extern "C" {
-int __cxa_atexit(AtExitCallback *callback, void *payload, void *) {
+LIBC_SHARED_INTERNAL int __cxa_atexit(AtExitCallback *callback, void *payload,
+ void *) {
return add_atexit_unit(atexit_callbacks, {callback, payload});
}
-void __cxa_finalize(void *dso) {
+LIBC_SHARED_INTERNAL void __cxa_finalize(void *dso) {
if (!dso)
call_exit_callbacks(atexit_callbacks);
}
diff --git a/libc/src/stdlib/exit.cpp b/libc/src/stdlib/exit.cpp
index a479019e20e114..d0eb590ea7b4cf 100644
--- a/libc/src/stdlib/exit.cpp
+++ b/libc/src/stdlib/exit.cpp
@@ -26,7 +26,11 @@
namespace LIBC_NAMESPACE_DECL {
-extern "C" void __cxa_finalize(void *);
+// Defined with the rest of the exit handlers. Visibility is declared here as
+// well as there: the parts of the library are merged into one object before
+// it becomes a shared one, and the narrower of the two would otherwise win
+// and keep it in.
+extern "C" LIBC_SHARED_INTERNAL void __cxa_finalize(void *);
// TODO: use recursive mutex to protect this routine.
[[noreturn]] LLVM_LIBC_FUNCTION(void, exit, (int status)) {
>From 0d8b66acb96cf6296f84bfc24a817a4ee78e7d69 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 11 Sep 2026 21:09:38 +0100
Subject: [PATCH 09/46] [Runtimes] Only use libc-headers when LLVM libc is
built in the tree
With RUNTIMES_USE_LIBC=llvm-libc, HandleLibC links runtimes-libc-headers
against libc-headers, a target which exists only when LLVM libc is built
in the same tree. A runtime built on its own against an LLVM libc that
is already installed, the way a distribution builds libc++, gets the
name on its link line as -llibc-headers and fails to link. The
-nostdlibinc added with it also hides the installed headers, which are
the only ones there are.
When the target does not exist, neither is done, and the headers come
from where the compiler looks for them, as they would for any other C
library. This matches how the same file already treats the libc and libm
targets.
libc++ 23.1.1 built this way against an installed LLVM libc, with the
change applied, and selected LLVM libc's locale support.
Assisted-by: Claude
---
runtimes/cmake/Modules/HandleLibC.cmake | 19 +++++++++++++------
1 file changed, 13 insertions(+), 6 deletions(-)
diff --git a/runtimes/cmake/Modules/HandleLibC.cmake b/runtimes/cmake/Modules/HandleLibC.cmake
index 9b6749dd2615ee..fdf176f3fe874b 100644
--- a/runtimes/cmake/Modules/HandleLibC.cmake
+++ b/runtimes/cmake/Modules/HandleLibC.cmake
@@ -30,12 +30,19 @@ if (RUNTIMES_USE_LIBC STREQUAL "system")
# Link against the in-tree LLVM libc
elseif (RUNTIMES_USE_LIBC STREQUAL "llvm-libc")
add_library(runtimes-libc-headers INTERFACE)
- target_link_libraries(runtimes-libc-headers INTERFACE libc-headers)
- check_cxx_compiler_flag(-nostdlibinc CXX_SUPPORTS_NOSTDLIBINC_FLAG)
- if(CXX_SUPPORTS_NOSTDLIBINC_FLAG)
- target_compile_options(runtimes-libc-headers INTERFACE "-nostdlibinc")
- if(LIBC_KERNEL_HEADERS)
- target_compile_options(runtimes-libc-headers INTERFACE "-idirafter${LIBC_KERNEL_HEADERS}")
+ # libc-headers is a target only when LLVM libc is built in the same tree.
+ # A runtime built against an LLVM libc that is already installed takes the
+ # headers from where the compiler looks for them, as it would for any other
+ # C library. Naming the missing target would put -llibc-headers on the link
+ # line, and -nostdlibinc would hide the headers it needs.
+ if (TARGET libc-headers)
+ target_link_libraries(runtimes-libc-headers INTERFACE libc-headers)
+ check_cxx_compiler_flag(-nostdlibinc CXX_SUPPORTS_NOSTDLIBINC_FLAG)
+ if(CXX_SUPPORTS_NOSTDLIBINC_FLAG)
+ target_compile_options(runtimes-libc-headers INTERFACE "-nostdlibinc")
+ if(LIBC_KERNEL_HEADERS)
+ target_compile_options(runtimes-libc-headers INTERFACE "-idirafter${LIBC_KERNEL_HEADERS}")
+ endif()
endif()
endif()
>From ea12491bd039591a3f452cae182969e1f1876517 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Sat, 12 Sep 2026 00:52:22 +0100
Subject: [PATCH 10/46] [libc] Install Scrt1.o with the shared library
Clang links position independent executables against Scrt1.o rather than
crt1.o, so without it no PIE can be linked against libc.so. In a shared
build the startup objects are already position independent, so crt1.o is
installed under both names.
Assisted-by: Claude
---
libc/startup/linux/CMakeLists.txt | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/libc/startup/linux/CMakeLists.txt b/libc/startup/linux/CMakeLists.txt
index 18379a9b4e4489..c0b924ae25908a 100644
--- a/libc/startup/linux/CMakeLists.txt
+++ b/libc/startup/linux/CMakeLists.txt
@@ -171,3 +171,12 @@ foreach(target IN LISTS startup_components)
RENAME $<TARGET_PROPERTY:${fq_target_name},OUTPUT_NAME>
COMPONENT libc)
endforeach()
+
+# Position independent executables start from Scrt1.o. The objects of a shared
+# build are position independent already, so crt1.o serves under both names.
+if(LIBC_ENABLE_SHARED)
+ install(FILES $<TARGET_OBJECTS:libc.startup.linux.crt1>
+ DESTINATION ${LIBC_INSTALL_LIBRARY_DIR}
+ RENAME Scrt1.o
+ COMPONENT libc)
+endif()
>From 13cb96cb534854124fda34aef0b4fd9bf4adda8f Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Sat, 12 Sep 2026 01:20:41 +0100
Subject: [PATCH 11/46] [libc] Collect object files behind header libraries
collect_object_file_deps followed object libraries and entrypoints but
had no case for header libraries, so an object library reached only
through a header library was left out of the library being assembled. A
static link hides that, because the program's final link finds the
missing object in libc.a. A shared library is linked on its own and
fails with an undefined hidden symbol instead.
Header libraries are dense in the dependency graph, and walking them
without remembering what has been visited makes configure take many
minutes. Each target now keeps the result of its walk, so later visits
are lookups, and a target reached again during its own walk ends the
cycle. Object library aliases are CMake ALIAS targets, which cannot
carry properties, so the walk resolves them to the target they name
first.
Assisted-by: Claude
---
libc/cmake/modules/LLVMLibCLibraryRules.cmake | 69 ++++++++++---------
1 file changed, 37 insertions(+), 32 deletions(-)
diff --git a/libc/cmake/modules/LLVMLibCLibraryRules.cmake b/libc/cmake/modules/LLVMLibCLibraryRules.cmake
index 46ba4cae98f19b..355d4dfd06275f 100644
--- a/libc/cmake/modules/LLVMLibCLibraryRules.cmake
+++ b/libc/cmake/modules/LLVMLibCLibraryRules.cmake
@@ -1,52 +1,57 @@
function(collect_object_file_deps target result)
# NOTE: This function does add entrypoint targets to |result|.
# It is expected that the caller adds them separately.
- set(all_deps "")
- get_target_property(target_type ${target} "TARGET_TYPE")
- if(NOT target_type)
+ #
+ # Header libraries are dense in the dependency graph, so each target keeps
+ # the result of its walk and every later visit is a lookup. A target reached
+ # again while its own walk is still in progress adds nothing, which ends a
+ # cycle. A dependency named but never defined, as some header libraries
+ # have, has nothing to add either.
+ if(NOT TARGET ${target})
+ set(${result} "" PARENT_SCOPE)
return()
endif()
-
- if(${target_type} STREQUAL ${OBJECT_LIBRARY_TARGET_TYPE})
- list(APPEND all_deps ${target})
- get_target_property(deps ${target} "DEPS")
- foreach(dep IN LISTS deps)
- collect_object_file_deps(${dep} dep_targets)
- list(APPEND all_deps ${dep_targets})
- endforeach()
- list(REMOVE_DUPLICATES all_deps)
+ get_target_property(aliased ${target} "ALIASED_TARGET")
+ if(aliased)
+ set(target ${aliased})
+ endif()
+ get_target_property(done ${target} "COLLECTED_OBJECT_DEPS_DONE")
+ if(done)
+ get_target_property(all_deps ${target} "COLLECTED_OBJECT_DEPS")
set(${result} ${all_deps} PARENT_SCOPE)
return()
endif()
+ get_target_property(in_progress ${target} "COLLECTING_OBJECT_DEPS")
+ get_target_property(target_type ${target} "TARGET_TYPE")
+ if(in_progress OR NOT target_type)
+ set(${result} "" PARENT_SCOPE)
+ return()
+ endif()
+ set_target_properties(${target} PROPERTIES COLLECTING_OBJECT_DEPS TRUE)
- if(${target_type} STREQUAL ${ENTRYPOINT_OBJ_TARGET_TYPE})
- set(entrypoint_target ${target})
- get_target_property(is_alias ${entrypoint_target} "IS_ALIAS")
- if(is_alias)
- get_target_property(aliasee ${entrypoint_target} "DEPS")
- if(NOT aliasee)
- message(FATAL_ERROR
- "Entrypoint alias ${entrypoint_target} does not have an aliasee.")
- endif()
- set(entrypoint_target ${aliasee})
+ set(all_deps "")
+ get_target_property(deps ${target} "DEPS")
+ if(${target_type} STREQUAL ${ENTRYPOINT_EXT_TARGET_TYPE})
+ # It is not possible to recursively extract deps of external dependencies.
+ # So, we just accumulate the direct dep and return.
+ set(all_deps ${deps})
+ elseif((${target_type} STREQUAL ${OBJECT_LIBRARY_TARGET_TYPE}) OR
+ (${target_type} STREQUAL ${ENTRYPOINT_OBJ_TARGET_TYPE}) OR
+ (${target_type} STREQUAL ${HDR_LIBRARY_TARGET_TYPE}))
+ if(${target_type} STREQUAL ${OBJECT_LIBRARY_TARGET_TYPE})
+ list(APPEND all_deps ${target})
endif()
- get_target_property(deps ${target} "DEPS")
foreach(dep IN LISTS deps)
collect_object_file_deps(${dep} dep_targets)
list(APPEND all_deps ${dep_targets})
endforeach()
list(REMOVE_DUPLICATES all_deps)
- set(${result} ${all_deps} PARENT_SCOPE)
- return()
endif()
- if(${target_type} STREQUAL ${ENTRYPOINT_EXT_TARGET_TYPE})
- # It is not possible to recursively extract deps of external dependencies.
- # So, we just accumulate the direct dep and return.
- get_target_property(deps ${target} "DEPS")
- set(${result} ${deps} PARENT_SCOPE)
- return()
- endif()
+ set_target_properties(${target} PROPERTIES
+ COLLECTED_OBJECT_DEPS "${all_deps}"
+ COLLECTED_OBJECT_DEPS_DONE TRUE)
+ set(${result} ${all_deps} PARENT_SCOPE)
endfunction()
function(get_all_object_file_deps result fq_deps_list)
>From 4848899ba2c1c86f53b2514164446b6108e04fef Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Sat, 12 Sep 2026 01:10:50 +0100
Subject: [PATCH 12/46] [libc] Add the libc_assert dependency of fx_bits
fx_bits.h uses LIBC_ASSERT, which in a full build calls internal::exit,
but the header library did not declare it. A static link never notices,
as the program's final link finds internal::exit in libc.a. libm.so is
linked on its own, and without NDEBUG it failed on an undefined hidden
symbol from the fixed point division entrypoints.
Assisted-by: Claude
---
libc/src/__support/fixed_point/CMakeLists.txt | 1 +
1 file changed, 1 insertion(+)
diff --git a/libc/src/__support/fixed_point/CMakeLists.txt b/libc/src/__support/fixed_point/CMakeLists.txt
index c1b0ab30e0307b..f3a252f3e277a7 100644
--- a/libc/src/__support/fixed_point/CMakeLists.txt
+++ b/libc/src/__support/fixed_point/CMakeLists.txt
@@ -22,6 +22,7 @@ add_header_library(
libc.src.__support.CPP.type_traits
libc.src.__support.CPP.bit
libc.src.__support.CPP.limits
+ libc.src.__support.libc_assert
libc.src.__support.math_extras
libc.src.__support.uint128
)
>From 62b6874adb9fc877ab5e04b925e3abf1f2aa098a Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Sat, 12 Sep 2026 01:10:50 +0100
Subject: [PATCH 13/46] [libc] Add the size_t dependency of pwd.h
pwd.yaml declares size_t, but the header target did not depend on it, so
building pwd.h before anything else had installed size_t.h left the
generated header including a file that was not there.
Assisted-by: Claude
---
libc/include/CMakeLists.txt | 1 +
1 file changed, 1 insertion(+)
diff --git a/libc/include/CMakeLists.txt b/libc/include/CMakeLists.txt
index 0130ba1d72e0e1..59fafb6065f764 100644
--- a/libc/include/CMakeLists.txt
+++ b/libc/include/CMakeLists.txt
@@ -623,6 +623,7 @@ add_header_macro(
.llvm-libc-types.struct_passwd
.llvm-libc-types.uid_t
.llvm-libc-types.gid_t
+ .llvm-libc-types.size_t
.llvm_libc_common_h
)
>From 1ff979337bd87c9df338e59ca0eed68be96a334a Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Thu, 24 Sep 2026 03:06:44 +0100
Subject: [PATCH 14/46] [libc] Keep the TLS routines in crt1.o for shared
builds
Upstream moved them into libc.a, where a program linked against libc.so cannot reach them because they are hidden.
Assisted-by: Claude
---
libc/startup/linux/CMakeLists.txt | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/libc/startup/linux/CMakeLists.txt b/libc/startup/linux/CMakeLists.txt
index c0b924ae25908a..566b82e943cdec 100644
--- a/libc/startup/linux/CMakeLists.txt
+++ b/libc/startup/linux/CMakeLists.txt
@@ -140,10 +140,20 @@ add_object_library(
-fno-stack-protector # stack protect canary is not available yet.
)
+# A program linked against libc.so cannot reach the TLS routines there, which
+# are hidden, so a shared build carries its own copy of them in crt1.o. A
+# static link takes them from crt1.o in the same way and leaves libc.a's.
+set(crt1_tls "")
+if(LIBC_ENABLE_SHARED)
+ set(crt1_tls
+ libc.src.__support.threads.linux.${LIBC_TARGET_ARCHITECTURE}.tls)
+endif()
+
# TODO: factor out crt1 into multiple objects
merge_relocatable_object(
crt1
.${LIBC_TARGET_ARCHITECTURE}.start
+ ${crt1_tls}
.${LIBC_TARGET_ARCHITECTURE}.irelative
.do_start
.gnu_property_section
>From 403d6ad8106f4e71a33d2c36eed304f652c7a444 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Tue, 29 Sep 2026 19:09:53 +0100
Subject: [PATCH 15/46] [libc] Keep the preinit_array entry in the init/fini
test under LTO
Nothing refers to the entry, so LTO treated it as dead and the pre-init
function never ran.
Assisted-by: Claude
---
libc/test/integration/startup/linux/init_fini_array_test.cpp | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/libc/test/integration/startup/linux/init_fini_array_test.cpp b/libc/test/integration/startup/linux/init_fini_array_test.cpp
index ed14e03a60c05e..ada64332857147 100644
--- a/libc/test/integration/startup/linux/init_fini_array_test.cpp
+++ b/libc/test/integration/startup/linux/init_fini_array_test.cpp
@@ -52,8 +52,9 @@ __attribute__((destructor(2))) void reset_preinitval() {
preinitval = 0;
}
+// Nothing refers to the entry, so LTO drops it unless it is marked as used.
using PreInitFunc = void();
-__attribute__((section(".preinit_array"))) PreInitFunc *preinit_func_ptr =
+__attribute__((section(".preinit_array"), used)) PreInitFunc *preinit_func_ptr =
&set_preinitval;
TEST_MAIN() {
>From 90b92ef1c4f64aaadbc8f50f72ad6d0a7cf2c443 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Tue, 29 Sep 2026 19:26:13 +0100
Subject: [PATCH 16/46] [libc] Keep LTO out of the compiler feature checks
The checks that link with -nostdlib to see whether the compiler calls a
runtime library never saw the calls under LTO: the functions in them are
unused, so they were dropped before code generation and each check passed.
libc then took the compiler's own float16 conversion, which raises no
exceptions, in place of its own.
Assisted-by: Claude
---
libc/cmake/modules/CheckCompilerFeatures.cmake | 3 +++
1 file changed, 3 insertions(+)
diff --git a/libc/cmake/modules/CheckCompilerFeatures.cmake b/libc/cmake/modules/CheckCompilerFeatures.cmake
index eaf702710b1cce..856e529b955c40 100644
--- a/libc/cmake/modules/CheckCompilerFeatures.cmake
+++ b/libc/cmake/modules/CheckCompilerFeatures.cmake
@@ -78,6 +78,9 @@ foreach(feature IN LISTS ALL_COMPILER_FEATURES)
elseif(${feature} MATCHES "^builtin_" OR
${feature} STREQUAL "float16_conversion")
set(compile_options ${LIBC_COMPILE_OPTIONS_DEFAULT})
+ # Under LTO nothing in the test is used, so it is dropped before the
+ # compiler picks calls to a library, and every check passes.
+ list(APPEND compile_options -fno-lto)
set(link_options -nostdlib)
# The compiler might handle calls to math builtins by generating calls to
# the respective libc math functions, in which case we cannot use these
>From fbd1efb862d20d8a82b4648db980aa41c3b92f60 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Thu, 8 Oct 2026 18:32:14 +0100
Subject: [PATCH 17/46] [libc] Compile the startup objects without LTO
The startup objects are linked into every program. Built with -flto they
are installed as bitcode, which a link without LTO cannot use, and which
lld rejects as an inconsistent LTO unit when the program's own objects
use -fsplit-lto-unit, as CFI builds do.
Assisted-by: Claude
---
libc/startup/linux/CMakeLists.txt | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/libc/startup/linux/CMakeLists.txt b/libc/startup/linux/CMakeLists.txt
index 566b82e943cdec..df42f787c2bfdd 100644
--- a/libc/startup/linux/CMakeLists.txt
+++ b/libc/startup/linux/CMakeLists.txt
@@ -65,7 +65,9 @@ function(add_startup_object name)
${name}
SRCS ${ADD_STARTUP_OBJECT_SRC}
DEPENDS ${ADD_STARTUP_OBJECT_DEPENDS}
- COMPILE_OPTIONS ${ADD_STARTUP_OBJECT_COMPILE_OPTIONS}
+ COMPILE_OPTIONS
+ ${ADD_STARTUP_OBJECT_COMPILE_OPTIONS}
+ -fno-lto
)
set_target_properties(
${fq_target_name}
>From 7af75c5d7d6168bbf2456ef9c1d50e93f7b3f113 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Sat, 10 Oct 2026 03:44:52 +0100
Subject: [PATCH 18/46] [libc] Build getentropy once
sys/random and unistd both provide it, and a shared library does not take two
definitions of one symbol. unistd's has no header of its own.
Assisted-by: Claude
---
libc/config/linux/aarch64/entrypoints.txt | 1 -
libc/config/linux/riscv/entrypoints.txt | 1 -
libc/config/linux/x86_64/entrypoints.txt | 1 -
3 files changed, 3 deletions(-)
diff --git a/libc/config/linux/aarch64/entrypoints.txt b/libc/config/linux/aarch64/entrypoints.txt
index 6963896a9ed6d2..de0d6a589b386f 100644
--- a/libc/config/linux/aarch64/entrypoints.txt
+++ b/libc/config/linux/aarch64/entrypoints.txt
@@ -473,7 +473,6 @@ set(TARGET_LIBC_ENTRYPOINTS
libc.src.unistd.ftruncate
libc.src.unistd.getcwd
libc.src.unistd.getegid
- libc.src.unistd.getentropy
libc.src.unistd.geteuid
libc.src.unistd.gethostname
libc.src.unistd.getpagesize
diff --git a/libc/config/linux/riscv/entrypoints.txt b/libc/config/linux/riscv/entrypoints.txt
index 18c6209434a6a3..cdcc29a8d47329 100644
--- a/libc/config/linux/riscv/entrypoints.txt
+++ b/libc/config/linux/riscv/entrypoints.txt
@@ -497,7 +497,6 @@ set(TARGET_LIBC_ENTRYPOINTS
libc.src.unistd.ftruncate
libc.src.unistd.getcwd
libc.src.unistd.getegid
- libc.src.unistd.getentropy
libc.src.unistd.geteuid
libc.src.unistd.gethostname
libc.src.unistd.getpagesize
diff --git a/libc/config/linux/x86_64/entrypoints.txt b/libc/config/linux/x86_64/entrypoints.txt
index b2747d2dfeb929..fba943b2d267c4 100644
--- a/libc/config/linux/x86_64/entrypoints.txt
+++ b/libc/config/linux/x86_64/entrypoints.txt
@@ -517,7 +517,6 @@ set(TARGET_LIBC_ENTRYPOINTS
libc.src.unistd.ftruncate
libc.src.unistd.getcwd
libc.src.unistd.getegid
- libc.src.unistd.getentropy
libc.src.unistd.geteuid
libc.src.unistd.gethostname
libc.src.unistd.getpagesize
>From fdf5b51823441234c6046741939031f01fa3ccd6 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Sun, 11 Oct 2026 11:22:27 +0100
Subject: [PATCH 19/46] [libc] Align hermetic test allocations to max_align_t
The hermetic allocator handed out blocks aligned to a pointer. The compiler
may assume operator new returns memory aligned for the largest scalar type,
so an object zeroed with aligned vector stores faulted: dir_scan_impl_test
crashed in ReadFailsAtStart.
Assisted-by: Claude
---
libc/test/UnitTest/HermeticTestUtils.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/libc/test/UnitTest/HermeticTestUtils.cpp b/libc/test/UnitTest/HermeticTestUtils.cpp
index 4a5cbd8040b52d..dbd53be4cb53ae 100644
--- a/libc/test/UnitTest/HermeticTestUtils.cpp
+++ b/libc/test/UnitTest/HermeticTestUtils.cpp
@@ -30,7 +30,7 @@ int atexit(void (*func)(void));
} // namespace LIBC_NAMESPACE_DECL
-constexpr uint64_t ALIGNMENT = alignof(uintptr_t);
+constexpr uint64_t ALIGNMENT = alignof(max_align_t);
namespace {
>From ddfb8c56fecff5221a35f9b2a9cc59a161749b5a Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Mon, 12 Oct 2026 04:42:02 +0100
Subject: [PATCH 20/46] [scudo] Serve from the secondary when the primary
cannot reserve its regions
The primary reserves address space for every size class at start. Under
RLIMIT_AS that fails and the process was aborted on its first malloc.
Fall back to the secondary for every request instead, as other
allocators fail only the allocations that do not fit.
Assisted-by: Claude
---
compiler-rt/lib/scudo/standalone/combined.h | 2 +-
compiler-rt/lib/scudo/standalone/primary32.h | 1 +
compiler-rt/lib/scudo/standalone/primary64.h | 27 +++++++++++++-------
3 files changed, 20 insertions(+), 10 deletions(-)
diff --git a/compiler-rt/lib/scudo/standalone/combined.h b/compiler-rt/lib/scudo/standalone/combined.h
index b90492175d6007..860faf90d353eb 100644
--- a/compiler-rt/lib/scudo/standalone/combined.h
+++ b/compiler-rt/lib/scudo/standalone/combined.h
@@ -409,7 +409,7 @@ class Allocator {
void *Block = nullptr;
uptr ClassId = 0;
uptr SecondaryBlockEnd = 0;
- if (LIKELY(PrimaryT::canAllocate(NeededSize))) {
+ if (LIKELY(PrimaryT::canAllocate(NeededSize) && Primary.isUsable())) {
ClassId = SizeClassMap::getClassIdBySize(NeededSize);
DCHECK_NE(ClassId, 0U);
typename TSDRegistryT::ScopedTSD TSD(TSDRegistry);
diff --git a/compiler-rt/lib/scudo/standalone/primary32.h b/compiler-rt/lib/scudo/standalone/primary32.h
index 10a786e81a7ab3..359941cc51244b 100644
--- a/compiler-rt/lib/scudo/standalone/primary32.h
+++ b/compiler-rt/lib/scudo/standalone/primary32.h
@@ -76,6 +76,7 @@ template <typename Config> class SizeClassAllocator32 {
}
static bool canAllocate(uptr Size) { return Size <= SizeClassMap::MaxSize; }
+ bool isUsable() const { return true; }
void init(s32 ReleaseToOsInterval) NO_THREAD_SAFETY_ANALYSIS;
diff --git a/compiler-rt/lib/scudo/standalone/primary64.h b/compiler-rt/lib/scudo/standalone/primary64.h
index d14e03c3a7a942..2acaf0bc3cc578 100644
--- a/compiler-rt/lib/scudo/standalone/primary64.h
+++ b/compiler-rt/lib/scudo/standalone/primary64.h
@@ -86,6 +86,10 @@ template <typename Config> class SizeClassAllocator64 {
}
static bool canAllocate(uptr Size) { return Size <= SizeClassMap::MaxSize; }
+ // False when the address space for the regions could not be reserved, which
+ // is what a limit on it (RLIMIT_AS) does. Every request is then served by
+ // the secondary, one mapping each, rather than the process being aborted.
+ bool isUsable() const { return !Unavailable; }
static constexpr bool conditionVariableEnabled() {
return Config::hasConditionVariableT();
}
@@ -140,6 +144,7 @@ template <typename Config> class SizeClassAllocator64 {
AtomicOptions Options;
private:
+ bool Unavailable = false;
static const uptr RegionSize = 1UL << RegionSizeLog;
static const uptr NumClasses = SizeClassMap::NumClasses;
static const uptr MapSizeIncrement = Config::getMapSizeIncrement();
@@ -387,18 +392,22 @@ void SizeClassAllocator64<Config>::init(s32 ReleaseToOsInterval)
if (Config::getEnableContiguousRegions()) {
ReservedMemoryT ReservedMemory = {};
// Reserve the space required for the Primary.
- CHECK(ReservedMemory.create(/*Addr=*/0U, RegionSize * NumClasses,
- "scudo:primary_reserve"));
- const uptr PrimaryBase = ReservedMemory.getBase();
+ if (UNLIKELY(!ReservedMemory.create(/*Addr=*/0U, RegionSize * NumClasses,
+ "scudo:primary_reserve",
+ MAP_ALLOWNOMEM))) {
+ Unavailable = true;
+ } else {
+ const uptr PrimaryBase = ReservedMemory.getBase();
- for (uptr I = 0; I < NumClasses; I++) {
- MemMapT RegionMemMap = ReservedMemory.dispatch(
- PrimaryBase + (I << RegionSizeLog), RegionSize);
- RegionInfo *Region = getRegionInfo(I);
+ for (uptr I = 0; I < NumClasses; I++) {
+ MemMapT RegionMemMap = ReservedMemory.dispatch(
+ PrimaryBase + (I << RegionSizeLog), RegionSize);
+ RegionInfo *Region = getRegionInfo(I);
- initRegion(Region, I, RegionMemMap, Config::getEnableRandomOffset());
+ initRegion(Region, I, RegionMemMap, Config::getEnableRandomOffset());
+ }
+ shuffle(RegionInfoArray, NumClasses, &Seed);
}
- shuffle(RegionInfoArray, NumClasses, &Seed);
}
if constexpr (SCUDO_DEBUG && conditionVariableEnabled()) {
>From fd367fb54cb9a3abdefb8446adc0056ce316e3cc Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 04:09:13 +0100
Subject: [PATCH 21/46] [libc] Report the correct load bias from
dl_iterate_phdr
dlpi_addr was hardcoded to zero with a TODO, which is only correct for a
non-PIE executable. Callers use dlpi_addr + p_vaddr to find a segment at
runtime, so for a PIE executable every such address came out wrong. The
vDSO case already computed the bias correctly; hoist that into a helper
and use it for both modules.
Also give the vDSO its usual name instead of an empty string, which the
main executable is meant to have exclusively.
Wire runtimes-libc-shared up to the shared objects now that
LIBC_ENABLE_SHARED can produce them, replacing the TODO in
HandleLibC.cmake. It stays empty when the option is off.
Assisted-by: Claude
---
libc/src/link/CMakeLists.txt | 1 -
libc/src/link/dl_iterate_phdr.cpp | 40 ++++++++++---------
libc/test/src/link/dl_iterate_phdr_test.cpp | 43 +++++++++++++++++++++
runtimes/cmake/Modules/HandleLibC.cmake | 12 +++++-
4 files changed, 76 insertions(+), 20 deletions(-)
diff --git a/libc/src/link/CMakeLists.txt b/libc/src/link/CMakeLists.txt
index c3fcdb3c1656ed..19c1ba645840ff 100644
--- a/libc/src/link/CMakeLists.txt
+++ b/libc/src/link/CMakeLists.txt
@@ -20,6 +20,5 @@ add_entrypoint_object(
libc.hdr.stdint_proxy
libc.include.elf
libc.include.llvm-libc-macros.link_macros
- libc.src.__support.CPP.span
libc.src.__support.OSUtil.linux.auxv
)
diff --git a/libc/src/link/dl_iterate_phdr.cpp b/libc/src/link/dl_iterate_phdr.cpp
index 166f50fdfee867..01d11de3673e9b 100644
--- a/libc/src/link/dl_iterate_phdr.cpp
+++ b/libc/src/link/dl_iterate_phdr.cpp
@@ -14,7 +14,7 @@
#include "dl_iterate_phdr.h"
#include "llvm-libc-macros/link-macros.h"
-#include "src/__support/CPP/span.h"
+#include "src/__support/CPP/mutex.h"
#include "src/__support/OSUtil/linux/auxv.h"
#include "src/__support/common.h"
#include "src/__support/macros/config.h"
@@ -25,15 +25,26 @@ extern "C" void *__ehdr_start;
namespace LIBC_NAMESPACE_DECL {
-struct dl_phdr_info create_executable_info(ElfW(Ehdr) * executable_header) {
- // TODO: Calculate dlpi_addr in the PIE case and set dlpi_name for VDSO.
+// The name the kernel gives the vDSO in the link map.
+constexpr const char *VDSO_NAME = "linux-vdso.so.1";
+
+// The load bias is the distance between where the first PT_LOAD segment was
+// linked and where it actually ended up. It is zero for a non-PIE executable
+// and non-zero for a PIE one or for the vDSO.
+static ElfW(Addr) load_bias(ElfW(Ehdr) * header, const ElfW(Phdr) * phdrs) {
+ for (ElfW(Half) i = 0; i < header->e_phnum; ++i)
+ if (phdrs[i].p_type == PT_LOAD)
+ return reinterpret_cast<ElfW(Addr)>(header) - phdrs[i].p_vaddr;
+ return 0;
+}
+
+struct dl_phdr_info create_module_info(ElfW(Ehdr) * header, const char *name) {
struct dl_phdr_info to_return;
- to_return.dlpi_addr = 0;
- to_return.dlpi_name = "";
to_return.dlpi_phdr = reinterpret_cast<ElfW(Phdr) *>(
- reinterpret_cast<uintptr_t>(executable_header) +
- executable_header->e_phoff);
- to_return.dlpi_phnum = executable_header->e_phnum;
+ reinterpret_cast<uintptr_t>(header) + header->e_phoff);
+ to_return.dlpi_phnum = header->e_phnum;
+ to_return.dlpi_addr = load_bias(header, to_return.dlpi_phdr);
+ to_return.dlpi_name = name;
to_return.dlpi_adds = 0;
to_return.dlpi_subs = 0;
to_return.dlpi_tls_modid = 0;
@@ -44,8 +55,9 @@ struct dl_phdr_info create_executable_info(ElfW(Ehdr) * executable_header) {
LLVM_LIBC_FUNCTION(int, dl_iterate_phdr,
(__dl_iterate_phdr_callback_t callback, void *arg)) {
ElfW(Ehdr) *executable_header = reinterpret_cast<ElfW(Ehdr) *>(&__ehdr_start);
+ // The main executable is always reported with an empty name.
struct dl_phdr_info executable_info =
- create_executable_info(executable_header);
+ create_module_info(executable_header, "");
int executable_return_code =
callback(&executable_info, sizeof(executable_info), arg);
if (executable_return_code != 0)
@@ -57,15 +69,7 @@ LLVM_LIBC_FUNCTION(int, dl_iterate_phdr,
ElfW(Ehdr) *vdso_header = reinterpret_cast<ElfW(Ehdr) *>(*vdso_start_address);
if (vdso_header == nullptr)
return 0;
- struct dl_phdr_info vdso_info = create_executable_info(vdso_header);
- for (auto elf_headers :
- cpp::span<const ElfW(Phdr)>(vdso_info.dlpi_phdr, vdso_info.dlpi_phnum)) {
- if (elf_headers.p_type == PT_LOAD) {
- vdso_info.dlpi_addr =
- reinterpret_cast<ElfW(Addr)>(vdso_header) - elf_headers.p_vaddr;
- break;
- }
- }
+ struct dl_phdr_info vdso_info = create_module_info(vdso_header, VDSO_NAME);
return callback(&vdso_info, sizeof(vdso_info), arg);
}
diff --git a/libc/test/src/link/dl_iterate_phdr_test.cpp b/libc/test/src/link/dl_iterate_phdr_test.cpp
index c3c8a5e89663c8..f8f3a7a4417e7b 100644
--- a/libc/test/src/link/dl_iterate_phdr_test.cpp
+++ b/libc/test/src/link/dl_iterate_phdr_test.cpp
@@ -15,6 +15,10 @@
#include "src/link/dl_iterate_phdr.h"
#include "test/UnitTest/Test.h"
+#include <elf.h>
+
+extern "C" void *__ehdr_start;
+
int save_return_1(struct dl_phdr_info *info, [[maybe_unused]] size_t info_size,
void *arg) {
*static_cast<int *>(arg) = info->dlpi_phnum;
@@ -40,3 +44,42 @@ TEST(LlvmLibcLinkDlIteratePhdrTest, BothExecutableAndVDSO) {
LIBC_NAMESPACE::dl_iterate_phdr(save_return_0, &program_header_count), 0);
EXPECT_GT(program_header_count, 0);
}
+
+// dlpi_addr must be the load bias, so that dlpi_addr + p_vaddr gives the
+// runtime address of a segment. Previously it was hardcoded to zero, which is
+// only correct for a non-PIE executable.
+struct BiasCheck {
+ bool saw_executable = false;
+ bool executable_bias_correct = false;
+ bool all_names_non_null = true;
+};
+
+int check_bias(struct dl_phdr_info *info, [[maybe_unused]] size_t info_size,
+ void *arg) {
+ BiasCheck *state = static_cast<BiasCheck *>(arg);
+ if (info->dlpi_name == nullptr) {
+ state->all_names_non_null = false;
+ return 0;
+ }
+ // The first module reported is the main executable, named "".
+ if (!state->saw_executable) {
+ state->saw_executable = true;
+ for (ElfW(Half) i = 0; i < info->dlpi_phnum; ++i) {
+ if (info->dlpi_phdr[i].p_type == PT_LOAD) {
+ ElfW(Addr) runtime = info->dlpi_addr + info->dlpi_phdr[i].p_vaddr;
+ state->executable_bias_correct =
+ runtime == reinterpret_cast<ElfW(Addr)>(&__ehdr_start);
+ break;
+ }
+ }
+ }
+ return 0;
+}
+
+TEST(LlvmLibcLinkDlIteratePhdrTest, LoadBiasIsCorrect) {
+ BiasCheck state;
+ EXPECT_EQ(LIBC_NAMESPACE::dl_iterate_phdr(check_bias, &state), 0);
+ EXPECT_TRUE(state.saw_executable);
+ EXPECT_TRUE(state.executable_bias_correct);
+ EXPECT_TRUE(state.all_names_non_null);
+}
diff --git a/runtimes/cmake/Modules/HandleLibC.cmake b/runtimes/cmake/Modules/HandleLibC.cmake
index fdf176f3fe874b..97bc875913bdca 100644
--- a/runtimes/cmake/Modules/HandleLibC.cmake
+++ b/runtimes/cmake/Modules/HandleLibC.cmake
@@ -57,6 +57,16 @@ elseif (RUNTIMES_USE_LIBC STREQUAL "llvm-libc")
target_link_options(runtimes-libc-static INTERFACE "-nolibc")
endif()
- # TODO: There's no support for building LLVM libc as a shared library yet.
+ # LLVM libc only builds shared objects when LIBC_ENABLE_SHARED is set, so
+ # this stays empty otherwise and the runtimes fall back to a static link.
add_library(runtimes-libc-shared INTERFACE)
+ if (TARGET libc-shared)
+ target_link_libraries(runtimes-libc-shared INTERFACE libc-shared)
+ endif()
+ if (TARGET libm-shared)
+ target_link_libraries(runtimes-libc-shared INTERFACE libm-shared)
+ endif()
+ if (CXX_SUPPORTS_NOLIBC_FLAG AND (TARGET libc-shared OR TARGET libm-shared))
+ target_link_options(runtimes-libc-shared INTERFACE "-nolibc")
+ endif()
endif()
>From f0086a5fff712dbb7f4de206889339ff9b5ce819 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 04:13:16 +0100
Subject: [PATCH 22/46] [libc] Add a reader for a module's PT_DYNAMIC array
First piece of the ELF support a dynamic loader needs. DynamicTable
walks the DT_NULL terminated array a PT_DYNAMIC segment points at.
Tag values are either a plain integer or an address that was recorded
relative to how the module was linked. Mixing those up is an easy way to
get a loader wrong, so value() and address() are separate and only
address() applies the load bias. for_each() covers the tags that may
repeat, DT_NEEDED in particular.
Header only, no allocation, so it is usable from the startup path where
there is no libc yet.
Assisted-by: Claude
---
libc/src/__support/CMakeLists.txt | 1 +
libc/src/__support/elf/CMakeLists.txt | 13 +++
libc/src/__support/elf/dynamic.h | 98 ++++++++++++++++++++
libc/test/src/__support/CMakeLists.txt | 1 +
libc/test/src/__support/elf/CMakeLists.txt | 11 +++
libc/test/src/__support/elf/dynamic_test.cpp | 93 +++++++++++++++++++
6 files changed, 217 insertions(+)
create mode 100644 libc/src/__support/elf/CMakeLists.txt
create mode 100644 libc/src/__support/elf/dynamic.h
create mode 100644 libc/test/src/__support/elf/CMakeLists.txt
create mode 100644 libc/test/src/__support/elf/dynamic_test.cpp
diff --git a/libc/src/__support/CMakeLists.txt b/libc/src/__support/CMakeLists.txt
index dfb1ed0e13ff39..cb03582f8f14c4 100644
--- a/libc/src/__support/CMakeLists.txt
+++ b/libc/src/__support/CMakeLists.txt
@@ -497,6 +497,7 @@ add_header_library(
libc.src.__support.macros.config
)
+add_subdirectory(elf)
add_subdirectory(FPUtil)
add_subdirectory(OSUtil)
add_subdirectory(StringUtil)
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
new file mode 100644
index 00000000000000..4c39881aaeb341
--- /dev/null
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -0,0 +1,13 @@
+add_header_library(
+ dynamic
+ HDRS
+ dynamic.h
+ DEPENDS
+ libc.hdr.elf_macros
+ libc.hdr.elf_proxy
+ libc.hdr.link_macros
+ libc.hdr.stdint_proxy
+ libc.src.__support.CPP.optional
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+)
diff --git a/libc/src/__support/elf/dynamic.h b/libc/src/__support/elf/dynamic.h
new file mode 100644
index 00000000000000..561a02cb5e3c06
--- /dev/null
+++ b/libc/src/__support/elf/dynamic.h
@@ -0,0 +1,98 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Reader for a module's PT_DYNAMIC array
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_DYNAMIC_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_DYNAMIC_H
+
+#include "hdr/elf_macros.h"
+#include "hdr/elf_proxy.h"
+#include "hdr/link_macros.h"
+#include "hdr/stdint_proxy.h"
+#include "src/__support/CPP/optional.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// A view over the DT_NULL terminated array a PT_DYNAMIC segment points at.
+//
+// Tag values are either a plain integer or an address. An address is stored
+// relative to how the module was linked, so it has to be adjusted by the load
+// bias to be usable at runtime. The two accessors keep that distinction
+// explicit, since applying the bias to an integer is a common way to get a
+// loader subtly wrong.
+class DynamicTable {
+public:
+ LIBC_INLINE constexpr DynamicTable() = default;
+ LIBC_INLINE constexpr DynamicTable(const ElfW(Dyn) * entries,
+ ElfW(Addr) load_bias)
+ : entries_(entries), load_bias_(load_bias) {}
+
+ LIBC_INLINE constexpr bool empty() const { return entries_ == nullptr; }
+ LIBC_INLINE constexpr ElfW(Addr) load_bias() const { return load_bias_; }
+
+ // The d_un.d_val of the first entry with this tag.
+ LIBC_INLINE cpp::optional<ElfW(Xword)> value(ElfW(Sxword) tag) const {
+ if (const ElfW(Dyn) *entry = find(tag))
+ return entry->d_un.d_val;
+ return cpp::nullopt;
+ }
+
+ // The d_un.d_ptr of the first entry with this tag, adjusted by the load
+ // bias so that it is valid in this process.
+ LIBC_INLINE cpp::optional<ElfW(Addr)> address(ElfW(Sxword) tag) const {
+ if (const ElfW(Dyn) *entry = find(tag))
+ return entry->d_un.d_ptr + load_bias_;
+ return cpp::nullopt;
+ }
+
+ LIBC_INLINE bool contains(ElfW(Sxword) tag) const {
+ return find(tag) != nullptr;
+ }
+
+ // Calls callback(const ElfW(Dyn) &) for every entry with this tag. DT_NEEDED
+ // is the reason this exists: a module may carry any number of them.
+ template <typename F>
+ LIBC_INLINE void for_each(ElfW(Sxword) tag, F callback) const {
+ if (entries_ == nullptr)
+ return;
+ for (const ElfW(Dyn) *entry = entries_; entry->d_tag != DT_NULL; ++entry)
+ if (entry->d_tag == tag)
+ callback(*entry);
+ }
+
+ LIBC_INLINE size_t count(ElfW(Sxword) tag) const {
+ size_t found = 0;
+ for_each(tag, [&found](const ElfW(Dyn) &) { ++found; });
+ return found;
+ }
+
+private:
+ LIBC_INLINE const ElfW(Dyn) * find(ElfW(Sxword) tag) const {
+ if (entries_ == nullptr)
+ return nullptr;
+ for (const ElfW(Dyn) *entry = entries_; entry->d_tag != DT_NULL; ++entry)
+ if (entry->d_tag == tag)
+ return entry;
+ return nullptr;
+ }
+
+ const ElfW(Dyn) *entries_ = nullptr;
+ ElfW(Addr) load_bias_ = 0;
+};
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_DYNAMIC_H
diff --git a/libc/test/src/__support/CMakeLists.txt b/libc/test/src/__support/CMakeLists.txt
index 3f52faa8061087..e254ed77304f20 100644
--- a/libc/test/src/__support/CMakeLists.txt
+++ b/libc/test/src/__support/CMakeLists.txt
@@ -362,6 +362,7 @@ add_libc_test(
libc.src.__support.macros.properties.os
)
+add_subdirectory(elf)
add_subdirectory(CPP)
add_subdirectory(File)
add_subdirectory(pwd)
diff --git a/libc/test/src/__support/elf/CMakeLists.txt b/libc/test/src/__support/elf/CMakeLists.txt
new file mode 100644
index 00000000000000..6291adf3de4811
--- /dev/null
+++ b/libc/test/src/__support/elf/CMakeLists.txt
@@ -0,0 +1,11 @@
+add_libc_testsuite(libc-elf-tests)
+
+add_libc_test(
+ dynamic_test
+ SUITE
+ libc-support-tests
+ SRCS
+ dynamic_test.cpp
+ DEPENDS
+ libc.src.__support.elf.dynamic
+)
diff --git a/libc/test/src/__support/elf/dynamic_test.cpp b/libc/test/src/__support/elf/dynamic_test.cpp
new file mode 100644
index 00000000000000..9178bc6c2e30bf
--- /dev/null
+++ b/libc/test/src/__support/elf/dynamic_test.cpp
@@ -0,0 +1,93 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Unittests for the PT_DYNAMIC reader
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/elf/dynamic.h"
+#include "test/UnitTest/Test.h"
+
+using LIBC_NAMESPACE::elf::DynamicTable;
+
+namespace {
+
+// A synthetic dynamic array, so the test does not depend on how the test
+// binary itself was linked.
+constexpr ElfW(Addr) LOAD_BIAS = 0x400000;
+
+// Aggregate initialised at namespace scope so there is no guard variable.
+const ElfW(Dyn) SAMPLE[] = {
+ {DT_NEEDED, {1}},
+ {DT_SONAME, {17}},
+ {DT_STRTAB, {0x1000}},
+ {DT_SYMTAB, {0x2000}},
+ {DT_NEEDED, {33}},
+ {DT_RELASZ, {0x180}},
+ {DT_NEEDED, {49}},
+ {DT_NULL, {0}},
+ // Anything past DT_NULL must not be seen.
+ {DT_RPATH, {99}},
+};
+
+} // anonymous namespace
+
+TEST(LlvmLibcElfDynamicTest, DefaultIsEmpty) {
+ DynamicTable table;
+ EXPECT_TRUE(table.empty());
+ EXPECT_FALSE(table.value(DT_SONAME).has_value());
+ EXPECT_FALSE(table.address(DT_STRTAB).has_value());
+ EXPECT_FALSE(table.contains(DT_NEEDED));
+ EXPECT_EQ(table.count(DT_NEEDED), size_t(0));
+}
+
+TEST(LlvmLibcElfDynamicTest, ValueIsNotBiased) {
+ DynamicTable table(SAMPLE, LOAD_BIAS);
+ ASSERT_TRUE(table.value(DT_RELASZ).has_value());
+ EXPECT_EQ(*table.value(DT_RELASZ), ElfW(Xword)(0x180));
+ ASSERT_TRUE(table.value(DT_SONAME).has_value());
+ EXPECT_EQ(*table.value(DT_SONAME), ElfW(Xword)(17));
+}
+
+TEST(LlvmLibcElfDynamicTest, AddressIsBiased) {
+ DynamicTable table(SAMPLE, LOAD_BIAS);
+ ASSERT_TRUE(table.address(DT_STRTAB).has_value());
+ EXPECT_EQ(*table.address(DT_STRTAB), ElfW(Addr)(LOAD_BIAS + 0x1000));
+ ASSERT_TRUE(table.address(DT_SYMTAB).has_value());
+ EXPECT_EQ(*table.address(DT_SYMTAB), ElfW(Addr)(LOAD_BIAS + 0x2000));
+}
+
+TEST(LlvmLibcElfDynamicTest, FindsFirstMatchOnly) {
+ DynamicTable table(SAMPLE, LOAD_BIAS);
+ ASSERT_TRUE(table.value(DT_NEEDED).has_value());
+ EXPECT_EQ(*table.value(DT_NEEDED), ElfW(Xword)(1));
+}
+
+TEST(LlvmLibcElfDynamicTest, IteratesEveryMatch) {
+ DynamicTable table(SAMPLE, LOAD_BIAS);
+ EXPECT_EQ(table.count(DT_NEEDED), size_t(3));
+
+ ElfW(Xword) total = 0;
+ table.for_each(DT_NEEDED,
+ [&total](const ElfW(Dyn) &e) { total += e.d_un.d_val; });
+ EXPECT_EQ(total, ElfW(Xword)(1 + 33 + 49));
+}
+
+TEST(LlvmLibcElfDynamicTest, StopsAtDtNull) {
+ DynamicTable table(SAMPLE, LOAD_BIAS);
+ EXPECT_FALSE(table.contains(DT_RPATH));
+ EXPECT_EQ(table.count(DT_RPATH), size_t(0));
+}
+
+TEST(LlvmLibcElfDynamicTest, MissingTagsAreAbsent) {
+ DynamicTable table(SAMPLE, LOAD_BIAS);
+ EXPECT_FALSE(table.contains(DT_JMPREL));
+ EXPECT_FALSE(table.value(DT_JMPREL).has_value());
+ EXPECT_FALSE(table.address(DT_JMPREL).has_value());
+}
>From c8ad7efb4e14cd7f5ea0959bbad45a6df0f172d6 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 04:16:12 +0100
Subject: [PATCH 23/46] [libc] Add the ELF symbol hash functions
Second piece of the ELF support a dynamic loader needs. sysv_hash backs
DT_HASH and gnu_hash backs DT_GNU_HASH.
Both are constexpr so a lookup of a fixed name can hash at compile time.
The test pins known values; the gnu_hash ones were confirmed by walking
glibc's own .gnu.hash bucket and chain arrays and checking that each
hash resolves to the expected entry in .dynsym.
Characters are taken as unsigned, which the tests cover, since sign
extending a byte above 0x7f gives the wrong hash for any non-ASCII
symbol name.
Assisted-by: Claude
---
libc/src/__support/elf/CMakeLists.txt | 10 ++++
libc/src/__support/elf/hash.h | 49 +++++++++++++++++++
libc/test/src/__support/elf/CMakeLists.txt | 10 ++++
libc/test/src/__support/elf/hash_test.cpp | 55 ++++++++++++++++++++++
4 files changed, 124 insertions(+)
create mode 100644 libc/src/__support/elf/hash.h
create mode 100644 libc/test/src/__support/elf/hash_test.cpp
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
index 4c39881aaeb341..a074bea930f0a4 100644
--- a/libc/src/__support/elf/CMakeLists.txt
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -11,3 +11,13 @@ add_header_library(
libc.src.__support.macros.attributes
libc.src.__support.macros.config
)
+
+add_header_library(
+ hash
+ HDRS
+ hash.h
+ DEPENDS
+ libc.hdr.stdint_proxy
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+)
diff --git a/libc/src/__support/elf/hash.h b/libc/src/__support/elf/hash.h
new file mode 100644
index 00000000000000..a093f4364740b2
--- /dev/null
+++ b/libc/src/__support/elf/hash.h
@@ -0,0 +1,49 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// ELF symbol hash functions
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_HASH_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_HASH_H
+
+#include "hdr/stdint_proxy.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// The hash used by DT_HASH, defined by the System V ABI. The result is 28
+// bits: the top nibble is folded back in and then cleared on every round.
+LIBC_INLINE constexpr uint32_t sysv_hash(const char *name) {
+ uint32_t h = 0;
+ for (; *name != '\0'; ++name) {
+ h = (h << 4) + static_cast<unsigned char>(*name);
+ uint32_t high = h & 0xf0000000u;
+ if (high != 0)
+ h ^= high >> 24;
+ h &= ~high;
+ }
+ return h;
+}
+
+// The hash used by DT_GNU_HASH, a djb2 variant.
+LIBC_INLINE constexpr uint32_t gnu_hash(const char *name) {
+ uint32_t h = 5381;
+ for (; *name != '\0'; ++name)
+ h = h * 33 + static_cast<unsigned char>(*name);
+ return h;
+}
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_HASH_H
diff --git a/libc/test/src/__support/elf/CMakeLists.txt b/libc/test/src/__support/elf/CMakeLists.txt
index 6291adf3de4811..4d7e9cfc656557 100644
--- a/libc/test/src/__support/elf/CMakeLists.txt
+++ b/libc/test/src/__support/elf/CMakeLists.txt
@@ -9,3 +9,13 @@ add_libc_test(
DEPENDS
libc.src.__support.elf.dynamic
)
+
+add_libc_test(
+ hash_test
+ SUITE
+ libc-support-tests
+ SRCS
+ hash_test.cpp
+ DEPENDS
+ libc.src.__support.elf.hash
+)
diff --git a/libc/test/src/__support/elf/hash_test.cpp b/libc/test/src/__support/elf/hash_test.cpp
new file mode 100644
index 00000000000000..001a17289cf44f
--- /dev/null
+++ b/libc/test/src/__support/elf/hash_test.cpp
@@ -0,0 +1,55 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Unittests for the ELF symbol hash functions
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/elf/hash.h"
+#include "test/UnitTest/Test.h"
+
+using LIBC_NAMESPACE::elf::gnu_hash;
+using LIBC_NAMESPACE::elf::sysv_hash;
+
+TEST(LlvmLibcElfHashTest, SysVKnownValues) {
+ EXPECT_EQ(sysv_hash(""), uint32_t(0x00000000));
+ EXPECT_EQ(sysv_hash("printf"), uint32_t(0x077905a6));
+ EXPECT_EQ(sysv_hash("malloc"), uint32_t(0x07383353));
+ EXPECT_EQ(sysv_hash("_init"), uint32_t(0x00660504));
+ EXPECT_EQ(sysv_hash("__libc_start_main"), uint32_t(0x0177ff8e));
+}
+
+TEST(LlvmLibcElfHashTest, SysVResultFitsIn28Bits) {
+ // The top nibble is cleared every round, so no input can set it.
+ EXPECT_EQ(sysv_hash("glibc_hash_collision_probe") & 0xf0000000u, uint32_t(0));
+ EXPECT_EQ(sysv_hash("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa") & 0xf0000000u,
+ uint32_t(0));
+}
+
+TEST(LlvmLibcElfHashTest, GnuKnownValues) {
+ // The empty string yields the djb2 seed.
+ EXPECT_EQ(gnu_hash(""), uint32_t(5381));
+ EXPECT_EQ(gnu_hash("printf"), uint32_t(0x156b2bb8));
+ EXPECT_EQ(gnu_hash("malloc"), uint32_t(0x0d39ad3d));
+ EXPECT_EQ(gnu_hash("_init"), uint32_t(0x0ef18db8));
+ EXPECT_EQ(gnu_hash("__libc_start_main"), uint32_t(0xf63d4e2e));
+}
+
+TEST(LlvmLibcElfHashTest, HashesAreUsableAtCompileTime) {
+ static_assert(sysv_hash("printf") == 0x077905a6u);
+ static_assert(gnu_hash("printf") == 0x156b2bb8u);
+ EXPECT_TRUE(true);
+}
+
+TEST(LlvmLibcElfHashTest, HighBytesAreNotSignExtended) {
+ // A char above 0x7f must be treated as unsigned.
+ const char high[] = {static_cast<char>(0xff), '\0'};
+ EXPECT_EQ(sysv_hash(high), uint32_t(0xff));
+ EXPECT_EQ(gnu_hash(high), uint32_t(5381u * 33u + 0xffu));
+}
>From 1cb493d18c0f267a9e59b9e9f338eabc709c5dc2 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 04:19:13 +0100
Subject: [PATCH 24/46] [libc] Add GNU hash symbol lookup
Third piece of the ELF support a dynamic loader needs. GnuSymbolTable
resolves a name against one module's .dynsym through its DT_GNU_HASH
table.
The Bloom filter is consulted first. It can only give a false positive,
so a miss there is conclusive and skips the chain walk, which is what
makes GNU hash faster than DT_HASH. The chain array stores each hash
with its low bit used as an end of chain marker, so the comparison masks
that bit off rather than treating it as part of the hash.
The test data is a hash table laid out the way a linker emits one, with
two buckets in use and four symbols sharing a chain, so the walk is
actually exercised rather than every lookup landing on a bucket head.
Assisted-by: Claude
---
libc/src/__support/elf/CMakeLists.txt | 14 +++
libc/src/__support/elf/symbol.h | 115 ++++++++++++++++++++
libc/test/src/__support/elf/CMakeLists.txt | 10 ++
libc/test/src/__support/elf/symbol_test.cpp | 101 +++++++++++++++++
4 files changed, 240 insertions(+)
create mode 100644 libc/src/__support/elf/symbol.h
create mode 100644 libc/test/src/__support/elf/symbol_test.cpp
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
index a074bea930f0a4..b63a8a1f023f6b 100644
--- a/libc/src/__support/elf/CMakeLists.txt
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -21,3 +21,17 @@ add_header_library(
libc.src.__support.macros.attributes
libc.src.__support.macros.config
)
+
+add_header_library(
+ symbol
+ HDRS
+ symbol.h
+ DEPENDS
+ libc.hdr.elf_macros
+ libc.hdr.elf_proxy
+ libc.hdr.link_macros
+ libc.hdr.stdint_proxy
+ libc.src.__support.elf.hash
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+)
diff --git a/libc/src/__support/elf/symbol.h b/libc/src/__support/elf/symbol.h
new file mode 100644
index 00000000000000..3be6d15cf6525e
--- /dev/null
+++ b/libc/src/__support/elf/symbol.h
@@ -0,0 +1,115 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Symbol lookup in a loaded ELF module
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_SYMBOL_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_SYMBOL_H
+
+#include "hdr/elf_macros.h"
+#include "hdr/elf_proxy.h"
+#include "hdr/link_macros.h"
+#include "hdr/stdint_proxy.h"
+#include "src/__support/elf/hash.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// The header of a DT_GNU_HASH table, followed by the Bloom filter, the bucket
+// array and the hash array.
+struct GnuHashHeader {
+ uint32_t nbuckets;
+ // Symbols below this index are not in the hash table at all.
+ uint32_t symoffset;
+ uint32_t bloom_size;
+ uint32_t bloom_shift;
+};
+
+// Looks names up in one module's .dynsym using its DT_GNU_HASH table.
+//
+// The Bloom filter is checked first. It can only give a false positive, never
+// a false negative, so a miss there is conclusive and skips the chain walk
+// entirely. That is the whole reason GNU hash is faster than DT_HASH.
+class GnuSymbolTable {
+public:
+ LIBC_INLINE constexpr GnuSymbolTable() = default;
+ LIBC_INLINE GnuSymbolTable(const void *hash_table, const ElfW(Sym) * symtab,
+ const char *strtab)
+ : symtab_(symtab), strtab_(strtab) {
+ if (hash_table == nullptr)
+ return;
+ header_ = static_cast<const GnuHashHeader *>(hash_table);
+ bloom_ = reinterpret_cast<const ElfW(Addr) *>(header_ + 1);
+ buckets_ = reinterpret_cast<const uint32_t *>(bloom_ + header_->bloom_size);
+ chain_ = buckets_ + header_->nbuckets;
+ }
+
+ LIBC_INLINE constexpr bool empty() const { return header_ == nullptr; }
+
+ // Returns the matching symbol, or nullptr. Defined only for names this
+ // module actually exports; callers walk modules in search order themselves.
+ LIBC_INLINE const ElfW(Sym) * lookup(const char *name) const {
+ if (header_ == nullptr || header_->nbuckets == 0)
+ return nullptr;
+
+ const uint32_t hash = gnu_hash(name);
+ if (!bloom_may_contain(hash))
+ return nullptr;
+
+ uint32_t index = buckets_[hash % header_->nbuckets];
+ if (index < header_->symoffset)
+ return nullptr;
+
+ for (;; ++index) {
+ const uint32_t entry = chain_[index - header_->symoffset];
+ // The low bit is the end-of-chain marker, so it is masked out of the
+ // comparison rather than being part of the hash.
+ if ((entry | 1) == (hash | 1) && matches(index, name))
+ return &symtab_[index];
+ if (entry & 1)
+ return nullptr;
+ }
+ }
+
+private:
+ LIBC_INLINE bool bloom_may_contain(uint32_t hash) const {
+ if (header_->bloom_size == 0)
+ return true;
+ constexpr uint32_t BITS = 8 * sizeof(ElfW(Addr));
+ const ElfW(Addr) word = bloom_[(hash / BITS) % header_->bloom_size];
+ const ElfW(Addr) mask = (ElfW(Addr))1 << (hash % BITS) |
+ (ElfW(Addr))1
+ << ((hash >> header_->bloom_shift) % BITS);
+ return (word & mask) == mask;
+ }
+
+ LIBC_INLINE bool matches(uint32_t index, const char *name) const {
+ const char *candidate = strtab_ + symtab_[index].st_name;
+ for (; *candidate == *name; ++candidate, ++name)
+ if (*candidate == '\0')
+ return true;
+ return false;
+ }
+
+ const GnuHashHeader *header_ = nullptr;
+ const ElfW(Addr) *bloom_ = nullptr;
+ const uint32_t *buckets_ = nullptr;
+ const uint32_t *chain_ = nullptr;
+ const ElfW(Sym) *symtab_ = nullptr;
+ const char *strtab_ = nullptr;
+};
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_SYMBOL_H
diff --git a/libc/test/src/__support/elf/CMakeLists.txt b/libc/test/src/__support/elf/CMakeLists.txt
index 4d7e9cfc656557..c64d1b112d0349 100644
--- a/libc/test/src/__support/elf/CMakeLists.txt
+++ b/libc/test/src/__support/elf/CMakeLists.txt
@@ -19,3 +19,13 @@ add_libc_test(
DEPENDS
libc.src.__support.elf.hash
)
+
+add_libc_test(
+ symbol_test
+ SUITE
+ libc-support-tests
+ SRCS
+ symbol_test.cpp
+ DEPENDS
+ libc.src.__support.elf.symbol
+)
diff --git a/libc/test/src/__support/elf/symbol_test.cpp b/libc/test/src/__support/elf/symbol_test.cpp
new file mode 100644
index 00000000000000..66bfedd4bc434f
--- /dev/null
+++ b/libc/test/src/__support/elf/symbol_test.cpp
@@ -0,0 +1,101 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Unittests for GNU hash symbol lookup
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/elf/symbol.h"
+#include "test/UnitTest/Test.h"
+
+using LIBC_NAMESPACE::elf::GnuSymbolTable;
+
+namespace {
+
+// A DT_GNU_HASH table laid out the way a linker emits one: header, Bloom
+// filter, buckets, then the hash array. Index 0 of .dynsym is the reserved
+// null entry, so symoffset is 1 and the hashed symbols are sorted by bucket.
+//
+// Symbols, in bucket order: strlen, printf, malloc, free, fopen.
+struct Table {
+ uint32_t nbuckets = 3;
+ uint32_t symoffset = 1;
+ uint32_t bloom_size = 2;
+ uint32_t bloom_shift = 6;
+ ElfW(Addr) bloom[2] = { 0x2110400020000084ULL, 0x2000200000000000ULL };
+ uint32_t buckets[3] = {1, 2, 0};
+ uint32_t chain[5] = {0x1c93bb9d, 0x156b2bb8, 0x0d39ad3c, 0x7c96f086,
+ 0x0f738b7d};
+};
+
+const Table TABLE;
+
+// Offsets into STRTAB below.
+const char STRTAB[] = "\0strlen\0printf\0malloc\0free\0fopen";
+
+ElfW(Sym) make_sym(uint32_t name_offset, ElfW(Addr) value) {
+ ElfW(Sym) s{};
+ s.st_name = name_offset;
+ s.st_value = value;
+ s.st_info = ELF64_ST_INFO(STB_GLOBAL, STT_FUNC);
+ s.st_shndx = 1;
+ return s;
+}
+
+const ElfW(Sym) SYMTAB[] = {
+ make_sym(0, 0), // reserved null entry
+ make_sym(1, 0x1000), // strlen
+ make_sym(8, 0x2000), // printf
+ make_sym(15, 0x3000), // malloc
+ make_sym(22, 0x4000), // free
+ make_sym(27, 0x5000), // fopen
+};
+
+GnuSymbolTable table() { return GnuSymbolTable(&TABLE, SYMTAB, STRTAB); }
+
+} // anonymous namespace
+
+TEST(LlvmLibcElfSymbolTest, DefaultIsEmpty) {
+ GnuSymbolTable empty;
+ EXPECT_TRUE(empty.empty());
+ EXPECT_EQ(empty.lookup("printf"), static_cast<const ElfW(Sym) *>(nullptr));
+}
+
+TEST(LlvmLibcElfSymbolTest, FindsEverySymbol) {
+ GnuSymbolTable t = table();
+ ASSERT_FALSE(t.empty());
+ // Each name must resolve to its own entry, including the ones that share a
+ // bucket and are only separated by walking the chain.
+ ASSERT_TRUE(t.lookup("strlen") != nullptr);
+ EXPECT_EQ(t.lookup("strlen")->st_value, ElfW(Addr)(0x1000));
+ ASSERT_TRUE(t.lookup("printf") != nullptr);
+ EXPECT_EQ(t.lookup("printf")->st_value, ElfW(Addr)(0x2000));
+ ASSERT_TRUE(t.lookup("malloc") != nullptr);
+ EXPECT_EQ(t.lookup("malloc")->st_value, ElfW(Addr)(0x3000));
+ ASSERT_TRUE(t.lookup("free") != nullptr);
+ EXPECT_EQ(t.lookup("free")->st_value, ElfW(Addr)(0x4000));
+ ASSERT_TRUE(t.lookup("fopen") != nullptr);
+ EXPECT_EQ(t.lookup("fopen")->st_value, ElfW(Addr)(0x5000));
+}
+
+TEST(LlvmLibcElfSymbolTest, MissingNamesReturnNull) {
+ GnuSymbolTable t = table();
+ EXPECT_EQ(t.lookup("calloc"), static_cast<const ElfW(Sym) *>(nullptr));
+ EXPECT_EQ(t.lookup(""), static_cast<const ElfW(Sym) *>(nullptr));
+ EXPECT_EQ(t.lookup("qsort"), static_cast<const ElfW(Sym) *>(nullptr));
+}
+
+TEST(LlvmLibcElfSymbolTest, PrefixesDoNotMatch) {
+ // A chain walk that compared only the hash, or compared with a prefix match,
+ // would wrongly resolve these.
+ GnuSymbolTable t = table();
+ EXPECT_EQ(t.lookup("printf_"), static_cast<const ElfW(Sym) *>(nullptr));
+ EXPECT_EQ(t.lookup("print"), static_cast<const ElfW(Sym) *>(nullptr));
+ EXPECT_EQ(t.lookup("fre"), static_cast<const ElfW(Sym) *>(nullptr));
+}
>From 18dc735be5cd602f953796b1d7c8ba97c389cb00 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 04:22:42 +0100
Subject: [PATCH 25/46] [libc] Add relative relocation processing and its type
constants
Fourth piece of the ELF support a dynamic loader needs. RelaTable views
a DT_RELA table, sized from DT_RELASZ, and applies the relative
relocations.
Relative is the only kind a module can apply to itself before any symbol
lookup is possible, so it is the first pass a loader runs and the one
the startup code needs to relocate the loader itself. Every other type
is left alone. apply_relative runs before the module is relocated, so it
must not go through the GOT or PLT.
elf-macros.h carried no relocation type constants at all, so add the
RELATIVE, GLOB_DAT, JUMP_SLOT and IRELATIVE types for the supported
architectures. The startup code was picking these up from the host
elf.h, which a full build should not depend on.
Assisted-by: Claude
---
libc/include/llvm-libc-macros/elf-macros.h | 29 +++++
libc/src/__support/elf/CMakeLists.txt | 15 +++
libc/src/__support/elf/relocation.h | 95 ++++++++++++++++
libc/test/src/__support/elf/CMakeLists.txt | 10 ++
.../src/__support/elf/relocation_test.cpp | 107 ++++++++++++++++++
5 files changed, 256 insertions(+)
create mode 100644 libc/src/__support/elf/relocation.h
create mode 100644 libc/test/src/__support/elf/relocation_test.cpp
diff --git a/libc/include/llvm-libc-macros/elf-macros.h b/libc/include/llvm-libc-macros/elf-macros.h
index 6857b6681bfa2a..7ab790dd7ef4a9 100644
--- a/libc/include/llvm-libc-macros/elf-macros.h
+++ b/libc/include/llvm-libc-macros/elf-macros.h
@@ -32,4 +32,33 @@
#define ELF64_R_TYPE(i) ((i) & 0xffffffffL)
#define ELF64_R_INFO(s, t) (((s) << 32) + ((t) & 0xffffffffL))
+// Per architecture relocation types.
+//
+// Only the types the startup code and the dynamic loader apply are listed.
+// RELATIVE adds the load bias to the addend; IRELATIVE additionally calls the
+// resolver at that address and stores what it returns.
+
+// x86_64
+#define R_X86_64_RELATIVE 8
+#define R_X86_64_GLOB_DAT 6
+#define R_X86_64_JUMP_SLOT 7
+#define R_X86_64_IRELATIVE 37
+
+// aarch64
+#define R_AARCH64_RELATIVE 1027
+#define R_AARCH64_GLOB_DAT 1025
+#define R_AARCH64_JUMP_SLOT 1026
+#define R_AARCH64_IRELATIVE 1032
+
+// arm
+#define R_ARM_RELATIVE 23
+#define R_ARM_GLOB_DAT 21
+#define R_ARM_JUMP_SLOT 22
+#define R_ARM_IRELATIVE 160
+
+// riscv
+#define R_RISCV_RELATIVE 3
+#define R_RISCV_JUMP_SLOT 5
+#define R_RISCV_IRELATIVE 58
+
#endif // LLVM_LIBC_MACROS_ELF_MACROS_H
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
index b63a8a1f023f6b..20a1e375432938 100644
--- a/libc/src/__support/elf/CMakeLists.txt
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -35,3 +35,18 @@ add_header_library(
libc.src.__support.macros.attributes
libc.src.__support.macros.config
)
+
+add_header_library(
+ relocation
+ HDRS
+ relocation.h
+ DEPENDS
+ libc.hdr.elf_macros
+ libc.hdr.types.size_t
+ libc.hdr.elf_proxy
+ libc.hdr.link_macros
+ libc.hdr.stdint_proxy
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+ libc.src.__support.macros.properties.architectures
+)
diff --git a/libc/src/__support/elf/relocation.h b/libc/src/__support/elf/relocation.h
new file mode 100644
index 00000000000000..83bc70448eca9b
--- /dev/null
+++ b/libc/src/__support/elf/relocation.h
@@ -0,0 +1,95 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Relative relocations for a loaded ELF module
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_RELOCATION_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_RELOCATION_H
+
+#include "hdr/elf_macros.h"
+#include "hdr/elf_proxy.h"
+#include "hdr/link_macros.h"
+#include "hdr/stdint_proxy.h"
+#include "hdr/types/size_t.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+#include "src/__support/macros/properties/architectures.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// The relocation type that means "add the load bias to the addend and store
+// it", which is the only kind a module can apply to itself before any symbol
+// lookup is possible.
+#if defined(LIBC_TARGET_ARCH_IS_X86_64)
+constexpr uint32_t RELATIVE_RELOC = R_X86_64_RELATIVE;
+#elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
+constexpr uint32_t RELATIVE_RELOC = R_AARCH64_RELATIVE;
+#elif defined(LIBC_TARGET_ARCH_IS_ANY_RISCV)
+constexpr uint32_t RELATIVE_RELOC = R_RISCV_RELATIVE;
+#elif defined(LIBC_TARGET_ARCH_IS_ARM)
+constexpr uint32_t RELATIVE_RELOC = R_ARM_RELATIVE;
+#else
+#error "Relative relocation type is not known for this architecture"
+#endif
+
+// ELF64_R_TYPE and ELF32_R_TYPE differ, and ElfW() picks the class, so this
+// keeps the choice in one place.
+LIBC_INLINE constexpr uint32_t reloc_type(ElfW(Xword) info) {
+#ifdef __LP64__
+ return static_cast<uint32_t>(ELF64_R_TYPE(info));
+#else
+ return static_cast<uint32_t>(ELF32_R_TYPE(info));
+#endif
+}
+
+// A view over a DT_RELA table, sized by DT_RELASZ.
+class RelaTable {
+public:
+ LIBC_INLINE constexpr RelaTable() = default;
+ LIBC_INLINE constexpr RelaTable(const ElfW(Rela) * entries, size_t size_bytes)
+ : entries_(entries), count_(size_bytes / sizeof(ElfW(Rela))) {}
+
+ LIBC_INLINE constexpr size_t size() const { return count_; }
+ LIBC_INLINE constexpr bool empty() const { return count_ == 0; }
+ LIBC_INLINE constexpr const ElfW(Rela) * begin() const { return entries_; }
+ LIBC_INLINE constexpr const ElfW(Rela) * end() const {
+ return entries_ + count_;
+ }
+
+ // Applies every relative relocation in the table, and returns how many were
+ // applied. Other relocation types are left alone: they need a symbol table,
+ // which is not available until this pass has run.
+ //
+ // This runs before the module is relocated, so it must not call anything
+ // through the GOT or PLT.
+ LIBC_INLINE size_t apply_relative(ElfW(Addr) load_bias) const {
+ size_t applied = 0;
+ for (size_t i = 0; i < count_; ++i) {
+ const ElfW(Rela) &rela = entries_[i];
+ if (reloc_type(rela.r_info) != RELATIVE_RELOC)
+ continue;
+ auto *target = reinterpret_cast<ElfW(Addr) *>(load_bias + rela.r_offset);
+ *target = load_bias + static_cast<ElfW(Addr)>(rela.r_addend);
+ ++applied;
+ }
+ return applied;
+ }
+
+private:
+ const ElfW(Rela) *entries_ = nullptr;
+ size_t count_ = 0;
+};
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_RELOCATION_H
diff --git a/libc/test/src/__support/elf/CMakeLists.txt b/libc/test/src/__support/elf/CMakeLists.txt
index c64d1b112d0349..a5520e13594443 100644
--- a/libc/test/src/__support/elf/CMakeLists.txt
+++ b/libc/test/src/__support/elf/CMakeLists.txt
@@ -29,3 +29,13 @@ add_libc_test(
DEPENDS
libc.src.__support.elf.symbol
)
+
+add_libc_test(
+ relocation_test
+ SUITE
+ libc-support-tests
+ SRCS
+ relocation_test.cpp
+ DEPENDS
+ libc.src.__support.elf.relocation
+)
diff --git a/libc/test/src/__support/elf/relocation_test.cpp b/libc/test/src/__support/elf/relocation_test.cpp
new file mode 100644
index 00000000000000..dca6ede8c9e8e9
--- /dev/null
+++ b/libc/test/src/__support/elf/relocation_test.cpp
@@ -0,0 +1,107 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Unittests for relative relocation processing
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/elf/relocation.h"
+#include "test/UnitTest/Test.h"
+
+using LIBC_NAMESPACE::elf::RelaTable;
+using LIBC_NAMESPACE::elf::RELATIVE_RELOC;
+
+namespace {
+
+// The targets a relocation writes into. They are addressed as an offset from
+// a pretend load bias, so the test mirrors how a real module is laid out.
+ElfW(Addr) SLOTS[4];
+
+ElfW(Addr) bias() {
+ // r_offset is added to the bias, so treat SLOTS as if it sat at offset 0.
+ return reinterpret_cast<ElfW(Addr)>(&SLOTS[0]);
+}
+
+ElfW(Rela) make(ElfW(Addr) offset, uint32_t type, ElfW(Sxword) addend) {
+ ElfW(Rela) r{};
+ r.r_offset = offset;
+#ifdef __LP64__
+ r.r_info = ELF64_R_INFO(ElfW(Xword)(0), ElfW(Xword)(type));
+#else
+ r.r_info = ELF32_R_INFO(0, type);
+#endif
+ r.r_addend = addend;
+ return r;
+}
+
+} // anonymous namespace
+
+TEST(LlvmLibcElfRelocationTest, EmptyTable) {
+ RelaTable table;
+ EXPECT_TRUE(table.empty());
+ EXPECT_EQ(table.size(), size_t(0));
+ EXPECT_EQ(table.apply_relative(0x1000), size_t(0));
+}
+
+TEST(LlvmLibcElfRelocationTest, SizeComesFromByteCount) {
+ ElfW(Rela) entries[3] = {};
+ RelaTable table(entries, sizeof(entries));
+ EXPECT_EQ(table.size(), size_t(3));
+ EXPECT_FALSE(table.empty());
+ EXPECT_EQ(table.end() - table.begin(), ptrdiff_t(3));
+}
+
+TEST(LlvmLibcElfRelocationTest, AppliesRelativeRelocations) {
+ for (auto &slot : SLOTS)
+ slot = 0xdeadbeef;
+
+ const ElfW(Addr) load_bias = bias();
+ ElfW(Rela) entries[] = {
+ make(0 * sizeof(ElfW(Addr)), RELATIVE_RELOC, 0x10),
+ make(1 * sizeof(ElfW(Addr)), RELATIVE_RELOC, 0x20),
+ make(3 * sizeof(ElfW(Addr)), RELATIVE_RELOC, 0x30),
+ };
+
+ RelaTable table(entries, sizeof(entries));
+ EXPECT_EQ(table.apply_relative(load_bias), size_t(3));
+
+ EXPECT_EQ(SLOTS[0], ElfW(Addr)(load_bias + 0x10));
+ EXPECT_EQ(SLOTS[1], ElfW(Addr)(load_bias + 0x20));
+ // Slot 2 had no relocation and must be untouched.
+ EXPECT_EQ(SLOTS[2], ElfW(Addr)(0xdeadbeef));
+ EXPECT_EQ(SLOTS[3], ElfW(Addr)(load_bias + 0x30));
+}
+
+TEST(LlvmLibcElfRelocationTest, IgnoresOtherRelocationTypes) {
+ for (auto &slot : SLOTS)
+ slot = 0xdeadbeef;
+
+ // A type that is deliberately not the relative one. Applying it here would
+ // be wrong: it needs a symbol table that does not exist yet.
+ constexpr uint32_t SOME_OTHER_TYPE = RELATIVE_RELOC + 1;
+ ElfW(Rela) entries[] = {
+ make(0 * sizeof(ElfW(Addr)), SOME_OTHER_TYPE, 0x10),
+ make(1 * sizeof(ElfW(Addr)), RELATIVE_RELOC, 0x20),
+ };
+
+ RelaTable table(entries, sizeof(entries));
+ EXPECT_EQ(table.apply_relative(bias()), size_t(1));
+ EXPECT_EQ(SLOTS[0], ElfW(Addr)(0xdeadbeef));
+ EXPECT_EQ(SLOTS[1], ElfW(Addr)(bias() + 0x20));
+}
+
+TEST(LlvmLibcElfRelocationTest, NegativeAddendsWork) {
+ for (auto &slot : SLOTS)
+ slot = 0;
+ const ElfW(Addr) load_bias = bias();
+ ElfW(Rela) entries[] = {make(0, RELATIVE_RELOC, -16)};
+ RelaTable table(entries, sizeof(entries));
+ EXPECT_EQ(table.apply_relative(load_bias), size_t(1));
+ EXPECT_EQ(SLOTS[0], ElfW(Addr)(load_bias - 16));
+}
>From 7bf53ee70afbd0e8c31f0316673e99311381cec6 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 04:28:51 +0100
Subject: [PATCH 26/46] [libc] Add a description of one loaded ELF module
Ties the previous pieces together. Module is built once from a module's
program headers and load bias, and then only read: it hands out the
dynamic table, the string and symbol tables, the SONAME, the DT_NEEDED
list in load order, and the relocation tables.
It is plain data with no ownership and no mutable state, so it can be
handed to whatever resolves symbols at runtime and to a debugger without
either needing to call back into the loader that produced it. That is
the shape the loader is being built towards, where the startup code
publishes a description of the module set and then stops being involved.
PT_TLS is exposed rather than kept private because sizing the thread
block needs every module's TLS segment, which is the thing a shared libc
currently gets wrong.
DT_JMPREL is kept separate from DT_RELA since the two are bound at
different times, and it is only accepted when DT_PLTREL says the table
is in RELA form.
Assisted-by: Claude
---
libc/src/__support/elf/CMakeLists.txt | 17 +++
libc/src/__support/elf/module.h | 147 ++++++++++++++++++++
libc/test/src/__support/elf/CMakeLists.txt | 10 ++
libc/test/src/__support/elf/module_test.cpp | 114 +++++++++++++++
4 files changed, 288 insertions(+)
create mode 100644 libc/src/__support/elf/module.h
create mode 100644 libc/test/src/__support/elf/module_test.cpp
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
index 20a1e375432938..9e2554c382b2e9 100644
--- a/libc/src/__support/elf/CMakeLists.txt
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -50,3 +50,20 @@ add_header_library(
libc.src.__support.macros.config
libc.src.__support.macros.properties.architectures
)
+
+add_header_library(
+ module
+ HDRS
+ module.h
+ DEPENDS
+ libc.hdr.elf_macros
+ libc.hdr.elf_proxy
+ libc.hdr.link_macros
+ libc.hdr.stdint_proxy
+ libc.hdr.types.size_t
+ libc.src.__support.elf.dynamic
+ libc.src.__support.elf.relocation
+ libc.src.__support.elf.symbol
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+)
diff --git a/libc/src/__support/elf/module.h b/libc/src/__support/elf/module.h
new file mode 100644
index 00000000000000..ca7394ae2a707b
--- /dev/null
+++ b/libc/src/__support/elf/module.h
@@ -0,0 +1,147 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Description of one loaded ELF module
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_MODULE_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_MODULE_H
+
+#include "hdr/elf_macros.h"
+#include "hdr/elf_proxy.h"
+#include "hdr/link_macros.h"
+#include "hdr/stdint_proxy.h"
+#include "hdr/types/size_t.h"
+#include "src/__support/elf/dynamic.h"
+#include "src/__support/elf/relocation.h"
+#include "src/__support/elf/symbol.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// Everything about one loaded ELF object that a consumer needs in order to
+// resolve a symbol in it or to describe it to a debugger.
+//
+// This is deliberately plain data with no ownership and no mutable state. It
+// is built once from a module's program headers and then only read, so it can
+// be published to libdl and to a debugger without any of them needing to call
+// back into the code that produced it.
+class Module {
+public:
+ LIBC_INLINE constexpr Module() = default;
+
+ // Builds the description from a module's program headers. `load_bias` is the
+ // distance between where the module was linked and where it was mapped.
+ LIBC_INLINE Module(const ElfW(Phdr) * phdrs, ElfW(Half) phnum,
+ ElfW(Addr) load_bias, const char *name)
+ : phdrs_(phdrs), phnum_(phnum), load_bias_(load_bias), name_(name) {
+ for (ElfW(Half) i = 0; i < phnum; ++i) {
+ switch (phdrs[i].p_type) {
+ case PT_DYNAMIC:
+ dynamic_ = DynamicTable(
+ reinterpret_cast<const ElfW(Dyn) *>(load_bias + phdrs[i].p_vaddr),
+ load_bias);
+ break;
+ case PT_TLS:
+ tls_ = &phdrs[i];
+ break;
+ default:
+ break;
+ }
+ }
+ }
+
+ LIBC_INLINE constexpr const char *name() const { return name_; }
+ LIBC_INLINE constexpr ElfW(Addr) load_bias() const { return load_bias_; }
+ LIBC_INLINE constexpr const ElfW(Phdr) * phdrs() const { return phdrs_; }
+ LIBC_INLINE constexpr ElfW(Half) phnum() const { return phnum_; }
+ LIBC_INLINE constexpr const DynamicTable &dynamic() const { return dynamic_; }
+
+ // The PT_TLS header, or null if the module has no thread local storage.
+ // A loader has to size the thread block from every module's PT_TLS, which
+ // is why this is exposed rather than kept private.
+ LIBC_INLINE constexpr const ElfW(Phdr) * tls() const { return tls_; }
+
+ LIBC_INLINE const char *strtab() const {
+ if (auto addr = dynamic_.address(DT_STRTAB))
+ return reinterpret_cast<const char *>(*addr);
+ return nullptr;
+ }
+
+ LIBC_INLINE const ElfW(Sym) * symtab() const {
+ if (auto addr = dynamic_.address(DT_SYMTAB))
+ return reinterpret_cast<const ElfW(Sym) *>(*addr);
+ return nullptr;
+ }
+
+ // The module's SONAME, or null if it has none. An executable normally does
+ // not carry one.
+ LIBC_INLINE const char *soname() const {
+ const char *strings = strtab();
+ auto offset = dynamic_.value(DT_SONAME);
+ if (strings == nullptr || !offset)
+ return nullptr;
+ return strings + *offset;
+ }
+
+ LIBC_INLINE GnuSymbolTable symbols() const {
+ auto hash = dynamic_.address(DT_GNU_HASH);
+ if (!hash)
+ return GnuSymbolTable();
+ return GnuSymbolTable(reinterpret_cast<const void *>(*hash), symtab(),
+ strtab());
+ }
+
+ // The DT_RELA table. DT_JMPREL is kept separate because a loader may bind
+ // it lazily while the rest has to be applied up front.
+ LIBC_INLINE RelaTable relocations() const {
+ auto addr = dynamic_.address(DT_RELA);
+ auto size = dynamic_.value(DT_RELASZ);
+ if (!addr || !size)
+ return RelaTable();
+ return RelaTable(reinterpret_cast<const ElfW(Rela) *>(*addr), *size);
+ }
+
+ LIBC_INLINE RelaTable plt_relocations() const {
+ auto addr = dynamic_.address(DT_JMPREL);
+ auto size = dynamic_.value(DT_PLTRELSZ);
+ // DT_PLTREL says which form DT_JMPREL takes; only RELA is supported.
+ auto form = dynamic_.value(DT_PLTREL);
+ if (!addr || !size || !form || *form != ElfW(Xword)(DT_RELA))
+ return RelaTable();
+ return RelaTable(reinterpret_cast<const ElfW(Rela) *>(*addr), *size);
+ }
+
+ // Calls callback(const char *) for each DT_NEEDED name, in the order the
+ // linker recorded them, which is the order they must be loaded in.
+ template <typename F> LIBC_INLINE void for_each_needed(F callback) const {
+ const char *strings = strtab();
+ if (strings == nullptr)
+ return;
+ dynamic_.for_each(DT_NEEDED, [strings, &callback](const ElfW(Dyn) &entry) {
+ callback(strings + entry.d_un.d_val);
+ });
+ }
+
+private:
+ const ElfW(Phdr) *phdrs_ = nullptr;
+ ElfW(Half) phnum_ = 0;
+ ElfW(Addr) load_bias_ = 0;
+ const char *name_ = nullptr;
+ const ElfW(Phdr) *tls_ = nullptr;
+ DynamicTable dynamic_;
+};
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_MODULE_H
diff --git a/libc/test/src/__support/elf/CMakeLists.txt b/libc/test/src/__support/elf/CMakeLists.txt
index a5520e13594443..694fb04d37087a 100644
--- a/libc/test/src/__support/elf/CMakeLists.txt
+++ b/libc/test/src/__support/elf/CMakeLists.txt
@@ -39,3 +39,13 @@ add_libc_test(
DEPENDS
libc.src.__support.elf.relocation
)
+
+add_libc_test(
+ module_test
+ SUITE
+ libc-support-tests
+ SRCS
+ module_test.cpp
+ DEPENDS
+ libc.src.__support.elf.module
+)
diff --git a/libc/test/src/__support/elf/module_test.cpp b/libc/test/src/__support/elf/module_test.cpp
new file mode 100644
index 00000000000000..7af6ce14d2fb09
--- /dev/null
+++ b/libc/test/src/__support/elf/module_test.cpp
@@ -0,0 +1,114 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Unittests for the loaded module description
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/elf/module.h"
+#include "test/UnitTest/Test.h"
+
+using LIBC_NAMESPACE::elf::Module;
+
+namespace {
+
+const char STRTAB[] = "\0libtest.so.1\0libc.so.0\0libm.so.0";
+// Offsets into STRTAB: 1 = libtest.so.1, 14 = libc.so.0, 24 = libm.so.0.
+
+ElfW(Rela) RELAS[4];
+
+const ElfW(Dyn) DYNAMIC[] = {
+ {DT_NEEDED, {14}},
+ {DT_NEEDED, {24}},
+ {DT_SONAME, {1}},
+ {DT_STRTAB, {0}}, // patched to &STRTAB at runtime
+ {DT_RELA, {0}}, // patched to &RELAS at runtime
+ {DT_RELASZ, {sizeof(RELAS)}},
+ {DT_NULL, {0}},
+};
+
+// The dynamic array records addresses relative to the link address, and
+// Module adds the load bias back on. Using a zero bias with absolute values
+// keeps the arithmetic in the test trivial.
+ElfW(Dyn) PATCHED[sizeof(DYNAMIC) / sizeof(DYNAMIC[0])];
+
+ElfW(Phdr) PHDRS[2];
+
+Module build() {
+ for (size_t i = 0; i < sizeof(PATCHED) / sizeof(PATCHED[0]); ++i) {
+ PATCHED[i] = DYNAMIC[i];
+ if (PATCHED[i].d_tag == DT_STRTAB)
+ PATCHED[i].d_un.d_ptr = reinterpret_cast<ElfW(Addr)>(STRTAB);
+ if (PATCHED[i].d_tag == DT_RELA)
+ PATCHED[i].d_un.d_ptr = reinterpret_cast<ElfW(Addr)>(RELAS);
+ }
+
+ PHDRS[0].p_type = PT_DYNAMIC;
+ PHDRS[0].p_vaddr = reinterpret_cast<ElfW(Addr)>(PATCHED);
+ PHDRS[1].p_type = PT_TLS;
+ PHDRS[1].p_vaddr = 0x9000;
+ PHDRS[1].p_memsz = 0x40;
+
+ return Module(PHDRS, 2, /*load_bias=*/0, "libtest.so.1");
+}
+
+} // anonymous namespace
+
+TEST(LlvmLibcElfModuleTest, DefaultIsInert) {
+ Module m;
+ EXPECT_TRUE(m.dynamic().empty());
+ EXPECT_EQ(m.phnum(), ElfW(Half)(0));
+ EXPECT_EQ(m.strtab(), static_cast<const char *>(nullptr));
+ EXPECT_EQ(m.soname(), static_cast<const char *>(nullptr));
+ EXPECT_EQ(m.tls(), static_cast<const ElfW(Phdr) *>(nullptr));
+ EXPECT_TRUE(m.relocations().empty());
+ EXPECT_TRUE(m.symbols().empty());
+}
+
+TEST(LlvmLibcElfModuleTest, FindsDynamicAndTlsSegments) {
+ Module m = build();
+ EXPECT_FALSE(m.dynamic().empty());
+ ASSERT_TRUE(m.tls() != nullptr);
+ EXPECT_EQ(m.tls()->p_memsz, ElfW(Xword)(0x40));
+}
+
+TEST(LlvmLibcElfModuleTest, ResolvesSoname) {
+ Module m = build();
+ ASSERT_TRUE(m.soname() != nullptr);
+ EXPECT_STREQ(m.soname(), "libtest.so.1");
+}
+
+TEST(LlvmLibcElfModuleTest, ListsNeededInOrder) {
+ Module m = build();
+ const char *seen[4] = {};
+ int count = 0;
+ m.for_each_needed([&](const char *name) {
+ if (count < 4)
+ seen[count] = name;
+ ++count;
+ });
+ ASSERT_EQ(count, 2);
+ EXPECT_STREQ(seen[0], "libc.so.0");
+ EXPECT_STREQ(seen[1], "libm.so.0");
+}
+
+TEST(LlvmLibcElfModuleTest, SizesRelocationTable) {
+ Module m = build();
+ EXPECT_EQ(m.relocations().size(), size_t(4));
+ // No DT_JMPREL in this module.
+ EXPECT_TRUE(m.plt_relocations().empty());
+}
+
+TEST(LlvmLibcElfModuleTest, NoGnuHashMeansNoSymbolTable) {
+ // The sample carries no DT_GNU_HASH, so lookups must not be attempted.
+ Module m = build();
+ EXPECT_TRUE(m.symbols().empty());
+ EXPECT_EQ(m.symbols().lookup("printf"),
+ static_cast<const ElfW(Sym) *>(nullptr));
+}
>From a3d0ed2decaa7445e2eb792f5e9a4b14f41a8b44 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 04:49:29 +0100
Subject: [PATCH 27/46] [libc] Add the static TLS layout calculation
Computes where each module's thread local block sits relative to the
thread pointer, and how large the whole block has to be.
This is the calculation a shared libc currently gets wrong. crt1.o sizes
the thread block from the main executable's PT_TLS alone, so a libc.so
carrying a PT_TLS segment of its own has nowhere to put it, and its
thread local accesses land outside the allocation. Sizing across every
loaded module is the loader's job, and this is that part of it.
Both ABI variants are handled: x86 puts the blocks below the thread
pointer and returns negative offsets, everything else puts them above a
two word thread control block and returns positive ones. The tests
assert the invariants that hold either way, so they do not have to be
written twice.
Assisted-by: Claude
---
libc/src/__support/elf/CMakeLists.txt | 13 ++
libc/src/__support/elf/tls_layout.h | 103 +++++++++++++++
libc/test/src/__support/elf/CMakeLists.txt | 10 ++
.../src/__support/elf/tls_layout_test.cpp | 117 ++++++++++++++++++
4 files changed, 243 insertions(+)
create mode 100644 libc/src/__support/elf/tls_layout.h
create mode 100644 libc/test/src/__support/elf/tls_layout_test.cpp
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
index 9e2554c382b2e9..dec1d693a56ff2 100644
--- a/libc/src/__support/elf/CMakeLists.txt
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -67,3 +67,16 @@ add_header_library(
libc.src.__support.macros.attributes
libc.src.__support.macros.config
)
+
+add_header_library(
+ tls_layout
+ HDRS
+ tls_layout.h
+ DEPENDS
+ libc.hdr.link_macros
+ libc.hdr.stdint_proxy
+ libc.hdr.types.size_t
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+ libc.src.__support.macros.properties.architectures
+)
diff --git a/libc/src/__support/elf/tls_layout.h b/libc/src/__support/elf/tls_layout.h
new file mode 100644
index 00000000000000..0b7590ddd40fd6
--- /dev/null
+++ b/libc/src/__support/elf/tls_layout.h
@@ -0,0 +1,103 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Static TLS layout across loaded modules
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_TLS_LAYOUT_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_TLS_LAYOUT_H
+
+#include "hdr/link_macros.h"
+#include "hdr/stdint_proxy.h"
+#include "hdr/types/size_t.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+#include "src/__support/macros/properties/architectures.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// Which side of the thread pointer the TLS blocks live on.
+//
+// Variant 2 puts them below the thread pointer, so a module's offset is
+// subtracted from it. Variant 1 puts them above, after a two word thread
+// control block. x86 uses variant 2 and everything else here uses variant 1.
+#if defined(LIBC_TARGET_ARCH_IS_X86_64) || defined(LIBC_TARGET_ARCH_IS_X86_32)
+constexpr bool TLS_VARIANT_2 = true;
+#else
+constexpr bool TLS_VARIANT_2 = false;
+#endif
+
+// Size of the thread control block that variant 1 reserves before the first
+// module's block.
+constexpr size_t TLS_TCB_SIZE = 2 * sizeof(void *);
+
+LIBC_INLINE constexpr size_t round_up(size_t value, size_t alignment) {
+ if (alignment <= 1)
+ return value;
+ return (value + alignment - 1) / alignment * alignment;
+}
+
+// Accumulates the static TLS layout as modules are added, in load order.
+//
+// This is the calculation a shared libc currently gets wrong: crt1.o sizes the
+// thread block from the main executable's PT_TLS alone, so a libc.so with a
+// PT_TLS segment of its own has nowhere to put it.
+class TlsLayout {
+public:
+ LIBC_INLINE constexpr TlsLayout() = default;
+
+ // Adds one module's PT_TLS and returns its offset from the thread pointer.
+ // Under variant 2 the returned value is subtracted from the thread pointer;
+ // under variant 1 it is added. A module with no TLS should not be added.
+ LIBC_INLINE intptr_t add(size_t memsz, size_t alignment) {
+ if (alignment == 0)
+ alignment = 1;
+ if (alignment > max_align_)
+ max_align_ = alignment;
+
+ if (TLS_VARIANT_2) {
+ // Offsets grow downwards, so the running total is rounded up after the
+ // new block is included.
+ offset_ = round_up(offset_ + memsz, alignment);
+ ++count_;
+ return -static_cast<intptr_t>(offset_);
+ }
+
+ if (count_ == 0)
+ offset_ = round_up(TLS_TCB_SIZE, alignment);
+ else
+ offset_ = round_up(offset_, alignment);
+ const size_t here = offset_;
+ offset_ += memsz;
+ ++count_;
+ return static_cast<intptr_t>(here);
+ }
+
+ // Total bytes the thread block needs, thread control block included.
+ LIBC_INLINE constexpr size_t size() const {
+ if (TLS_VARIANT_2)
+ return offset_ + TLS_TCB_SIZE;
+ return offset_;
+ }
+
+ LIBC_INLINE constexpr size_t alignment() const { return max_align_; }
+ LIBC_INLINE constexpr size_t module_count() const { return count_; }
+
+private:
+ size_t offset_ = 0;
+ size_t max_align_ = TLS_TCB_SIZE;
+ size_t count_ = 0;
+};
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_TLS_LAYOUT_H
diff --git a/libc/test/src/__support/elf/CMakeLists.txt b/libc/test/src/__support/elf/CMakeLists.txt
index 694fb04d37087a..b5bdddc97ef66c 100644
--- a/libc/test/src/__support/elf/CMakeLists.txt
+++ b/libc/test/src/__support/elf/CMakeLists.txt
@@ -49,3 +49,13 @@ add_libc_test(
DEPENDS
libc.src.__support.elf.module
)
+
+add_libc_test(
+ tls_layout_test
+ SUITE
+ libc-support-tests
+ SRCS
+ tls_layout_test.cpp
+ DEPENDS
+ libc.src.__support.elf.tls_layout
+)
diff --git a/libc/test/src/__support/elf/tls_layout_test.cpp b/libc/test/src/__support/elf/tls_layout_test.cpp
new file mode 100644
index 00000000000000..559cc3ce40974d
--- /dev/null
+++ b/libc/test/src/__support/elf/tls_layout_test.cpp
@@ -0,0 +1,117 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Unittests for the static TLS layout
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/elf/tls_layout.h"
+#include "test/UnitTest/Test.h"
+
+using LIBC_NAMESPACE::elf::round_up;
+using LIBC_NAMESPACE::elf::TLS_TCB_SIZE;
+using LIBC_NAMESPACE::elf::TLS_VARIANT_2;
+using LIBC_NAMESPACE::elf::TlsLayout;
+
+TEST(LlvmLibcElfTlsLayoutTest, RoundUp) {
+ EXPECT_EQ(round_up(0, 8), size_t(0));
+ EXPECT_EQ(round_up(1, 8), size_t(8));
+ EXPECT_EQ(round_up(8, 8), size_t(8));
+ EXPECT_EQ(round_up(9, 8), size_t(16));
+ // An alignment of zero or one must leave the value alone.
+ EXPECT_EQ(round_up(7, 1), size_t(7));
+ EXPECT_EQ(round_up(7, 0), size_t(7));
+}
+
+TEST(LlvmLibcElfTlsLayoutTest, EmptyLayout) {
+ TlsLayout layout;
+ EXPECT_EQ(layout.module_count(), size_t(0));
+ EXPECT_EQ(layout.alignment(), TLS_TCB_SIZE);
+ if (TLS_VARIANT_2)
+ EXPECT_EQ(layout.size(), TLS_TCB_SIZE);
+ else
+ EXPECT_EQ(layout.size(), size_t(0));
+}
+
+TEST(LlvmLibcElfTlsLayoutTest, OffsetsAreOnTheRightSide) {
+ TlsLayout layout;
+ intptr_t first = layout.add(32, 8);
+ intptr_t second = layout.add(16, 8);
+ if (TLS_VARIANT_2) {
+ // Below the thread pointer, and each module further down than the last.
+ EXPECT_LT(first, intptr_t(0));
+ EXPECT_LT(second, first);
+ } else {
+ // Above the thread pointer, past the thread control block.
+ EXPECT_GE(first, intptr_t(TLS_TCB_SIZE));
+ EXPECT_GT(second, first);
+ }
+ EXPECT_EQ(layout.module_count(), size_t(2));
+}
+
+TEST(LlvmLibcElfTlsLayoutTest, BlocksDoNotOverlap) {
+ // Three modules with awkward sizes and alignments. Whatever the variant,
+ // each block must fit entirely inside the total and not touch its
+ // neighbours.
+ struct Mod {
+ size_t size, align;
+ };
+ const Mod mods[] = {{40, 8}, {1, 1}, {96, 64}};
+
+ TlsLayout layout;
+ intptr_t offsets[3];
+ for (int i = 0; i < 3; ++i)
+ offsets[i] = layout.add(mods[i].size, mods[i].align);
+
+ for (int i = 0; i < 3; ++i) {
+ // Each block must respect its own alignment requirement.
+ size_t magnitude =
+ static_cast<size_t>(offsets[i] < 0 ? -offsets[i] : offsets[i]);
+ EXPECT_EQ(magnitude % mods[i].align, size_t(0));
+ // And must lie inside the block that gets allocated.
+ EXPECT_LE(magnitude, layout.size());
+ }
+
+ for (int i = 0; i + 1 < 3; ++i) {
+ size_t a = static_cast<size_t>(offsets[i] < 0 ? -offsets[i] : offsets[i]);
+ size_t b = static_cast<size_t>(offsets[i + 1] < 0 ? -offsets[i + 1]
+ : offsets[i + 1]);
+ if (TLS_VARIANT_2)
+ // b is further from the thread pointer and must clear a's whole block.
+ EXPECT_GE(b, a + mods[i + 1].size);
+ else
+ EXPECT_GE(b, a + mods[i].size);
+ }
+}
+
+TEST(LlvmLibcElfTlsLayoutTest, AlignmentIsTheLargestSeen) {
+ TlsLayout layout;
+ layout.add(8, 8);
+ layout.add(8, 64);
+ layout.add(8, 16);
+ EXPECT_EQ(layout.alignment(), size_t(64));
+}
+
+TEST(LlvmLibcElfTlsLayoutTest, SizeCoversEveryModule) {
+ // The case that motivated this: a small executable followed by a libc.so
+ // with a large TLS segment. The total has to cover both.
+ TlsLayout layout;
+ layout.add(16, 8);
+ layout.add(52904, 64);
+ EXPECT_GE(layout.size(), size_t(16 + 52904));
+ EXPECT_EQ(layout.module_count(), size_t(2));
+}
+
+TEST(LlvmLibcElfTlsLayoutTest, ZeroAlignmentIsTreatedAsOne) {
+ TlsLayout layout;
+ intptr_t offset = layout.add(24, 0);
+ EXPECT_EQ(layout.module_count(), size_t(1));
+ size_t magnitude = static_cast<size_t>(offset < 0 ? -offset : offset);
+ EXPECT_GE(layout.size(), magnitude);
+}
>From 088159c9b7d3e5055e72872b6c436150377478ae Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 04:52:49 +0100
Subject: [PATCH 28/46] [libc] Add the PT_LOAD address space layout calculation
Works out how much address space a module needs and where its segments
sit in it. A loader reserves the whole span in one mapping and then maps
the segments into it, rather than mapping each one separately: the
segments have to keep their relative offsets for the module's own
relocations to hold, and a separate mapping would let something else
take an address in the middle of the range.
PT_LOAD entries are not required to be sorted, so the span is taken as a
minimum and maximum rather than from the first and last entries, which
the tests cover along with a non-PIE base and an unaligned segment end.
Mapping p_flags onto mmap protection bits is left to the caller, since
llvm-libc does not define the PROT_ macros yet and they belong next to
the mmap call rather than here.
Assisted-by: Claude
---
libc/src/__support/elf/CMakeLists.txt | 14 +++
libc/src/__support/elf/load_layout.h | 99 +++++++++++++++++
libc/test/src/__support/elf/CMakeLists.txt | 10 ++
.../src/__support/elf/load_layout_test.cpp | 100 ++++++++++++++++++
4 files changed, 223 insertions(+)
create mode 100644 libc/src/__support/elf/load_layout.h
create mode 100644 libc/test/src/__support/elf/load_layout_test.cpp
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
index dec1d693a56ff2..82810c41f16612 100644
--- a/libc/src/__support/elf/CMakeLists.txt
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -80,3 +80,17 @@ add_header_library(
libc.src.__support.macros.config
libc.src.__support.macros.properties.architectures
)
+
+add_header_library(
+ load_layout
+ HDRS
+ load_layout.h
+ DEPENDS
+ libc.hdr.elf_macros
+ libc.hdr.elf_proxy
+ libc.hdr.link_macros
+ libc.hdr.stdint_proxy
+ libc.hdr.types.size_t
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+)
diff --git a/libc/src/__support/elf/load_layout.h b/libc/src/__support/elf/load_layout.h
new file mode 100644
index 00000000000000..06dd7403024d67
--- /dev/null
+++ b/libc/src/__support/elf/load_layout.h
@@ -0,0 +1,99 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Address space layout of a module's PT_LOAD segments
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_LOAD_LAYOUT_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_LOAD_LAYOUT_H
+
+#include "hdr/elf_macros.h"
+#include "hdr/elf_proxy.h"
+#include "hdr/link_macros.h"
+#include "hdr/stdint_proxy.h"
+#include "hdr/types/size_t.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// How much address space a module needs, and where its segments sit in it.
+//
+// A loader has to reserve the whole span in one mapping first and then map the
+// segments into it. Mapping each segment separately would let something else
+// take an address in the middle of the range, and the segments have to keep
+// their relative offsets for the module's own relocations to be correct.
+class LoadLayout {
+public:
+ LIBC_INLINE constexpr LoadLayout() = default;
+
+ LIBC_INLINE LoadLayout(const ElfW(Phdr) * phdrs, ElfW(Half) phnum) {
+ bool seen = false;
+ for (ElfW(Half) i = 0; i < phnum; ++i) {
+ if (phdrs[i].p_type != PT_LOAD)
+ continue;
+ const ElfW(Addr) start = phdrs[i].p_vaddr;
+ const ElfW(Addr) end = phdrs[i].p_vaddr + phdrs[i].p_memsz;
+ if (!seen) {
+ first_ = start;
+ last_ = end;
+ seen = true;
+ } else {
+ if (start < first_)
+ first_ = start;
+ if (end > last_)
+ last_ = end;
+ }
+ if (phdrs[i].p_align > align_)
+ align_ = phdrs[i].p_align;
+ ++count_;
+ }
+ }
+
+ LIBC_INLINE constexpr bool empty() const { return count_ == 0; }
+ LIBC_INLINE constexpr size_t segment_count() const { return count_; }
+
+ // Lowest and highest link time addresses the module occupies. A shared
+ // object normally starts at zero, an executable at its fixed base.
+ LIBC_INLINE constexpr ElfW(Addr) first_vaddr() const { return first_; }
+ LIBC_INLINE constexpr ElfW(Addr) last_vaddr() const { return last_; }
+
+ // Bytes of address space to reserve, from the start of the first segment's
+ // page to the end of the last.
+ LIBC_INLINE size_t reservation_size(size_t page_size) const {
+ if (count_ == 0)
+ return 0;
+ return page_align_up(last_, page_size) - page_align_down(first_, page_size);
+ }
+
+ LIBC_INLINE constexpr ElfW(Xword) alignment() const { return align_; }
+
+ LIBC_INLINE static ElfW(Addr)
+ page_align_down(ElfW(Addr) value, size_t page_size) {
+ return value & ~static_cast<ElfW(Addr)>(page_size - 1);
+ }
+
+ LIBC_INLINE static ElfW(Addr)
+ page_align_up(ElfW(Addr) value, size_t page_size) {
+ return page_align_down(value + page_size - 1, page_size);
+ }
+
+private:
+ ElfW(Addr) first_ = 0;
+ ElfW(Addr) last_ = 0;
+ ElfW(Xword) align_ = 0;
+ size_t count_ = 0;
+};
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_LOAD_LAYOUT_H
diff --git a/libc/test/src/__support/elf/CMakeLists.txt b/libc/test/src/__support/elf/CMakeLists.txt
index b5bdddc97ef66c..74598684cb21b0 100644
--- a/libc/test/src/__support/elf/CMakeLists.txt
+++ b/libc/test/src/__support/elf/CMakeLists.txt
@@ -59,3 +59,13 @@ add_libc_test(
DEPENDS
libc.src.__support.elf.tls_layout
)
+
+add_libc_test(
+ load_layout_test
+ SUITE
+ libc-support-tests
+ SRCS
+ load_layout_test.cpp
+ DEPENDS
+ libc.src.__support.elf.load_layout
+)
diff --git a/libc/test/src/__support/elf/load_layout_test.cpp b/libc/test/src/__support/elf/load_layout_test.cpp
new file mode 100644
index 00000000000000..933a7c450c4c95
--- /dev/null
+++ b/libc/test/src/__support/elf/load_layout_test.cpp
@@ -0,0 +1,100 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Unittests for the PT_LOAD address space layout
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/elf/load_layout.h"
+#include "test/UnitTest/Test.h"
+
+using LIBC_NAMESPACE::elf::LoadLayout;
+
+namespace {
+
+constexpr size_t PAGE = 4096;
+
+ElfW(Phdr) make(ElfW(Word) type, ElfW(Addr) vaddr, ElfW(Xword) memsz,
+ ElfW(Xword) align) {
+ ElfW(Phdr) p{};
+ p.p_type = type;
+ p.p_vaddr = vaddr;
+ p.p_memsz = memsz;
+ p.p_align = align;
+ return p;
+}
+
+} // anonymous namespace
+
+TEST(LlvmLibcElfLoadLayoutTest, EmptyWhenNoLoadSegments) {
+ ElfW(Phdr) phdrs[] = {make(PT_DYNAMIC, 0x1000, 0x100, 8),
+ make(PT_TLS, 0x2000, 0x40, 8)};
+ LoadLayout layout(phdrs, 2);
+ EXPECT_TRUE(layout.empty());
+ EXPECT_EQ(layout.segment_count(), size_t(0));
+ EXPECT_EQ(layout.reservation_size(PAGE), size_t(0));
+}
+
+TEST(LlvmLibcElfLoadLayoutTest, SpansEverySegment) {
+ // A typical shared object: text, rodata, then data, with other segment
+ // types interleaved to make sure they are skipped.
+ ElfW(Phdr) phdrs[] = {
+ make(PT_PHDR, 0x40, 0x1c0, 8), make(PT_LOAD, 0x0000, 0x2000, PAGE),
+ make(PT_LOAD, 0x2000, 0x1000, PAGE), make(PT_DYNAMIC, 0x2f00, 0x100, 8),
+ make(PT_LOAD, 0x4000, 0x0800, PAGE),
+ };
+ LoadLayout layout(phdrs, 5);
+ EXPECT_FALSE(layout.empty());
+ EXPECT_EQ(layout.segment_count(), size_t(3));
+ EXPECT_EQ(layout.first_vaddr(), ElfW(Addr)(0x0000));
+ EXPECT_EQ(layout.last_vaddr(), ElfW(Addr)(0x4800));
+ // 0x0000 rounded down, 0x4800 rounded up to 0x5000.
+ EXPECT_EQ(layout.reservation_size(PAGE), size_t(0x5000));
+ EXPECT_EQ(layout.alignment(), ElfW(Xword)(PAGE));
+}
+
+TEST(LlvmLibcElfLoadLayoutTest, HandlesNonZeroBase) {
+ // A non-PIE executable starts at a fixed address, so the reservation must
+ // be measured from that base rather than from zero.
+ ElfW(Phdr) phdrs[] = {
+ make(PT_LOAD, 0x400000, 0x1000, PAGE),
+ make(PT_LOAD, 0x402000, 0x1000, PAGE),
+ };
+ LoadLayout layout(phdrs, 2);
+ EXPECT_EQ(layout.first_vaddr(), ElfW(Addr)(0x400000));
+ EXPECT_EQ(layout.last_vaddr(), ElfW(Addr)(0x403000));
+ EXPECT_EQ(layout.reservation_size(PAGE), size_t(0x3000));
+}
+
+TEST(LlvmLibcElfLoadLayoutTest, SegmentsOutOfOrder) {
+ // Nothing requires PT_LOAD entries to be sorted, so the span must be taken
+ // as a minimum and maximum rather than from the first and last entries.
+ ElfW(Phdr) phdrs[] = {
+ make(PT_LOAD, 0x4000, 0x1000, PAGE),
+ make(PT_LOAD, 0x0000, 0x1000, PAGE),
+ };
+ LoadLayout layout(phdrs, 2);
+ EXPECT_EQ(layout.first_vaddr(), ElfW(Addr)(0x0000));
+ EXPECT_EQ(layout.last_vaddr(), ElfW(Addr)(0x5000));
+ EXPECT_EQ(layout.reservation_size(PAGE), size_t(0x5000));
+}
+
+TEST(LlvmLibcElfLoadLayoutTest, UnalignedEndsAreRoundedUp) {
+ ElfW(Phdr) phdrs[] = {make(PT_LOAD, 0x1001, 0x0001, PAGE)};
+ LoadLayout layout(phdrs, 1);
+ // Start page is 0x1000, end 0x1002 rounds up to 0x2000.
+ EXPECT_EQ(layout.reservation_size(PAGE), size_t(0x1000));
+}
+
+TEST(LlvmLibcElfLoadLayoutTest, PageAlignHelpers) {
+ EXPECT_EQ(LoadLayout::page_align_down(0x1fff, PAGE), ElfW(Addr)(0x1000));
+ EXPECT_EQ(LoadLayout::page_align_down(0x1000, PAGE), ElfW(Addr)(0x1000));
+ EXPECT_EQ(LoadLayout::page_align_up(0x1001, PAGE), ElfW(Addr)(0x2000));
+ EXPECT_EQ(LoadLayout::page_align_up(0x1000, PAGE), ElfW(Addr)(0x1000));
+}
>From 82c706a2a5ec66e4b8fcd0b4f179c20889948831 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 08:03:34 +0100
Subject: [PATCH 29/46] [libc] Map a module's PT_LOAD segments into memory
Reserve the whole span with a single PROT_NONE mapping and place the
segments inside it. Mapping them separately would let something else
take an address in the middle of the range, and the segments must keep
their relative offsets for the module's own relocations to hold. On
failure the reservation is released, so a caller never has to unmap a
partly built module.
Anything past p_filesz is .bss. The remainder of the last file backed
page still holds what the file had there, so it is zeroed explicitly,
and whole pages beyond it are mapped anonymously. Both cases are tested,
since a mapper that handles only whole pages appears correct until a
segment's filesz ends mid page.
Assisted-by: Claude
---
libc/src/__support/elf/CMakeLists.txt | 20 ++
libc/src/__support/elf/map_segments.h | 164 +++++++++++++++++
libc/test/src/__support/elf/CMakeLists.txt | 14 ++
.../src/__support/elf/map_segments_test.cpp | 173 ++++++++++++++++++
4 files changed, 371 insertions(+)
create mode 100644 libc/src/__support/elf/map_segments.h
create mode 100644 libc/test/src/__support/elf/map_segments_test.cpp
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
index 82810c41f16612..ae415f271150e7 100644
--- a/libc/src/__support/elf/CMakeLists.txt
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -94,3 +94,23 @@ add_header_library(
libc.src.__support.macros.attributes
libc.src.__support.macros.config
)
+
+add_header_library(
+ map_segments
+ HDRS
+ map_segments.h
+ DEPENDS
+ libc.hdr.elf_macros
+ libc.hdr.elf_proxy
+ libc.hdr.errno_macros
+ libc.hdr.link_macros
+ libc.hdr.stdint_proxy
+ libc.hdr.sys_mman_macros
+ libc.hdr.types.size_t
+ libc.src.__support.OSUtil.linux.syscall_wrappers.mmap
+ libc.src.__support.OSUtil.linux.syscall_wrappers.munmap
+ libc.src.__support.elf.load_layout
+ libc.src.__support.error_or
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+)
diff --git a/libc/src/__support/elf/map_segments.h b/libc/src/__support/elf/map_segments.h
new file mode 100644
index 00000000000000..4cba95ba92f940
--- /dev/null
+++ b/libc/src/__support/elf/map_segments.h
@@ -0,0 +1,164 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Mapping a module's PT_LOAD segments into memory
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_MAP_SEGMENTS_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_MAP_SEGMENTS_H
+
+#include "hdr/elf_macros.h"
+#include "hdr/elf_proxy.h"
+#include "hdr/errno_macros.h"
+#include "hdr/link_macros.h"
+#include "hdr/stdint_proxy.h"
+#include "hdr/sys_mman_macros.h"
+#include "hdr/types/size_t.h"
+#include "src/__support/OSUtil/linux/syscall_wrappers/mmap.h"
+#include "src/__support/OSUtil/linux/syscall_wrappers/munmap.h"
+#include "src/__support/elf/load_layout.h"
+#include "src/__support/error_or.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// A module that has been mapped into this process.
+struct MappedModule {
+ void *reservation = nullptr;
+ size_t reservation_size = 0;
+ ElfW(Addr) load_bias = 0;
+};
+
+LIBC_INLINE int segment_protection(ElfW(Word) flags) {
+ int prot = 0;
+ if (flags & PF_R)
+ prot |= PROT_READ;
+ if (flags & PF_W)
+ prot |= PROT_WRITE;
+ if (flags & PF_X)
+ prot |= PROT_EXEC;
+ return prot;
+}
+
+// Maps every PT_LOAD segment of an open module into one contiguous region.
+//
+// The whole span is reserved first with a single PROT_NONE mapping and the
+// segments are then placed inside it. Mapping them one at a time would let
+// something else take an address in the middle of the range, and the segments
+// have to keep their relative offsets for the module's own relocations to be
+// correct.
+//
+// On failure the reservation is released, so a caller never has to unmap a
+// partially built module.
+LIBC_INLINE ErrorOr<MappedModule> map_segments(int fd, const ElfW(Phdr) * phdrs,
+ ElfW(Half) phnum,
+ size_t page_size) {
+ LoadLayout layout(phdrs, phnum);
+ if (layout.empty())
+ return Error(ENOEXEC);
+
+ const size_t span = layout.reservation_size(page_size);
+ auto reservation = linux_syscalls::mmap(nullptr, span, PROT_NONE,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ if (!reservation.has_value())
+ return Error(reservation.error());
+
+ const ElfW(Addr) base = reinterpret_cast<ElfW(Addr)>(reservation.value());
+ const ElfW(Addr) first =
+ LoadLayout::page_align_down(layout.first_vaddr(), page_size);
+ const ElfW(Addr) bias = base - first;
+
+ auto fail = [&](int error) -> ErrorOr<MappedModule> {
+ linux_syscalls::munmap(reservation.value(), span);
+ return Error(error);
+ };
+
+ // How far the segments mapped so far reach, so that a segment which shares
+ // a page with the one before it does not map over what is already there.
+ ElfW(Addr) mapped_to = 0;
+
+ for (ElfW(Half) i = 0; i < phnum; ++i) {
+ const ElfW(Phdr) &phdr = phdrs[i];
+ if (phdr.p_type != PT_LOAD || phdr.p_memsz == 0)
+ continue;
+
+ const int prot = segment_protection(phdr.p_flags);
+ const ElfW(Addr) start =
+ LoadLayout::page_align_down(bias + phdr.p_vaddr, page_size);
+ // The file offset has to be aligned by the same amount the address was.
+ const ElfW(Off) offset =
+ phdr.p_offset -
+ (phdr.p_vaddr - LoadLayout::page_align_down(phdr.p_vaddr, page_size));
+ const ElfW(Addr) file_end = bias + phdr.p_vaddr + phdr.p_filesz;
+
+ if (phdr.p_filesz != 0) {
+ const size_t length =
+ LoadLayout::page_align_up(file_end, page_size) - start;
+ auto mapped = linux_syscalls::mmap(reinterpret_cast<void *>(start),
+ length, prot, MAP_PRIVATE | MAP_FIXED,
+ fd, static_cast<off_t>(offset));
+ if (!mapped.has_value())
+ return fail(mapped.error());
+ mapped_to = start + length;
+ }
+
+ if (phdr.p_memsz <= phdr.p_filesz)
+ continue;
+
+ // Anything past p_filesz is .bss. The remainder of the last file backed
+ // page still holds whatever the file had there, so it is zeroed, and any
+ // whole pages beyond it are mapped anonymously.
+ if (phdr.p_filesz != 0 && (prot & PROT_WRITE) != 0) {
+ const ElfW(Addr) page_end =
+ LoadLayout::page_align_up(file_end, page_size);
+ for (ElfW(Addr) p = file_end; p < page_end; ++p)
+ *reinterpret_cast<unsigned char *>(p) = 0;
+ }
+
+ // A segment with nothing in the file behind it is all .bss, and the page
+ // its first byte falls in is its own to map: rounding up from where the
+ // file part ended would skip that page and leave the segment with no
+ // memory at all. Where an earlier segment already reached into that page,
+ // the mapping starts after it instead of over it.
+ ElfW(Addr) bss_start = phdr.p_filesz != 0
+ ? LoadLayout::page_align_up(file_end, page_size)
+ : start;
+ if (bss_start < mapped_to)
+ bss_start = mapped_to;
+ const ElfW(Addr) bss_end = LoadLayout::page_align_up(
+ bias + phdr.p_vaddr + phdr.p_memsz, page_size);
+ if (bss_end > bss_start) {
+ auto zeroed = linux_syscalls::mmap(
+ reinterpret_cast<void *>(bss_start), bss_end - bss_start, prot,
+ MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0);
+ if (!zeroed.has_value())
+ return fail(zeroed.error());
+ mapped_to = bss_end;
+ }
+ }
+
+ MappedModule module;
+ module.reservation = reservation.value();
+ module.reservation_size = span;
+ module.load_bias = bias;
+ return module;
+}
+
+LIBC_INLINE void unmap_module(const MappedModule &module) {
+ if (module.reservation != nullptr)
+ linux_syscalls::munmap(module.reservation, module.reservation_size);
+}
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_MAP_SEGMENTS_H
diff --git a/libc/test/src/__support/elf/CMakeLists.txt b/libc/test/src/__support/elf/CMakeLists.txt
index 74598684cb21b0..dcc1ac02e92664 100644
--- a/libc/test/src/__support/elf/CMakeLists.txt
+++ b/libc/test/src/__support/elf/CMakeLists.txt
@@ -69,3 +69,17 @@ add_libc_test(
DEPENDS
libc.src.__support.elf.load_layout
)
+
+add_libc_test(
+ map_segments_test
+ SUITE
+ libc-support-tests
+ SRCS
+ map_segments_test.cpp
+ DEPENDS
+ libc.hdr.fcntl_macros
+ libc.src.__support.OSUtil.linux.syscall_wrappers.close
+ libc.src.__support.OSUtil.linux.syscall_wrappers.open
+ libc.src.__support.OSUtil.linux.syscall_wrappers.write
+ libc.src.__support.elf.map_segments
+)
diff --git a/libc/test/src/__support/elf/map_segments_test.cpp b/libc/test/src/__support/elf/map_segments_test.cpp
new file mode 100644
index 00000000000000..476edd74e34050
--- /dev/null
+++ b/libc/test/src/__support/elf/map_segments_test.cpp
@@ -0,0 +1,173 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Unittests for PT_LOAD segment mapping
+///
+//===----------------------------------------------------------------------===//
+
+#include "hdr/fcntl_macros.h"
+#include "src/__support/OSUtil/linux/syscall_wrappers/close.h"
+#include "src/__support/OSUtil/linux/syscall_wrappers/open.h"
+#include "src/__support/OSUtil/linux/syscall_wrappers/write.h"
+#include "src/__support/elf/map_segments.h"
+#include "test/UnitTest/Test.h"
+
+using LIBC_NAMESPACE::elf::map_segments;
+using LIBC_NAMESPACE::elf::MappedModule;
+using LIBC_NAMESPACE::elf::unmap_module;
+
+namespace {
+
+constexpr size_t PAGE = 4096;
+constexpr char PATH[] = "testdata/map_segments.bin";
+
+// Two pages of distinguishable bytes, so a wrongly computed file offset shows
+// up as the wrong page rather than as a crash.
+constexpr unsigned char FIRST_BYTE = 0xaa;
+constexpr unsigned char SECOND_BYTE = 0xbb;
+
+int write_backing_file() {
+ auto fd = LIBC_NAMESPACE::linux_syscalls::open(
+ PATH, O_CREAT | O_TRUNC | O_WRONLY, 0600);
+ if (!fd.has_value())
+ return -1;
+ static unsigned char page[PAGE];
+ for (size_t i = 0; i < PAGE; ++i)
+ page[i] = FIRST_BYTE;
+ LIBC_NAMESPACE::linux_syscalls::write(fd.value(), page, PAGE);
+ for (size_t i = 0; i < PAGE; ++i)
+ page[i] = SECOND_BYTE;
+ LIBC_NAMESPACE::linux_syscalls::write(fd.value(), page, PAGE);
+ LIBC_NAMESPACE::linux_syscalls::close(fd.value());
+
+ auto rd = LIBC_NAMESPACE::linux_syscalls::open(PATH, O_RDONLY, 0);
+ return rd.has_value() ? rd.value() : -1;
+}
+
+ElfW(Phdr) load(ElfW(Addr) vaddr, ElfW(Off) offset, ElfW(Xword) filesz,
+ ElfW(Xword) memsz, ElfW(Word) flags) {
+ ElfW(Phdr) p{};
+ p.p_type = PT_LOAD;
+ p.p_vaddr = vaddr;
+ p.p_offset = offset;
+ p.p_filesz = filesz;
+ p.p_memsz = memsz;
+ p.p_flags = flags;
+ p.p_align = PAGE;
+ return p;
+}
+
+} // anonymous namespace
+
+TEST(LlvmLibcElfMapSegmentsTest, RejectsModuleWithNoLoadSegments) {
+ ElfW(Phdr) phdrs[1]{};
+ phdrs[0].p_type = PT_DYNAMIC;
+ auto result = map_segments(-1, phdrs, 1, PAGE);
+ EXPECT_FALSE(result.has_value());
+}
+
+TEST(LlvmLibcElfMapSegmentsTest, MapsSegmentsAndZeroesBss) {
+ int fd = write_backing_file();
+ ASSERT_GE(fd, 0);
+
+ // Page 0 read only from file offset 0, page 1 read write from offset PAGE
+ // with a second page of .bss behind it.
+ ElfW(Phdr) phdrs[] = {
+ load(0, 0, PAGE, PAGE, PF_R),
+ load(PAGE, PAGE, PAGE, 2 * PAGE, PF_R | PF_W),
+ };
+
+ auto result = map_segments(fd, phdrs, 2, PAGE);
+ LIBC_NAMESPACE::linux_syscalls::close(fd);
+ ASSERT_TRUE(result.has_value());
+
+ MappedModule module = result.value();
+ ASSERT_TRUE(module.reservation != nullptr);
+ // Three pages: one read only, one read write, one .bss.
+ EXPECT_EQ(module.reservation_size, size_t(3 * PAGE));
+
+ const unsigned char *base =
+ reinterpret_cast<const unsigned char *>(module.load_bias);
+ // File backed contents landed on the right pages.
+ EXPECT_EQ(base[0], FIRST_BYTE);
+ EXPECT_EQ(base[PAGE - 1], FIRST_BYTE);
+ EXPECT_EQ(base[PAGE], SECOND_BYTE);
+ EXPECT_EQ(base[2 * PAGE - 1], SECOND_BYTE);
+ // The .bss page is zero, not a repeat of the file.
+ EXPECT_EQ(base[2 * PAGE], static_cast<unsigned char>(0));
+ EXPECT_EQ(base[3 * PAGE - 1], static_cast<unsigned char>(0));
+
+ unmap_module(module);
+}
+
+TEST(LlvmLibcElfMapSegmentsTest, PartialPageBssIsZeroed) {
+ int fd = write_backing_file();
+ ASSERT_GE(fd, 0);
+
+ // filesz stops halfway through the page, so the rest of that page still
+ // holds file bytes and has to be cleared.
+ ElfW(Phdr) phdrs[] = {load(0, 0, PAGE / 2, PAGE, PF_R | PF_W)};
+
+ auto result = map_segments(fd, phdrs, 1, PAGE);
+ LIBC_NAMESPACE::linux_syscalls::close(fd);
+ ASSERT_TRUE(result.has_value());
+
+ MappedModule module = result.value();
+ const unsigned char *base =
+ reinterpret_cast<const unsigned char *>(module.load_bias);
+ EXPECT_EQ(base[0], FIRST_BYTE);
+ EXPECT_EQ(base[PAGE / 2 - 1], FIRST_BYTE);
+ EXPECT_EQ(base[PAGE / 2], static_cast<unsigned char>(0));
+ EXPECT_EQ(base[PAGE - 1], static_cast<unsigned char>(0));
+
+ unmap_module(module);
+}
+
+TEST(LlvmLibcElfMapSegmentsTest, ASegmentWithNothingInTheFileStillGetsMemory) {
+ // A segment whose file size is zero is all .bss. Rounding up from where the
+ // file part ended would skip the page its first byte falls in and leave it
+ // with no memory at all, which is a write to an unmapped address the first
+ // time anything touches it.
+ ElfW(Phdr) headers[2] = {};
+ // Something file backed, ending part way into a page.
+ headers[0].p_type = PT_LOAD;
+ headers[0].p_flags = PF_R | PF_W;
+ headers[0].p_offset = 0;
+ headers[0].p_vaddr = 0;
+ headers[0].p_filesz = 64;
+ headers[0].p_memsz = 64;
+ headers[0].p_align = PAGE;
+ // And a segment of nothing but memory, in a page of its own.
+ headers[1].p_type = PT_LOAD;
+ headers[1].p_flags = PF_R | PF_W;
+ headers[1].p_offset = 64;
+ headers[1].p_vaddr = 2 * PAGE + 16;
+ headers[1].p_filesz = 0;
+ headers[1].p_memsz = 32;
+ headers[1].p_align = PAGE;
+
+ auto fd = LIBC_NAMESPACE::linux_syscalls::open(PATH, O_RDONLY, 0);
+ ASSERT_TRUE(fd.has_value());
+ auto mapped = map_segments(fd.value(), headers, 2, PAGE);
+ LIBC_NAMESPACE::linux_syscalls::close(fd.value());
+ ASSERT_TRUE(mapped.has_value());
+
+ // The whole of the second segment can be written to and reads back what was
+ // written, which it could not if it had never been mapped.
+ auto *bss = reinterpret_cast<unsigned char *>(mapped.value().load_bias +
+ headers[1].p_vaddr);
+ for (size_t i = 0; i < headers[1].p_memsz; ++i)
+ ASSERT_EQ(bss[i], static_cast<unsigned char>(0));
+ for (size_t i = 0; i < headers[1].p_memsz; ++i)
+ bss[i] = static_cast<unsigned char>(i + 1);
+ for (size_t i = 0; i < headers[1].p_memsz; ++i)
+ ASSERT_EQ(bss[i], static_cast<unsigned char>(i + 1));
+
+ unmap_module(mapped.value());
+}
>From f7bdf8aae95e4ba81fefd5b610c82662ed5f42f1 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 08:06:05 +0100
Subject: [PATCH 30/46] [libc] Load a shared object from a file
Open the file, validate the header, read the program headers, map the
segments, describe the result, and apply the relocations the module can
resolve on its own.
The module is described through its mapped program headers rather than
the copy read from the file, so the description remains valid for the
lifetime of the mapping.
Only relative relocations can be applied here. Anything naming a symbol
needs the rest of the module set, so binding is left out.
The test loads the libc.so this build produced and looks up printf,
memcpy and fopen, exercising mapping, dynamic section parsing and GNU
hash lookup against a real object. It runs only when LIBC_ENABLE_SHARED
provides a shared object.
e_phnum is capped so a malformed file cannot request an unbounded
buffer.
Assisted-by: Claude
---
libc/src/__support/elf/CMakeLists.txt | 24 ++++
libc/src/__support/elf/load_module.h | 131 ++++++++++++++++++
libc/src/__support/elf/map_segments.h | 39 +++++-
libc/test/src/__support/elf/CMakeLists.txt | 14 ++
.../src/__support/elf/load_module_test.cpp | 90 ++++++++++++
.../src/__support/elf/map_segments_test.cpp | 60 +++++++-
6 files changed, 348 insertions(+), 10 deletions(-)
create mode 100644 libc/src/__support/elf/load_module.h
create mode 100644 libc/test/src/__support/elf/load_module_test.cpp
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
index ae415f271150e7..08955ab3432bbe 100644
--- a/libc/src/__support/elf/CMakeLists.txt
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -114,3 +114,27 @@ add_header_library(
libc.src.__support.macros.attributes
libc.src.__support.macros.config
)
+
+add_header_library(
+ load_module
+ HDRS
+ load_module.h
+ DEPENDS
+ libc.hdr.elf_macros
+ libc.hdr.elf_proxy
+ libc.hdr.errno_macros
+ libc.hdr.fcntl_macros
+ libc.hdr.link_macros
+ libc.hdr.stdint_proxy
+ libc.hdr.stdio_macros
+ libc.hdr.types.size_t
+ libc.src.__support.OSUtil.linux.syscall_wrappers.close
+ libc.src.__support.OSUtil.linux.syscall_wrappers.lseek
+ libc.src.__support.OSUtil.linux.syscall_wrappers.open
+ libc.src.__support.OSUtil.linux.syscall_wrappers.read
+ libc.src.__support.elf.map_segments
+ libc.src.__support.elf.module
+ libc.src.__support.error_or
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+)
diff --git a/libc/src/__support/elf/load_module.h b/libc/src/__support/elf/load_module.h
new file mode 100644
index 00000000000000..4bc51631077084
--- /dev/null
+++ b/libc/src/__support/elf/load_module.h
@@ -0,0 +1,131 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Loading a shared object from a file
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_LOAD_MODULE_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_LOAD_MODULE_H
+
+#include "hdr/elf_macros.h"
+#include "hdr/elf_proxy.h"
+#include "hdr/errno_macros.h"
+#include "hdr/fcntl_macros.h"
+#include "hdr/link_macros.h"
+#include "hdr/stdint_proxy.h"
+#include "hdr/stdio_macros.h"
+#include "hdr/types/size_t.h"
+#include "src/__support/OSUtil/linux/syscall_wrappers/close.h"
+#include "src/__support/OSUtil/linux/syscall_wrappers/lseek.h"
+#include "src/__support/OSUtil/linux/syscall_wrappers/open.h"
+#include "src/__support/OSUtil/linux/syscall_wrappers/read.h"
+#include "src/__support/elf/map_segments.h"
+#include "src/__support/elf/module.h"
+#include "src/__support/error_or.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// A shared object that has been mapped and had its own relocations applied.
+//
+// It is not linked against anything yet: symbols it imports from other modules
+// still need binding, which needs the rest of the module set.
+struct LoadedModule {
+ MappedModule mapping;
+ Module module;
+};
+
+#ifdef __LP64__
+constexpr unsigned char ELFCLASS_NATIVE = ELFCLASS64;
+#else
+constexpr unsigned char ELFCLASS_NATIVE = ELFCLASS32;
+#endif
+
+// The largest program header table this will read, so a malformed file cannot
+// ask for an unbounded stack buffer.
+constexpr ElfW(Half) MAX_PHNUM = 128;
+
+LIBC_INLINE bool is_loadable_elf(const ElfW(Ehdr) & header) {
+ return header.e_ident[EI_MAG0] == ELFMAG0 &&
+ header.e_ident[EI_MAG1] == ELFMAG1 &&
+ header.e_ident[EI_MAG2] == ELFMAG2 &&
+ header.e_ident[EI_MAG3] == ELFMAG3 &&
+ header.e_ident[EI_CLASS] == ELFCLASS_NATIVE &&
+ (header.e_type == ET_DYN || header.e_type == ET_EXEC) &&
+ header.e_phentsize == sizeof(ElfW(Phdr)) && header.e_phnum != 0 &&
+ header.e_phnum <= MAX_PHNUM;
+}
+
+// Opens `path`, maps it, and applies the relocations it can resolve on its
+// own. The caller owns the mapping and releases it with unmap_module.
+LIBC_INLINE ErrorOr<LoadedModule> load_module(const char *path,
+ size_t page_size) {
+ auto fd = linux_syscalls::open(path, O_RDONLY, 0);
+ if (!fd.has_value())
+ return Error(fd.error());
+
+ auto close_fd = [&] { linux_syscalls::close(fd.value()); };
+
+ ElfW(Ehdr) header;
+ auto header_read = linux_syscalls::read(fd.value(), &header, sizeof(header));
+ if (!header_read.has_value() ||
+ static_cast<size_t>(header_read.value()) != sizeof(header)) {
+ close_fd();
+ return Error(ENOEXEC);
+ }
+ if (!is_loadable_elf(header)) {
+ close_fd();
+ return Error(ENOEXEC);
+ }
+
+ ElfW(Phdr) phdrs[MAX_PHNUM];
+ const size_t phdrs_size = header.e_phnum * sizeof(ElfW(Phdr));
+ if (!linux_syscalls::lseek(fd.value(), static_cast<off_t>(header.e_phoff),
+ SEEK_SET)
+ .has_value()) {
+ close_fd();
+ return Error(ENOEXEC);
+ }
+ auto phdrs_read = linux_syscalls::read(fd.value(), phdrs, phdrs_size);
+ if (!phdrs_read.has_value() ||
+ static_cast<size_t>(phdrs_read.value()) != phdrs_size) {
+ close_fd();
+ return Error(ENOEXEC);
+ }
+
+ auto mapped =
+ map_segments(fd.value(), phdrs, header.e_phnum, page_size, path);
+ close_fd();
+ if (!mapped.has_value())
+ return Error(mapped.error());
+
+ LoadedModule loaded;
+ loaded.mapping = mapped.value();
+
+ // Describe the module through its mapped program headers rather than the
+ // copy read from the file, so the description stays valid for as long as
+ // the mapping does.
+ const ElfW(Phdr) *mapped_phdrs = reinterpret_cast<const ElfW(Phdr) *>(
+ loaded.mapping.load_bias + header.e_phoff);
+ loaded.module = Module(mapped_phdrs, header.e_phnum, loaded.mapping.load_bias,
+ loaded.mapping.name);
+
+ // Relative relocations are the ones a module can apply to itself. Anything
+ // that names a symbol needs the other modules to be loaded first.
+ loaded.module.relocations().apply_relative(loaded.mapping.load_bias);
+ return loaded;
+}
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_LOAD_MODULE_H
diff --git a/libc/src/__support/elf/map_segments.h b/libc/src/__support/elf/map_segments.h
index 4cba95ba92f940..27c3b35064587f 100644
--- a/libc/src/__support/elf/map_segments.h
+++ b/libc/src/__support/elf/map_segments.h
@@ -36,6 +36,10 @@ struct MappedModule {
void *reservation = nullptr;
size_t reservation_size = 0;
ElfW(Addr) load_bias = 0;
+ // The path the module was loaded from, held inside the reservation so that
+ // it lasts exactly as long as the module does. Null where the path was too
+ // long to keep.
+ const char *name = nullptr;
};
LIBC_INLINE int segment_protection(ElfW(Word) flags) {
@@ -57,17 +61,24 @@ LIBC_INLINE int segment_protection(ElfW(Word) flags) {
// have to keep their relative offsets for the module's own relocations to be
// correct.
//
+// A copy of `path` is kept in a page of its own past the segments. The string
+// a caller passes in is often a buffer it is about to reuse or release, and
+// the module has to be able to say where it came from for as long as it is
+// loaded.
+//
// On failure the reservation is released, so a caller never has to unmap a
// partially built module.
LIBC_INLINE ErrorOr<MappedModule> map_segments(int fd, const ElfW(Phdr) * phdrs,
ElfW(Half) phnum,
- size_t page_size) {
+ size_t page_size,
+ const char *path) {
LoadLayout layout(phdrs, phnum);
if (layout.empty())
return Error(ENOEXEC);
const size_t span = layout.reservation_size(page_size);
- auto reservation = linux_syscalls::mmap(nullptr, span, PROT_NONE,
+ const size_t total = span + page_size;
+ auto reservation = linux_syscalls::mmap(nullptr, total, PROT_NONE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (!reservation.has_value())
return Error(reservation.error());
@@ -78,7 +89,7 @@ LIBC_INLINE ErrorOr<MappedModule> map_segments(int fd, const ElfW(Phdr) * phdrs,
const ElfW(Addr) bias = base - first;
auto fail = [&](int error) -> ErrorOr<MappedModule> {
- linux_syscalls::munmap(reservation.value(), span);
+ linux_syscalls::munmap(reservation.value(), total);
return Error(error);
};
@@ -146,10 +157,30 @@ LIBC_INLINE ErrorOr<MappedModule> map_segments(int fd, const ElfW(Phdr) * phdrs,
}
}
+ auto name_page = linux_syscalls::mmap(
+ reinterpret_cast<void *>(base + span), page_size, PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0);
+ if (!name_page.has_value())
+ return fail(name_page.error());
+
+ const char *stored = nullptr;
+ if (path != nullptr) {
+ char *out = reinterpret_cast<char *>(base + span);
+ size_t i = 0;
+ for (; path[i] != '\0' && i + 1 < page_size; ++i)
+ out[i] = path[i];
+ out[i] = '\0';
+ // A path with no room to be kept whole is better left unnamed than
+ // truncated: a truncated name could compare equal to another module's.
+ if (path[i] == '\0')
+ stored = out;
+ }
+
MappedModule module;
module.reservation = reservation.value();
- module.reservation_size = span;
+ module.reservation_size = total;
module.load_bias = bias;
+ module.name = stored;
return module;
}
diff --git a/libc/test/src/__support/elf/CMakeLists.txt b/libc/test/src/__support/elf/CMakeLists.txt
index dcc1ac02e92664..18d0a0d9b97474 100644
--- a/libc/test/src/__support/elf/CMakeLists.txt
+++ b/libc/test/src/__support/elf/CMakeLists.txt
@@ -83,3 +83,17 @@ add_libc_test(
libc.src.__support.OSUtil.linux.syscall_wrappers.write
libc.src.__support.elf.map_segments
)
+
+if(LIBC_ENABLE_SHARED)
+ add_libc_test(
+ load_module_test
+ SUITE
+ libc-support-tests
+ SRCS
+ load_module_test.cpp
+ COMPILE_OPTIONS
+ -DLIBC_TEST_SHARED_OBJECT="$<TARGET_FILE:libc-shared>"
+ DEPENDS
+ libc.src.__support.elf.load_module
+ )
+endif()
diff --git a/libc/test/src/__support/elf/load_module_test.cpp b/libc/test/src/__support/elf/load_module_test.cpp
new file mode 100644
index 00000000000000..41e2e245835a82
--- /dev/null
+++ b/libc/test/src/__support/elf/load_module_test.cpp
@@ -0,0 +1,90 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Unittests for loading a shared object
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/elf/load_module.h"
+#include "test/UnitTest/Test.h"
+
+using LIBC_NAMESPACE::elf::load_module;
+using LIBC_NAMESPACE::elf::LoadedModule;
+using LIBC_NAMESPACE::elf::unmap_module;
+
+namespace {
+constexpr size_t PAGE = 4096;
+// Set by the build to the libc.so this test was built alongside.
+constexpr const char *SO_PATH = LIBC_TEST_SHARED_OBJECT;
+} // anonymous namespace
+
+TEST(LlvmLibcElfLoadModuleTest, RejectsMissingFile) {
+ auto result = load_module("testdata/definitely-not-here.so", PAGE);
+ EXPECT_FALSE(result.has_value());
+}
+
+TEST(LlvmLibcElfLoadModuleTest, RejectsNonElfFile) {
+ // The test binary's own source tree always has this.
+ auto result = load_module("/etc/hostname", PAGE);
+ EXPECT_FALSE(result.has_value());
+}
+
+TEST(LlvmLibcElfLoadModuleTest, LoadsRealSharedObject) {
+ auto result = load_module(SO_PATH, PAGE);
+ ASSERT_TRUE(result.has_value());
+ LoadedModule loaded = result.value();
+
+ ASSERT_TRUE(loaded.mapping.reservation != nullptr);
+ EXPECT_GT(loaded.mapping.reservation_size, size_t(0));
+
+ // The ELF header landed at the load bias.
+ const unsigned char *base =
+ reinterpret_cast<const unsigned char *>(loaded.mapping.load_bias);
+ EXPECT_EQ(base[0], static_cast<unsigned char>(0x7f));
+ EXPECT_EQ(base[1], static_cast<unsigned char>('E'));
+ EXPECT_EQ(base[2], static_cast<unsigned char>('L'));
+ EXPECT_EQ(base[3], static_cast<unsigned char>('F'));
+
+ // A real shared object has a dynamic section and a string table.
+ EXPECT_FALSE(loaded.module.dynamic().empty());
+ EXPECT_TRUE(loaded.module.strtab() != nullptr);
+
+ unmap_module(loaded.mapping);
+}
+
+TEST(LlvmLibcElfLoadModuleTest, ResolvesSymbolsInLoadedObject) {
+ auto result = load_module(SO_PATH, PAGE);
+ ASSERT_TRUE(result.has_value());
+ LoadedModule loaded = result.value();
+
+ auto symbols = loaded.module.symbols();
+ ASSERT_FALSE(symbols.empty());
+
+ // These are exported by libc.so, and looking them up exercises the whole
+ // chain: mapping, dynamic section parsing, and GNU hash lookup.
+ const ElfW(Sym) *printf_sym = symbols.lookup("printf");
+ ASSERT_TRUE(printf_sym != nullptr);
+ EXPECT_NE(printf_sym->st_value, ElfW(Addr)(0));
+
+ EXPECT_TRUE(symbols.lookup("memcpy") != nullptr);
+ EXPECT_TRUE(symbols.lookup("fopen") != nullptr);
+ // Something that is definitely not there.
+ EXPECT_TRUE(symbols.lookup("definitely_not_a_libc_symbol") == nullptr);
+
+ unmap_module(loaded.mapping);
+}
+
+TEST(LlvmLibcElfLoadModuleTest, ReportsSoname) {
+ auto result = load_module(SO_PATH, PAGE);
+ ASSERT_TRUE(result.has_value());
+ LoadedModule loaded = result.value();
+ // libc.so is built with a SONAME, so it must come back non-null.
+ EXPECT_TRUE(loaded.module.soname() != nullptr);
+ unmap_module(loaded.mapping);
+}
diff --git a/libc/test/src/__support/elf/map_segments_test.cpp b/libc/test/src/__support/elf/map_segments_test.cpp
index 476edd74e34050..138f1ef70b8be2 100644
--- a/libc/test/src/__support/elf/map_segments_test.cpp
+++ b/libc/test/src/__support/elf/map_segments_test.cpp
@@ -68,7 +68,7 @@ ElfW(Phdr) load(ElfW(Addr) vaddr, ElfW(Off) offset, ElfW(Xword) filesz,
TEST(LlvmLibcElfMapSegmentsTest, RejectsModuleWithNoLoadSegments) {
ElfW(Phdr) phdrs[1]{};
phdrs[0].p_type = PT_DYNAMIC;
- auto result = map_segments(-1, phdrs, 1, PAGE);
+ auto result = map_segments(-1, phdrs, 1, PAGE, PATH);
EXPECT_FALSE(result.has_value());
}
@@ -83,14 +83,17 @@ TEST(LlvmLibcElfMapSegmentsTest, MapsSegmentsAndZeroesBss) {
load(PAGE, PAGE, PAGE, 2 * PAGE, PF_R | PF_W),
};
- auto result = map_segments(fd, phdrs, 2, PAGE);
+ auto result = map_segments(fd, phdrs, 2, PAGE, PATH);
LIBC_NAMESPACE::linux_syscalls::close(fd);
ASSERT_TRUE(result.has_value());
MappedModule module = result.value();
ASSERT_TRUE(module.reservation != nullptr);
- // Three pages: one read only, one read write, one .bss.
- EXPECT_EQ(module.reservation_size, size_t(3 * PAGE));
+ // Three pages: one read only, one read write, one .bss, plus the page the
+ // name is kept in.
+ EXPECT_EQ(module.reservation_size, size_t(4 * PAGE));
+ ASSERT_TRUE(module.name != nullptr);
+ EXPECT_STREQ(module.name, PATH);
const unsigned char *base =
reinterpret_cast<const unsigned char *>(module.load_bias);
@@ -114,7 +117,7 @@ TEST(LlvmLibcElfMapSegmentsTest, PartialPageBssIsZeroed) {
// holds file bytes and has to be cleared.
ElfW(Phdr) phdrs[] = {load(0, 0, PAGE / 2, PAGE, PF_R | PF_W)};
- auto result = map_segments(fd, phdrs, 1, PAGE);
+ auto result = map_segments(fd, phdrs, 1, PAGE, PATH);
LIBC_NAMESPACE::linux_syscalls::close(fd);
ASSERT_TRUE(result.has_value());
@@ -154,7 +157,7 @@ TEST(LlvmLibcElfMapSegmentsTest, ASegmentWithNothingInTheFileStillGetsMemory) {
auto fd = LIBC_NAMESPACE::linux_syscalls::open(PATH, O_RDONLY, 0);
ASSERT_TRUE(fd.has_value());
- auto mapped = map_segments(fd.value(), headers, 2, PAGE);
+ auto mapped = map_segments(fd.value(), headers, 2, PAGE, PATH);
LIBC_NAMESPACE::linux_syscalls::close(fd.value());
ASSERT_TRUE(mapped.has_value());
@@ -171,3 +174,48 @@ TEST(LlvmLibcElfMapSegmentsTest, ASegmentWithNothingInTheFileStillGetsMemory) {
unmap_module(mapped.value());
}
+
+TEST(LlvmLibcElfMapSegmentsTest, KeepsACopyOfTheName) {
+ int fd = write_backing_file();
+ ASSERT_GE(fd, 0);
+ ElfW(Phdr) phdrs[] = {load(0, 0, PAGE, PAGE, PF_R)};
+
+ // A caller is free to reuse the buffer it passed in as soon as the call
+ // returns, so what the module reports has to be somewhere else.
+ char path[sizeof(PATH)];
+ for (size_t i = 0; i < sizeof(PATH); ++i)
+ path[i] = PATH[i];
+
+ auto result = map_segments(fd, phdrs, 1, PAGE, path);
+ LIBC_NAMESPACE::linux_syscalls::close(fd);
+ ASSERT_TRUE(result.has_value());
+
+ MappedModule module = result.value();
+ ASSERT_TRUE(module.name != nullptr);
+ EXPECT_TRUE(module.name != path);
+ for (size_t i = 0; i < sizeof(path); ++i)
+ path[i] = 'x';
+ EXPECT_STREQ(module.name, PATH);
+
+ unmap_module(module);
+}
+
+TEST(LlvmLibcElfMapSegmentsTest, LeavesAModuleUnnamedRatherThanTruncateIt) {
+ int fd = write_backing_file();
+ ASSERT_GE(fd, 0);
+ ElfW(Phdr) phdrs[] = {load(0, 0, PAGE, PAGE, PF_R)};
+
+ char path[PAGE + 1];
+ for (size_t i = 0; i < PAGE; ++i)
+ path[i] = 'a';
+ path[PAGE] = '\0';
+
+ auto result = map_segments(fd, phdrs, 1, PAGE, path);
+ LIBC_NAMESPACE::linux_syscalls::close(fd);
+ ASSERT_TRUE(result.has_value());
+
+ MappedModule module = result.value();
+ EXPECT_TRUE(module.name == nullptr);
+
+ unmap_module(module);
+}
>From 77ae0cf950e9ee62db4c2a84edbb5add78c66129 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 08:09:47 +0100
Subject: [PATCH 31/46] [libc] Bind a module's symbol relocations across a
search order
Resolve the relocations that name a symbol, which is what a loaded
module needs beyond being mapped.
SearchOrder walks the modules in load order and takes the first
definition, which is how interposition works. A symbol a module only
references has no section, so SHN_UNDEF entries are skipped rather than
resolving to zero.
A weak undefined symbol may stay unresolved and is written as zero,
which is how a module tests for an optional dependency. A strong symbol
that cannot be found is counted rather than fatal, leaving the caller to
decide whether to fail the load.
The test loads the libm.so and libc.so this build produced, confirms
libm records a DT_NEEDED on libc, and binds libm against it with nothing
unresolved.
Also adds the absolute relocation types to elf-macros.h, alongside the
relative and PLT ones.
Assisted-by: Claude
---
libc/include/llvm-libc-macros/elf-macros.h | 4 +
libc/src/__support/elf/CMakeLists.txt | 17 +++
libc/src/__support/elf/bind.h | 158 +++++++++++++++++++++
libc/test/src/__support/elf/CMakeLists.txt | 19 +++
libc/test/src/__support/elf/bind_test.cpp | 93 ++++++++++++
5 files changed, 291 insertions(+)
create mode 100644 libc/src/__support/elf/bind.h
create mode 100644 libc/test/src/__support/elf/bind_test.cpp
diff --git a/libc/include/llvm-libc-macros/elf-macros.h b/libc/include/llvm-libc-macros/elf-macros.h
index 7ab790dd7ef4a9..d6322f7a9501a6 100644
--- a/libc/include/llvm-libc-macros/elf-macros.h
+++ b/libc/include/llvm-libc-macros/elf-macros.h
@@ -39,24 +39,28 @@
// resolver at that address and stores what it returns.
// x86_64
+#define R_X86_64_64 1
#define R_X86_64_RELATIVE 8
#define R_X86_64_GLOB_DAT 6
#define R_X86_64_JUMP_SLOT 7
#define R_X86_64_IRELATIVE 37
// aarch64
+#define R_AARCH64_ABS64 257
#define R_AARCH64_RELATIVE 1027
#define R_AARCH64_GLOB_DAT 1025
#define R_AARCH64_JUMP_SLOT 1026
#define R_AARCH64_IRELATIVE 1032
// arm
+#define R_ARM_ABS32 2
#define R_ARM_RELATIVE 23
#define R_ARM_GLOB_DAT 21
#define R_ARM_JUMP_SLOT 22
#define R_ARM_IRELATIVE 160
// riscv
+#define R_RISCV_64 2
#define R_RISCV_RELATIVE 3
#define R_RISCV_JUMP_SLOT 5
#define R_RISCV_IRELATIVE 58
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
index 08955ab3432bbe..df25b16179b370 100644
--- a/libc/src/__support/elf/CMakeLists.txt
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -138,3 +138,20 @@ add_header_library(
libc.src.__support.macros.attributes
libc.src.__support.macros.config
)
+
+add_header_library(
+ bind
+ HDRS
+ bind.h
+ DEPENDS
+ libc.hdr.elf_macros
+ libc.hdr.elf_proxy
+ libc.hdr.link_macros
+ libc.hdr.stdint_proxy
+ libc.hdr.types.size_t
+ libc.src.__support.CPP.optional
+ libc.src.__support.elf.module
+ libc.src.__support.elf.relocation
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+)
diff --git a/libc/src/__support/elf/bind.h b/libc/src/__support/elf/bind.h
new file mode 100644
index 00000000000000..0adaeabdb284e5
--- /dev/null
+++ b/libc/src/__support/elf/bind.h
@@ -0,0 +1,158 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Binding a module's symbol relocations
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_BIND_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_BIND_H
+
+#include "hdr/elf_macros.h"
+#include "hdr/elf_proxy.h"
+#include "hdr/link_macros.h"
+#include "hdr/stdint_proxy.h"
+#include "hdr/types/size_t.h"
+#include "src/__support/CPP/optional.h"
+#include "src/__support/elf/module.h"
+#include "src/__support/elf/relocation.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// Relocation types that take a symbol value. Anything else is either relative,
+// handled before this point, or unsupported.
+#if defined(LIBC_TARGET_ARCH_IS_X86_64)
+constexpr uint32_t RELOC_ABSOLUTE = R_X86_64_64;
+constexpr uint32_t RELOC_GLOB_DAT = R_X86_64_GLOB_DAT;
+constexpr uint32_t RELOC_JUMP_SLOT = R_X86_64_JUMP_SLOT;
+#elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
+constexpr uint32_t RELOC_ABSOLUTE = 257; // R_AARCH64_ABS64
+constexpr uint32_t RELOC_GLOB_DAT = R_AARCH64_GLOB_DAT;
+constexpr uint32_t RELOC_JUMP_SLOT = R_AARCH64_JUMP_SLOT;
+#elif defined(LIBC_TARGET_ARCH_IS_ANY_RISCV)
+constexpr uint32_t RELOC_ABSOLUTE = 2; // R_RISCV_64
+constexpr uint32_t RELOC_GLOB_DAT = R_RISCV_JUMP_SLOT;
+constexpr uint32_t RELOC_JUMP_SLOT = R_RISCV_JUMP_SLOT;
+#elif defined(LIBC_TARGET_ARCH_IS_ARM)
+constexpr uint32_t RELOC_ABSOLUTE = 2; // R_ARM_ABS32
+constexpr uint32_t RELOC_GLOB_DAT = R_ARM_GLOB_DAT;
+constexpr uint32_t RELOC_JUMP_SLOT = R_ARM_JUMP_SLOT;
+#else
+#error "Symbol relocation types are not known for this architecture"
+#endif
+
+LIBC_INLINE constexpr unsigned char symbol_binding(unsigned char info) {
+#ifdef __LP64__
+ return static_cast<unsigned char>(ELF64_ST_BIND(info));
+#else
+ return static_cast<unsigned char>(ELF32_ST_BIND(info));
+#endif
+}
+
+LIBC_INLINE constexpr uint32_t reloc_symbol(ElfW(Xword) info) {
+#ifdef __LP64__
+ return static_cast<uint32_t>(ELF64_R_SYM(info));
+#else
+ return static_cast<uint32_t>(ELF32_R_SYM(info));
+#endif
+}
+
+// An ordered set of modules, searched in load order the way a dynamic loader
+// does. The first definition found wins, which is what makes interposition
+// work.
+class SearchOrder {
+public:
+ LIBC_INLINE constexpr SearchOrder(const Module *modules, size_t count)
+ : modules_(modules), count_(count) {}
+
+ // Returns the runtime address of `name`, or nothing if no module defines it.
+ LIBC_INLINE cpp::optional<ElfW(Addr)> resolve(const char *name) const {
+ for (size_t i = 0; i < count_; ++i) {
+ const ElfW(Sym) *symbol = modules_[i].symbols().lookup(name);
+ if (symbol == nullptr)
+ continue;
+ // A symbol the module only references rather than defines has no
+ // section, and must not satisfy the lookup.
+ if (symbol->st_shndx == SHN_UNDEF)
+ continue;
+ return modules_[i].load_bias() + symbol->st_value;
+ }
+ return cpp::nullopt;
+ }
+
+private:
+ const Module *modules_;
+ size_t count_;
+};
+
+struct BindResult {
+ size_t bound = 0;
+ size_t unresolved = 0;
+};
+
+// Applies one table of symbol relocations against the search order.
+//
+// A strong symbol that cannot be found is counted rather than fatal, so the
+// caller decides whether to fail the load.
+LIBC_INLINE BindResult bind_relocations(const Module &target,
+ const RelaTable &table,
+ const SearchOrder &order) {
+ BindResult result;
+ const ElfW(Sym) *symtab = target.symtab();
+ const char *strtab = target.strtab();
+ if (symtab == nullptr || strtab == nullptr)
+ return result;
+
+ for (const ElfW(Rela) *rela = table.begin(); rela != table.end(); ++rela) {
+ const uint32_t type = reloc_type(rela->r_info);
+ if (type != RELOC_ABSOLUTE && type != RELOC_GLOB_DAT &&
+ type != RELOC_JUMP_SLOT)
+ continue;
+
+ const ElfW(Sym) &symbol = symtab[reloc_symbol(rela->r_info)];
+ const char *name = strtab + symbol.st_name;
+ auto found = order.resolve(name);
+
+ // A weak undefined symbol is allowed to stay unresolved and is written as
+ // zero, which is what lets a module test for an optional dependency.
+ ElfW(Addr) value = 0;
+ if (found) {
+ value = *found;
+ } else if (symbol_binding(symbol.st_info) != STB_WEAK) {
+ ++result.unresolved;
+ continue;
+ }
+
+ auto *where =
+ reinterpret_cast<ElfW(Addr) *>(target.load_bias() + rela->r_offset);
+ // Only the absolute form carries an addend that matters here; the GOT and
+ // PLT slots take the symbol value on its own.
+ *where = (type == RELOC_ABSOLUTE)
+ ? value + static_cast<ElfW(Addr)>(rela->r_addend)
+ : value;
+ ++result.bound;
+ }
+ return result;
+}
+
+// Binds both a module's ordinary and PLT relocations.
+LIBC_INLINE BindResult bind_module(const Module &target,
+ const SearchOrder &order) {
+ BindResult a = bind_relocations(target, target.relocations(), order);
+ BindResult b = bind_relocations(target, target.plt_relocations(), order);
+ return BindResult{a.bound + b.bound, a.unresolved + b.unresolved};
+}
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_BIND_H
diff --git a/libc/test/src/__support/elf/CMakeLists.txt b/libc/test/src/__support/elf/CMakeLists.txt
index 18d0a0d9b97474..5ad20060b7812e 100644
--- a/libc/test/src/__support/elf/CMakeLists.txt
+++ b/libc/test/src/__support/elf/CMakeLists.txt
@@ -1,5 +1,8 @@
add_libc_testsuite(libc-elf-tests)
+# map_segments_test writes a backing file to map.
+file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/testdata)
+
add_libc_test(
dynamic_test
SUITE
@@ -97,3 +100,19 @@ if(LIBC_ENABLE_SHARED)
libc.src.__support.elf.load_module
)
endif()
+
+if(LIBC_ENABLE_SHARED)
+ add_libc_test(
+ bind_test
+ SUITE
+ libc-support-tests
+ SRCS
+ bind_test.cpp
+ COMPILE_OPTIONS
+ -DLIBC_TEST_SHARED_OBJECT="$<TARGET_FILE:libc-shared>"
+ -DLIBM_TEST_SHARED_OBJECT="$<TARGET_FILE:libm-shared>"
+ DEPENDS
+ libc.src.__support.elf.bind
+ libc.src.__support.elf.load_module
+ )
+endif()
diff --git a/libc/test/src/__support/elf/bind_test.cpp b/libc/test/src/__support/elf/bind_test.cpp
new file mode 100644
index 00000000000000..0e70fc3c03ea4a
--- /dev/null
+++ b/libc/test/src/__support/elf/bind_test.cpp
@@ -0,0 +1,93 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Unittests for cross module symbol binding
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/elf/bind.h"
+#include "src/__support/elf/load_module.h"
+#include "test/UnitTest/Test.h"
+
+using LIBC_NAMESPACE::elf::bind_module;
+using LIBC_NAMESPACE::elf::BindResult;
+using LIBC_NAMESPACE::elf::load_module;
+using LIBC_NAMESPACE::elf::LoadedModule;
+using LIBC_NAMESPACE::elf::Module;
+using LIBC_NAMESPACE::elf::SearchOrder;
+using LIBC_NAMESPACE::elf::unmap_module;
+
+namespace {
+constexpr size_t PAGE = 4096;
+constexpr const char *LIBC_PATH = LIBC_TEST_SHARED_OBJECT;
+constexpr const char *LIBM_PATH = LIBM_TEST_SHARED_OBJECT;
+} // anonymous namespace
+
+TEST(LlvmLibcElfBindTest, ResolvesAcrossModules) {
+ auto libc = load_module(LIBC_PATH, PAGE);
+ ASSERT_TRUE(libc.has_value());
+ LoadedModule c = libc.value();
+
+ Module modules[] = {c.module};
+ SearchOrder order(modules, 1);
+
+ // Defined by libc.so, so the search order must find it at a real address.
+ auto printf_addr = order.resolve("printf");
+ ASSERT_TRUE(printf_addr.has_value());
+ EXPECT_GT(*printf_addr, ElfW(Addr)(0));
+ // Inside the mapping it was loaded into.
+ EXPECT_GE(*printf_addr, ElfW(Addr)(c.mapping.load_bias));
+
+ EXPECT_FALSE(order.resolve("definitely_not_a_libc_symbol").has_value());
+
+ unmap_module(c.mapping);
+}
+
+TEST(LlvmLibcElfBindTest, BindsLibmAgainstLibc) {
+ // libm.so records a DT_NEEDED on libc.so, so this is a real two module
+ // case rather than a synthetic one.
+ auto libc = load_module(LIBC_PATH, PAGE);
+ ASSERT_TRUE(libc.has_value());
+ auto libm = load_module(LIBM_PATH, PAGE);
+ ASSERT_TRUE(libm.has_value());
+
+ LoadedModule c = libc.value();
+ LoadedModule m = libm.value();
+
+ // libm names libc among its dependencies.
+ bool needs_libc = false;
+ m.module.for_each_needed([&needs_libc](const char *name) {
+ const char *p = name;
+ // Match the "libc.so" prefix without pulling in a string routine.
+ const char *want = "libc.so";
+ while (*want != '\0' && *p == *want) {
+ ++p;
+ ++want;
+ }
+ if (*want == '\0')
+ needs_libc = true;
+ });
+ EXPECT_TRUE(needs_libc);
+
+ // Search order is the target first, then its dependency, as a loader does.
+ Module modules[] = {m.module, c.module};
+ SearchOrder order(modules, 2);
+
+ BindResult result = bind_module(m.module, order);
+ // Every strong symbol libm imports must have been found.
+ EXPECT_EQ(result.unresolved, size_t(0));
+
+ unmap_module(m.mapping);
+ unmap_module(c.mapping);
+}
+
+TEST(LlvmLibcElfBindTest, EmptySearchOrderResolvesNothing) {
+ SearchOrder order(nullptr, 0);
+ EXPECT_FALSE(order.resolve("printf").has_value());
+}
>From 231abcd4b3bf0bb8adacee0be0b1eb0e6a14f7d7 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 08:13:47 +0100
Subject: [PATCH 32/46] [libc] Allocate and initialise a thread's static TLS
block
Allocate one mapping sized for every module with a PT_TLS segment, copy
each module's .tdata into it, and leave the .tbss remainder zero as
fresh anonymous memory already is.
Without this a shared libc is unusable: libc.so carries a PT_TLS segment
of its own, but crt1.o sizes the thread block from the main executable's
alone, so libc.so's thread local accesses land outside the allocation.
The caller can ask for each module's offset from the thread pointer,
which a DTV needs; recomputing them elsewhere would duplicate the layout
rules.
Two cases the tests cover: under variant 2 an empty layout still reports
the control block size, so the module count rather than the total
decides whether anything needs allocating; and a module's block sits at
its rounded up size rather than its raw p_memsz.
Assisted-by: Claude
---
libc/src/__support/elf/CMakeLists.txt | 22 +++
libc/src/__support/elf/tls_block.h | 130 ++++++++++++++++++
libc/test/src/__support/elf/CMakeLists.txt | 15 ++
.../test/src/__support/elf/tls_block_test.cpp | 113 +++++++++++++++
4 files changed, 280 insertions(+)
create mode 100644 libc/src/__support/elf/tls_block.h
create mode 100644 libc/test/src/__support/elf/tls_block_test.cpp
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
index df25b16179b370..37fc54059ca6b9 100644
--- a/libc/src/__support/elf/CMakeLists.txt
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -155,3 +155,25 @@ add_header_library(
libc.src.__support.macros.attributes
libc.src.__support.macros.config
)
+
+add_header_library(
+ tls_block
+ HDRS
+ tls_block.h
+ DEPENDS
+ libc.hdr.elf_macros
+ libc.hdr.elf_proxy
+ libc.hdr.errno_macros
+ libc.hdr.link_macros
+ libc.hdr.stdint_proxy
+ libc.hdr.sys_mman_macros
+ libc.hdr.types.size_t
+ libc.src.__support.OSUtil.linux.syscall_wrappers.mmap
+ libc.src.__support.OSUtil.linux.syscall_wrappers.munmap
+ libc.src.__support.elf.module
+ libc.src.__support.elf.tls_layout
+ libc.src.__support.error_or
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+ libc.src.string.memory_utils.inline_memcpy
+)
diff --git a/libc/src/__support/elf/tls_block.h b/libc/src/__support/elf/tls_block.h
new file mode 100644
index 00000000000000..c6ab3ac1c0447e
--- /dev/null
+++ b/libc/src/__support/elf/tls_block.h
@@ -0,0 +1,130 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Allocating a thread's static TLS block
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_TLS_BLOCK_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_TLS_BLOCK_H
+
+#include "hdr/elf_macros.h"
+#include "hdr/elf_proxy.h"
+#include "hdr/errno_macros.h"
+#include "hdr/link_macros.h"
+#include "hdr/stdint_proxy.h"
+#include "hdr/sys_mman_macros.h"
+#include "hdr/types/size_t.h"
+#include "src/__support/OSUtil/linux/syscall_wrappers/mmap.h"
+#include "src/__support/OSUtil/linux/syscall_wrappers/munmap.h"
+#include "src/__support/elf/module.h"
+#include "src/__support/elf/tls_layout.h"
+#include "src/__support/error_or.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+#include "src/string/memory_utils/inline_memcpy.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// A thread's static TLS, with the thread pointer the ABI expects.
+struct TlsBlock {
+ void *storage = nullptr;
+ size_t size = 0;
+ uintptr_t thread_pointer = 0;
+};
+
+// Allocates and initialises the static TLS for one thread, covering every
+// module in the list that has a PT_TLS segment.
+//
+// Sizing this from the main executable alone is the bug that makes a shared
+// libc unusable: libc.so carries a PT_TLS segment of its own, and its thread
+// local accesses land outside a block that did not account for it.
+//
+// Each module's .tdata is copied in and the .tbss remainder is left zero,
+// which mmap already guarantees for fresh anonymous memory.
+// `offsets`, when given, receives each module's offset from the thread
+// pointer, indexed the same as `modules`. A module without TLS gets zero. A
+// caller building a DTV needs these, and recomputing them would mean keeping
+// the layout rules in two places.
+LIBC_INLINE ErrorOr<TlsBlock> allocate_tls_block(const Module *modules,
+ size_t count,
+ intptr_t *offsets = nullptr) {
+ TlsLayout layout;
+ for (size_t i = 0; i < count; ++i) {
+ const ElfW(Phdr) *tls = modules[i].tls();
+ if (tls != nullptr)
+ layout.add(tls->p_memsz, tls->p_align);
+ }
+
+ // Under variant 2 an empty layout still reports the control block size, so
+ // the module count is what says whether anything needs allocating.
+ if (layout.module_count() == 0) {
+ for (size_t i = 0; i < count; ++i)
+ if (offsets != nullptr)
+ offsets[i] = 0;
+ return TlsBlock{};
+ }
+ // Each module's offset is a multiple of its own alignment, which only
+ // places it correctly if the thread pointer is a multiple of the strictest
+ // one. Under variant 2 the pointer sits at the top, so the blocks below it
+ // are padded until it does; the mapping itself is page aligned.
+ const size_t below_tp =
+ TLS_VARIANT_2
+ ? round_up(layout.size() - TLS_TCB_SIZE, layout.alignment())
+ : 0;
+ const size_t total = TLS_VARIANT_2 ? below_tp + TLS_TCB_SIZE : layout.size();
+
+ auto storage = linux_syscalls::mmap(nullptr, total, PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ if (!storage.has_value())
+ return Error(storage.error());
+
+ const uintptr_t base = reinterpret_cast<uintptr_t>(storage.value());
+ // Variant 2 puts the thread control block at the top with the module blocks
+ // below it; variant 1 puts it at the bottom with the modules above.
+ const uintptr_t tp = TLS_VARIANT_2 ? base + below_tp : base;
+
+ TlsLayout again;
+ for (size_t i = 0; i < count; ++i) {
+ const ElfW(Phdr) *tls = modules[i].tls();
+ if (tls == nullptr) {
+ if (offsets != nullptr)
+ offsets[i] = 0;
+ continue;
+ }
+ const intptr_t offset = again.add(tls->p_memsz, tls->p_align);
+ if (offsets != nullptr)
+ offsets[i] = offset;
+ unsigned char *dest =
+ reinterpret_cast<unsigned char *>(static_cast<intptr_t>(tp) + offset);
+ if (tls->p_filesz != 0) {
+ const void *image =
+ reinterpret_cast<const void *>(modules[i].load_bias() + tls->p_vaddr);
+ inline_memcpy(dest, image, tls->p_filesz);
+ }
+ // The rest is .tbss and is already zero from the fresh mapping.
+ }
+
+ TlsBlock block;
+ block.storage = storage.value();
+ block.size = total;
+ block.thread_pointer = tp;
+ return block;
+}
+
+LIBC_INLINE void free_tls_block(const TlsBlock &block) {
+ if (block.storage != nullptr)
+ linux_syscalls::munmap(block.storage, block.size);
+}
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_TLS_BLOCK_H
diff --git a/libc/test/src/__support/elf/CMakeLists.txt b/libc/test/src/__support/elf/CMakeLists.txt
index 5ad20060b7812e..ad1de00c8e3aeb 100644
--- a/libc/test/src/__support/elf/CMakeLists.txt
+++ b/libc/test/src/__support/elf/CMakeLists.txt
@@ -116,3 +116,18 @@ if(LIBC_ENABLE_SHARED)
libc.src.__support.elf.load_module
)
endif()
+
+if(LIBC_ENABLE_SHARED)
+ add_libc_test(
+ tls_block_test
+ SUITE
+ libc-support-tests
+ SRCS
+ tls_block_test.cpp
+ COMPILE_OPTIONS
+ -DLIBC_TEST_SHARED_OBJECT="$<TARGET_FILE:libc-shared>"
+ DEPENDS
+ libc.src.__support.elf.load_module
+ libc.src.__support.elf.tls_block
+ )
+endif()
diff --git a/libc/test/src/__support/elf/tls_block_test.cpp b/libc/test/src/__support/elf/tls_block_test.cpp
new file mode 100644
index 00000000000000..b54b0c247731c3
--- /dev/null
+++ b/libc/test/src/__support/elf/tls_block_test.cpp
@@ -0,0 +1,113 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Unittests for static TLS block allocation
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/elf/load_module.h"
+#include "src/__support/elf/tls_block.h"
+#include "test/UnitTest/Test.h"
+
+using LIBC_NAMESPACE::elf::allocate_tls_block;
+using LIBC_NAMESPACE::elf::free_tls_block;
+using LIBC_NAMESPACE::elf::load_module;
+using LIBC_NAMESPACE::elf::LoadedModule;
+using LIBC_NAMESPACE::elf::Module;
+using LIBC_NAMESPACE::elf::TLS_TCB_SIZE;
+using LIBC_NAMESPACE::elf::TLS_VARIANT_2;
+using LIBC_NAMESPACE::elf::TlsBlock;
+using LIBC_NAMESPACE::elf::unmap_module;
+
+namespace {
+constexpr size_t PAGE = 4096;
+constexpr const char *LIBC_PATH = LIBC_TEST_SHARED_OBJECT;
+} // anonymous namespace
+
+TEST(LlvmLibcElfTlsBlockTest, NoModulesMeansNoBlock) {
+ auto result = allocate_tls_block(nullptr, 0);
+ ASSERT_TRUE(result.has_value());
+ EXPECT_EQ(result.value().storage, static_cast<void *>(nullptr));
+ EXPECT_EQ(result.value().size, size_t(0));
+}
+
+TEST(LlvmLibcElfTlsBlockTest, ModuleWithoutTlsContributesNothing) {
+ Module bare;
+ auto result = allocate_tls_block(&bare, 1);
+ ASSERT_TRUE(result.has_value());
+ EXPECT_EQ(result.value().size, size_t(0));
+}
+
+TEST(LlvmLibcElfTlsBlockTest, CoversLibcOwnTlsSegment) {
+ // This is the case that motivated the whole thing: libc.so has a PT_TLS
+ // segment of its own, and a block sized from the executable alone has
+ // nowhere to put it.
+ auto loaded = load_module(LIBC_PATH, PAGE);
+ ASSERT_TRUE(loaded.has_value());
+ LoadedModule c = loaded.value();
+
+ const ElfW(Phdr) *tls = c.module.tls();
+ ASSERT_TRUE(tls != nullptr);
+ EXPECT_GT(tls->p_memsz, ElfW(Xword)(0));
+
+ Module modules[] = {c.module};
+ auto result = allocate_tls_block(modules, 1);
+ ASSERT_TRUE(result.has_value());
+ TlsBlock block = result.value();
+
+ ASSERT_TRUE(block.storage != nullptr);
+ // The block has to hold libc's whole TLS segment plus the control block.
+ EXPECT_GE(block.size, size_t(tls->p_memsz) + TLS_TCB_SIZE);
+
+ // The thread pointer sits inside the allocation, at the top under variant 2
+ // and at the bottom under variant 1.
+ const uintptr_t base = reinterpret_cast<uintptr_t>(block.storage);
+ EXPECT_GE(block.thread_pointer, base);
+ EXPECT_LE(block.thread_pointer, base + block.size);
+ if (TLS_VARIANT_2)
+ EXPECT_EQ(block.thread_pointer, base + block.size - TLS_TCB_SIZE);
+ else
+ EXPECT_EQ(block.thread_pointer, base);
+
+ free_tls_block(block);
+ unmap_module(c.mapping);
+}
+
+TEST(LlvmLibcElfTlsBlockTest, InitialisedDataIsCopiedIn) {
+ auto loaded = load_module(LIBC_PATH, PAGE);
+ ASSERT_TRUE(loaded.has_value());
+ LoadedModule c = loaded.value();
+ const ElfW(Phdr) *tls = c.module.tls();
+ ASSERT_TRUE(tls != nullptr);
+
+ Module modules[] = {c.module};
+ intptr_t offsets[1] = {0};
+ auto result = allocate_tls_block(modules, 1, offsets);
+ ASSERT_TRUE(result.has_value());
+ TlsBlock block = result.value();
+
+ if (tls->p_filesz != 0) {
+ // The module's .tdata image must appear at the offset the allocator
+ // reported, byte for byte.
+ const unsigned char *image = reinterpret_cast<const unsigned char *>(
+ c.module.load_bias() + tls->p_vaddr);
+ const unsigned char *placed = reinterpret_cast<const unsigned char *>(
+ static_cast<intptr_t>(block.thread_pointer) + offsets[0]);
+ bool same = true;
+ for (ElfW(Xword) i = 0; i < tls->p_filesz; ++i)
+ if (placed[i] != image[i]) {
+ same = false;
+ break;
+ }
+ EXPECT_TRUE(same);
+ }
+
+ free_tls_block(block);
+ unmap_module(c.mapping);
+}
>From d967bd5c28d7d85b7bfcfd2ccfb05e765000da94 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 08:16:23 +0100
Subject: [PATCH 33/46] [libc] Add a registry of modules loaded at runtime
Track what is loaded, in load order, which is the order symbol lookup
walks. dlopen needs to know whether an object is already open, so that
opening it twice shares one mapping, and what the current search order
is.
The table is a fixed size array rather than an allocation, so the
registry is usable from the startup path where there is no allocator.
Entries carry a reference count for dlclose and a permanent flag for the
objects a process starts with, which are never unloaded.
collect() writes the search order into caller supplied storage so
building it does not allocate either.
Assisted-by: Claude
---
libc/src/__support/elf/CMakeLists.txt | 17 +++
libc/src/__support/elf/registry.h | 114 ++++++++++++++++++
libc/test/src/__support/elf/CMakeLists.txt | 15 +++
libc/test/src/__support/elf/registry_test.cpp | 95 +++++++++++++++
4 files changed, 241 insertions(+)
create mode 100644 libc/src/__support/elf/registry.h
create mode 100644 libc/test/src/__support/elf/registry_test.cpp
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
index 37fc54059ca6b9..d9ada44bd620ae 100644
--- a/libc/src/__support/elf/CMakeLists.txt
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -177,3 +177,20 @@ add_header_library(
libc.src.__support.macros.config
libc.src.string.memory_utils.inline_memcpy
)
+
+add_header_library(
+ registry
+ HDRS
+ registry.h
+ DEPENDS
+ libc.hdr.elf_macros
+ libc.hdr.elf_proxy
+ libc.hdr.link_macros
+ libc.hdr.types.size_t
+ libc.src.__support.CPP.optional
+ libc.src.__support.elf.bind
+ libc.src.__support.elf.load_module
+ libc.src.__support.elf.module
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+)
diff --git a/libc/src/__support/elf/registry.h b/libc/src/__support/elf/registry.h
new file mode 100644
index 00000000000000..bf50c958a941b2
--- /dev/null
+++ b/libc/src/__support/elf/registry.h
@@ -0,0 +1,114 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Registry of modules loaded at runtime
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_REGISTRY_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_REGISTRY_H
+
+#include "hdr/elf_macros.h"
+#include "hdr/elf_proxy.h"
+#include "hdr/link_macros.h"
+#include "hdr/types/size_t.h"
+#include "src/__support/CPP/optional.h"
+#include "src/__support/elf/bind.h"
+#include "src/__support/elf/load_module.h"
+#include "src/__support/elf/module.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// How many objects may be open at once. A fixed table keeps the registry
+// usable from contexts where the allocator is not available yet.
+constexpr size_t MAX_LOADED_MODULES = 64;
+
+struct RegistryEntry {
+ LoadedModule loaded;
+ size_t references = 0;
+ bool in_use = false;
+ // Set for objects the process was started with, which are never unloaded.
+ bool permanent = false;
+};
+
+// The set of objects currently loaded, in the order they were loaded, which
+// is the order symbol lookup walks them in.
+class Registry {
+public:
+ LIBC_INLINE constexpr Registry() = default;
+
+ LIBC_INLINE size_t count() const { return count_; }
+
+ LIBC_INLINE RegistryEntry *begin() { return entries_; }
+ LIBC_INLINE RegistryEntry *at(size_t index) {
+ return index < count_ ? &entries_[index] : nullptr;
+ }
+
+ // Finds an already loaded object by the name it was opened under, so that
+ // opening the same object twice shares one mapping.
+ LIBC_INLINE RegistryEntry *find(const char *name) {
+ for (size_t i = 0; i < count_; ++i) {
+ if (!entries_[i].in_use)
+ continue;
+ const char *have = entries_[i].loaded.module.name();
+ if (have != nullptr && equal(have, name))
+ return &entries_[i];
+ }
+ return nullptr;
+ }
+
+ LIBC_INLINE RegistryEntry *add(const LoadedModule &loaded) {
+ for (size_t i = 0; i < MAX_LOADED_MODULES; ++i) {
+ if (entries_[i].in_use)
+ continue;
+ entries_[i].loaded = loaded;
+ entries_[i].references = 1;
+ entries_[i].in_use = true;
+ entries_[i].permanent = false;
+ if (i >= count_)
+ count_ = i + 1;
+ return &entries_[i];
+ }
+ return nullptr;
+ }
+
+ LIBC_INLINE void remove(RegistryEntry *entry) {
+ entry->in_use = false;
+ entry->references = 0;
+ }
+
+ // Builds the search order over everything currently loaded. The caller
+ // supplies the storage so this stays allocation free.
+ LIBC_INLINE size_t collect(Module *out, size_t capacity) {
+ size_t written = 0;
+ for (size_t i = 0; i < count_ && written < capacity; ++i)
+ if (entries_[i].in_use)
+ out[written++] = entries_[i].loaded.module;
+ return written;
+ }
+
+private:
+ LIBC_INLINE static bool equal(const char *a, const char *b) {
+ for (; *a == *b; ++a, ++b)
+ if (*a == '\0')
+ return true;
+ return false;
+ }
+
+ RegistryEntry entries_[MAX_LOADED_MODULES] = {};
+ size_t count_ = 0;
+};
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_REGISTRY_H
diff --git a/libc/test/src/__support/elf/CMakeLists.txt b/libc/test/src/__support/elf/CMakeLists.txt
index ad1de00c8e3aeb..1d5f1a46e506c4 100644
--- a/libc/test/src/__support/elf/CMakeLists.txt
+++ b/libc/test/src/__support/elf/CMakeLists.txt
@@ -131,3 +131,18 @@ if(LIBC_ENABLE_SHARED)
libc.src.__support.elf.tls_block
)
endif()
+
+if(LIBC_ENABLE_SHARED)
+ add_libc_test(
+ registry_test
+ SUITE
+ libc-support-tests
+ SRCS
+ registry_test.cpp
+ COMPILE_OPTIONS
+ -DLIBC_TEST_SHARED_OBJECT="$<TARGET_FILE:libc-shared>"
+ -DLIBM_TEST_SHARED_OBJECT="$<TARGET_FILE:libm-shared>"
+ DEPENDS
+ libc.src.__support.elf.registry
+ )
+endif()
diff --git a/libc/test/src/__support/elf/registry_test.cpp b/libc/test/src/__support/elf/registry_test.cpp
new file mode 100644
index 00000000000000..03a4eb52c131cd
--- /dev/null
+++ b/libc/test/src/__support/elf/registry_test.cpp
@@ -0,0 +1,95 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Unittests for the loaded module registry
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/elf/registry.h"
+#include "test/UnitTest/Test.h"
+
+using LIBC_NAMESPACE::elf::load_module;
+using LIBC_NAMESPACE::elf::LoadedModule;
+using LIBC_NAMESPACE::elf::Module;
+using LIBC_NAMESPACE::elf::Registry;
+using LIBC_NAMESPACE::elf::RegistryEntry;
+using LIBC_NAMESPACE::elf::unmap_module;
+
+namespace {
+constexpr size_t PAGE = 4096;
+constexpr const char *LIBC_PATH = LIBC_TEST_SHARED_OBJECT;
+constexpr const char *LIBM_PATH = LIBM_TEST_SHARED_OBJECT;
+} // anonymous namespace
+
+TEST(LlvmLibcElfRegistryTest, StartsEmpty) {
+ Registry registry;
+ EXPECT_EQ(registry.count(), size_t(0));
+ EXPECT_TRUE(registry.find("anything") == nullptr);
+ EXPECT_TRUE(registry.at(0) == nullptr);
+}
+
+TEST(LlvmLibcElfRegistryTest, TracksLoadedObjects) {
+ Registry registry;
+ auto libc = load_module(LIBC_PATH, PAGE);
+ ASSERT_TRUE(libc.has_value());
+ auto libm = load_module(LIBM_PATH, PAGE);
+ ASSERT_TRUE(libm.has_value());
+
+ RegistryEntry *c = registry.add(libc.value());
+ RegistryEntry *m = registry.add(libm.value());
+ ASSERT_TRUE(c != nullptr);
+ ASSERT_TRUE(m != nullptr);
+ EXPECT_EQ(registry.count(), size_t(2));
+ EXPECT_EQ(c->references, size_t(1));
+
+ // Looking up by the name each was opened under finds the same entry back.
+ EXPECT_EQ(registry.find(LIBC_PATH), c);
+ EXPECT_EQ(registry.find(LIBM_PATH), m);
+ EXPECT_TRUE(registry.find("/not/loaded.so") == nullptr);
+
+ // The search order covers everything loaded, in load order.
+ Module order[4];
+ size_t written = registry.collect(order, 4);
+ EXPECT_EQ(written, size_t(2));
+
+ unmap_module(m->loaded.mapping);
+ unmap_module(c->loaded.mapping);
+}
+
+TEST(LlvmLibcElfRegistryTest, RemovedEntriesLeaveTheSearchOrder) {
+ Registry registry;
+ auto libc = load_module(LIBC_PATH, PAGE);
+ ASSERT_TRUE(libc.has_value());
+ RegistryEntry *c = registry.add(libc.value());
+ ASSERT_TRUE(c != nullptr);
+
+ Module order[2];
+ EXPECT_EQ(registry.collect(order, 2), size_t(1));
+
+ unmap_module(c->loaded.mapping);
+ registry.remove(c);
+ EXPECT_EQ(registry.collect(order, 2), size_t(0));
+ EXPECT_TRUE(registry.find(LIBC_PATH) == nullptr);
+}
+
+TEST(LlvmLibcElfRegistryTest, CollectRespectsCapacity) {
+ Registry registry;
+ auto libc = load_module(LIBC_PATH, PAGE);
+ ASSERT_TRUE(libc.has_value());
+ auto libm = load_module(LIBM_PATH, PAGE);
+ ASSERT_TRUE(libm.has_value());
+ RegistryEntry *c = registry.add(libc.value());
+ RegistryEntry *m = registry.add(libm.value());
+
+ Module order[1];
+ EXPECT_EQ(registry.collect(order, 1), size_t(1));
+
+ unmap_module(m->loaded.mapping);
+ unmap_module(c->loaded.mapping);
+}
>From 727929cd39f8525a9ef1b845f526729035e5cd4d Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 08:22:51 +0100
Subject: [PATCH 34/46] [libc] Parse the stack the kernel hands a new process
Provide a view of argc, argv, envp and the auxiliary vector, which the
kernel lays out contiguously at the stack pointer. A loader entry point
reads these before anything else is available, so this is a view rather
than a copy.
executable_from() turns the auxiliary vector into a description of the
main executable, so a loader need not read it back from disk: the kernel
has already mapped it and reports where the program headers landed.
AT_BASE distinguishes a program started through an interpreter.
A mismatched AT_PHENT is rejected rather than trusted, since walking the
table with the wrong stride reads past the end of every entry.
Assisted-by: Claude
---
libc/src/__support/elf/CMakeLists.txt | 15 +++
libc/src/__support/elf/startup_stack.h | 122 ++++++++++++++++++
libc/test/src/__support/elf/CMakeLists.txt | 10 ++
.../src/__support/elf/startup_stack_test.cpp | 112 ++++++++++++++++
4 files changed, 259 insertions(+)
create mode 100644 libc/src/__support/elf/startup_stack.h
create mode 100644 libc/test/src/__support/elf/startup_stack_test.cpp
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
index d9ada44bd620ae..b90a16e7b959d1 100644
--- a/libc/src/__support/elf/CMakeLists.txt
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -194,3 +194,18 @@ add_header_library(
libc.src.__support.macros.attributes
libc.src.__support.macros.config
)
+
+add_header_library(
+ startup_stack
+ HDRS
+ startup_stack.h
+ DEPENDS
+ libc.hdr.elf_macros
+ libc.hdr.elf_proxy
+ libc.hdr.link_macros
+ libc.hdr.stdint_proxy
+ libc.hdr.types.size_t
+ libc.src.__support.CPP.optional
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+)
diff --git a/libc/src/__support/elf/startup_stack.h b/libc/src/__support/elf/startup_stack.h
new file mode 100644
index 00000000000000..c3ab9af287f1d4
--- /dev/null
+++ b/libc/src/__support/elf/startup_stack.h
@@ -0,0 +1,122 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// The stack the kernel hands a new process
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_STARTUP_STACK_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_STARTUP_STACK_H
+
+#include "hdr/elf_macros.h"
+#include "hdr/elf_proxy.h"
+#include "hdr/link_macros.h"
+#include "hdr/stdint_proxy.h"
+#include "hdr/types/size_t.h"
+#include "src/__support/CPP/optional.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// The layout at the stack pointer when the kernel starts a process:
+//
+// argc, argv[0..argc-1], NULL, envp[0..], NULL, auxv[0..], AT_NULL
+//
+// This is the first thing a loader entry point reads, before anything else is
+// available. It is a view, not a copy: everything points into that stack.
+class StartupStack {
+public:
+ LIBC_INLINE constexpr StartupStack() = default;
+
+ LIBC_INLINE static StartupStack from(void *stack_pointer) {
+ StartupStack result;
+ auto *words = reinterpret_cast<uintptr_t *>(stack_pointer);
+ result.argc_ = words[0];
+ result.argv_ = reinterpret_cast<char **>(&words[1]);
+ // envp begins one past argv's terminating null.
+ char **cursor = result.argv_ + result.argc_;
+ result.envp_ = cursor + 1;
+ cursor = result.envp_;
+ while (*cursor != nullptr)
+ ++cursor;
+ result.auxv_ = reinterpret_cast<ElfW(auxv_t) *>(cursor + 1);
+ return result;
+ }
+
+ LIBC_INLINE constexpr uintptr_t argc() const { return argc_; }
+ LIBC_INLINE constexpr char **argv() const { return argv_; }
+ LIBC_INLINE constexpr char **envp() const { return envp_; }
+ LIBC_INLINE constexpr ElfW(auxv_t) * auxv() const { return auxv_; }
+
+ // The value of one auxiliary vector entry, if the kernel supplied it.
+ LIBC_INLINE cpp::optional<unsigned long> auxval(unsigned long type) const {
+ if (auxv_ == nullptr)
+ return cpp::nullopt;
+ for (ElfW(auxv_t) *entry = auxv_; entry->a_type != AT_NULL; ++entry)
+ if (entry->a_type == type)
+ return entry->a_un.a_val;
+ return cpp::nullopt;
+ }
+
+ LIBC_INLINE size_t envc() const {
+ size_t count = 0;
+ if (envp_ != nullptr)
+ for (char **e = envp_; *e != nullptr; ++e)
+ ++count;
+ return count;
+ }
+
+private:
+ uintptr_t argc_ = 0;
+ char **argv_ = nullptr;
+ char **envp_ = nullptr;
+ ElfW(auxv_t) *auxv_ = nullptr;
+};
+
+// Where the kernel put the main executable, taken from the auxiliary vector.
+//
+// A loader needs this to describe the executable without reading it back off
+// disk: the kernel has already mapped it and says where its program headers
+// landed.
+struct ExecutableImage {
+ const ElfW(Phdr) *phdrs = nullptr;
+ ElfW(Half) phnum = 0;
+ ElfW(Addr) entry = 0;
+ // Where the interpreter itself was mapped. Zero when the program was
+ // started directly rather than through an interpreter.
+ ElfW(Addr) interpreter_base = 0;
+};
+
+LIBC_INLINE cpp::optional<ExecutableImage>
+executable_from(const StartupStack &stack) {
+ auto phdr = stack.auxval(AT_PHDR);
+ auto phnum = stack.auxval(AT_PHNUM);
+ auto phent = stack.auxval(AT_PHENT);
+ if (!phdr || !phnum || !phent)
+ return cpp::nullopt;
+ // A mismatched entry size means the vector does not describe headers this
+ // build understands, and walking them would read past each one.
+ if (*phent != sizeof(ElfW(Phdr)))
+ return cpp::nullopt;
+
+ ExecutableImage image;
+ image.phdrs = reinterpret_cast<const ElfW(Phdr) *>(*phdr);
+ image.phnum = static_cast<ElfW(Half)>(*phnum);
+ image.entry = static_cast<ElfW(Addr)>(stack.auxval(AT_ENTRY).value_or(0));
+ image.interpreter_base =
+ static_cast<ElfW(Addr)>(stack.auxval(AT_BASE).value_or(0));
+ return image;
+}
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_STARTUP_STACK_H
diff --git a/libc/test/src/__support/elf/CMakeLists.txt b/libc/test/src/__support/elf/CMakeLists.txt
index 1d5f1a46e506c4..de1ed0bd61c33f 100644
--- a/libc/test/src/__support/elf/CMakeLists.txt
+++ b/libc/test/src/__support/elf/CMakeLists.txt
@@ -146,3 +146,13 @@ if(LIBC_ENABLE_SHARED)
libc.src.__support.elf.registry
)
endif()
+
+add_libc_test(
+ startup_stack_test
+ SUITE
+ libc-support-tests
+ SRCS
+ startup_stack_test.cpp
+ DEPENDS
+ libc.src.__support.elf.startup_stack
+)
diff --git a/libc/test/src/__support/elf/startup_stack_test.cpp b/libc/test/src/__support/elf/startup_stack_test.cpp
new file mode 100644
index 00000000000000..60b95c41348af3
--- /dev/null
+++ b/libc/test/src/__support/elf/startup_stack_test.cpp
@@ -0,0 +1,112 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Unittests for the process startup stack
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/elf/startup_stack.h"
+#include "test/UnitTest/Test.h"
+
+using LIBC_NAMESPACE::elf::executable_from;
+using LIBC_NAMESPACE::elf::StartupStack;
+
+namespace {
+
+char ARG0[] = "prog";
+char ARG1[] = "--flag";
+char ENV0[] = "PATH=/bin";
+char ENV1[] = "HOME=/root";
+
+ElfW(Phdr) FAKE_PHDRS[3];
+
+// A stack laid out exactly as the kernel builds one.
+uintptr_t *build_stack(bool with_auxv, unsigned long phent) {
+ static uintptr_t words[32];
+ size_t i = 0;
+ words[i++] = 2; // argc
+ words[i++] = reinterpret_cast<uintptr_t>(ARG0);
+ words[i++] = reinterpret_cast<uintptr_t>(ARG1);
+ words[i++] = 0; // end of argv
+ words[i++] = reinterpret_cast<uintptr_t>(ENV0);
+ words[i++] = reinterpret_cast<uintptr_t>(ENV1);
+ words[i++] = 0; // end of envp
+ if (with_auxv) {
+ words[i++] = AT_PHDR;
+ words[i++] = reinterpret_cast<uintptr_t>(FAKE_PHDRS);
+ words[i++] = AT_PHNUM;
+ words[i++] = 3;
+ words[i++] = AT_PHENT;
+ words[i++] = phent;
+ words[i++] = AT_ENTRY;
+ words[i++] = 0x401000;
+ words[i++] = AT_PAGESZ;
+ words[i++] = 4096;
+ }
+ words[i++] = AT_NULL;
+ words[i++] = 0;
+ return words;
+}
+
+} // anonymous namespace
+
+TEST(LlvmLibcElfStartupStackTest, ParsesArgvAndEnvp) {
+ StartupStack stack =
+ StartupStack::from(build_stack(true, sizeof(ElfW(Phdr))));
+ EXPECT_EQ(stack.argc(), uintptr_t(2));
+ ASSERT_TRUE(stack.argv() != nullptr);
+ EXPECT_STREQ(stack.argv()[0], "prog");
+ EXPECT_STREQ(stack.argv()[1], "--flag");
+ EXPECT_TRUE(stack.argv()[2] == nullptr);
+
+ ASSERT_TRUE(stack.envp() != nullptr);
+ EXPECT_STREQ(stack.envp()[0], "PATH=/bin");
+ EXPECT_STREQ(stack.envp()[1], "HOME=/root");
+ EXPECT_TRUE(stack.envp()[2] == nullptr);
+ EXPECT_EQ(stack.envc(), size_t(2));
+}
+
+TEST(LlvmLibcElfStartupStackTest, ReadsAuxiliaryVector) {
+ StartupStack stack =
+ StartupStack::from(build_stack(true, sizeof(ElfW(Phdr))));
+ ASSERT_TRUE(stack.auxval(AT_PAGESZ).has_value());
+ EXPECT_EQ(*stack.auxval(AT_PAGESZ), 4096UL);
+ EXPECT_EQ(*stack.auxval(AT_PHNUM), 3UL);
+ EXPECT_EQ(*stack.auxval(AT_ENTRY), 0x401000UL);
+ // Not supplied, so absent rather than zero.
+ EXPECT_FALSE(stack.auxval(AT_BASE).has_value());
+}
+
+TEST(LlvmLibcElfStartupStackTest, EmptyAuxiliaryVector) {
+ StartupStack stack =
+ StartupStack::from(build_stack(false, sizeof(ElfW(Phdr))));
+ EXPECT_EQ(stack.argc(), uintptr_t(2));
+ EXPECT_FALSE(stack.auxval(AT_PHDR).has_value());
+ EXPECT_FALSE(executable_from(stack).has_value());
+}
+
+TEST(LlvmLibcElfStartupStackTest, DescribesTheExecutable) {
+ StartupStack stack =
+ StartupStack::from(build_stack(true, sizeof(ElfW(Phdr))));
+ auto image = executable_from(stack);
+ ASSERT_TRUE(image.has_value());
+ EXPECT_EQ(image->phdrs, static_cast<const ElfW(Phdr) *>(FAKE_PHDRS));
+ EXPECT_EQ(image->phnum, ElfW(Half)(3));
+ EXPECT_EQ(image->entry, ElfW(Addr)(0x401000));
+ // AT_BASE absent means the program was not started through an interpreter.
+ EXPECT_EQ(image->interpreter_base, ElfW(Addr)(0));
+}
+
+TEST(LlvmLibcElfStartupStackTest, RejectsMismatchedPhdrSize) {
+ // A vector describing headers of a different size must be refused rather
+ // than walked, since each step would read past the end of an entry.
+ StartupStack stack =
+ StartupStack::from(build_stack(true, sizeof(ElfW(Phdr)) + 8));
+ EXPECT_FALSE(executable_from(stack).has_value());
+}
>From 53147e3e70d3b53d7be52b0e252b254377ce08a1 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 08:40:37 +0100
Subject: [PATCH 35/46] [libc] Add the ELF machine types and absolute
relocation constants
scripts/sorttable.c and scripts/recordmcount.c in the Linux kernel
switch over the machine and relocation type of whatever object they are
given, so they need the constants for every architecture the kernel
supports.
elf.h carried seven EM_ values and no absolute relocation types. Add the
machine types the kernel names, and the no-op and absolute data
relocations for x86, arm, aarch64, mips, powerpc, s390, sh, sparc and
loongarch.
Every value matches the system elf.h. Two that have no standard spelling
there are omitted rather than invented.
Assisted-by: Claude
---
libc/include/llvm-libc-macros/elf-macros.h | 29 ++++++++++++
libc/include/llvm-libc-proxy/elf_proxy.yaml | 52 +++++++++++++++++++++
2 files changed, 81 insertions(+)
diff --git a/libc/include/llvm-libc-macros/elf-macros.h b/libc/include/llvm-libc-macros/elf-macros.h
index d6322f7a9501a6..3ddcfb94d53ebe 100644
--- a/libc/include/llvm-libc-macros/elf-macros.h
+++ b/libc/include/llvm-libc-macros/elf-macros.h
@@ -65,4 +65,33 @@
#define R_RISCV_JUMP_SLOT 5
#define R_RISCV_IRELATIVE 58
+// The no-op relocation is type zero on every architecture, and the absolute
+// data relocations below are what object file tools match against.
+
+#define R_386_NONE 0
+#define R_386_32 1
+
+#define R_X86_64_NONE 0
+
+#define R_AARCH64_NONE 0
+
+#define R_ARM_NONE 0
+
+#define R_MIPS_32 2
+#define R_MIPS_64 18
+
+#define R_PPC_ADDR32 1
+#define R_PPC64_ADDR64 38
+
+#define R_390_64 22
+
+#define R_SH_DIR32 1
+
+#define R_SPARC_64 32
+
+#define R_LARCH_32 1
+#define R_LARCH_64 2
+#define R_LARCH_MARK_LA 20
+#define R_LARCH_SOP_PUSH_PLT_PCREL 29
+
#endif // LLVM_LIBC_MACROS_ELF_MACROS_H
diff --git a/libc/include/llvm-libc-proxy/elf_proxy.yaml b/libc/include/llvm-libc-proxy/elf_proxy.yaml
index 42788f3b3f071c..46307968574d71 100644
--- a/libc/include/llvm-libc-proxy/elf_proxy.yaml
+++ b/libc/include/llvm-libc-proxy/elf_proxy.yaml
@@ -75,18 +75,70 @@ macros:
macro_value: '0xffff'
- macro_name: EM_NONE
macro_value: 0
+ - macro_name: EM_M32
+ macro_value: 1
+ - macro_name: EM_SPARC
+ macro_value: 2
- macro_name: EM_386
macro_value: 3
+ - macro_name: EM_68K
+ macro_value: 4
+ - macro_name: EM_88K
+ macro_value: 5
+ - macro_name: EM_860
+ macro_value: 7
+ - macro_name: EM_MIPS
+ macro_value: 8
+ - macro_name: EM_S370
+ macro_value: 9
+ - macro_name: EM_MIPS_RS3_LE
+ macro_value: 10
+ - macro_name: EM_PARISC
+ macro_value: 15
+ - macro_name: EM_SPARC32PLUS
+ macro_value: 18
+ - macro_name: EM_PPC
+ macro_value: 20
- macro_name: EM_PPC64
macro_value: 21
+ - macro_name: EM_S390
+ macro_value: 22
+ - macro_name: EM_SPU
+ macro_value: 23
+ - macro_name: EM_SH
+ macro_value: 42
+ - macro_name: EM_SPARCV9
+ macro_value: 43
+ - macro_name: EM_IA_64
+ macro_value: 50
- macro_name: EM_ARM
macro_value: 40
- macro_name: EM_X86_64
macro_value: 62
- macro_name: EM_AARCH64
macro_value: 183
+ - macro_name: EM_MICROBLAZE
+ macro_value: 189
+ - macro_name: EM_XTENSA
+ macro_value: 94
+ - macro_name: EM_BLACKFIN
+ macro_value: 106
+ - macro_name: EM_ALTERA_NIOS2
+ macro_value: 113
+ - macro_name: EM_TI_C6000
+ macro_value: 140
+ - macro_name: EM_TILEPRO
+ macro_value: 188
+ - macro_name: EM_TILEGX
+ macro_value: 191
- macro_name: EM_RISCV
macro_value: 243
+ - macro_name: EM_BPF
+ macro_value: 247
+ - macro_name: EM_CSKY
+ macro_value: 252
+ - macro_name: EM_LOONGARCH
+ macro_value: 258
- macro_name: EV_NONE
macro_value: 0
- macro_name: EV_CURRENT
>From ca1db9bbf7f603ab2fc25dedae60760bd7f685a9 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 08:55:28 +0100
Subject: [PATCH 36/46] [libc] Complete the x86 relocation types and add those
other tools match
arch/x86/tools/relocs.c and scripts/mod/modpost.c in the Linux kernel
switch over every relocation type an input object might carry, not only
those the running build emits, so a partial set stops them compiling.
Add the complete i386 and x86_64 sets, the arm, aarch64, mips, riscv and
loongarch types those two tools name, and STT_SPARC_REGISTER.
All 131 values match the system elf.h. R_ARM_THM_CALL is the current
name for the value glibc still spells R_ARM_THM_PC22.
Assisted-by: Claude
---
libc/include/llvm-libc-macros/elf-macros.h | 137 +++++++++++++++++++--
1 file changed, 130 insertions(+), 7 deletions(-)
diff --git a/libc/include/llvm-libc-macros/elf-macros.h b/libc/include/llvm-libc-macros/elf-macros.h
index 3ddcfb94d53ebe..f08d37a359c8f7 100644
--- a/libc/include/llvm-libc-macros/elf-macros.h
+++ b/libc/include/llvm-libc-macros/elf-macros.h
@@ -39,11 +39,6 @@
// resolver at that address and stores what it returns.
// x86_64
-#define R_X86_64_64 1
-#define R_X86_64_RELATIVE 8
-#define R_X86_64_GLOB_DAT 6
-#define R_X86_64_JUMP_SLOT 7
-#define R_X86_64_IRELATIVE 37
// aarch64
#define R_AARCH64_ABS64 257
@@ -68,10 +63,94 @@
// The no-op relocation is type zero on every architecture, and the absolute
// data relocations below are what object file tools match against.
-#define R_386_NONE 0
+// x86, complete: object file tools switch over every relocation type an
+// input might carry, not just the ones this build emits.
+#define R_386_16 20
#define R_386_32 1
-
+#define R_386_32PLT 11
+#define R_386_8 22
+#define R_386_COPY 5
+#define R_386_GLOB_DAT 6
+#define R_386_GOT32 3
+#define R_386_GOT32X 43
+#define R_386_GOTOFF 9
+#define R_386_GOTPC 10
+#define R_386_IRELATIVE 42
+#define R_386_JMP_SLOT 7
+#define R_386_NONE 0
+#define R_386_NUM 44
+#define R_386_PC16 21
+#define R_386_PC32 2
+#define R_386_PC8 23
+#define R_386_PLT32 4
+#define R_386_RELATIVE 8
+#define R_386_SIZE32 38
+#define R_386_TLS_DESC 41
+#define R_386_TLS_DESC_CALL 40
+#define R_386_TLS_DTPMOD32 35
+#define R_386_TLS_DTPOFF32 36
+#define R_386_TLS_GD 18
+#define R_386_TLS_GD_32 24
+#define R_386_TLS_GD_CALL 26
+#define R_386_TLS_GD_POP 27
+#define R_386_TLS_GD_PUSH 25
+#define R_386_TLS_GOTDESC 39
+#define R_386_TLS_GOTIE 16
+#define R_386_TLS_IE 15
+#define R_386_TLS_IE_32 33
+#define R_386_TLS_LDM 19
+#define R_386_TLS_LDM_32 28
+#define R_386_TLS_LDM_CALL 30
+#define R_386_TLS_LDM_POP 31
+#define R_386_TLS_LDM_PUSH 29
+#define R_386_TLS_LDO_32 32
+#define R_386_TLS_LE 17
+#define R_386_TLS_LE_32 34
+#define R_386_TLS_TPOFF 14
+#define R_386_TLS_TPOFF32 37
+
+#define R_X86_64_16 12
+#define R_X86_64_32 10
+#define R_X86_64_32S 11
+#define R_X86_64_64 1
+#define R_X86_64_8 14
+#define R_X86_64_COPY 5
+#define R_X86_64_DTPMOD64 16
+#define R_X86_64_DTPOFF32 21
+#define R_X86_64_DTPOFF64 17
+#define R_X86_64_GLOB_DAT 6
+#define R_X86_64_GOT32 3
+#define R_X86_64_GOT64 27
+#define R_X86_64_GOTOFF64 25
+#define R_X86_64_GOTPC32 26
+#define R_X86_64_GOTPC32_TLSDESC 34
+#define R_X86_64_GOTPC64 29
+#define R_X86_64_GOTPCREL 9
+#define R_X86_64_GOTPCREL64 28
+#define R_X86_64_GOTPCRELX 41
+#define R_X86_64_GOTPLT64 30
+#define R_X86_64_GOTTPOFF 22
+#define R_X86_64_IRELATIVE 37
+#define R_X86_64_JUMP_SLOT 7
#define R_X86_64_NONE 0
+#define R_X86_64_NUM 43
+#define R_X86_64_PC16 13
+#define R_X86_64_PC32 2
+#define R_X86_64_PC64 24
+#define R_X86_64_PC8 15
+#define R_X86_64_PLT32 4
+#define R_X86_64_PLTOFF64 31
+#define R_X86_64_RELATIVE 8
+#define R_X86_64_RELATIVE64 38
+#define R_X86_64_REX_GOTPCRELX 42
+#define R_X86_64_SIZE32 32
+#define R_X86_64_SIZE64 33
+#define R_X86_64_TLSDESC 36
+#define R_X86_64_TLSDESC_CALL 35
+#define R_X86_64_TLSGD 19
+#define R_X86_64_TLSLD 20
+#define R_X86_64_TPOFF32 23
+#define R_X86_64_TPOFF64 18
#define R_AARCH64_NONE 0
@@ -94,4 +173,48 @@
#define R_LARCH_MARK_LA 20
#define R_LARCH_SOP_PUSH_PLT_PCREL 29
+// Additional relocation and symbol types that object file tools match
+// against. R_ARM_THM_CALL is the current name for the value glibc still
+// spells R_ARM_THM_PC22.
+#define R_ARM_CALL 28
+#define R_ARM_JUMP24 29
+#define R_ARM_MOVT_ABS 44
+#define R_ARM_MOVW_ABS_NC 43
+#define R_ARM_PC24 1
+#define R_ARM_REL32 3
+#define R_ARM_THM_CALL 10
+#define R_ARM_THM_JUMP19 51
+#define R_ARM_THM_JUMP24 30
+#define R_ARM_THM_MOVT_ABS 48
+#define R_ARM_THM_MOVW_ABS_NC 47
+#define R_ARM_THM_PC22 10
+
+#define R_AARCH64_CALL26 283
+
+#define R_MIPS_26 4
+#define R_MIPS_LO16 6
+
+#define R_RISCV_SUB32 39
+
+#define R_LARCH_ALIGN 102
+#define R_LARCH_RELAX 100
+#define R_LARCH_SUB32 55
+
+#define STT_SPARC_REGISTER 13
+
+// The thirty two bit absolute data relocation of each remaining
+// architecture, which is what a tool rewriting debug information matches
+// against to find a value it has to adjust.
+#define R_390_32 4
+#define R_68K_32 1
+#define R_ALPHA_REFLONG 1
+#define R_IA64_SECREL32LSB 0x65
+#define R_PARISC_DIR32 1
+#define R_PPC_UADDR32 24
+#define R_SPARC_32 3
+#define R_SPARC_UA32 23
+
+// The section a MIPS object keeps its debug information in.
+#define SHT_MIPS_DWARF 0x7000001e
+
#endif // LLVM_LIBC_MACROS_ELF_MACROS_H
>From ca2a9e92c1b49537def0fb9d448d590f718185e3 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 09:21:12 +0100
Subject: [PATCH 37/46] [libc] Add the startup dynamic linker
Build the loader as ld-llvm-libc-<arch>.so.1. It is its own interpreter
and has no dependencies, so a program linked against libc.so can run.
The loader relocates itself, reads the process stack, describes the
executable from the auxiliary vector and applies its relative
relocations, loads the dependency graph, lays out thread local storage
across all modules, binds symbols, installs the thread pointer, runs the
initialisers of the objects it loaded, and returns the entry point.
A bare SONAME is searched for under the requesting object's DT_RUNPATH
or DT_RPATH, then LD_LIBRARY_PATH, then /lib and /usr/lib. There is no
ld.so.cache.
Four defects that only a dynamic link exposes are fixed here:
- The kernel applies no relocations, so a position independent
executable needs its own relative relocations applied.
- internal::self was an inline variable, giving the executable and
libc.so separate copies. Startup set the attributes on one and exit
read the other. Declare and define it once.
- app was defined in crt1.o, which an executable does not export, so
libc.so could not see it. Move it into libc with a strong
declaration so a static link still pulls it from the archive.
- The word at the thread pointer must point at itself. Compiler
generated thread local accesses read it to find their base.
crt1 skips its own thread local setup when AT_BASE reports an
interpreter, since only the interpreter can size the block across all
modules, and no longer assumes the executable has a PT_TLS.
The images are part of a module's data and so are subject to its
relocations: a thread local initialised with the address of anything is
written by one. They were copied into the thread's blocks while laying
out the storage, which happens before binding, so what each thread got
was the value the linker left rather than the address of anything.
Anything reached through such a pointer jumped to zero. MPFR keeps its
constant caches this way, which is why it could not raise a number to a
power, and why GCC died folding one.
Only the layout has to precede binding, because a relocation against a
thread local stores an offset the layout fixes. The copying is now a step
of its own, run once the modules are bound.
The loader reported that a module could not be linked without saying
what was missing from it, which leaves the reader to find that out with
a disassembler. It now names the first symbol it could not resolve, as
every other loader does.
Assisted-by: Claude
---
libc/CMakeLists.txt | 1 +
libc/config/linux/app.h | 6 +
libc/include/llvm-libc-macros/elf-macros.h | 2 +
libc/lib/CMakeLists.txt | 17 +-
libc/loader/CMakeLists.txt | 10 +
libc/loader/linux/CMakeLists.txt | 57 ++++
libc/loader/linux/loader_main.cpp | 63 ++++
libc/loader/linux/loader_main.h | 29 ++
libc/loader/linux/x86_64/start.cpp | 55 ++++
libc/src/__support/elf/bind.h | 102 ++++++-
libc/src/__support/elf/self_relocate.h | 73 +++++
libc/src/__support/elf/startup_linker.h | 319 +++++++++++++++++++++
libc/src/__support/elf/thread_pointer.h | 56 ++++
libc/src/__support/elf/tls_block.h | 43 ++-
libc/startup/linux/CMakeLists.txt | 11 +
libc/startup/linux/app.cpp | 28 ++
libc/startup/linux/do_start.cpp | 31 +-
libc/test/src/__support/elf/bind_test.cpp | 2 +-
18 files changed, 875 insertions(+), 30 deletions(-)
create mode 100644 libc/loader/CMakeLists.txt
create mode 100644 libc/loader/linux/CMakeLists.txt
create mode 100644 libc/loader/linux/loader_main.cpp
create mode 100644 libc/loader/linux/loader_main.h
create mode 100644 libc/loader/linux/x86_64/start.cpp
create mode 100644 libc/src/__support/elf/self_relocate.h
create mode 100644 libc/src/__support/elf/startup_linker.h
create mode 100644 libc/src/__support/elf/thread_pointer.h
create mode 100644 libc/startup/linux/app.cpp
diff --git a/libc/CMakeLists.txt b/libc/CMakeLists.txt
index 4464a27817cdda..c067e2fd809069 100644
--- a/libc/CMakeLists.txt
+++ b/libc/CMakeLists.txt
@@ -619,6 +619,7 @@ if(LLVM_LIBC_FULL_BUILD AND NOT LIBC_TARGET_ARCHITECTURE_IS_SPIRV)
# The startup system can potentially depend on the library components so add
# it after the library implementation directories.
add_subdirectory(startup)
+add_subdirectory(loader)
endif()
# The lib and test directories are added at the very end as tests
diff --git a/libc/config/linux/app.h b/libc/config/linux/app.h
index 35d14f24df4617..2a03f50b747c38 100644
--- a/libc/config/linux/app.h
+++ b/libc/config/linux/app.h
@@ -62,7 +62,13 @@ struct AppProperties {
uintptr_t *env_ptr;
};
+#ifdef LIBC_COPT_SHARED_LIBRARY
+// Defined in libc rather than in crt1.o, so libc.so can reach it too. An
+// inline variable would give each of them a copy of its own.
+extern LIBC_SHARED_INTERNAL AppProperties app;
+#else
LIBC_INLINE_VAR AppProperties app;
+#endif
// The descriptor of a thread's TLS area.
struct TLSDescriptor {
diff --git a/libc/include/llvm-libc-macros/elf-macros.h b/libc/include/llvm-libc-macros/elf-macros.h
index f08d37a359c8f7..b4021ff3930bd9 100644
--- a/libc/include/llvm-libc-macros/elf-macros.h
+++ b/libc/include/llvm-libc-macros/elf-macros.h
@@ -57,6 +57,7 @@
// riscv
#define R_RISCV_64 2
#define R_RISCV_RELATIVE 3
+#define R_RISCV_TLS_TPREL64 11
#define R_RISCV_JUMP_SLOT 5
#define R_RISCV_IRELATIVE 58
@@ -153,6 +154,7 @@
#define R_X86_64_TPOFF64 18
#define R_AARCH64_NONE 0
+#define R_AARCH64_TLS_TPREL64 1030
#define R_ARM_NONE 0
diff --git a/libc/lib/CMakeLists.txt b/libc/lib/CMakeLists.txt
index bba7023b8c4e7d..8bfee65b6533bd 100644
--- a/libc/lib/CMakeLists.txt
+++ b/libc/lib/CMakeLists.txt
@@ -59,6 +59,14 @@ foreach(archive IN ZIP_LISTS
add_dependencies(${archive_1} libc-startup)
endif()
endif()
+ # With LIBC_ENABLE_SHARED the definition of `app` moves out of crt1.o and
+ # into libc, so the archive needs it too or a static link is left with the
+ # weak declaration resolving to nothing.
+ if(LIBC_ENABLE_SHARED AND archive_1 STREQUAL "libc"
+ AND TARGET libc.startup.${LIBC_TARGET_OS}.app)
+ target_sources(${archive_1} PRIVATE
+ $<TARGET_OBJECTS:libc.startup.${LIBC_TARGET_OS}.app>)
+ endif()
list(APPEND added_archive_targets ${archive_1})
if(LIBC_ENABLE_SHARED)
@@ -78,9 +86,12 @@ foreach(archive IN ZIP_LISTS
if(TARGET libc-startup)
add_dependencies(${archive_1}-shared libc-startup)
endif()
- if(archive_1 STREQUAL "libc" AND TARGET ${tls_object})
- target_sources(${archive_1}-shared
- PRIVATE $<TARGET_OBJECTS:${tls_object}>)
+ set(app_object libc.startup.${LIBC_TARGET_OS}.app)
+ if(archive_1 STREQUAL "libc")
+ if(TARGET ${app_object})
+ target_sources(${archive_1}-shared
+ PRIVATE $<TARGET_OBJECTS:${app_object}>)
+ endif()
else()
# libm and libmvec reference errno and other libc internals.
target_link_libraries(${archive_1}-shared PRIVATE libc-shared)
diff --git a/libc/loader/CMakeLists.txt b/libc/loader/CMakeLists.txt
new file mode 100644
index 00000000000000..b987abfd9851c0
--- /dev/null
+++ b/libc/loader/CMakeLists.txt
@@ -0,0 +1,10 @@
+# The startup dynamic linker. It is only built when there are shared objects
+# for it to load.
+if(NOT LIBC_ENABLE_SHARED)
+ return()
+endif()
+if(NOT EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${LIBC_TARGET_OS})
+ message(STATUS "Skipping the loader for ${LIBC_TARGET_OS}")
+ return()
+endif()
+add_subdirectory(${LIBC_TARGET_OS})
diff --git a/libc/loader/linux/CMakeLists.txt b/libc/loader/linux/CMakeLists.txt
new file mode 100644
index 00000000000000..e79a973cbef4b9
--- /dev/null
+++ b/libc/loader/linux/CMakeLists.txt
@@ -0,0 +1,57 @@
+if(NOT EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${LIBC_TARGET_ARCHITECTURE})
+ message(STATUS "Skipping the loader for ${LIBC_TARGET_ARCHITECTURE}")
+ return()
+endif()
+
+# The loader relocates itself before anything else runs, so its own code must
+# not need a relocation applied to reach a global or make a call. Hidden
+# visibility and no PLT are what keep the compiler from emitting one.
+set(LIBC_LOADER_COMPILE_OPTIONS
+ -fvisibility=hidden
+ -fno-plt
+ -fno-stack-protector
+ -fno-exceptions
+ -fno-rtti
+ -ffreestanding
+ -fno-builtin
+ -ftls-model=initial-exec
+)
+
+add_library(libc-loader-objects OBJECT
+ ${CMAKE_CURRENT_SOURCE_DIR}/loader_main.cpp
+ ${CMAKE_CURRENT_SOURCE_DIR}/${LIBC_TARGET_ARCHITECTURE}/start.cpp
+)
+target_include_directories(libc-loader-objects
+ SYSTEM PRIVATE ${LIBC_INCLUDE_DIR})
+target_include_directories(libc-loader-objects PRIVATE ${LIBC_SOURCE_DIR})
+target_compile_options(libc-loader-objects PRIVATE
+ ${LIBC_LOADER_COMPILE_OPTIONS} -DLIBC_COPT_PUBLIC_PACKAGING)
+set_target_properties(libc-loader-objects
+ PROPERTIES POSITION_INDEPENDENT_CODE ON)
+add_dependencies(libc-loader-objects libc-headers)
+
+add_library(libc-loader SHARED $<TARGET_OBJECTS:libc-loader-objects>)
+add_dependencies(libc-loader libc-loader-objects)
+set_target_properties(libc-loader
+ PROPERTIES
+ OUTPUT_NAME ld-llvm-libc-${LIBC_TARGET_ARCHITECTURE}.so.1
+ PREFIX ""
+ SUFFIX ""
+ LIBRARY_OUTPUT_DIRECTORY ${LIBC_LIBRARY_DIR})
+
+# The loader is its own interpreter and depends on nothing, so it links
+# against no library at all and its entry point is the stub rather than a
+# shared object's usual one.
+target_link_options(libc-loader PRIVATE
+ -nostdlib
+ -Wl,-e,_start
+ -Wl,--no-undefined
+ -Wl,-z,now
+)
+if(LIBC_COMPILER_RT_BUILTINS)
+ target_link_libraries(libc-loader PRIVATE ${LIBC_COMPILER_RT_BUILTINS})
+endif()
+
+install(TARGETS libc-loader
+ LIBRARY DESTINATION ${LIBC_INSTALL_LIBRARY_DIR}
+ COMPONENT libc)
diff --git a/libc/loader/linux/loader_main.cpp b/libc/loader/linux/loader_main.cpp
new file mode 100644
index 00000000000000..1edb9f3367e100
--- /dev/null
+++ b/libc/loader/linux/loader_main.cpp
@@ -0,0 +1,63 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Entry point of the startup dynamic linker
+///
+//===----------------------------------------------------------------------===//
+
+#include "loader/linux/loader_main.h"
+
+#include "src/__support/OSUtil/io.h"
+#include "src/__support/OSUtil/syscall.h"
+
+#include "src/__support/elf/startup_linker.h"
+#include "src/__support/elf/startup_stack.h"
+#include "src/__support/macros/config.h"
+#include <sys/syscall.h>
+
+namespace LIBC_NAMESPACE_DECL {
+
+namespace {
+
+// The kernel always supplies AT_PAGESZ, but a sane value costs nothing.
+constexpr size_t FALLBACK_PAGE_SIZE = 4096;
+
+// The loader deliberately depends on nothing outside itself, so this is the
+// syscall rather than libc's exit: there is no libc yet.
+[[noreturn]] void die(const char *message) {
+ write_to_stderr(message);
+ for (;;)
+ syscall_impl<long>(SYS_exit_group, 127);
+}
+
+} // anonymous namespace
+
+uintptr_t loader_link(void *stack_pointer) {
+ elf::StartupStack stack = elf::StartupStack::from(stack_pointer);
+
+ auto image = elf::executable_from(stack);
+ if (!image)
+ die("loader: the kernel did not describe the executable\n");
+
+ // A program run directly rather than through this loader has no entry for
+ // us to hand control to.
+ if (image->entry == 0)
+ die("loader: no entry point\n");
+
+ const size_t page_size =
+ static_cast<size_t>(stack.auxval(AT_PAGESZ).value_or(FALLBACK_PAGE_SIZE));
+
+ elf::StartupLinker linker(page_size, stack.envp());
+ if (!linker.link(stack, *image))
+ die("loader: linking failed\n");
+
+ return image->entry;
+}
+
+} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/loader/linux/loader_main.h b/libc/loader/linux/loader_main.h
new file mode 100644
index 00000000000000..d7f577f1d4c425
--- /dev/null
+++ b/libc/loader/linux/loader_main.h
@@ -0,0 +1,29 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Entry point of the startup dynamic linker
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_LOADER_LINUX_LOADER_MAIN_H
+#define LLVM_LIBC_LOADER_LINUX_LOADER_MAIN_H
+
+#include "hdr/stdint_proxy.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+
+// Links the process and returns the address to transfer control to. The entry
+// stub jumps there with the stack exactly as the kernel left it, which is what
+// the program's own startup code expects to find.
+uintptr_t loader_link(void *stack_pointer);
+
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_LOADER_LINUX_LOADER_MAIN_H
diff --git a/libc/loader/linux/x86_64/start.cpp b/libc/loader/linux/x86_64/start.cpp
new file mode 100644
index 00000000000000..034c364cd595fe
--- /dev/null
+++ b/libc/loader/linux/x86_64/start.cpp
@@ -0,0 +1,55 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Entry stub of the startup dynamic linker (x86_64)
+///
+//===----------------------------------------------------------------------===//
+
+#include "loader/linux/loader_main.h"
+
+#include "hdr/link_macros.h"
+#include "src/__support/elf/self_relocate.h"
+#include "src/__support/macros/config.h"
+
+// Supplied by the linker: the loader's own dynamic section and the address it
+// was actually mapped at.
+extern "C" const ElfW(Dyn) _DYNAMIC[];
+extern "C" const char __ehdr_start[];
+
+namespace LIBC_NAMESPACE_DECL {
+
+// Runs before the loader has been relocated, so it may not read a global or
+// call anything through the GOT. Both symbols it needs are resolved
+// pc-relatively by the linker, which works without any relocation applied.
+extern "C" [[gnu::used]] uintptr_t __libc_loader_start(void *stack_pointer) {
+ const ElfW(Addr) load_bias = reinterpret_cast<ElfW(Addr)>(__ehdr_start);
+ elf::self_relocate(_DYNAMIC, load_bias);
+ // Everything from here on has a usable address.
+ return loader_link(stack_pointer);
+}
+
+} // namespace LIBC_NAMESPACE_DECL
+
+// The kernel enters here with the stack pointer at argc and nothing else set
+// up. The stack must reach the program's entry point untouched, so the
+// pointer is passed along and then restored before the jump.
+__asm__(".text\n"
+ ".globl _start\n"
+ ".type _start, at function\n"
+ "_start:\n"
+ " xor %rbp, %rbp\n"
+ " mov %rsp, %rdi\n" // the stack is the only argument
+ " and $-16, %rsp\n" // the ABI wants it aligned for the call
+ " push %rdi\n"
+ " push %rdi\n" // keep the alignment
+ " call __libc_loader_start\n"
+ " pop %rdi\n"
+ " pop %rsp\n" // hand the program the original stack
+ " jmp *%rax\n" // and the entry point it returned
+ ".size _start,.-_start\n");
diff --git a/libc/src/__support/elf/bind.h b/libc/src/__support/elf/bind.h
index 0adaeabdb284e5..1d8376b261d280 100644
--- a/libc/src/__support/elf/bind.h
+++ b/libc/src/__support/elf/bind.h
@@ -34,18 +34,23 @@ namespace elf {
constexpr uint32_t RELOC_ABSOLUTE = R_X86_64_64;
constexpr uint32_t RELOC_GLOB_DAT = R_X86_64_GLOB_DAT;
constexpr uint32_t RELOC_JUMP_SLOT = R_X86_64_JUMP_SLOT;
+constexpr uint32_t RELOC_TLS_OFFSET = R_X86_64_TPOFF64;
#elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
-constexpr uint32_t RELOC_ABSOLUTE = 257; // R_AARCH64_ABS64
+constexpr uint32_t RELOC_ABSOLUTE = R_AARCH64_ABS64;
constexpr uint32_t RELOC_GLOB_DAT = R_AARCH64_GLOB_DAT;
constexpr uint32_t RELOC_JUMP_SLOT = R_AARCH64_JUMP_SLOT;
+constexpr uint32_t RELOC_TLS_OFFSET = R_AARCH64_TLS_TPREL64;
#elif defined(LIBC_TARGET_ARCH_IS_ANY_RISCV)
-constexpr uint32_t RELOC_ABSOLUTE = 2; // R_RISCV_64
+constexpr uint32_t RELOC_ABSOLUTE = R_RISCV_64;
constexpr uint32_t RELOC_GLOB_DAT = R_RISCV_JUMP_SLOT;
constexpr uint32_t RELOC_JUMP_SLOT = R_RISCV_JUMP_SLOT;
+constexpr uint32_t RELOC_TLS_OFFSET = R_RISCV_TLS_TPREL64;
#elif defined(LIBC_TARGET_ARCH_IS_ARM)
-constexpr uint32_t RELOC_ABSOLUTE = 2; // R_ARM_ABS32
+constexpr uint32_t RELOC_ABSOLUTE = R_ARM_ABS32;
constexpr uint32_t RELOC_GLOB_DAT = R_ARM_GLOB_DAT;
constexpr uint32_t RELOC_JUMP_SLOT = R_ARM_JUMP_SLOT;
+// ARM uses a different TLS model; not handled here yet.
+constexpr uint32_t RELOC_TLS_OFFSET = 0xffffffffu;
#else
#error "Symbol relocation types are not known for this architecture"
#endif
@@ -74,8 +79,15 @@ class SearchOrder {
LIBC_INLINE constexpr SearchOrder(const Module *modules, size_t count)
: modules_(modules), count_(count) {}
- // Returns the runtime address of `name`, or nothing if no module defines it.
- LIBC_INLINE cpp::optional<ElfW(Addr)> resolve(const char *name) const {
+ struct Resolved {
+ ElfW(Addr) address;
+ // Which module defined it, so a thread local can be placed relative to
+ // that module's block rather than this one's.
+ size_t module;
+ ElfW(Addr) value;
+ };
+
+ LIBC_INLINE cpp::optional<Resolved> find(const char *name) const {
for (size_t i = 0; i < count_; ++i) {
const ElfW(Sym) *symbol = modules_[i].symbols().lookup(name);
if (symbol == nullptr)
@@ -84,19 +96,39 @@ class SearchOrder {
// section, and must not satisfy the lookup.
if (symbol->st_shndx == SHN_UNDEF)
continue;
- return modules_[i].load_bias() + symbol->st_value;
+ return Resolved{modules_[i].load_bias() + symbol->st_value, i,
+ symbol->st_value};
}
return cpp::nullopt;
}
+ // Returns the runtime address of `name`, or nothing if no module defines it.
+ LIBC_INLINE cpp::optional<ElfW(Addr)> resolve(const char *name) const {
+ if (auto found = find(name))
+ return found->address;
+ return cpp::nullopt;
+ }
+
+ // Offsets of each module's thread local block from the thread pointer, as
+ // the loader laid them out. Needed to place a thread local symbol.
+ LIBC_INLINE void set_tls_offsets(const intptr_t *offsets) {
+ tls_offsets_ = offsets;
+ }
+ LIBC_INLINE const intptr_t *tls_offsets() const { return tls_offsets_; }
+
private:
const Module *modules_;
size_t count_;
+ const intptr_t *tls_offsets_ = nullptr;
};
struct BindResult {
size_t bound = 0;
size_t unresolved = 0;
+ // The first symbol that could not be found, so that whoever gives up can
+ // say which one it was. A count on its own leaves the reader to work that
+ // out with a disassembler.
+ const char *missing = nullptr;
};
// Applies one table of symbol relocations against the search order.
@@ -104,6 +136,7 @@ struct BindResult {
// A strong symbol that cannot be found is counted rather than fatal, so the
// caller decides whether to fail the load.
LIBC_INLINE BindResult bind_relocations(const Module &target,
+ size_t target_index,
const RelaTable &table,
const SearchOrder &order) {
BindResult result;
@@ -114,12 +147,52 @@ LIBC_INLINE BindResult bind_relocations(const Module &target,
for (const ElfW(Rela) *rela = table.begin(); rela != table.end(); ++rela) {
const uint32_t type = reloc_type(rela->r_info);
+ const bool is_tls = type == RELOC_TLS_OFFSET;
if (type != RELOC_ABSOLUTE && type != RELOC_GLOB_DAT &&
- type != RELOC_JUMP_SLOT)
+ type != RELOC_JUMP_SLOT && !is_tls)
continue;
- const ElfW(Sym) &symbol = symtab[reloc_symbol(rela->r_info)];
+ const uint32_t symbol_index = reloc_symbol(rela->r_info);
+ const ElfW(Sym) &symbol = symtab[symbol_index];
const char *name = strtab + symbol.st_name;
+
+ if (is_tls) {
+ // A thread local is addressed relative to the thread pointer, so what
+ // gets stored is the defining module's block offset plus the symbol's
+ // offset inside it, never an absolute address.
+ const intptr_t *offsets = order.tls_offsets();
+ if (offsets == nullptr) {
+ ++result.unresolved;
+ if (result.missing == nullptr)
+ result.missing = name;
+ continue;
+ }
+ intptr_t block = 0;
+ ElfW(Addr) value = 0;
+ if (symbol_index == 0 || symbol.st_shndx != SHN_UNDEF) {
+ // Either no symbol at all, where the addend alone gives the offset
+ // within this module's block, or one defined here. Most of a libc's
+ // own thread locals are hidden, so they are in no hash table and can
+ // only be resolved here rather than through the search order.
+ block = offsets[target_index];
+ value = symbol_index == 0 ? 0 : symbol.st_value;
+ } else if (auto defined = order.find(name)) {
+ block = offsets[defined->module];
+ value = defined->value;
+ } else {
+ ++result.unresolved;
+ if (result.missing == nullptr)
+ result.missing = name;
+ continue;
+ }
+ auto *slot =
+ reinterpret_cast<ElfW(Addr) *>(target.load_bias() + rela->r_offset);
+ *slot = static_cast<ElfW(Addr)>(block + static_cast<intptr_t>(value) +
+ rela->r_addend);
+ ++result.bound;
+ continue;
+ }
+
auto found = order.resolve(name);
// A weak undefined symbol is allowed to stay unresolved and is written as
@@ -129,6 +202,8 @@ LIBC_INLINE BindResult bind_relocations(const Module &target,
value = *found;
} else if (symbol_binding(symbol.st_info) != STB_WEAK) {
++result.unresolved;
+ if (result.missing == nullptr)
+ result.missing = name;
continue;
}
@@ -145,11 +220,14 @@ LIBC_INLINE BindResult bind_relocations(const Module &target,
}
// Binds both a module's ordinary and PLT relocations.
-LIBC_INLINE BindResult bind_module(const Module &target,
+LIBC_INLINE BindResult bind_module(const Module &target, size_t target_index,
const SearchOrder &order) {
- BindResult a = bind_relocations(target, target.relocations(), order);
- BindResult b = bind_relocations(target, target.plt_relocations(), order);
- return BindResult{a.bound + b.bound, a.unresolved + b.unresolved};
+ BindResult a =
+ bind_relocations(target, target_index, target.relocations(), order);
+ BindResult b =
+ bind_relocations(target, target_index, target.plt_relocations(), order);
+ return BindResult{a.bound + b.bound, a.unresolved + b.unresolved,
+ a.missing != nullptr ? a.missing : b.missing};
}
} // namespace elf
diff --git a/libc/src/__support/elf/self_relocate.h b/libc/src/__support/elf/self_relocate.h
new file mode 100644
index 00000000000000..f3efcd8988385c
--- /dev/null
+++ b/libc/src/__support/elf/self_relocate.h
@@ -0,0 +1,73 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Relocating the loader itself
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_SELF_RELOCATE_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_SELF_RELOCATE_H
+
+#include "hdr/elf_macros.h"
+#include "hdr/elf_proxy.h"
+#include "hdr/link_macros.h"
+#include "src/__support/elf/relocation.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// Applies the loader's own relative relocations.
+//
+// This is the first thing that runs, before anything in the loader has a
+// usable address. It must therefore not call through the GOT or the PLT, read
+// a global, or touch thread local storage: none of those work until it has
+// finished. Everything it needs is passed in, and it is marked always inline
+// so there is not even a call to make.
+//
+// `dynamic` and `load_bias` come from the entry stub, which can compute both
+// with pc-relative addressing before any relocation has been applied.
+[[gnu::always_inline]] inline void self_relocate(const ElfW(Dyn) * dynamic,
+ ElfW(Addr) load_bias) {
+ const ElfW(Rela) *rela = nullptr;
+ size_t rela_size = 0;
+
+ for (const ElfW(Dyn) *entry = dynamic; entry->d_tag != DT_NULL; ++entry) {
+ switch (entry->d_tag) {
+ case DT_RELA:
+ rela =
+ reinterpret_cast<const ElfW(Rela) *>(entry->d_un.d_ptr + load_bias);
+ break;
+ case DT_RELASZ:
+ rela_size = entry->d_un.d_val;
+ break;
+ default:
+ break;
+ }
+ }
+
+ if (rela == nullptr || rela_size == 0)
+ return;
+
+ // Written out rather than reusing RelaTable so that nothing here depends on
+ // a member function being callable before relocation.
+ const size_t count = rela_size / sizeof(ElfW(Rela));
+ for (size_t i = 0; i < count; ++i) {
+ if (reloc_type(rela[i].r_info) != RELATIVE_RELOC)
+ continue;
+ auto *target = reinterpret_cast<ElfW(Addr) *>(load_bias + rela[i].r_offset);
+ *target = load_bias + static_cast<ElfW(Addr)>(rela[i].r_addend);
+ }
+}
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_SELF_RELOCATE_H
diff --git a/libc/src/__support/elf/startup_linker.h b/libc/src/__support/elf/startup_linker.h
new file mode 100644
index 00000000000000..27059bb66e04b6
--- /dev/null
+++ b/libc/src/__support/elf/startup_linker.h
@@ -0,0 +1,319 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// The startup dynamic linker
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_STARTUP_LINKER_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_STARTUP_LINKER_H
+
+#include "hdr/elf_macros.h"
+#include "hdr/elf_proxy.h"
+#include "hdr/link_macros.h"
+#include "hdr/types/size_t.h"
+#include "src/__support/OSUtil/io.h"
+#include "src/__support/elf/bind.h"
+#include "src/__support/elf/load_module.h"
+#include "src/__support/elf/module.h"
+#include "src/__support/elf/startup_stack.h"
+#include "src/__support/elf/thread_pointer.h"
+#include "src/__support/elf/tls_block.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// How many objects one process may load at startup.
+constexpr size_t MAX_STARTUP_MODULES = 64;
+// Where a bare SONAME is looked for. There is no ld.so.cache: musl manages
+// without one and a glibc compatible cache format would buy nothing here.
+constexpr const char *DEFAULT_SEARCH_PATHS[] = {"/lib", "/usr/lib"};
+
+class StartupLinker {
+public:
+ LIBC_INLINE StartupLinker(size_t page_size, char **envp)
+ : page_size_(page_size), library_path_(find_library_path(envp)) {}
+
+ // Loads the executable's dependency graph, binds it, sets up thread local
+ // storage and runs the initialisers. Returns false with a message already
+ // written to stderr if anything failed.
+ LIBC_INLINE bool link(const StartupStack &stack,
+ const ExecutableImage &image) {
+ // The executable is already mapped; describe it where the kernel put it.
+ const ElfW(Addr) bias = executable_bias(image);
+ modules_[count_++] = Module(image.phdrs, image.phnum, bias, "");
+
+ // The kernel maps the executable but applies nothing. A position
+ // independent one still has relative relocations of its own, and without
+ // them even its init array bounds point at the wrong addresses.
+ modules_[0].relocations().apply_relative(bias);
+
+ // Breadth first, so a dependency named earlier is loaded earlier, which
+ // is the order symbol lookup then walks.
+ for (size_t i = 0; i < count_; ++i) {
+ bool failed = false;
+ const Module requester = modules_[i];
+ requester.for_each_needed([&](const char *name) {
+ if (failed || already_loaded(name))
+ return;
+ if (!load_dependency(requester, name))
+ failed = true;
+ });
+ if (failed)
+ return false;
+ }
+
+ // Thread local storage is laid out before binding, because a relocation
+ // against a thread local stores an offset from the thread pointer and
+ // that offset is not known until the layout is fixed. Only the layout
+ // though: what goes in the blocks is copied once binding is done.
+ if (!setup_tls())
+ return false;
+
+ SearchOrder order(modules_, count_);
+ order.set_tls_offsets(tls_offsets_);
+ for (size_t i = 0; i < count_; ++i) {
+ BindResult result = bind_module(modules_[i], i, order);
+ if (result.unresolved != 0) {
+ report("symbol not found while linking ");
+ report(modules_[i].name());
+ if (result.missing != nullptr) {
+ report(": ");
+ report(result.missing);
+ }
+ report("\n");
+ return false;
+ }
+ }
+
+ // The initialisation images are part of the modules' data, so they are
+ // only what they should be once the modules have been relocated. A
+ // thread local holding the address of anything is written by one.
+ copy_tls_images(modules_, count_, tls_block_.thread_pointer, tls_offsets_);
+
+ // Only once everything is bound may any of it run.
+ if (!install_thread_pointer())
+ return false;
+
+ run_initialisers();
+ return true;
+ }
+
+ LIBC_INLINE size_t module_count() const { return count_; }
+
+private:
+ // The kernel maps the executable but does not say what bias it used, so it
+ // is recovered the same way as for any other module.
+ LIBC_INLINE static ElfW(Addr) executable_bias(const ExecutableImage &image) {
+ for (ElfW(Half) i = 0; i < image.phnum; ++i)
+ if (image.phdrs[i].p_type == PT_PHDR)
+ return reinterpret_cast<ElfW(Addr)>(image.phdrs) -
+ image.phdrs[i].p_vaddr;
+ return 0;
+ }
+
+ LIBC_INLINE bool already_loaded(const char *name) const {
+ for (size_t i = 0; i < count_; ++i) {
+ const char *soname = modules_[i].soname();
+ if (soname != nullptr && equal(soname, name))
+ return true;
+ }
+ return false;
+ }
+
+ LIBC_INLINE static bool equal(const char *a, const char *b) {
+ for (; *a == *b; ++a, ++b)
+ if (*a == '\0')
+ return true;
+ return false;
+ }
+
+ // LD_LIBRARY_PATH, without the value being copied anywhere: the environment
+ // outlives the loader.
+ LIBC_INLINE static const char *find_library_path(char **envp) {
+ if (envp == nullptr)
+ return nullptr;
+ const char key[] = "LD_LIBRARY_PATH=";
+ for (char **e = envp; *e != nullptr; ++e) {
+ const char *p = *e;
+ size_t i = 0;
+ for (; key[i] != '\0' && p[i] == key[i]; ++i)
+ ;
+ if (key[i] == '\0')
+ return p + i;
+ }
+ return nullptr;
+ }
+
+ // Tries `name` under each colon separated directory in `list`.
+ LIBC_INLINE bool try_path_list(const char *list, const char *name,
+ LoadedModule &out) {
+ if (list == nullptr)
+ return false;
+ for (const char *segment = list; segment != nullptr;) {
+ const char *end = segment;
+ while (*end != '\0' && *end != ':')
+ ++end;
+ char dir[256];
+ const size_t length = static_cast<size_t>(end - segment);
+ if (length > 0 && length < sizeof(dir)) {
+ for (size_t i = 0; i < length; ++i)
+ dir[i] = segment[i];
+ dir[length] = '\0';
+ char path[256];
+ if (join(dir, name, path, sizeof(path))) {
+ auto loaded = load_module(path, page_size_);
+ if (loaded.has_value()) {
+ out = loaded.value();
+ return true;
+ }
+ }
+ }
+ segment = (*end == '\0') ? nullptr : end + 1;
+ }
+ return false;
+ }
+
+ // The search order a loader is expected to use: the requesting object's own
+ // run path, then LD_LIBRARY_PATH, then the system directories.
+ LIBC_INLINE bool load_dependency(const Module &from, const char *name) {
+ if (count_ >= MAX_STARTUP_MODULES) {
+ report("too many shared objects\n");
+ return false;
+ }
+ for (const char *p = name; *p != '\0'; ++p) {
+ if (*p == '/') {
+ auto loaded = load_module(name, page_size_);
+ if (!loaded.has_value())
+ return missing(name);
+ return remember(loaded.value());
+ }
+ }
+
+ LoadedModule loaded;
+ if (try_path_list(run_path(from), name, loaded))
+ return remember(loaded);
+ if (try_path_list(library_path_, name, loaded))
+ return remember(loaded);
+ for (const char *dir : DEFAULT_SEARCH_PATHS) {
+ char path[256];
+ if (!join(dir, name, path, sizeof(path)))
+ continue;
+ auto found = load_module(path, page_size_);
+ if (found.has_value())
+ return remember(found.value());
+ }
+ return missing(name);
+ }
+
+ // DT_RUNPATH, or the older DT_RPATH when a module still carries one.
+ LIBC_INLINE static const char *run_path(const Module &module) {
+ const char *strings = module.strtab();
+ if (strings == nullptr)
+ return nullptr;
+ if (auto offset = module.dynamic().value(DT_RUNPATH))
+ return strings + *offset;
+ if (auto offset = module.dynamic().value(DT_RPATH))
+ return strings + *offset;
+ return nullptr;
+ }
+
+ LIBC_INLINE bool remember(const LoadedModule &loaded) {
+ mappings_[count_] = loaded.mapping;
+ modules_[count_] = loaded.module;
+ ++count_;
+ return true;
+ }
+
+ LIBC_INLINE bool missing(const char *name) {
+ report("cannot open shared object ");
+ report(name);
+ report("\n");
+ return false;
+ }
+
+ LIBC_INLINE static bool join(const char *dir, const char *name, char *out,
+ size_t capacity) {
+ size_t n = 0;
+ for (const char *p = dir; *p != '\0'; ++p) {
+ if (n + 2 >= capacity)
+ return false;
+ out[n++] = *p;
+ }
+ out[n++] = '/';
+ for (const char *p = name; *p != '\0'; ++p) {
+ if (n + 1 >= capacity)
+ return false;
+ out[n++] = *p;
+ }
+ out[n] = '\0';
+ return true;
+ }
+
+ LIBC_INLINE bool setup_tls() {
+ auto block = allocate_tls_block(modules_, count_, tls_offsets_,
+ /*copy_images=*/false);
+ if (!block.has_value()) {
+ report("cannot allocate thread local storage\n");
+ return false;
+ }
+ tls_block_ = block.value();
+ return true;
+ }
+
+ LIBC_INLINE bool install_thread_pointer() {
+ // Nothing in the process uses thread local storage.
+ if (tls_block_.storage == nullptr)
+ return true;
+ if (!set_thread_pointer(tls_block_.thread_pointer)) {
+ report("cannot set the thread pointer\n");
+ return false;
+ }
+ return true;
+ }
+
+ // Dependencies are initialised before the objects that need them, so the
+ // list is walked backwards. The executable is index zero and is skipped:
+ // its own startup code runs its initialisers, and running them here as well
+ // would run them twice.
+ LIBC_INLINE void run_initialisers() {
+ for (size_t i = count_; i > 1; --i)
+ run_init_array(modules_[i - 1]);
+ }
+
+ LIBC_INLINE static void run_init_array(const Module &module) {
+ auto array = module.dynamic().address(DT_INIT_ARRAY);
+ auto size = module.dynamic().value(DT_INIT_ARRAYSZ);
+ if (!array || !size)
+ return;
+ auto **functions = reinterpret_cast<void (**)()>(*array);
+ for (size_t i = 0; i < *size / sizeof(void *); ++i)
+ if (functions[i] != nullptr)
+ functions[i]();
+ }
+
+ LIBC_INLINE static void report(const char *message) {
+ write_to_stderr(message);
+ }
+
+ size_t page_size_;
+ const char *library_path_;
+ Module modules_[MAX_STARTUP_MODULES];
+ MappedModule mappings_[MAX_STARTUP_MODULES];
+ intptr_t tls_offsets_[MAX_STARTUP_MODULES] = {};
+ TlsBlock tls_block_;
+ size_t count_ = 0;
+};
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_STARTUP_LINKER_H
diff --git a/libc/src/__support/elf/thread_pointer.h b/libc/src/__support/elf/thread_pointer.h
new file mode 100644
index 00000000000000..9734e97fe68f8b
--- /dev/null
+++ b/libc/src/__support/elf/thread_pointer.h
@@ -0,0 +1,56 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Installing the thread pointer
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_THREAD_POINTER_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_THREAD_POINTER_H
+
+#include "hdr/stdint_proxy.h"
+#include "src/__support/OSUtil/syscall.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+#include "src/__support/macros/properties/architectures.h"
+
+#if defined(LIBC_TARGET_ARCH_IS_X86_64)
+#include <asm/prctl.h>
+#include <sys/syscall.h>
+#endif
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// Points the thread register at a block prepared by the loader. Everything
+// thread local in every module is addressed relative to this, so nothing that
+// touches thread local storage may run before it.
+LIBC_INLINE bool set_thread_pointer(uintptr_t value) {
+#if defined(LIBC_TARGET_ARCH_IS_X86_64)
+ // The register is not writable from user mode, so the kernel sets it.
+ return syscall_impl(SYS_arch_prctl, ARCH_SET_FS, value) != -1;
+#elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
+#if __has_builtin(__builtin_arm_wsr64)
+ __builtin_arm_wsr64("tpidr_el0", value);
+#else
+ __asm__ __volatile__("msr tpidr_el0, %0" : : "r"(value));
+#endif
+ return true;
+#elif defined(LIBC_TARGET_ARCH_IS_ANY_RISCV)
+ __asm__ __volatile__("mv tp, %0\n\t" : : "r"(value));
+ return true;
+#else
+#error "Setting the thread pointer is not implemented for this architecture"
+#endif
+}
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_THREAD_POINTER_H
diff --git a/libc/src/__support/elf/tls_block.h b/libc/src/__support/elf/tls_block.h
index c6ab3ac1c0447e..241c9d181315a5 100644
--- a/libc/src/__support/elf/tls_block.h
+++ b/libc/src/__support/elf/tls_block.h
@@ -53,9 +53,32 @@ struct TlsBlock {
// pointer, indexed the same as `modules`. A module without TLS gets zero. A
// caller building a DTV needs these, and recomputing them would mean keeping
// the layout rules in two places.
+// Copies each module's thread local initialisation image into a block laid
+// out by allocate_tls_block.
+//
+// This is separate because the images are part of the modules' data and so
+// are subject to their relocations: a thread local initialised with the
+// address of anything is written by one. Copying before they are applied
+// takes the value the linker left, which is not the address of anything.
+LIBC_INLINE void copy_tls_images(const Module *modules, size_t count,
+ uintptr_t tp, const intptr_t *offsets) {
+ for (size_t i = 0; i < count; ++i) {
+ const ElfW(Phdr) *tls = modules[i].tls();
+ if (tls == nullptr || tls->p_filesz == 0)
+ continue;
+ auto *dest = reinterpret_cast<unsigned char *>(
+ static_cast<intptr_t>(tp) + offsets[i]);
+ const void *image =
+ reinterpret_cast<const void *>(modules[i].load_bias() + tls->p_vaddr);
+ inline_memcpy(dest, image, tls->p_filesz);
+ // The rest is .tbss and is already zero from the fresh mapping.
+ }
+}
+
LIBC_INLINE ErrorOr<TlsBlock> allocate_tls_block(const Module *modules,
size_t count,
- intptr_t *offsets = nullptr) {
+ intptr_t *offsets = nullptr,
+ bool copy_images = true) {
TlsLayout layout;
for (size_t i = 0; i < count; ++i) {
const ElfW(Phdr) *tls = modules[i].tls();
@@ -91,6 +114,12 @@ LIBC_INLINE ErrorOr<TlsBlock> allocate_tls_block(const Module *modules,
// below it; variant 1 puts it at the bottom with the modules above.
const uintptr_t tp = TLS_VARIANT_2 ? base + below_tp : base;
+ // The ABI requires the first word of the thread control block to point at
+ // the thread pointer itself. Compiler generated thread local accesses read
+ // it to find their own base, so leaving it zero makes every one of them
+ // dereference an offset from address zero.
+ *reinterpret_cast<uintptr_t *>(tp) = tp;
+
TlsLayout again;
for (size_t i = 0; i < count; ++i) {
const ElfW(Phdr) *tls = modules[i].tls();
@@ -102,13 +131,13 @@ LIBC_INLINE ErrorOr<TlsBlock> allocate_tls_block(const Module *modules,
const intptr_t offset = again.add(tls->p_memsz, tls->p_align);
if (offsets != nullptr)
offsets[i] = offset;
- unsigned char *dest =
+ if (!copy_images || tls->p_filesz == 0)
+ continue;
+ auto *dest =
reinterpret_cast<unsigned char *>(static_cast<intptr_t>(tp) + offset);
- if (tls->p_filesz != 0) {
- const void *image =
- reinterpret_cast<const void *>(modules[i].load_bias() + tls->p_vaddr);
- inline_memcpy(dest, image, tls->p_filesz);
- }
+ const void *image =
+ reinterpret_cast<const void *>(modules[i].load_bias() + tls->p_vaddr);
+ inline_memcpy(dest, image, tls->p_filesz);
// The rest is .tbss and is already zero from the fresh mapping.
}
diff --git a/libc/startup/linux/CMakeLists.txt b/libc/startup/linux/CMakeLists.txt
index df42f787c2bfdd..c3b5d44d47a2e8 100644
--- a/libc/startup/linux/CMakeLists.txt
+++ b/libc/startup/linux/CMakeLists.txt
@@ -90,6 +90,15 @@ endif()
add_subdirectory(${LIBC_TARGET_ARCHITECTURE})
+add_object_library(
+ app
+ SRCS
+ app.cpp
+ DEPENDS
+ libc.config.app_h
+ libc.src.__support.macros.config
+)
+
add_object_library(
gnu_property_section
SRCS
@@ -111,6 +120,7 @@ add_object_library(
do_start.h
irelative.h
DEPENDS
+ .app
libc.config.app_h
libc.hdr.elf_proxy
libc.hdr.link_macros
@@ -184,6 +194,7 @@ foreach(target IN LISTS startup_components)
COMPONENT libc)
endforeach()
+
# Position independent executables start from Scrt1.o. The objects of a shared
# build are position independent already, so crt1.o serves under both names.
if(LIBC_ENABLE_SHARED)
diff --git a/libc/startup/linux/app.cpp b/libc/startup/linux/app.cpp
new file mode 100644
index 00000000000000..a24ac318464c31
--- /dev/null
+++ b/libc/startup/linux/app.cpp
@@ -0,0 +1,28 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Definition of the application properties
+///
+//===----------------------------------------------------------------------===//
+
+#include "config/app.h"
+
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+
+#ifdef LIBC_COPT_SHARED_LIBRARY
+// The startup code fills this in, but libc.so reads it, and an executable
+// does not export its symbols to the objects it loads. So in a shared build
+// the definition lives here, inside libc.so, where both can reach it. The
+// static build keeps it in crt1 with the code that populates it.
+AppProperties app;
+#endif
+
+} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/startup/linux/do_start.cpp b/libc/startup/linux/do_start.cpp
index 504411971f43f1..41609884770415 100644
--- a/libc/startup/linux/do_start.cpp
+++ b/libc/startup/linux/do_start.cpp
@@ -119,6 +119,9 @@ static TLSDescriptor tls;
unsigned long hwcap = 0;
unsigned long hwcap2 = 0;
const char *execfn = nullptr;
+ // Non-zero when the program was started through an interpreter, which then
+ // owns thread local storage for the whole process.
+ unsigned long interpreter_base = 0;
auxv::Vector::initialize_unsafe(
reinterpret_cast<const auxv::Entry *>(env_end_marker + 1));
auxv::Vector auxvec;
@@ -142,6 +145,9 @@ static TLSDescriptor tls;
case AT_EXECFN:
execfn = reinterpret_cast<const char *>(aux_entry.val);
break;
+ case AT_BASE:
+ interpreter_base = aux_entry.val;
+ break;
default:
break; // TODO: Read other useful entries from the aux vector.
}
@@ -190,7 +196,10 @@ static TLSDescriptor tls;
reinterpret_cast<uintptr_t>(__rela_iplt_end))
apply_irelative_relocs(base, hwcap, hwcap2);
- if (tls_phdr) {
+ // A dynamically linked executable often has no thread local storage of its
+ // own, because what it uses lives in libc.so instead. There is then no
+ // PT_TLS here to describe.
+ if (tls_phdr != nullptr) {
app.tls.address = tls_phdr->p_vaddr + base;
app.tls.size = tls_phdr->p_memsz;
app.tls.init_size = tls_phdr->p_filesz;
@@ -199,13 +208,21 @@ static TLSDescriptor tls;
app.tls.align = 1;
}
- // This descriptor has to be static since its cleanup function cannot
- // capture the context.
- init_tls(tls);
- if (!set_thread_ptr(tls.tp))
- linux_syscalls::exit_group(1);
+ // Thread local storage has to be sized across every loaded module, which
+ // only the interpreter can see. When there is one it has already set the
+ // thread pointer, and setting it again here would replace a block sized for
+ // the whole process with one sized for the executable alone.
+ if (interpreter_base == 0) {
+ // This descriptor has to be static since its cleanup function cannot
+ // capture the context.
+ init_tls(tls);
+ if (!set_thread_ptr(tls.tp))
+ linux_syscalls::exit_group(1);
+ }
- get_tcb(tls.tp)->attrib = &main_thread_attrib;
+ // Whichever set the thread pointer, the loader or the code above, the
+ // control block is reached through it.
+ get_tcb()->attrib = &main_thread_attrib;
// We want the fini array callbacks to be run after other atexit
// callbacks are run. So, we register them before running the init
diff --git a/libc/test/src/__support/elf/bind_test.cpp b/libc/test/src/__support/elf/bind_test.cpp
index 0e70fc3c03ea4a..e0ca6e31cf071e 100644
--- a/libc/test/src/__support/elf/bind_test.cpp
+++ b/libc/test/src/__support/elf/bind_test.cpp
@@ -79,7 +79,7 @@ TEST(LlvmLibcElfBindTest, BindsLibmAgainstLibc) {
Module modules[] = {m.module, c.module};
SearchOrder order(modules, 2);
- BindResult result = bind_module(m.module, order);
+ BindResult result = bind_module(m.module, 0, order);
// Every strong symbol libm imports must have been found.
EXPECT_EQ(result.unresolved, size_t(0));
>From 44bed3cf32ce310d3f90acaf63e07d6fe923ddb5 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Fri, 4 Sep 2026 13:23:38 +0100
Subject: [PATCH 38/46] [libc] Implement dlopen, dlsym, dlclose and dlerror
The dlfcn functions were conforming no-op stubs. They now operate on the
module description the startup linker publishes.
The linker records what it loaded into storage inside libc and then
stops being involved; dlopen reads and extends that record rather than
calling back into a loader that has finished. The storage lives in libc
rather than in the loader because the loader is gone by the time
anything looks, and it is found by symbol lookup rather than by linking,
since the loader cannot link against one of the objects it loads.
dlopen returns an existing handle when the object is already loaded, so
the same object opened twice shares one mapping and one reference count.
A handle is an index rather than a pointer, so a caller cannot
dereference it into loader state.
Two cases are refused. An object with a PT_TLS segment is rejected:
giving thread local storage to threads that already exist requires a
dynamic thread vector. In a process not linked by this loader there is
no module set to add to, and dlopen reports that.
dlclose unmaps only the most recently opened object, since a handle is
an index and removing from the middle would renumber the rest. An
earlier object keeps its mapping until the process ends, which is
permitted: dlclose is not required to unload anything.
The loader's lock is recursive. dlopen runs the initialisers of what it
loaded while it holds the lock, and an initialiser is entitled to call
dlopen or dlsym itself; glibc's load lock is recursive for the same
reason.
Assisted-by: Claude
---
libc/lib/CMakeLists.txt | 6 +-
libc/src/__support/elf/CMakeLists.txt | 13 ++++
libc/src/__support/elf/module.h | 13 ++++
libc/src/__support/elf/passive_abi.h | 76 +++++++++++++++++++
libc/src/__support/elf/startup_linker.h | 25 ++++++-
libc/src/dlfcn/CMakeLists.txt | 39 ++++++++++
libc/src/dlfcn/dl_internal.cpp | 49 +++++++++++++
libc/src/dlfcn/dl_internal.h | 64 ++++++++++++++++
libc/src/dlfcn/dlclose.cpp | 42 ++++++++++-
libc/src/dlfcn/dlerror.cpp | 12 ++-
libc/src/dlfcn/dlopen.cpp | 98 ++++++++++++++++++++++++-
libc/src/dlfcn/dlsym.cpp | 37 +++++++++-
libc/startup/linux/CMakeLists.txt | 8 ++
libc/startup/linux/passive_abi.cpp | 45 ++++++++++++
14 files changed, 512 insertions(+), 15 deletions(-)
create mode 100644 libc/src/__support/elf/passive_abi.h
create mode 100644 libc/src/dlfcn/dl_internal.cpp
create mode 100644 libc/src/dlfcn/dl_internal.h
create mode 100644 libc/startup/linux/passive_abi.cpp
diff --git a/libc/lib/CMakeLists.txt b/libc/lib/CMakeLists.txt
index 8bfee65b6533bd..4ac36c84c2db71 100644
--- a/libc/lib/CMakeLists.txt
+++ b/libc/lib/CMakeLists.txt
@@ -65,7 +65,8 @@ foreach(archive IN ZIP_LISTS
if(LIBC_ENABLE_SHARED AND archive_1 STREQUAL "libc"
AND TARGET libc.startup.${LIBC_TARGET_OS}.app)
target_sources(${archive_1} PRIVATE
- $<TARGET_OBJECTS:libc.startup.${LIBC_TARGET_OS}.app>)
+ $<TARGET_OBJECTS:libc.startup.${LIBC_TARGET_OS}.app>
+ $<TARGET_OBJECTS:libc.startup.${LIBC_TARGET_OS}.passive_abi>)
endif()
list(APPEND added_archive_targets ${archive_1})
@@ -90,7 +91,8 @@ foreach(archive IN ZIP_LISTS
if(archive_1 STREQUAL "libc")
if(TARGET ${app_object})
target_sources(${archive_1}-shared
- PRIVATE $<TARGET_OBJECTS:${app_object}>)
+ PRIVATE $<TARGET_OBJECTS:${app_object}>
+ $<TARGET_OBJECTS:libc.startup.${LIBC_TARGET_OS}.passive_abi>)
endif()
else()
# libm and libmvec reference errno and other libc internals.
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
index b90a16e7b959d1..2110c6c3e1c6b3 100644
--- a/libc/src/__support/elf/CMakeLists.txt
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -209,3 +209,16 @@ add_header_library(
libc.src.__support.macros.attributes
libc.src.__support.macros.config
)
+
+add_header_library(
+ passive_abi
+ HDRS
+ passive_abi.h
+ DEPENDS
+ libc.hdr.stdint_proxy
+ libc.hdr.types.size_t
+ libc.src.__support.elf.map_segments
+ libc.src.__support.elf.module
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+)
diff --git a/libc/src/__support/elf/module.h b/libc/src/__support/elf/module.h
index ca7394ae2a707b..ac6c3ab1d4e19d 100644
--- a/libc/src/__support/elf/module.h
+++ b/libc/src/__support/elf/module.h
@@ -141,6 +141,19 @@ class Module {
DynamicTable dynamic_;
};
+// Runs a module's DT_FINI_ARRAY, in reverse of the order it was built, which
+// is what the ABI requires.
+LIBC_INLINE void run_fini_array(const Module &module) {
+ auto array = module.dynamic().address(DT_FINI_ARRAY);
+ auto size = module.dynamic().value(DT_FINI_ARRAYSZ);
+ if (!array || !size)
+ return;
+ auto **functions = reinterpret_cast<void (**)()>(*array);
+ for (size_t i = *size / sizeof(void *); i > 0; --i)
+ if (functions[i - 1] != nullptr)
+ functions[i - 1]();
+}
+
} // namespace elf
} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/src/__support/elf/passive_abi.h b/libc/src/__support/elf/passive_abi.h
new file mode 100644
index 00000000000000..4927feaccdbfc8
--- /dev/null
+++ b/libc/src/__support/elf/passive_abi.h
@@ -0,0 +1,76 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// What the loader publishes about the loaded modules
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_PASSIVE_ABI_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_PASSIVE_ABI_H
+
+#include "hdr/stdint_proxy.h"
+#include "hdr/types/size_t.h"
+#include "src/__support/elf/map_segments.h"
+#include "src/__support/elf/module.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// How many objects a process may have loaded at once, startup and dlopen
+// together.
+constexpr size_t MAX_PROCESS_MODULES = 64;
+
+// The description of the loaded module set.
+//
+// The startup linker fills this in and then stops being involved. Everything
+// afterwards, dlopen included, reads and extends it. Keeping it as plain data
+// rather than a call back into the loader is what lets the startup code
+// finish and go away: there is no interface to keep alive, only a record of
+// what was loaded.
+//
+// The storage lives in libc rather than in the loader, because the loader is
+// gone by the time anything else wants to look.
+struct ModuleSet {
+ Module *modules;
+ MappedModule *mappings;
+ intptr_t *tls_offsets;
+ // Reference counts, so dlclose only unmaps what nothing else is using.
+ // Objects loaded at startup begin at one and are never released.
+ size_t *references;
+ size_t count;
+ size_t capacity;
+ size_t page_size;
+ // Set once the startup linker has finished, so a caller can tell a linked
+ // process from one that was started some other way.
+ bool linked;
+};
+
+// Given C linkage and a stable name so the loader can find it by symbol
+// lookup. It cannot link against libc: libc is one of the things it loads,
+// and the loader has to be self contained to run before anything else.
+constexpr const char *MODULE_SET_SYMBOL = "__llvm_libc_process_modules";
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+extern "C" [[gnu::visibility("default")]]
+LIBC_NAMESPACE::elf::ModuleSet __llvm_libc_process_modules;
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// The same object, under the name the rest of libc uses.
+LIBC_INLINE ModuleSet &loaded_modules() { return __llvm_libc_process_modules; }
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_PASSIVE_ABI_H
diff --git a/libc/src/__support/elf/startup_linker.h b/libc/src/__support/elf/startup_linker.h
index 27059bb66e04b6..81c0dc400d6161 100644
--- a/libc/src/__support/elf/startup_linker.h
+++ b/libc/src/__support/elf/startup_linker.h
@@ -22,6 +22,7 @@
#include "src/__support/elf/bind.h"
#include "src/__support/elf/load_module.h"
#include "src/__support/elf/module.h"
+#include "src/__support/elf/passive_abi.h"
#include "src/__support/elf/startup_stack.h"
#include "src/__support/elf/thread_pointer.h"
#include "src/__support/elf/tls_block.h"
@@ -32,7 +33,7 @@ namespace LIBC_NAMESPACE_DECL {
namespace elf {
// How many objects one process may load at startup.
-constexpr size_t MAX_STARTUP_MODULES = 64;
+constexpr size_t MAX_STARTUP_MODULES = MAX_PROCESS_MODULES;
// Where a bare SONAME is looked for. There is no ld.so.cache: musl manages
// without one and a glibc compatible cache format would buy nothing here.
constexpr const char *DEFAULT_SEARCH_PATHS[] = {"/lib", "/usr/lib"};
@@ -104,6 +105,8 @@ class StartupLinker {
return false;
run_initialisers();
+
+ publish(order);
return true;
}
@@ -304,6 +307,26 @@ class StartupLinker {
write_to_stderr(message);
}
+ // Copies what was loaded into libc's storage, found by symbol lookup since
+ // the loader cannot link against the thing it loads. From here the loader
+ // is finished and anything that wants to know reads that instead.
+ LIBC_INLINE void publish(const SearchOrder &order) {
+ auto address = order.resolve(MODULE_SET_SYMBOL);
+ if (!address)
+ return; // Nothing in the process wants to know.
+ auto *set = reinterpret_cast<ModuleSet *>(*address);
+ const size_t n = count_ < set->capacity ? count_ : set->capacity;
+ for (size_t i = 0; i < n; ++i) {
+ set->modules[i] = modules_[i];
+ set->mappings[i] = mappings_[i];
+ set->tls_offsets[i] = tls_offsets_[i];
+ set->references[i] = 1;
+ }
+ set->count = n;
+ set->page_size = page_size_;
+ set->linked = true;
+ }
+
size_t page_size_;
const char *library_path_;
Module modules_[MAX_STARTUP_MODULES];
diff --git a/libc/src/dlfcn/CMakeLists.txt b/libc/src/dlfcn/CMakeLists.txt
index 876d915c84064a..f97f5905947349 100644
--- a/libc/src/dlfcn/CMakeLists.txt
+++ b/libc/src/dlfcn/CMakeLists.txt
@@ -1,9 +1,28 @@
+add_object_library(
+ dl_internal
+ SRCS
+ dl_internal.cpp
+ HDRS
+ dl_internal.h
+ DEPENDS
+ libc.hdr.types.size_t
+ libc.src.__support.elf.passive_abi
+ libc.src.__support.threads.mutex
+)
+
add_entrypoint_object(
dlclose
SRCS
dlclose.cpp
HDRS
dlclose.h
+ DEPENDS
+ libc.include.dlfcn
+ libc.src.__support.CPP.mutex
+ libc.src.__support.elf.map_segments
+ libc.src.__support.elf.passive_abi
+ libc.src.__support.threads.mutex
+ libc.src.dlfcn.dl_internal
)
add_entrypoint_object(
@@ -13,6 +32,15 @@ add_entrypoint_object(
HDRS
dlerror.h
DEPENDS
+ libc.src.__support.CPP.mutex
+ libc.src.__support.elf.passive_abi
+ libc.src.__support.threads.mutex
+ libc.src.dlfcn.dl_internal
+ libc.src.__support.CPP.mutex
+ libc.src.__support.elf.passive_abi
+ libc.src.__support.threads.mutex
+ libc.src.dlfcn.dl_internal
+ libc.src.__support.elf.map_segments
libc.include.dlfcn
)
@@ -23,6 +51,12 @@ add_entrypoint_object(
HDRS
dlopen.h
DEPENDS
+ libc.src.__support.CPP.mutex
+ libc.src.__support.elf.passive_abi
+ libc.src.__support.threads.mutex
+ libc.src.dlfcn.dl_internal
+ libc.src.__support.elf.bind
+ libc.src.__support.elf.load_module
libc.include.dlfcn
)
@@ -33,6 +67,11 @@ add_entrypoint_object(
HDRS
dlsym.h
DEPENDS
+ libc.src.__support.CPP.mutex
+ libc.src.__support.elf.passive_abi
+ libc.src.__support.threads.mutex
+ libc.src.dlfcn.dl_internal
+ libc.src.__support.elf.bind
libc.include.dlfcn
)
diff --git a/libc/src/dlfcn/dl_internal.cpp b/libc/src/dlfcn/dl_internal.cpp
new file mode 100644
index 00000000000000..8ed98491a18a27
--- /dev/null
+++ b/libc/src/dlfcn/dl_internal.cpp
@@ -0,0 +1,49 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Shared state of the dlfcn functions
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/dlfcn/dl_internal.h"
+
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace dl {
+
+// Recursive, because the initialisers of what was just loaded run while it
+// is held, and an initialiser is entitled to call back into the loader:
+// dlopen for something it needs, or dlsym for something it was given. A
+// plain mutex leaves the one thread waiting on itself, which is what
+// systemd did as PID 1, with nothing left to wake it. glibc's load lock is
+// recursive for the same reason.
+Mutex dl_mutex(false, /*is_recursive=*/true, false, false);
+Mutex dl_write_lock(false, /*is_recursive=*/true, false, false);
+const char *dl_last_error = nullptr;
+
+namespace {
+char dl_error_text[4352];
+} // anonymous namespace
+
+void set_error(const char *subject, const char *message) {
+ size_t used = 0;
+ auto append = [&used](const char *text) {
+ for (; *text != '\0' && used + 1 < sizeof(dl_error_text); ++text)
+ dl_error_text[used++] = *text;
+ };
+ append(subject);
+ append(": ");
+ append(message);
+ dl_error_text[used] = '\0';
+ dl_last_error = dl_error_text;
+}
+
+} // namespace dl
+} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/src/dlfcn/dl_internal.h b/libc/src/dlfcn/dl_internal.h
new file mode 100644
index 00000000000000..c41a418595caee
--- /dev/null
+++ b/libc/src/dlfcn/dl_internal.h
@@ -0,0 +1,64 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Shared state of the dlfcn functions
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC_DLFCN_DL_INTERNAL_H
+#define LLVM_LIBC_SRC_DLFCN_DL_INTERNAL_H
+
+#include "hdr/types/size_t.h"
+#include "src/__support/elf/passive_abi.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+#include "src/__support/threads/mutex.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace dl {
+
+// A handle is the module's index plus one, so that it is never null and
+// carries no pointer the caller could dereference.
+LIBC_INLINE void *handle_for(size_t index) {
+ return reinterpret_cast<void *>(index + 1);
+}
+
+// Returns false when the handle did not come from dlopen.
+LIBC_INLINE bool index_for(void *handle, size_t &index) {
+ const size_t value = reinterpret_cast<size_t>(handle);
+ if (value == 0 || value > elf::loaded_modules().count)
+ return false;
+ index = value - 1;
+ return true;
+}
+
+extern Mutex dl_mutex;
+
+// Held while the module set is being changed, and by anything that reads it
+// without holding dl_mutex. The load lock above is held across the
+// initialisers of whatever was opened, which can wait on another thread, so
+// a thread that is only walking the modules, an unwinder in
+// dl_iterate_phdr, must not wait on that one.
+extern Mutex dl_write_lock;
+
+// The message dlerror will report, or null when there is nothing to report.
+// Either a string literal or the text in dl_error_text.
+extern const char *dl_last_error;
+
+LIBC_INLINE void set_error(const char *message) { dl_last_error = message; }
+
+// A message that names what it is about, as glibc's do: "libfoo.so: cannot
+// open shared object file". The parts come from the caller, so they are
+// copied, and cut short if they do not fit.
+void set_error(const char *subject, const char *message);
+
+} // namespace dl
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC_DLFCN_DL_INTERNAL_H
diff --git a/libc/src/dlfcn/dlclose.cpp b/libc/src/dlfcn/dlclose.cpp
index e76eb74af681a8..d1672893f53dac 100644
--- a/libc/src/dlfcn/dlclose.cpp
+++ b/libc/src/dlfcn/dlclose.cpp
@@ -8,12 +8,50 @@
#include "dlclose.h"
+#include "src/__support/CPP/mutex.h"
#include "src/__support/common.h"
+#include "src/__support/elf/map_segments.h"
+#include "src/__support/elf/passive_abi.h"
#include "src/__support/macros/config.h"
+#include "src/dlfcn/dl_internal.h"
namespace LIBC_NAMESPACE_DECL {
-// TODO(@izaakschroeder): https://github.com/llvm/llvm-project/issues/97917
-LLVM_LIBC_FUNCTION(int, dlclose, (void *)) { return -1; }
+LLVM_LIBC_FUNCTION(int, dlclose, (void *handle)) {
+ elf::ModuleSet &set = elf::loaded_modules();
+ cpp::lock_guard lock(dl::dl_mutex);
+
+ size_t index = 0;
+ if (!set.linked || !dl::index_for(handle, index)) {
+ dl::set_error("dlclose was given a handle that dlopen did not return");
+ return -1;
+ }
+ if (set.references[index] == 0) {
+ dl::set_error("shared object is already closed");
+ return -1;
+ }
+
+ if (--set.references[index] != 0)
+ return 0;
+
+ // The objects the process started with are never unloaded, so their count
+ // is left where it is rather than allowed to reach zero.
+ if (index == 0) {
+ set.references[index] = 1;
+ return 0;
+ }
+
+ // Only the last module can be dropped from the set, since a handle is an
+ // index into it and removing from the middle would renumber the rest. An
+ // earlier one keeps its mapping until the process ends, which is allowed:
+ // dlclose is not required to unload anything. Nor is it finalized, since
+ // opening it again hands back the same module without initializing it.
+ if (index + 1 == set.count) {
+ elf::run_fini_array(set.modules[index]);
+ elf::unmap_module(set.mappings[index]);
+ set.count = index;
+ }
+ return 0;
+}
} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/src/dlfcn/dlerror.cpp b/libc/src/dlfcn/dlerror.cpp
index cb2f2f50e2ef6f..e91e2f8abcf4d9 100644
--- a/libc/src/dlfcn/dlerror.cpp
+++ b/libc/src/dlfcn/dlerror.cpp
@@ -1,4 +1,4 @@
-//===-- Implementation of delerror ----------------------------------------===//
+//===-- Implementation of dlerror -----------------------------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
@@ -8,14 +8,20 @@
#include "dlerror.h"
+#include "src/__support/CPP/mutex.h"
#include "src/__support/common.h"
#include "src/__support/macros/config.h"
+#include "src/dlfcn/dl_internal.h"
namespace LIBC_NAMESPACE_DECL {
-// TODO(@izaakschroeder): https://github.com/llvm/llvm-project/issues/97918
LLVM_LIBC_FUNCTION(char *, dlerror, ()) {
- return const_cast<char *>("unsupported");
+ cpp::lock_guard lock(dl::dl_mutex);
+ // Reading the error clears it, so a second call with nothing in between
+ // reports no error rather than repeating the last one.
+ const char *message = dl::dl_last_error;
+ dl::dl_last_error = nullptr;
+ return const_cast<char *>(message);
}
} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/src/dlfcn/dlopen.cpp b/libc/src/dlfcn/dlopen.cpp
index 9e8d58e347635d..7028cf588196b3 100644
--- a/libc/src/dlfcn/dlopen.cpp
+++ b/libc/src/dlfcn/dlopen.cpp
@@ -1,4 +1,4 @@
-//===-- Implementation of dlopen -----------------------------------------===//
+//===-- Implementation of dlopen ------------------------------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
@@ -8,12 +8,104 @@
#include "dlopen.h"
+#include "src/__support/CPP/mutex.h"
#include "src/__support/common.h"
+#include "src/__support/elf/bind.h"
+#include "src/__support/elf/load_module.h"
+#include "src/__support/elf/passive_abi.h"
#include "src/__support/macros/config.h"
+#include "src/dlfcn/dl_internal.h"
namespace LIBC_NAMESPACE_DECL {
-// TODO(@izaakschroeder): https://github.com/llvm/llvm-project/issues/97919
-LLVM_LIBC_FUNCTION(void *, dlopen, (const char *, int)) { return nullptr; }
+namespace {
+
+bool same_name(const char *a, const char *b) {
+ if (a == nullptr || b == nullptr)
+ return false;
+ for (; *a == *b; ++a, ++b)
+ if (*a == '\0')
+ return true;
+ return false;
+}
+
+void run_init_array(const elf::Module &module) {
+ auto array = module.dynamic().address(DT_INIT_ARRAY);
+ auto size = module.dynamic().value(DT_INIT_ARRAYSZ);
+ if (!array || !size)
+ return;
+ auto **functions = reinterpret_cast<void (**)()>(*array);
+ for (size_t i = 0; i < *size / sizeof(void *); ++i)
+ if (functions[i] != nullptr)
+ functions[i]();
+}
+
+} // anonymous namespace
+
+LLVM_LIBC_FUNCTION(void *, dlopen, (const char *path, int)) {
+ elf::ModuleSet &set = elf::loaded_modules();
+ cpp::lock_guard lock(dl::dl_mutex);
+
+ // Without a loader there is no module set to add to, and nothing the caller
+ // opened could be linked against the program.
+ if (!set.linked) {
+ dl::set_error("dlopen is only available in a dynamically linked process");
+ return nullptr;
+ }
+ // A null path asks for a handle to the main program.
+ if (path == nullptr)
+ return dl::handle_for(0);
+
+ for (size_t i = 0; i < set.count; ++i) {
+ if (same_name(set.modules[i].name(), path) ||
+ same_name(set.modules[i].soname(), path)) {
+ ++set.references[i];
+ return dl::handle_for(i);
+ }
+ }
+
+ if (set.count >= set.capacity) {
+ dl::set_error("too many shared objects are open");
+ return nullptr;
+ }
+
+ auto loaded = elf::load_module(path, set.page_size);
+ if (!loaded.has_value()) {
+ dl::set_error(path, "cannot open shared object file");
+ return nullptr;
+ }
+
+ // Thread local storage in a newly opened object would have to be added to
+ // every thread that already exists, which needs a dynamic thread vector.
+ // Refusing is better than handing back a module whose thread locals alias
+ // something else.
+ if (loaded.value().module.tls() != nullptr) {
+ elf::unmap_module(loaded.value().mapping);
+ dl::set_error("shared object uses thread local storage");
+ return nullptr;
+ }
+
+ const size_t index = set.count;
+ set.modules[index] = loaded.value().module;
+ set.mappings[index] = loaded.value().mapping;
+ set.tls_offsets[index] = 0;
+ set.references[index] = 1;
+ set.count = index + 1;
+
+ // Bind against everything now loaded, the new object included, so it can
+ // refer to itself as well as to what was already there.
+ elf::SearchOrder order(set.modules, set.count);
+ order.set_tls_offsets(set.tls_offsets);
+ elf::BindResult result = elf::bind_module(set.modules[index], index, order);
+ if (result.unresolved != 0) {
+ set.count = index;
+ elf::unmap_module(loaded.value().mapping);
+ dl::set_error("shared object has undefined symbols");
+ return nullptr;
+ }
+
+ run_init_array(set.modules[index]);
+ return dl::handle_for(index);
+}
} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/src/dlfcn/dlsym.cpp b/libc/src/dlfcn/dlsym.cpp
index dc0da7d6d120f5..74487f08dbe06c 100644
--- a/libc/src/dlfcn/dlsym.cpp
+++ b/libc/src/dlfcn/dlsym.cpp
@@ -1,4 +1,4 @@
-//===-- Implementation of dlsym ------------------------------------------===//
+//===-- Implementation of dlsym -------------------------------------------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
@@ -8,14 +8,43 @@
#include "dlsym.h"
+#include "src/__support/CPP/mutex.h"
#include "src/__support/common.h"
+#include "src/__support/elf/bind.h"
+#include "src/__support/elf/passive_abi.h"
#include "src/__support/macros/config.h"
+#include "src/dlfcn/dl_internal.h"
namespace LIBC_NAMESPACE_DECL {
-// TODO(@izaakschroeder): https://github.com/llvm/llvm-project/issues/97920
-LLVM_LIBC_FUNCTION(void *, dlsym, (void *__restrict, const char *__restrict)) {
- return nullptr;
+LLVM_LIBC_FUNCTION(void *, dlsym,
+ (void *__restrict handle, const char *__restrict name)) {
+ elf::ModuleSet &set = elf::loaded_modules();
+ cpp::lock_guard lock(dl::dl_mutex);
+
+ if (!set.linked) {
+ dl::set_error("dlsym is only available in a dynamically linked process");
+ return nullptr;
+ }
+ if (name == nullptr) {
+ dl::set_error("dlsym was given no symbol name");
+ return nullptr;
+ }
+
+ size_t index = 0;
+ if (!dl::index_for(handle, index)) {
+ dl::set_error("dlsym was given a handle that dlopen did not return");
+ return nullptr;
+ }
+
+ const elf::Module &module = set.modules[index];
+ const ElfW(Sym) *symbol = module.symbols().lookup(name);
+ // A symbol the object only references does not count as finding it.
+ if (symbol == nullptr || symbol->st_shndx == SHN_UNDEF) {
+ dl::set_error("symbol not found");
+ return nullptr;
+ }
+ return reinterpret_cast<void *>(module.load_bias() + symbol->st_value);
}
} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/startup/linux/CMakeLists.txt b/libc/startup/linux/CMakeLists.txt
index c3b5d44d47a2e8..d3f1a2d8763cd6 100644
--- a/libc/startup/linux/CMakeLists.txt
+++ b/libc/startup/linux/CMakeLists.txt
@@ -194,6 +194,14 @@ foreach(target IN LISTS startup_components)
COMPONENT libc)
endforeach()
+add_object_library(
+ passive_abi
+ SRCS
+ passive_abi.cpp
+ DEPENDS
+ libc.src.__support.elf.passive_abi
+ libc.src.__support.macros.config
+)
# Position independent executables start from Scrt1.o. The objects of a shared
# build are position independent already, so crt1.o serves under both names.
diff --git a/libc/startup/linux/passive_abi.cpp b/libc/startup/linux/passive_abi.cpp
new file mode 100644
index 00000000000000..b6786fc98da23f
--- /dev/null
+++ b/libc/startup/linux/passive_abi.cpp
@@ -0,0 +1,45 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Storage for the loaded module set
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/elf/passive_abi.h"
+
+#include "src/__support/macros/config.h"
+
+#ifdef LIBC_COPT_SHARED_LIBRARY
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+namespace {
+
+Module storage_modules[MAX_PROCESS_MODULES];
+MappedModule storage_mappings[MAX_PROCESS_MODULES];
+intptr_t storage_tls_offsets[MAX_PROCESS_MODULES];
+size_t storage_references[MAX_PROCESS_MODULES];
+
+} // anonymous namespace
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+// `linked` starts false, so a process that was never linked by this loader
+// sees an empty set rather than something that looks usable.
+LIBC_NAMESPACE::elf::ModuleSet __llvm_libc_process_modules = {
+ LIBC_NAMESPACE::elf::storage_modules,
+ LIBC_NAMESPACE::elf::storage_mappings,
+ LIBC_NAMESPACE::elf::storage_tls_offsets,
+ LIBC_NAMESPACE::elf::storage_references,
+ /*count=*/0,
+ LIBC_NAMESPACE::elf::MAX_PROCESS_MODULES,
+ /*page_size=*/4096,
+ /*linked=*/false};
+
+#endif // LIBC_COPT_SHARED_LIBRARY
>From db1b7b071288159e76277d84524fb38013575cd6 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Sat, 5 Sep 2026 09:18:01 +0100
Subject: [PATCH 39/46] [libc] Support the general dynamic TLS model
A shared library cannot know where in a thread's storage its own thread
locals will sit, so the compiler emits a module and an offset and a call
to __tls_get_addr rather than a thread pointer relative address. Nothing
implemented that, so any shared library with a thread local failed to
link.
Assisted-by: Claude
---
libc/config/linux/aarch64/entrypoints.txt | 3 ++
libc/config/linux/riscv/entrypoints.txt | 3 ++
libc/config/linux/x86_64/entrypoints.txt | 3 ++
libc/lib/CMakeLists.txt | 9 ++++
libc/src/CMakeLists.txt | 1 +
libc/src/__support/elf/CMakeLists.txt | 13 +++++
libc/src/__support/elf/bind.h | 44 +++++++++++++++-
libc/src/__support/elf/passive_abi.h | 29 ++++++++++-
libc/src/__support/elf/thread_pointer.h | 24 +++++++++
libc/src/__support/elf/tls_block.h | 15 +++++-
libc/src/__support/elf/tls_layout.h | 16 +++++-
.../threads/linux/aarch64/CMakeLists.txt | 3 ++
.../__support/threads/linux/aarch64/tls.cpp | 27 ++++++++++
.../threads/linux/riscv/CMakeLists.txt | 3 ++
.../src/__support/threads/linux/riscv/tls.cpp | 27 ++++++++++
.../threads/linux/x86_64/CMakeLists.txt | 3 ++
.../__support/threads/linux/x86_64/tls.cpp | 28 +++++++++-
libc/src/tls/CMakeLists.txt | 12 +++++
libc/src/tls/tls_get_addr.cpp | 52 +++++++++++++++++++
libc/src/tls/tls_get_addr.h | 33 ++++++++++++
.../test/src/__support/elf/tls_block_test.cpp | 29 +++++++++--
.../src/__support/elf/tls_layout_test.cpp | 3 +-
22 files changed, 369 insertions(+), 11 deletions(-)
create mode 100644 libc/src/tls/CMakeLists.txt
create mode 100644 libc/src/tls/tls_get_addr.cpp
create mode 100644 libc/src/tls/tls_get_addr.h
diff --git a/libc/config/linux/aarch64/entrypoints.txt b/libc/config/linux/aarch64/entrypoints.txt
index de0d6a589b386f..21d1dc087b0c69 100644
--- a/libc/config/linux/aarch64/entrypoints.txt
+++ b/libc/config/linux/aarch64/entrypoints.txt
@@ -83,6 +83,9 @@ set(TARGET_LIBC_ENTRYPOINTS
libc.src.sched.sched_setscheduler
libc.src.sched.sched_yield
+ # The general dynamic thread local model
+ libc.src.tls.__tls_get_addr
+
# string.h entrypoints
libc.src.string.memccpy
libc.src.string.memchr
diff --git a/libc/config/linux/riscv/entrypoints.txt b/libc/config/linux/riscv/entrypoints.txt
index cdcc29a8d47329..1ecbcb540e2625 100644
--- a/libc/config/linux/riscv/entrypoints.txt
+++ b/libc/config/linux/riscv/entrypoints.txt
@@ -108,6 +108,9 @@ set(TARGET_LIBC_ENTRYPOINTS
libc.src.sched.sched_setscheduler
libc.src.sched.sched_yield
+ # The general dynamic thread local model
+ libc.src.tls.__tls_get_addr
+
# string.h entrypoints
libc.src.string.memccpy
libc.src.string.memchr
diff --git a/libc/config/linux/x86_64/entrypoints.txt b/libc/config/linux/x86_64/entrypoints.txt
index fba943b2d267c4..e3dd35b49e30cb 100644
--- a/libc/config/linux/x86_64/entrypoints.txt
+++ b/libc/config/linux/x86_64/entrypoints.txt
@@ -108,6 +108,9 @@ set(TARGET_LIBC_ENTRYPOINTS
libc.src.sched.sched_setscheduler
libc.src.sched.sched_yield
+ # The general dynamic thread local model
+ libc.src.tls.__tls_get_addr
+
# string.h entrypoints
libc.src.string.memccpy
libc.src.string.memchr
diff --git a/libc/lib/CMakeLists.txt b/libc/lib/CMakeLists.txt
index 4ac36c84c2db71..5ec5c320e84a96 100644
--- a/libc/lib/CMakeLists.txt
+++ b/libc/lib/CMakeLists.txt
@@ -194,12 +194,20 @@ if(LLVM_LIBC_FULL_BUILD)
set(header_install_target install-libc-headers)
endif()
+# The loader is installed alongside the shared libraries it exists to load,
+# so it has to be built before the install step runs.
+set(loader_target "")
+if(TARGET libc-loader)
+ set(loader_target libc-loader)
+endif()
+
add_custom_target(install-libc
DEPENDS ${added_archive_targets}
${added_shared_targets}
${added_bitcode_targets}
${added_empty_archives}
${startup_target}
+ ${loader_target}
${header_install_target}
COMMAND "${CMAKE_COMMAND}"
-DCMAKE_INSTALL_COMPONENT=libc
@@ -210,6 +218,7 @@ add_custom_target(install-libc-stripped
${added_bitcode_targets}
${added_empty_archives}
${startup_target}
+ ${loader_target}
${header_install_target}
COMMAND "${CMAKE_COMMAND}"
-DCMAKE_INSTALL_COMPONENT=libc
diff --git a/libc/src/CMakeLists.txt b/libc/src/CMakeLists.txt
index 485259980f7643..c2ec9468e5b604 100644
--- a/libc/src/CMakeLists.txt
+++ b/libc/src/CMakeLists.txt
@@ -24,6 +24,7 @@ add_subdirectory(stdlib)
add_subdirectory(string)
add_subdirectory(strings)
add_subdirectory(time)
+add_subdirectory(tls)
add_subdirectory(wchar)
add_subdirectory(wctype)
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
index 2110c6c3e1c6b3..18887e2b031be0 100644
--- a/libc/src/__support/elf/CMakeLists.txt
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -1,3 +1,15 @@
+add_header_library(
+ thread_pointer
+ HDRS
+ thread_pointer.h
+ DEPENDS
+ libc.hdr.stdint_proxy
+ libc.include.sys_syscall
+ libc.src.__support.OSUtil.osutil
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+)
+
add_header_library(
dynamic
HDRS
@@ -168,6 +180,7 @@ add_header_library(
libc.hdr.stdint_proxy
libc.hdr.sys_mman_macros
libc.hdr.types.size_t
+ libc.src.__support.OSUtil.linux.syscall_wrappers.getrandom
libc.src.__support.OSUtil.linux.syscall_wrappers.mmap
libc.src.__support.OSUtil.linux.syscall_wrappers.munmap
libc.src.__support.elf.module
diff --git a/libc/src/__support/elf/bind.h b/libc/src/__support/elf/bind.h
index 1d8376b261d280..dc490372e4d197 100644
--- a/libc/src/__support/elf/bind.h
+++ b/libc/src/__support/elf/bind.h
@@ -35,22 +35,33 @@ constexpr uint32_t RELOC_ABSOLUTE = R_X86_64_64;
constexpr uint32_t RELOC_GLOB_DAT = R_X86_64_GLOB_DAT;
constexpr uint32_t RELOC_JUMP_SLOT = R_X86_64_JUMP_SLOT;
constexpr uint32_t RELOC_TLS_OFFSET = R_X86_64_TPOFF64;
+// The general dynamic model, which a shared library uses because it does not
+// know at link time where in the thread's blocks its own will sit. The pair
+// is read at run time by __tls_get_addr.
+constexpr uint32_t RELOC_TLS_MODULE = R_X86_64_DTPMOD64;
+constexpr uint32_t RELOC_TLS_MODULE_OFFSET = R_X86_64_DTPOFF64;
#elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
constexpr uint32_t RELOC_ABSOLUTE = R_AARCH64_ABS64;
constexpr uint32_t RELOC_GLOB_DAT = R_AARCH64_GLOB_DAT;
constexpr uint32_t RELOC_JUMP_SLOT = R_AARCH64_JUMP_SLOT;
constexpr uint32_t RELOC_TLS_OFFSET = R_AARCH64_TLS_TPREL64;
+constexpr uint32_t RELOC_TLS_MODULE = R_AARCH64_TLS_DTPMOD;
+constexpr uint32_t RELOC_TLS_MODULE_OFFSET = R_AARCH64_TLS_DTPREL;
#elif defined(LIBC_TARGET_ARCH_IS_ANY_RISCV)
constexpr uint32_t RELOC_ABSOLUTE = R_RISCV_64;
constexpr uint32_t RELOC_GLOB_DAT = R_RISCV_JUMP_SLOT;
constexpr uint32_t RELOC_JUMP_SLOT = R_RISCV_JUMP_SLOT;
constexpr uint32_t RELOC_TLS_OFFSET = R_RISCV_TLS_TPREL64;
+constexpr uint32_t RELOC_TLS_MODULE = R_RISCV_TLS_DTPMOD64;
+constexpr uint32_t RELOC_TLS_MODULE_OFFSET = R_RISCV_TLS_DTPREL64;
#elif defined(LIBC_TARGET_ARCH_IS_ARM)
constexpr uint32_t RELOC_ABSOLUTE = R_ARM_ABS32;
constexpr uint32_t RELOC_GLOB_DAT = R_ARM_GLOB_DAT;
constexpr uint32_t RELOC_JUMP_SLOT = R_ARM_JUMP_SLOT;
// ARM uses a different TLS model; not handled here yet.
constexpr uint32_t RELOC_TLS_OFFSET = 0xffffffffu;
+constexpr uint32_t RELOC_TLS_MODULE = 0xfffffffeu;
+constexpr uint32_t RELOC_TLS_MODULE_OFFSET = 0xfffffffdu;
#else
#error "Symbol relocation types are not known for this architecture"
#endif
@@ -148,14 +159,45 @@ LIBC_INLINE BindResult bind_relocations(const Module &target,
for (const ElfW(Rela) *rela = table.begin(); rela != table.end(); ++rela) {
const uint32_t type = reloc_type(rela->r_info);
const bool is_tls = type == RELOC_TLS_OFFSET;
+ const bool is_tls_module = type == RELOC_TLS_MODULE;
+ const bool is_tls_module_offset = type == RELOC_TLS_MODULE_OFFSET;
if (type != RELOC_ABSOLUTE && type != RELOC_GLOB_DAT &&
- type != RELOC_JUMP_SLOT && !is_tls)
+ type != RELOC_JUMP_SLOT && !is_tls && !is_tls_module &&
+ !is_tls_module_offset)
continue;
const uint32_t symbol_index = reloc_symbol(rela->r_info);
const ElfW(Sym) &symbol = symtab[symbol_index];
const char *name = strtab + symbol.st_name;
+ if (is_tls_module || is_tls_module_offset) {
+ // The general dynamic model. What is stored is not an address but a
+ // module and an offset within that module's block, which
+ // __tls_get_addr turns into an address once it knows which thread is
+ // asking.
+ size_t module = target_index;
+ ElfW(Addr) value = 0;
+ if (symbol_index != 0 && symbol.st_shndx == SHN_UNDEF) {
+ auto defined = order.find(name);
+ if (!defined) {
+ ++result.unresolved;
+ if (result.missing == nullptr)
+ result.missing = name;
+ continue;
+ }
+ module = defined->module;
+ value = defined->value;
+ } else if (symbol_index != 0) {
+ value = symbol.st_value;
+ }
+ auto *slot =
+ reinterpret_cast<ElfW(Addr) *>(target.load_bias() + rela->r_offset);
+ *slot = is_tls_module ? static_cast<ElfW(Addr)>(module)
+ : static_cast<ElfW(Addr)>(value + rela->r_addend);
+ ++result.bound;
+ continue;
+ }
+
if (is_tls) {
// A thread local is addressed relative to the thread pointer, so what
// gets stored is the defining module's block offset plus the symbol's
diff --git a/libc/src/__support/elf/passive_abi.h b/libc/src/__support/elf/passive_abi.h
index 4927feaccdbfc8..deeba1d7034fdc 100644
--- a/libc/src/__support/elf/passive_abi.h
+++ b/libc/src/__support/elf/passive_abi.h
@@ -61,15 +61,40 @@ constexpr const char *MODULE_SET_SYMBOL = "__llvm_libc_process_modules";
} // namespace elf
} // namespace LIBC_NAMESPACE_DECL
-extern "C" [[gnu::visibility("default")]]
+// Declared weak because the startup files refer to it and are also linked
+// into programs built against a static libc, where nothing defines it. Such a
+// program has no loader and no module set, and the reference resolves to
+// nothing rather than failing the link.
+extern "C" [[gnu::weak, gnu::visibility("default")]]
LIBC_NAMESPACE::elf::ModuleSet __llvm_libc_process_modules;
namespace LIBC_NAMESPACE_DECL {
namespace elf {
-// The same object, under the name the rest of libc uses.
+// The same object, under the name the rest of libc uses. Only for code that
+// is always linked with the definition, which is everything inside libc and
+// the loader.
LIBC_INLINE ModuleSet &loaded_modules() { return __llvm_libc_process_modules; }
+// Some readers of the set do not hold a lock: __tls_get_addr and the creation
+// of a thread cannot wait on the loader. What they may rely on is that an
+// array they have seen is never freed, only replaced by a larger one (see
+// make_room in dlopen), and that an entry is written before the count which
+// covers it. These are the two halves of that, for the code that changes the
+// set and the code that reads it unlocked.
+LIBC_INLINE size_t module_count(const ModuleSet &set) {
+ return __atomic_load_n(&set.count, __ATOMIC_ACQUIRE);
+}
+LIBC_INLINE void set_module_count(ModuleSet &set, size_t count) {
+ __atomic_store_n(&set.count, count, __ATOMIC_RELEASE);
+}
+
+// The record, or null where it was not linked in at all. Callers that may run
+// in a statically linked program have to ask this way.
+LIBC_INLINE ModuleSet *process_modules() {
+ return &__llvm_libc_process_modules;
+}
+
} // namespace elf
} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/src/__support/elf/thread_pointer.h b/libc/src/__support/elf/thread_pointer.h
index 9734e97fe68f8b..af7c4288f132e9 100644
--- a/libc/src/__support/elf/thread_pointer.h
+++ b/libc/src/__support/elf/thread_pointer.h
@@ -50,6 +50,30 @@ LIBC_INLINE bool set_thread_pointer(uintptr_t value) {
#endif
}
+// Where the thread's blocks are, which is what a thread local's offset is
+// measured from.
+LIBC_INLINE uintptr_t thread_pointer() {
+#if __has_builtin(__builtin_thread_pointer)
+ return reinterpret_cast<uintptr_t>(__builtin_thread_pointer());
+#elif defined(LIBC_TARGET_ARCH_IS_X86_64)
+ // The register reads back through itself; the first word of the thread
+ // control block holds its own address so that this works.
+ uintptr_t value;
+ __asm__ __volatile__("mov %%fs:0, %0" : "=r"(value));
+ return value;
+#elif defined(LIBC_TARGET_ARCH_IS_AARCH64)
+ uintptr_t value;
+ __asm__ __volatile__("mrs %0, tpidr_el0" : "=r"(value));
+ return value;
+#elif defined(LIBC_TARGET_ARCH_IS_ANY_RISCV)
+ uintptr_t value;
+ __asm__ __volatile__("mv %0, tp" : "=r"(value));
+ return value;
+#else
+#error "Reading the thread pointer is not implemented for this architecture"
+#endif
+}
+
} // namespace elf
} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/src/__support/elf/tls_block.h b/libc/src/__support/elf/tls_block.h
index 241c9d181315a5..99a1b39877892a 100644
--- a/libc/src/__support/elf/tls_block.h
+++ b/libc/src/__support/elf/tls_block.h
@@ -21,6 +21,7 @@
#include "hdr/stdint_proxy.h"
#include "hdr/sys_mman_macros.h"
#include "hdr/types/size_t.h"
+#include "src/__support/OSUtil/linux/syscall_wrappers/getrandom.h"
#include "src/__support/OSUtil/linux/syscall_wrappers/mmap.h"
#include "src/__support/OSUtil/linux/syscall_wrappers/munmap.h"
#include "src/__support/elf/module.h"
@@ -100,9 +101,9 @@ LIBC_INLINE ErrorOr<TlsBlock> allocate_tls_block(const Module *modules,
// are padded until it does; the mapping itself is page aligned.
const size_t below_tp =
TLS_VARIANT_2
- ? round_up(layout.size() - TLS_TCB_SIZE, layout.alignment())
+ ? round_up(layout.size() - TLS_TCB_RESERVE, layout.alignment())
: 0;
- const size_t total = TLS_VARIANT_2 ? below_tp + TLS_TCB_SIZE : layout.size();
+ const size_t total = TLS_VARIANT_2 ? below_tp + TLS_TCB_RESERVE : layout.size();
auto storage = linux_syscalls::mmap(nullptr, total, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
@@ -120,6 +121,16 @@ LIBC_INLINE ErrorOr<TlsBlock> allocate_tls_block(const Module *modules,
// dereference an offset from address zero.
*reinterpret_cast<uintptr_t *>(tp) = tp;
+ // Code built with -fstack-protector reads its guard from a fixed offset
+ // from the thread pointer, so every thread needs one before any such code
+ // runs. A failed read leaves the value at zero, which still compares equal
+ // to itself and so costs correctness nothing.
+ if (TLS_VARIANT_2) {
+ auto *guard = reinterpret_cast<uintptr_t *>(tp + TLS_STACK_GUARD_OFFSET);
+ if (!linux_syscalls::getrandom(guard, sizeof(uintptr_t), 0).has_value())
+ *guard = 0;
+ }
+
TlsLayout again;
for (size_t i = 0; i < count; ++i) {
const ElfW(Phdr) *tls = modules[i].tls();
diff --git a/libc/src/__support/elf/tls_layout.h b/libc/src/__support/elf/tls_layout.h
index 0b7590ddd40fd6..04ae7a56dabf33 100644
--- a/libc/src/__support/elf/tls_layout.h
+++ b/libc/src/__support/elf/tls_layout.h
@@ -39,6 +39,20 @@ constexpr bool TLS_VARIANT_2 = false;
// module's block.
constexpr size_t TLS_TCB_SIZE = 2 * sizeof(void *);
+// Bytes to keep at the thread pointer itself. Variant 1 needs only the
+// control block, which its offsets already skip. Variant 2 has to leave room
+// for what x86 addresses through the thread pointer by fixed offset: the self
+// pointer at zero, the stack guard at 0x28 and the pointer guard at 0x30.
+#if defined(LIBC_TARGET_ARCH_IS_X86_64) || defined(LIBC_TARGET_ARCH_IS_X86_32)
+constexpr size_t TLS_TCB_RESERVE = 64;
+#else
+constexpr size_t TLS_TCB_RESERVE = TLS_TCB_SIZE;
+#endif
+
+// Where the stack guard sits, relative to the thread pointer, for code built
+// with -fstack-protector.
+constexpr size_t TLS_STACK_GUARD_OFFSET = 0x28;
+
LIBC_INLINE constexpr size_t round_up(size_t value, size_t alignment) {
if (alignment <= 1)
return value;
@@ -84,7 +98,7 @@ class TlsLayout {
// Total bytes the thread block needs, thread control block included.
LIBC_INLINE constexpr size_t size() const {
if (TLS_VARIANT_2)
- return offset_ + TLS_TCB_SIZE;
+ return offset_ + TLS_TCB_RESERVE;
return offset_;
}
diff --git a/libc/src/__support/threads/linux/aarch64/CMakeLists.txt b/libc/src/__support/threads/linux/aarch64/CMakeLists.txt
index a2320f8d015c38..1dacfda2e753ac 100644
--- a/libc/src/__support/threads/linux/aarch64/CMakeLists.txt
+++ b/libc/src/__support/threads/linux/aarch64/CMakeLists.txt
@@ -8,6 +8,9 @@ add_object_library(
libc.include.sys_syscall
libc.src.__support.OSUtil.osutil
libc.src.__support.OSUtil.linux.syscall_wrappers.exit
+
+ libc.src.__support.elf.passive_abi
+ libc.src.__support.elf.tls_block
libc.src.__support.OSUtil.linux.syscall_wrappers.mmap
libc.src.__support.OSUtil.linux.syscall_wrappers.munmap
libc.src.__support.threads.linux.tcb
diff --git a/libc/src/__support/threads/linux/aarch64/tls.cpp b/libc/src/__support/threads/linux/aarch64/tls.cpp
index bf922aaf0b2cf0..0e00dde382141e 100644
--- a/libc/src/__support/threads/linux/aarch64/tls.cpp
+++ b/libc/src/__support/threads/linux/aarch64/tls.cpp
@@ -12,6 +12,8 @@
#include "src/__support/OSUtil/linux/syscall_wrappers/mmap.h"
#include "src/__support/OSUtil/linux/syscall_wrappers/munmap.h"
#include "src/__support/OSUtil/syscall.h"
+#include "src/__support/elf/passive_abi.h"
+#include "src/__support/elf/tls_block.h"
#include "src/__support/macros/config.h"
#include "src/__support/threads/linux/tcb.h"
#include "src/string/memory_utils/inline_memcpy.h"
@@ -24,7 +26,32 @@
namespace LIBC_NAMESPACE_DECL {
+// Builds the thread's static TLS from the loaded module set.
+//
+// Every module loaded at startup has a place in the one block each thread is
+// given, and where each one sits was decided when the process was linked.
+// Sizing this from the main executable's PT_TLS alone leaves libc.so's own
+// thread locals, and those of every shared library, pointing outside it.
+LIBC_INLINE bool init_tls_from_modules(TLSDescriptor &tls_descriptor) {
+ const elf::ModuleSet *modules = elf::process_modules();
+ if (modules == nullptr || !modules->linked)
+ return false;
+ // Only what was loaded at startup has a place in the block. Anything opened
+ // since takes a block of its own the first time a thread asks for it.
+ auto block = elf::allocate_tls_block(
+ __atomic_load_n(&modules->modules, __ATOMIC_ACQUIRE),
+ modules->static_count);
+ if (!block.has_value())
+ linux_syscalls::exit(1);
+ tls_descriptor = {block->size, reinterpret_cast<uintptr_t>(block->storage),
+ block->thread_pointer};
+ return true;
+}
+
[[gnu::flatten]] void init_tls(TLSDescriptor &tls_descriptor) {
+ if (init_tls_from_modules(tls_descriptor))
+ return;
+
// aarch64 follows the variant 1 TLS layout:
//
// 1. First entry is the dynamic thread vector pointer
diff --git a/libc/src/__support/threads/linux/riscv/CMakeLists.txt b/libc/src/__support/threads/linux/riscv/CMakeLists.txt
index a2320f8d015c38..1dacfda2e753ac 100644
--- a/libc/src/__support/threads/linux/riscv/CMakeLists.txt
+++ b/libc/src/__support/threads/linux/riscv/CMakeLists.txt
@@ -8,6 +8,9 @@ add_object_library(
libc.include.sys_syscall
libc.src.__support.OSUtil.osutil
libc.src.__support.OSUtil.linux.syscall_wrappers.exit
+
+ libc.src.__support.elf.passive_abi
+ libc.src.__support.elf.tls_block
libc.src.__support.OSUtil.linux.syscall_wrappers.mmap
libc.src.__support.OSUtil.linux.syscall_wrappers.munmap
libc.src.__support.threads.linux.tcb
diff --git a/libc/src/__support/threads/linux/riscv/tls.cpp b/libc/src/__support/threads/linux/riscv/tls.cpp
index 88a45a09df1043..2a7ff006a57c52 100644
--- a/libc/src/__support/threads/linux/riscv/tls.cpp
+++ b/libc/src/__support/threads/linux/riscv/tls.cpp
@@ -12,6 +12,8 @@
#include "src/__support/OSUtil/linux/syscall_wrappers/mmap.h"
#include "src/__support/OSUtil/linux/syscall_wrappers/munmap.h"
#include "src/__support/OSUtil/syscall.h"
+#include "src/__support/elf/passive_abi.h"
+#include "src/__support/elf/tls_block.h"
#include "src/__support/macros/config.h"
#include "src/__support/threads/linux/tcb.h"
#include "src/string/memory_utils/inline_memcpy.h"
@@ -19,7 +21,32 @@
namespace LIBC_NAMESPACE_DECL {
+// Builds the thread's static TLS from the loaded module set.
+//
+// Every module loaded at startup has a place in the one block each thread is
+// given, and where each one sits was decided when the process was linked.
+// Sizing this from the main executable's PT_TLS alone leaves libc.so's own
+// thread locals, and those of every shared library, pointing outside it.
+LIBC_INLINE bool init_tls_from_modules(TLSDescriptor &tls_descriptor) {
+ const elf::ModuleSet *modules = elf::process_modules();
+ if (modules == nullptr || !modules->linked)
+ return false;
+ // Only what was loaded at startup has a place in the block. Anything opened
+ // since takes a block of its own the first time a thread asks for it.
+ auto block = elf::allocate_tls_block(
+ __atomic_load_n(&modules->modules, __ATOMIC_ACQUIRE),
+ modules->static_count);
+ if (!block.has_value())
+ linux_syscalls::exit(1);
+ tls_descriptor = {block->size, reinterpret_cast<uintptr_t>(block->storage),
+ block->thread_pointer};
+ return true;
+}
+
[[gnu::flatten]] void init_tls(TLSDescriptor &tls_descriptor) {
+ if (init_tls_from_modules(tls_descriptor))
+ return;
+
// riscv follows the variant 1 TLS layout:
const uintptr_t TCB_SIZE = sizeof(ThreadControlBlock);
uintptr_t padding = 0;
diff --git a/libc/src/__support/threads/linux/x86_64/CMakeLists.txt b/libc/src/__support/threads/linux/x86_64/CMakeLists.txt
index cfd58503320086..3ab662ed4055c8 100644
--- a/libc/src/__support/threads/linux/x86_64/CMakeLists.txt
+++ b/libc/src/__support/threads/linux/x86_64/CMakeLists.txt
@@ -8,6 +8,9 @@ add_object_library(
libc.include.sys_syscall
libc.src.__support.OSUtil.osutil
libc.src.__support.OSUtil.linux.syscall_wrappers.exit
+
+ libc.src.__support.elf.passive_abi
+ libc.src.__support.elf.tls_block
libc.src.__support.OSUtil.linux.syscall_wrappers.getrandom
libc.src.__support.OSUtil.linux.syscall_wrappers.mmap
libc.src.__support.OSUtil.linux.syscall_wrappers.munmap
diff --git a/libc/src/__support/threads/linux/x86_64/tls.cpp b/libc/src/__support/threads/linux/x86_64/tls.cpp
index af2e16740681be..454c74826064aa 100644
--- a/libc/src/__support/threads/linux/x86_64/tls.cpp
+++ b/libc/src/__support/threads/linux/x86_64/tls.cpp
@@ -12,6 +12,8 @@
#include "src/__support/OSUtil/linux/syscall_wrappers/getrandom.h"
#include "src/__support/OSUtil/linux/syscall_wrappers/mmap.h"
#include "src/__support/OSUtil/linux/syscall_wrappers/munmap.h"
+#include "src/__support/elf/passive_abi.h"
+#include "src/__support/elf/tls_block.h"
#include "src/__support/macros/config.h"
#include "src/__support/threads/linux/tcb.h"
#include "src/string/memory_utils/inline_memcpy.h"
@@ -21,8 +23,32 @@
namespace LIBC_NAMESPACE_DECL {
-// TODO: Also generalize this routine and handle dynamic loading properly.
+// Builds the thread's static TLS from the loaded module set.
+//
+// Every module loaded at startup has a place in the one block each thread is
+// given, and where each one sits was decided when the process was linked.
+// Sizing this from the main executable's PT_TLS alone leaves libc.so's own
+// thread locals, and those of every shared library, pointing outside it.
+LIBC_INLINE bool init_tls_from_modules(TLSDescriptor &tls_descriptor) {
+ const elf::ModuleSet *modules = elf::process_modules();
+ if (modules == nullptr || !modules->linked)
+ return false;
+ // Only what was loaded at startup has a place in the block. Anything opened
+ // since takes a block of its own the first time a thread asks for it.
+ auto block = elf::allocate_tls_block(
+ __atomic_load_n(&modules->modules, __ATOMIC_ACQUIRE),
+ modules->static_count);
+ if (!block.has_value())
+ linux_syscalls::exit(1);
+ tls_descriptor = {block->size, reinterpret_cast<uintptr_t>(block->storage),
+ block->thread_pointer};
+ return true;
+}
+
[[gnu::flatten]] void init_tls(TLSDescriptor &tls_descriptor) {
+ if (init_tls_from_modules(tls_descriptor))
+ return;
+
// We will assume the alignment is always a power of two.
uintptr_t tls_size = app.tls.size & -app.tls.align;
if (tls_size != app.tls.size)
diff --git a/libc/src/tls/CMakeLists.txt b/libc/src/tls/CMakeLists.txt
new file mode 100644
index 00000000000000..6b0986a59aaa5f
--- /dev/null
+++ b/libc/src/tls/CMakeLists.txt
@@ -0,0 +1,12 @@
+add_entrypoint_object(
+ __tls_get_addr
+ SRCS
+ tls_get_addr.cpp
+ HDRS
+ tls_get_addr.h
+ DEPENDS
+ libc.hdr.stdint_proxy
+ libc.hdr.types.size_t
+ libc.src.__support.elf.passive_abi
+ libc.src.__support.elf.thread_pointer
+)
diff --git a/libc/src/tls/tls_get_addr.cpp b/libc/src/tls/tls_get_addr.cpp
new file mode 100644
index 00000000000000..a583eac7e470b8
--- /dev/null
+++ b/libc/src/tls/tls_get_addr.cpp
@@ -0,0 +1,52 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Implementation of __tls_get_addr
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/tls/tls_get_addr.h"
+
+#include "hdr/stdint_proxy.h"
+#include "src/__support/common.h"
+#include "src/__support/elf/passive_abi.h"
+#include "src/__support/elf/thread_pointer.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+
+// Where a thread local lives for the thread which is asking.
+//
+// A shared library cannot know at link time where in a thread's blocks its
+// own will sit, so its thread locals are compiled to name the module and an
+// offset within that module's block and to call this. The module's offset
+// from the thread pointer is what the loader worked out when it laid the
+// blocks out, and it is in the record the loader left behind.
+//
+// Every module here is one loaded at startup, so they all have a place in
+// the one block each thread is given and the answer is a sum. A module
+// brought in later by dlopen would need a block of its own, allocated the
+// first time each thread asked for it; that is not supported yet, and such
+// a module's index is not in the record, so this reports it as nothing
+// rather than reading past the end.
+LLVM_LIBC_FUNCTION(void *, __tls_get_addr, (TlsIndex * index)) {
+ if (index == nullptr)
+ return nullptr;
+
+ const elf::ModuleSet &modules = elf::loaded_modules();
+ if (modules.tls_offsets == nullptr || index->ti_module >= modules.count)
+ return nullptr;
+
+ const uintptr_t base = elf::thread_pointer();
+ const intptr_t offset = modules.tls_offsets[index->ti_module];
+ return reinterpret_cast<void *>(base + static_cast<uintptr_t>(offset) +
+ index->ti_offset);
+}
+
+} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/src/tls/tls_get_addr.h b/libc/src/tls/tls_get_addr.h
new file mode 100644
index 00000000000000..3c18d93e7ccd66
--- /dev/null
+++ b/libc/src/tls/tls_get_addr.h
@@ -0,0 +1,33 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Implementation header for __tls_get_addr
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC_TLS_TLS_GET_ADDR_H
+#define LLVM_LIBC_SRC_TLS_TLS_GET_ADDR_H
+
+#include "hdr/types/size_t.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+
+// What the compiler passes: which module the thread local belongs to, and
+// where it sits inside that module's block. The layout is fixed by the ABI.
+struct TlsIndex {
+ unsigned long ti_module;
+ unsigned long ti_offset;
+};
+
+void *__tls_get_addr(TlsIndex *index);
+
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC_TLS_TLS_GET_ADDR_H
diff --git a/libc/test/src/__support/elf/tls_block_test.cpp b/libc/test/src/__support/elf/tls_block_test.cpp
index b54b0c247731c3..4e0fceca2affd9 100644
--- a/libc/test/src/__support/elf/tls_block_test.cpp
+++ b/libc/test/src/__support/elf/tls_block_test.cpp
@@ -20,7 +20,8 @@ using LIBC_NAMESPACE::elf::free_tls_block;
using LIBC_NAMESPACE::elf::load_module;
using LIBC_NAMESPACE::elf::LoadedModule;
using LIBC_NAMESPACE::elf::Module;
-using LIBC_NAMESPACE::elf::TLS_TCB_SIZE;
+using LIBC_NAMESPACE::elf::TLS_STACK_GUARD_OFFSET;
+using LIBC_NAMESPACE::elf::TLS_TCB_RESERVE;
using LIBC_NAMESPACE::elf::TLS_VARIANT_2;
using LIBC_NAMESPACE::elf::TlsBlock;
using LIBC_NAMESPACE::elf::unmap_module;
@@ -63,7 +64,7 @@ TEST(LlvmLibcElfTlsBlockTest, CoversLibcOwnTlsSegment) {
ASSERT_TRUE(block.storage != nullptr);
// The block has to hold libc's whole TLS segment plus the control block.
- EXPECT_GE(block.size, size_t(tls->p_memsz) + TLS_TCB_SIZE);
+ EXPECT_GE(block.size, size_t(tls->p_memsz) + TLS_TCB_RESERVE);
// The thread pointer sits inside the allocation, at the top under variant 2
// and at the bottom under variant 1.
@@ -71,7 +72,7 @@ TEST(LlvmLibcElfTlsBlockTest, CoversLibcOwnTlsSegment) {
EXPECT_GE(block.thread_pointer, base);
EXPECT_LE(block.thread_pointer, base + block.size);
if (TLS_VARIANT_2)
- EXPECT_EQ(block.thread_pointer, base + block.size - TLS_TCB_SIZE);
+ EXPECT_EQ(block.thread_pointer, base + block.size - TLS_TCB_RESERVE);
else
EXPECT_EQ(block.thread_pointer, base);
@@ -79,6 +80,28 @@ TEST(LlvmLibcElfTlsBlockTest, CoversLibcOwnTlsSegment) {
unmap_module(c.mapping);
}
+TEST(LlvmLibcElfTlsBlockTest, StackGuardIsSeeded) {
+ if (!TLS_VARIANT_2)
+ return;
+ auto loaded = load_module(LIBC_PATH, PAGE);
+ ASSERT_TRUE(loaded.has_value());
+ LoadedModule c = loaded.value();
+
+ Module modules[] = {c.module};
+ auto result = allocate_tls_block(modules, 1);
+ ASSERT_TRUE(result.has_value());
+ TlsBlock block = result.value();
+
+ // Where code built with -fstack-protector reads its guard from. Leaving it
+ // zero would make every such function compare zero against zero.
+ auto *guard = reinterpret_cast<uintptr_t *>(block.thread_pointer +
+ TLS_STACK_GUARD_OFFSET);
+ EXPECT_NE(*guard, uintptr_t(0));
+
+ free_tls_block(block);
+ unmap_module(c.mapping);
+}
+
TEST(LlvmLibcElfTlsBlockTest, InitialisedDataIsCopiedIn) {
auto loaded = load_module(LIBC_PATH, PAGE);
ASSERT_TRUE(loaded.has_value());
diff --git a/libc/test/src/__support/elf/tls_layout_test.cpp b/libc/test/src/__support/elf/tls_layout_test.cpp
index 559cc3ce40974d..a247bb7a9f7704 100644
--- a/libc/test/src/__support/elf/tls_layout_test.cpp
+++ b/libc/test/src/__support/elf/tls_layout_test.cpp
@@ -15,6 +15,7 @@
#include "test/UnitTest/Test.h"
using LIBC_NAMESPACE::elf::round_up;
+using LIBC_NAMESPACE::elf::TLS_TCB_RESERVE;
using LIBC_NAMESPACE::elf::TLS_TCB_SIZE;
using LIBC_NAMESPACE::elf::TLS_VARIANT_2;
using LIBC_NAMESPACE::elf::TlsLayout;
@@ -34,7 +35,7 @@ TEST(LlvmLibcElfTlsLayoutTest, EmptyLayout) {
EXPECT_EQ(layout.module_count(), size_t(0));
EXPECT_EQ(layout.alignment(), TLS_TCB_SIZE);
if (TLS_VARIANT_2)
- EXPECT_EQ(layout.size(), TLS_TCB_SIZE);
+ EXPECT_EQ(layout.size(), TLS_TCB_RESERVE);
else
EXPECT_EQ(layout.size(), size_t(0));
}
>From 5b2eea6a9ddb99045c300989aa710e10671fdf30 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Sat, 5 Sep 2026 13:57:10 +0100
Subject: [PATCH 40/46] [libc] Expand $ORIGIN in a run path
DT_RUNPATH commonly holds $ORIGIN, which stands for the directory
containing the object and is what lets a program find libraries beside
it wherever it is installed. The loader took the entry literally, so a
program built that way could not find its own libraries.
A library's $ORIGIN is the directory of the path it was loaded from. The
executable's module is named with the path the kernel was given, but its
$ORIGIN is the directory of the file it really is, which /proc/self/exe
names, as glibc and musl take it: a command is often started through a
link, such as a versioned tool's name in /usr/bin, and its libraries sit
beside the file the link leads to. Where there is no /proc, as early in
boot, the path the kernel was given stands in.
Assisted-by: Claude
---
libc/src/__support/elf/CMakeLists.txt | 11 ++
libc/src/__support/elf/run_path.h | 118 +++++++++++++++++
libc/src/__support/elf/startup_linker.h | 47 +++++--
libc/test/src/__support/elf/CMakeLists.txt | 12 ++
libc/test/src/__support/elf/run_path_test.cpp | 122 ++++++++++++++++++
5 files changed, 299 insertions(+), 11 deletions(-)
create mode 100644 libc/src/__support/elf/run_path.h
create mode 100644 libc/test/src/__support/elf/run_path_test.cpp
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
index 18887e2b031be0..691f61cac81757 100644
--- a/libc/src/__support/elf/CMakeLists.txt
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -1,3 +1,14 @@
+add_header_library(
+ run_path
+ HDRS
+ run_path.h
+ DEPENDS
+ libc.hdr.types.size_t
+ libc.src.__support.OSUtil.linux.syscall_wrappers.readlink
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+)
+
add_header_library(
thread_pointer
HDRS
diff --git a/libc/src/__support/elf/run_path.h b/libc/src/__support/elf/run_path.h
new file mode 100644
index 00000000000000..abd8a4ab3cf249
--- /dev/null
+++ b/libc/src/__support/elf/run_path.h
@@ -0,0 +1,118 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Expanding a run path entry
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_RUN_PATH_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_RUN_PATH_H
+
+#include "hdr/types/size_t.h"
+#include "src/__support/OSUtil/linux/syscall_wrappers/readlink.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+LIBC_INLINE bool starts_with(const char *text, const char *prefix) {
+ for (; *prefix != '\0'; ++text, ++prefix)
+ if (*text != *prefix)
+ return false;
+ return true;
+}
+
+// How much of a path names the directory holding it, with the separator
+// dropped. Zero when there is no directory part at all.
+LIBC_INLINE size_t directory_length(const char *path) {
+ size_t last = 0;
+ for (size_t i = 0; path[i] != '\0'; ++i)
+ if (path[i] == '/')
+ last = i;
+ // A path rooted at / has the slash itself as its directory.
+ if (last == 0 && path[0] == '/')
+ return 1;
+ return last;
+}
+
+// The longest path the executable is resolved to. A longer one keeps the path
+// the program was started by.
+constexpr size_t MAX_EXECUTABLE_PATH = 4096;
+
+// What $ORIGIN stands for in the executable's own run paths: the directory of
+// the file the program really is. A command is often started through a link,
+// such as a versioned tool's name under /usr/bin, and its libraries sit beside
+// the file the link leads to, which is where glibc and musl look as well. The
+// kernel says which file that is through /proc. Where there is no /proc to ask,
+// as early in boot, or the path does not fit, the path the program was started
+// by is kept.
+LIBC_INLINE const char *executable_origin(const char *started_as, char *buffer,
+ size_t capacity) {
+ if (capacity < 2)
+ return started_as;
+ auto length = linux_syscalls::readlink("/proc/self/exe", buffer, capacity);
+ if (!length.has_value() || length.value() <= 0 ||
+ static_cast<size_t>(length.value()) >= capacity)
+ return started_as;
+ buffer[length.value()] = '\0';
+ return buffer;
+}
+
+// Copies one run path entry into `out`, replacing $ORIGIN with the directory
+// holding the object whose run path it is. Both spellings the dynamic linkers
+// accept are handled. Returns false if it does not fit, if the entry is
+// empty, or if it needs an origin and there is none to give it.
+LIBC_INLINE bool expand_run_path(const char *entry, size_t length,
+ const char *origin, char *out,
+ size_t capacity) {
+ if (capacity == 0)
+ return false;
+ size_t n = 0;
+ for (size_t i = 0; i < length;) {
+ if (entry[i] == '$') {
+ const char *rest = entry + i + 1;
+ const size_t remaining = length - i - 1;
+ size_t token = 0;
+ if (remaining >= 6 && starts_with(rest, "ORIGIN"))
+ token = 7;
+ else if (remaining >= 8 && starts_with(rest, "{ORIGIN}"))
+ token = 9;
+ if (token != 0) {
+ if (origin == nullptr)
+ return false;
+ const size_t directory = directory_length(origin);
+ if (directory == 0) {
+ // The object was named without a directory, so it came from the
+ // working directory and that is what $ORIGIN means here.
+ if (n + 1 >= capacity)
+ return false;
+ out[n++] = '.';
+ } else {
+ if (n + directory >= capacity)
+ return false;
+ for (size_t j = 0; j < directory; ++j)
+ out[n++] = origin[j];
+ }
+ i += token;
+ continue;
+ }
+ }
+ if (n + 1 >= capacity)
+ return false;
+ out[n++] = entry[i++];
+ }
+ out[n] = '\0';
+ return n != 0;
+}
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_RUN_PATH_H
diff --git a/libc/src/__support/elf/startup_linker.h b/libc/src/__support/elf/startup_linker.h
index 81c0dc400d6161..b1539403f491b9 100644
--- a/libc/src/__support/elf/startup_linker.h
+++ b/libc/src/__support/elf/startup_linker.h
@@ -23,6 +23,7 @@
#include "src/__support/elf/load_module.h"
#include "src/__support/elf/module.h"
#include "src/__support/elf/passive_abi.h"
+#include "src/__support/elf/run_path.h"
#include "src/__support/elf/startup_stack.h"
#include "src/__support/elf/thread_pointer.h"
#include "src/__support/elf/tls_block.h"
@@ -50,13 +51,19 @@ class StartupLinker {
const ExecutableImage &image) {
// The executable is already mapped; describe it where the kernel put it.
const ElfW(Addr) bias = executable_bias(image);
- modules_[count_++] = Module(image.phdrs, image.phnum, bias, "");
+ modules_[count_++] =
+ Module(image.phdrs, image.phnum, bias, executable_path(stack));
// The kernel maps the executable but applies nothing. A position
// independent one still has relative relocations of its own, and without
// them even its init array bounds point at the wrong addresses.
modules_[0].relocations().apply_relative(bias);
+ // A program started through a link looks for its libraries beside the
+ // file it really is.
+ executable_origin_ = executable_origin(modules_[0].name(), executable_path_,
+ sizeof(executable_path_));
+
// Breadth first, so a dependency named earlier is loaded earlier, which
// is the order symbol lookup then walks.
for (size_t i = 0; i < count_; ++i) {
@@ -65,7 +72,8 @@ class StartupLinker {
requester.for_each_needed([&](const char *name) {
if (failed || already_loaded(name))
return;
- if (!load_dependency(requester, name))
+ if (!load_dependency(requester, i == 0 ? executable_origin_ : nullptr,
+ name))
failed = true;
});
if (failed)
@@ -123,6 +131,15 @@ class StartupLinker {
return 0;
}
+ // What the kernel was asked to run, which is where $ORIGIN points for the
+ // executable's own run path. It may be relative, in which case it is
+ // relative to the same directory the kernel resolved it against.
+ LIBC_INLINE static const char *executable_path(const StartupStack &stack) {
+ if (auto value = stack.auxval(AT_EXECFN))
+ return reinterpret_cast<const char *>(*value);
+ return "";
+ }
+
LIBC_INLINE bool already_loaded(const char *name) const {
for (size_t i = 0; i < count_; ++i) {
const char *soname = modules_[i].soname();
@@ -156,9 +173,12 @@ class StartupLinker {
return nullptr;
}
- // Tries `name` under each colon separated directory in `list`.
+ // Tries `name` under each colon separated directory in `list`. `origin` is
+ // what $ORIGIN stands for in that list, and is null where the list may not
+ // use it.
LIBC_INLINE bool try_path_list(const char *list, const char *name,
- LoadedModule &out) {
+ LoadedModule &out,
+ const char *origin = nullptr) {
if (list == nullptr)
return false;
for (const char *segment = list; segment != nullptr;) {
@@ -167,10 +187,8 @@ class StartupLinker {
++end;
char dir[256];
const size_t length = static_cast<size_t>(end - segment);
- if (length > 0 && length < sizeof(dir)) {
- for (size_t i = 0; i < length; ++i)
- dir[i] = segment[i];
- dir[length] = '\0';
+ if (length > 0 &&
+ expand_run_path(segment, length, origin, dir, sizeof(dir))) {
char path[256];
if (join(dir, name, path, sizeof(path))) {
auto loaded = load_module(path, page_size_);
@@ -186,8 +204,11 @@ class StartupLinker {
}
// The search order a loader is expected to use: the requesting object's own
- // run path, then LD_LIBRARY_PATH, then the system directories.
- LIBC_INLINE bool load_dependency(const Module &from, const char *name) {
+ // run path, then LD_LIBRARY_PATH, then the system directories. `origin` is
+ // what $ORIGIN in `from`'s run path stands for, or null for the directory of
+ // its own name.
+ LIBC_INLINE bool load_dependency(const Module &from, const char *origin,
+ const char *name) {
if (count_ >= MAX_STARTUP_MODULES) {
report("too many shared objects\n");
return false;
@@ -202,7 +223,8 @@ class StartupLinker {
}
LoadedModule loaded;
- if (try_path_list(run_path(from), name, loaded))
+ if (try_path_list(run_path(from), name, loaded,
+ origin != nullptr ? origin : from.name()))
return remember(loaded);
if (try_path_list(library_path_, name, loaded))
return remember(loaded);
@@ -329,6 +351,9 @@ class StartupLinker {
size_t page_size_;
const char *library_path_;
+ // Where the executable really is, for $ORIGIN in its run path.
+ char executable_path_[MAX_EXECUTABLE_PATH];
+ const char *executable_origin_ = nullptr;
Module modules_[MAX_STARTUP_MODULES];
MappedModule mappings_[MAX_STARTUP_MODULES];
intptr_t tls_offsets_[MAX_STARTUP_MODULES] = {};
diff --git a/libc/test/src/__support/elf/CMakeLists.txt b/libc/test/src/__support/elf/CMakeLists.txt
index de1ed0bd61c33f..c9406f88fe83d6 100644
--- a/libc/test/src/__support/elf/CMakeLists.txt
+++ b/libc/test/src/__support/elf/CMakeLists.txt
@@ -63,6 +63,7 @@ add_libc_test(
libc.src.__support.elf.tls_layout
)
+
add_libc_test(
load_layout_test
SUITE
@@ -156,3 +157,14 @@ add_libc_test(
DEPENDS
libc.src.__support.elf.startup_stack
)
+
+add_libc_test(
+ run_path_test
+ SUITE
+ libc-support-tests
+ SRCS
+ run_path_test.cpp
+ DEPENDS
+ libc.src.__support.CPP.string_view
+ libc.src.__support.elf.run_path
+)
diff --git a/libc/test/src/__support/elf/run_path_test.cpp b/libc/test/src/__support/elf/run_path_test.cpp
new file mode 100644
index 00000000000000..1c2d65e5136f0b
--- /dev/null
+++ b/libc/test/src/__support/elf/run_path_test.cpp
@@ -0,0 +1,122 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Unittests for run path expansion
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/CPP/string_view.h"
+#include "src/__support/elf/run_path.h"
+#include "test/UnitTest/Test.h"
+
+using LIBC_NAMESPACE::cpp::string_view;
+using LIBC_NAMESPACE::elf::directory_length;
+using LIBC_NAMESPACE::elf::executable_origin;
+using LIBC_NAMESPACE::elf::expand_run_path;
+
+namespace {
+
+// Expands a whole entry and reports what came out, or an empty view if it
+// could not be expanded.
+string_view expand(const char *entry, const char *origin, char *buffer,
+ size_t capacity) {
+ size_t length = 0;
+ while (entry[length] != '\0')
+ ++length;
+ if (!expand_run_path(entry, length, origin, buffer, capacity))
+ return string_view();
+ return string_view(buffer);
+}
+
+} // anonymous namespace
+
+TEST(LlvmLibcElfRunPathTest, EntryWithoutOriginIsCopied) {
+ char buffer[64];
+ EXPECT_TRUE(expand("/usr/lib", nullptr, buffer, sizeof(buffer)) ==
+ string_view("/usr/lib"));
+}
+
+TEST(LlvmLibcElfRunPathTest, OriginBecomesTheObjectsDirectory) {
+ char buffer[64];
+ EXPECT_TRUE(expand("$ORIGIN/../lib", "/opt/app/bin/tool", buffer,
+ sizeof(buffer)) == string_view("/opt/app/bin/../lib"));
+ EXPECT_TRUE(expand("${ORIGIN}/lib", "/opt/app/bin/tool", buffer,
+ sizeof(buffer)) == string_view("/opt/app/bin/lib"));
+}
+
+TEST(LlvmLibcElfRunPathTest, OriginOnItsOwn) {
+ char buffer[64];
+ EXPECT_TRUE(expand("$ORIGIN", "/opt/app/tool", buffer, sizeof(buffer)) ==
+ string_view("/opt/app"));
+}
+
+TEST(LlvmLibcElfRunPathTest, OriginAppearsMoreThanOnce) {
+ char buffer[64];
+ EXPECT_TRUE(expand("$ORIGIN/a:$ORIGIN/b", "/x/y/z", buffer, sizeof(buffer)) ==
+ string_view("/x/y/a:/x/y/b"));
+}
+
+TEST(LlvmLibcElfRunPathTest, ObjectInTheWorkingDirectory) {
+ char buffer[64];
+ // Nothing names a directory, so $ORIGIN is the directory the name was
+ // resolved against.
+ EXPECT_TRUE(expand("$ORIGIN/lib", "tool", buffer, sizeof(buffer)) ==
+ string_view("./lib"));
+}
+
+TEST(LlvmLibcElfRunPathTest, ObjectAtTheRoot) {
+ char buffer[64];
+ EXPECT_TRUE(expand("$ORIGIN/lib", "/tool", buffer, sizeof(buffer)) ==
+ string_view("//lib"));
+}
+
+TEST(LlvmLibcElfRunPathTest, DollarThatIsNotOrigin) {
+ char buffer[64];
+ EXPECT_TRUE(expand("/opt/$LIB", "/x/y", buffer, sizeof(buffer)) ==
+ string_view("/opt/$LIB"));
+ EXPECT_TRUE(expand("$ORIGINAL", "/x/y", buffer, sizeof(buffer)) ==
+ string_view("/xAL"));
+}
+
+TEST(LlvmLibcElfRunPathTest, RefusesWhatItCannotExpand) {
+ char buffer[64];
+ // No origin to put in.
+ EXPECT_TRUE(expand("$ORIGIN/lib", nullptr, buffer, sizeof(buffer)).empty());
+ // Nothing to expand at all.
+ EXPECT_TRUE(expand("", "/x/y", buffer, sizeof(buffer)).empty());
+ // Does not fit.
+ char small[8];
+ EXPECT_TRUE(
+ expand("$ORIGIN/lib", "/a/very/long/path/tool", small, sizeof(small))
+ .empty());
+}
+
+TEST(LlvmLibcElfRunPathTest, DirectoryLength) {
+ EXPECT_EQ(directory_length("/opt/app/tool"), size_t(8));
+ EXPECT_EQ(directory_length("tool"), size_t(0));
+ EXPECT_EQ(directory_length("/tool"), size_t(1));
+ EXPECT_EQ(directory_length(""), size_t(0));
+}
+
+// The executable's origin is the file it really is, which /proc gives as an
+// absolute path, rather than whatever it was started by.
+TEST(LlvmLibcElfRunPathTest, ExecutableOriginIsTheFileItReallyIs) {
+ char buffer[4096];
+ const char *origin =
+ executable_origin("/started/through/a/link", buffer, sizeof(buffer));
+ ASSERT_TRUE(origin == buffer);
+ EXPECT_EQ(origin[0], '/');
+}
+
+TEST(LlvmLibcElfRunPathTest, ExecutableOriginWithoutRoomIsHowItWasStarted) {
+ char buffer[4];
+ EXPECT_TRUE(
+ string_view(executable_origin("/bin/tool", buffer, sizeof(buffer))) ==
+ string_view("/bin/tool"));
+}
>From 2493d7037d44558c1b32c5068eaaf15800e35abb Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Sat, 5 Sep 2026 15:11:52 +0100
Subject: [PATCH 41/46] [libc] Let dlopen load a module with thread local
storage
dlopen refused any shared object with a PT_TLS segment, because the
threads that already existed were given blocks with no room for it and
there was nowhere to put its thread locals.
Each thread now keeps its own vector of blocks for modules opened after
it started, filling one in from the module's initial image the first
time it reaches for one of that module's thread locals. A thread that
never touches such a module allocates nothing. __tls_get_addr answers
from the thread's static block for a module loaded at startup, which has
a place in it, and from this vector for anything else.
The storage comes from the kernel rather than from malloc, because a
thread gives these back as it ends, which is also when the allocator is
taking its own per thread state apart, and neither should have to know
the order of the other.
A block stays where it is for as long as its module is loaded. The
address of a thread local can be kept anywhere, and a destructor
registered to run as the thread ends holds one. An index freed by
dlclose goes to the next dlopen, though, so the record carries a
generation which both bump, dlopen stamps each module it adds with the
value it reached, and a block records the value it was made at. A block
made before the module at its index was opened belongs to one that is no
longer there, and is made again from the new module's image.
Verified against glibc: a module with thread locals opened and used from
the main thread and from threads created after it, from threads that
already existed when it was opened, twenty rounds of opening, using from
four threads, and closing, and a thread that keeps using one module's
thread locals after opening and using another.
Assisted-by: Claude
---
libc/src/__support/elf/CMakeLists.txt | 34 +++
libc/src/__support/elf/dtv.h | 180 +++++++++++++++
libc/src/__support/elf/dynamic_blocks.cpp | 40 ++++
libc/src/__support/elf/dynamic_blocks.h | 35 +++
libc/src/__support/elf/map_segments.h | 5 +
libc/src/__support/elf/passive_abi.h | 8 +
libc/src/__support/elf/startup_linker.h | 1 +
.../__support/threads/linux/CMakeLists.txt | 1 +
libc/src/__support/threads/linux/thread.cpp | 6 +
libc/src/dlfcn/dlclose.cpp | 4 +-
libc/src/dlfcn/dlopen.cpp | 18 +-
libc/src/tls/CMakeLists.txt | 1 +
libc/src/tls/tls_get_addr.cpp | 38 ++--
libc/startup/linux/passive_abi.cpp | 2 +
libc/test/src/__support/elf/CMakeLists.txt | 16 ++
libc/test/src/__support/elf/dtv_test.cpp | 215 ++++++++++++++++++
16 files changed, 579 insertions(+), 25 deletions(-)
create mode 100644 libc/src/__support/elf/dtv.h
create mode 100644 libc/src/__support/elf/dynamic_blocks.cpp
create mode 100644 libc/src/__support/elf/dynamic_blocks.h
create mode 100644 libc/test/src/__support/elf/dtv_test.cpp
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
index 691f61cac81757..26263bc3edb16f 100644
--- a/libc/src/__support/elf/CMakeLists.txt
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -246,3 +246,37 @@ add_header_library(
libc.src.__support.macros.attributes
libc.src.__support.macros.config
)
+
+add_header_library(
+ dtv
+ HDRS
+ dtv.h
+ DEPENDS
+ libc.hdr.elf_macros
+ libc.hdr.elf_proxy
+ libc.hdr.sys_mman_macros
+ libc.src.__support.OSUtil.linux.syscall_wrappers.mmap
+ libc.src.__support.OSUtil.linux.syscall_wrappers.munmap
+ libc.hdr.link_macros
+ libc.hdr.stdint_proxy
+ libc.hdr.types.size_t
+ libc.src.__support.elf.module
+ libc.src.__support.elf.passive_abi
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+ libc.src.string.memory_utils.inline_memcpy
+)
+
+add_object_library(
+ dynamic_blocks
+ SRCS
+ dynamic_blocks.cpp
+ HDRS
+ dynamic_blocks.h
+ DEPENDS
+ libc.hdr.types.size_t
+ libc.src.__support.elf.dtv
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+)
+
diff --git a/libc/src/__support/elf/dtv.h b/libc/src/__support/elf/dtv.h
new file mode 100644
index 00000000000000..32d17c94da88bc
--- /dev/null
+++ b/libc/src/__support/elf/dtv.h
@@ -0,0 +1,180 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// A thread's dynamic thread vector
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_DTV_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_DTV_H
+
+#include "hdr/elf_macros.h"
+#include "hdr/elf_proxy.h"
+#include "hdr/link_macros.h"
+#include "hdr/stdint_proxy.h"
+#include "hdr/sys_mman_macros.h"
+#include "hdr/types/size_t.h"
+#include "src/__support/OSUtil/linux/syscall_wrappers/mmap.h"
+#include "src/__support/OSUtil/linux/syscall_wrappers/munmap.h"
+#include "src/__support/elf/module.h"
+#include "src/__support/elf/passive_abi.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+#include "src/string/memory_utils/inline_memcpy.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// Where a thread keeps the blocks for modules that were not there when it was
+// given its own.
+//
+// Everything loaded at startup has a place in the one block each thread gets,
+// at an offset fixed when the process was linked. A module opened later has
+// no such place: the threads that already exist were given blocks with no
+// room for it. So each thread grows this instead, a block at a time, the
+// first time it reaches for one of that module's thread locals.
+//
+// A block stays where it is for as long as its module is loaded. The address
+// of a thread local can be kept anywhere, a destructor registered to run as
+// the thread ends among them, so the only block given back early is one made
+// for a module that has since been closed and had its index given to another.
+//
+// The storage comes from the kernel rather than from malloc. A thread gives
+// these back as it ends, which is also when the allocator is taking its own
+// per thread state apart, and the order of the two is not something either
+// should have to know about.
+class DynamicThreadVector {
+public:
+ // The block for `module_index` in this thread, allocating and filling it in
+ // from the module's initial image if this is the first time. Null if there
+ // is no room to be had or the module has no thread local storage.
+ LIBC_INLINE void *block_for(const ModuleSet &modules, size_t module_index) {
+ if (module_index >= module_count(modules))
+ return nullptr;
+ if (module_index < count_) {
+ Entry &entry = entries_[module_index];
+ if (entry.storage != nullptr) {
+ const MappedModule *mappings =
+ __atomic_load_n(&modules.mappings, __ATOMIC_ACQUIRE);
+ if (entry.generation >= mappings[module_index].generation)
+ return entry.block;
+ // Made before the module now at this index was opened, so for the
+ // one that was there before it.
+ linux_syscalls::munmap(entry.storage, entry.size);
+ entry = Entry{};
+ }
+ }
+ return make_block(modules, module_index);
+ }
+
+ // Gives back everything this thread took. Called as the thread ends.
+ LIBC_INLINE void release() {
+ for (size_t i = 0; i < count_; ++i)
+ if (entries_[i].storage != nullptr)
+ linux_syscalls::munmap(entries_[i].storage, entries_[i].size);
+ if (entries_ != nullptr)
+ linux_syscalls::munmap(entries_, capacity_ * sizeof(Entry));
+ entries_ = nullptr;
+ count_ = 0;
+ capacity_ = 0;
+ }
+
+private:
+ struct Entry {
+ // What the kernel gave, and what it was asked for, so it can be given
+ // back.
+ void *storage;
+ size_t size;
+ // Where within it the module's thread locals begin, which is the first
+ // point far enough in to have the alignment they need.
+ void *block;
+ // The generation of the module set when this was made.
+ size_t generation;
+ };
+
+ LIBC_INLINE static void *map(size_t size) {
+ auto result = linux_syscalls::mmap(nullptr, size, PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ if (!result.has_value())
+ return nullptr;
+ return result.value();
+ }
+
+ LIBC_INLINE bool reserve(size_t wanted) {
+ if (wanted <= capacity_)
+ return true;
+ size_t next = capacity_ == 0 ? 16 : capacity_ * 2;
+ while (next < wanted)
+ next *= 2;
+ auto *grown = static_cast<Entry *>(map(next * sizeof(Entry)));
+ if (grown == nullptr)
+ return false;
+ // Fresh anonymous memory is already zero, so only what is in use has to
+ // be carried over.
+ for (size_t i = 0; i < count_; ++i)
+ grown[i] = entries_[i];
+ if (entries_ != nullptr)
+ linux_syscalls::munmap(entries_, capacity_ * sizeof(Entry));
+ entries_ = grown;
+ capacity_ = next;
+ return true;
+ }
+
+ LIBC_INLINE void *make_block(const ModuleSet &modules, size_t module_index) {
+ if (module_index >= module_count(modules))
+ return nullptr;
+ const Module *loaded = __atomic_load_n(&modules.modules, __ATOMIC_ACQUIRE);
+ const ElfW(Phdr) *tls = loaded[module_index].tls();
+ if (tls == nullptr)
+ return nullptr;
+
+ if (!reserve(module_index + 1))
+ return nullptr;
+ if (module_index >= count_)
+ count_ = module_index + 1;
+
+ // What the kernel gives back is page aligned, which covers any alignment
+ // a thread local can ask for, but the request is padded anyway so that a
+ // larger one would still be met.
+ const size_t alignment =
+ tls->p_align < 1 ? 1 : static_cast<size_t>(tls->p_align);
+ const size_t size = tls->p_memsz + alignment;
+ void *storage = map(size);
+ if (storage == nullptr)
+ return nullptr;
+
+ uintptr_t start = reinterpret_cast<uintptr_t>(storage);
+ start = (start + alignment - 1) & ~(alignment - 1);
+ auto *block = reinterpret_cast<unsigned char *>(start);
+
+ // The .tdata is copied in and the .tbss remainder is left zero, which the
+ // fresh mapping already is.
+ if (tls->p_filesz != 0) {
+ const void *image = reinterpret_cast<const void *>(
+ loaded[module_index].load_bias() + tls->p_vaddr);
+ inline_memcpy(block, image, tls->p_filesz);
+ }
+
+ entries_[module_index] =
+ Entry{storage, size, block,
+ __atomic_load_n(&modules.generation, __ATOMIC_ACQUIRE)};
+ return block;
+ }
+
+ // Indexed the same as the module set, holding what this thread took for
+ // each module that needed it.
+ Entry *entries_ = nullptr;
+ size_t count_ = 0;
+ size_t capacity_ = 0;
+};
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_DTV_H
diff --git a/libc/src/__support/elf/dynamic_blocks.cpp b/libc/src/__support/elf/dynamic_blocks.cpp
new file mode 100644
index 00000000000000..1f3b4b44f7232d
--- /dev/null
+++ b/libc/src/__support/elf/dynamic_blocks.cpp
@@ -0,0 +1,40 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// A thread's blocks for modules opened later
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/elf/dynamic_blocks.h"
+
+#include "src/__support/elf/dtv.h"
+#include "src/__support/elf/passive_abi.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+namespace {
+
+// It is itself a thread local of libc, which was loaded at startup, so it
+// lives in the block every thread already has and costs a thread that never
+// opens anything nothing at all.
+LIBC_THREAD_LOCAL DynamicThreadVector blocks;
+
+} // anonymous namespace
+
+void *dynamic_block_for(size_t module_index) {
+ return blocks.block_for(loaded_modules(), module_index);
+}
+
+void release_dynamic_thread_blocks() { blocks.release(); }
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/src/__support/elf/dynamic_blocks.h b/libc/src/__support/elf/dynamic_blocks.h
new file mode 100644
index 00000000000000..654f449d03d952
--- /dev/null
+++ b/libc/src/__support/elf/dynamic_blocks.h
@@ -0,0 +1,35 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// A thread's blocks for modules opened later
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_DYNAMIC_BLOCKS_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_DYNAMIC_BLOCKS_H
+
+#include "hdr/types/size_t.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// The calling thread's block for `module_index`, allocated and filled in from
+// the module's initial image if this is the first time it has asked. Null if
+// there is no room to be had or the module has no thread local storage.
+void *dynamic_block_for(size_t module_index);
+
+// Gives back what the calling thread allocated for modules opened after it
+// started. Called as the thread ends.
+void release_dynamic_thread_blocks();
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_DYNAMIC_BLOCKS_H
diff --git a/libc/src/__support/elf/map_segments.h b/libc/src/__support/elf/map_segments.h
index 27c3b35064587f..5ef53c45b1a617 100644
--- a/libc/src/__support/elf/map_segments.h
+++ b/libc/src/__support/elf/map_segments.h
@@ -40,6 +40,11 @@ struct MappedModule {
// it lasts exactly as long as the module does. Null where the path was too
// long to keep.
const char *name = nullptr;
+ // The generation of the module set that opening this reached. An index
+ // freed by dlclose goes to the next dlopen, and a thread compares this with
+ // when it made its block for the index to tell whether the block was made
+ // for this module or for one that used to be there.
+ size_t generation = 0;
};
LIBC_INLINE int segment_protection(ElfW(Word) flags) {
diff --git a/libc/src/__support/elf/passive_abi.h b/libc/src/__support/elf/passive_abi.h
index deeba1d7034fdc..c82b17e02c7080 100644
--- a/libc/src/__support/elf/passive_abi.h
+++ b/libc/src/__support/elf/passive_abi.h
@@ -46,6 +46,14 @@ struct ModuleSet {
// Objects loaded at startup begin at one and are never released.
size_t *references;
size_t count;
+ // How many of those were loaded at startup. Those have a place in the block
+ // each thread is given; anything past this was opened later and takes a
+ // block of its own per thread.
+ size_t static_count;
+ // Bumped whenever a module is added or dropped after startup. Each module
+ // opened later records the value its opening reached, and a thread's block
+ // for an index made before then was made for a module that is gone.
+ size_t generation;
size_t capacity;
size_t page_size;
// Set once the startup linker has finished, so a caller can tell a linked
diff --git a/libc/src/__support/elf/startup_linker.h b/libc/src/__support/elf/startup_linker.h
index b1539403f491b9..d852ba596fd2f6 100644
--- a/libc/src/__support/elf/startup_linker.h
+++ b/libc/src/__support/elf/startup_linker.h
@@ -345,6 +345,7 @@ class StartupLinker {
set->references[i] = 1;
}
set->count = n;
+ set->static_count = n;
set->page_size = page_size_;
set->linked = true;
}
diff --git a/libc/src/__support/threads/linux/CMakeLists.txt b/libc/src/__support/threads/linux/CMakeLists.txt
index 2e539d4895835b..7f9f6685ad812f 100644
--- a/libc/src/__support/threads/linux/CMakeLists.txt
+++ b/libc/src/__support/threads/linux/CMakeLists.txt
@@ -72,6 +72,7 @@ add_object_library(
libc.src.__support.CPP.stringstream
libc.src.__support.CPP.string_view
libc.src.__support.common
+ libc.src.__support.elf.dynamic_blocks
libc.src.__support.error_or
libc.src.__support.OSUtil.linux.sigmask
libc.src.__support.OSUtil.linux.syscall_wrappers.clone
diff --git a/libc/src/__support/threads/linux/thread.cpp b/libc/src/__support/threads/linux/thread.cpp
index db8f7ed103c778..75cf723e56b573 100644
--- a/libc/src/__support/threads/linux/thread.cpp
+++ b/libc/src/__support/threads/linux/thread.cpp
@@ -33,6 +33,7 @@
#include "src/__support/OSUtil/linux/syscall_wrappers/write.h"
#include "src/__support/OSUtil/syscall.h" // For syscall functions.
#include "src/__support/common.h"
+#include "src/__support/elf/dynamic_blocks.h"
#include "src/__support/error_or.h"
#include "src/__support/libc_errno.h" // For error macros
#include "src/__support/macros/config.h"
@@ -501,6 +502,11 @@ void thread_exit(ThreadReturnValue retval, ThreadStyle style) {
// be called by the thread which owns them.
internal::call_atexit_callbacks();
+ // Anything this thread allocated for a module opened after it started is
+ // its own, so it goes back now. The callbacks above may have touched such a
+ // module's thread locals, so this comes after them.
+ elf::release_dynamic_thread_blocks();
+
uint32_t joinable_state = uint32_t(DetachState::JOINABLE);
if (!attrib->detach_state.compare_exchange_strong(
joinable_state, uint32_t(DetachState::EXITING))) {
diff --git a/libc/src/dlfcn/dlclose.cpp b/libc/src/dlfcn/dlclose.cpp
index d1672893f53dac..4999e16280b531 100644
--- a/libc/src/dlfcn/dlclose.cpp
+++ b/libc/src/dlfcn/dlclose.cpp
@@ -49,7 +49,9 @@ LLVM_LIBC_FUNCTION(int, dlclose, (void *handle)) {
if (index + 1 == set.count) {
elf::run_fini_array(set.modules[index]);
elf::unmap_module(set.mappings[index]);
- set.count = index;
+ // The index goes back to the next dlopen, which stamps what it puts there
+ // with a later generation than any block a thread made for this one.
+ ++set.generation;
}
return 0;
}
diff --git a/libc/src/dlfcn/dlopen.cpp b/libc/src/dlfcn/dlopen.cpp
index 7028cf588196b3..56a347e258bc96 100644
--- a/libc/src/dlfcn/dlopen.cpp
+++ b/libc/src/dlfcn/dlopen.cpp
@@ -75,22 +75,20 @@ LLVM_LIBC_FUNCTION(void *, dlopen, (const char *path, int)) {
return nullptr;
}
- // Thread local storage in a newly opened object would have to be added to
- // every thread that already exists, which needs a dynamic thread vector.
- // Refusing is better than handing back a module whose thread locals alias
- // something else.
- if (loaded.value().module.tls() != nullptr) {
- elf::unmap_module(loaded.value().mapping);
- dl::set_error("shared object uses thread local storage");
- return nullptr;
- }
-
const size_t index = set.count;
set.modules[index] = loaded.value().module;
set.mappings[index] = loaded.value().mapping;
+ // Nothing loaded now has a place in the block each thread already has, so
+ // there is no offset from a thread pointer to record. Each thread takes a
+ // block of its own for it, the first time it asks.
set.tls_offsets[index] = 0;
set.references[index] = 1;
set.count = index + 1;
+ // A thread may still hold a block for a module that used to be at this
+ // index. Stamping it with the generation this reaches is what tells it that
+ // block is not for what is there now.
+ ++set.generation;
+ set.mappings[index].generation = set.generation;
// Bind against everything now loaded, the new object included, so it can
// refer to itself as well as to what was already there.
diff --git a/libc/src/tls/CMakeLists.txt b/libc/src/tls/CMakeLists.txt
index 6b0986a59aaa5f..1b479b7f642507 100644
--- a/libc/src/tls/CMakeLists.txt
+++ b/libc/src/tls/CMakeLists.txt
@@ -7,6 +7,7 @@ add_entrypoint_object(
DEPENDS
libc.hdr.stdint_proxy
libc.hdr.types.size_t
+ libc.src.__support.elf.dynamic_blocks
libc.src.__support.elf.passive_abi
libc.src.__support.elf.thread_pointer
)
diff --git a/libc/src/tls/tls_get_addr.cpp b/libc/src/tls/tls_get_addr.cpp
index a583eac7e470b8..83600515b3c14b 100644
--- a/libc/src/tls/tls_get_addr.cpp
+++ b/libc/src/tls/tls_get_addr.cpp
@@ -15,8 +15,10 @@
#include "hdr/stdint_proxy.h"
#include "src/__support/common.h"
+#include "src/__support/elf/dynamic_blocks.h"
#include "src/__support/elf/passive_abi.h"
#include "src/__support/elf/thread_pointer.h"
+#include "src/__support/macros/attributes.h"
#include "src/__support/macros/config.h"
namespace LIBC_NAMESPACE_DECL {
@@ -25,28 +27,36 @@ namespace LIBC_NAMESPACE_DECL {
//
// A shared library cannot know at link time where in a thread's blocks its
// own will sit, so its thread locals are compiled to name the module and an
-// offset within that module's block and to call this. The module's offset
-// from the thread pointer is what the loader worked out when it laid the
-// blocks out, and it is in the record the loader left behind.
+// offset within that module's block and to call this.
//
-// Every module here is one loaded at startup, so they all have a place in
-// the one block each thread is given and the answer is a sum. A module
-// brought in later by dlopen would need a block of its own, allocated the
-// first time each thread asked for it; that is not supported yet, and such
-// a module's index is not in the record, so this reports it as nothing
-// rather than reading past the end.
+// A module loaded at startup has a place in the one block each thread is
+// given, at an offset the loader worked out and left behind in its record, so
+// the answer is a sum. One opened later has no such place, because the
+// threads that already exist were given blocks with no room for it, and takes
+// a block of its own that this thread allocates the first time it asks.
LLVM_LIBC_FUNCTION(void *, __tls_get_addr, (TlsIndex * index)) {
if (index == nullptr)
return nullptr;
const elf::ModuleSet &modules = elf::loaded_modules();
- if (modules.tls_offsets == nullptr || index->ti_module >= modules.count)
+ if (index->ti_module >= elf::module_count(modules))
return nullptr;
- const uintptr_t base = elf::thread_pointer();
- const intptr_t offset = modules.tls_offsets[index->ti_module];
- return reinterpret_cast<void *>(base + static_cast<uintptr_t>(offset) +
- index->ti_offset);
+ if (index->ti_module < modules.static_count) {
+ const intptr_t *offsets =
+ __atomic_load_n(&modules.tls_offsets, __ATOMIC_ACQUIRE);
+ if (offsets == nullptr)
+ return nullptr;
+ const uintptr_t base = elf::thread_pointer();
+ const intptr_t offset = offsets[index->ti_module];
+ return reinterpret_cast<void *>(base + static_cast<uintptr_t>(offset) +
+ index->ti_offset);
+ }
+
+ void *block = elf::dynamic_block_for(index->ti_module);
+ if (block == nullptr)
+ return nullptr;
+ return reinterpret_cast<unsigned char *>(block) + index->ti_offset;
}
} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/startup/linux/passive_abi.cpp b/libc/startup/linux/passive_abi.cpp
index b6786fc98da23f..3e2c5b90942c86 100644
--- a/libc/startup/linux/passive_abi.cpp
+++ b/libc/startup/linux/passive_abi.cpp
@@ -38,6 +38,8 @@ LIBC_NAMESPACE::elf::ModuleSet __llvm_libc_process_modules = {
LIBC_NAMESPACE::elf::storage_tls_offsets,
LIBC_NAMESPACE::elf::storage_references,
/*count=*/0,
+ /*static_count=*/0,
+ /*generation=*/1,
LIBC_NAMESPACE::elf::MAX_PROCESS_MODULES,
/*page_size=*/4096,
/*linked=*/false};
diff --git a/libc/test/src/__support/elf/CMakeLists.txt b/libc/test/src/__support/elf/CMakeLists.txt
index c9406f88fe83d6..0a31a5aa726ec8 100644
--- a/libc/test/src/__support/elf/CMakeLists.txt
+++ b/libc/test/src/__support/elf/CMakeLists.txt
@@ -168,3 +168,19 @@ add_libc_test(
libc.src.__support.CPP.string_view
libc.src.__support.elf.run_path
)
+
+if(LIBC_ENABLE_SHARED)
+ add_libc_test(
+ dtv_test
+ SUITE
+ libc-support-tests
+ SRCS
+ dtv_test.cpp
+ DEPENDS
+ libc.src.__support.elf.dtv
+ libc.src.__support.elf.load_module
+ libc.src.__support.elf.passive_abi
+ COMPILE_OPTIONS
+ -DLIBC_TEST_SHARED_OBJECT="$<TARGET_FILE:libc-shared>"
+ )
+endif()
diff --git a/libc/test/src/__support/elf/dtv_test.cpp b/libc/test/src/__support/elf/dtv_test.cpp
new file mode 100644
index 00000000000000..ede8541d4ac605
--- /dev/null
+++ b/libc/test/src/__support/elf/dtv_test.cpp
@@ -0,0 +1,215 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Unittests for a thread's dynamic thread vector
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/elf/dtv.h"
+#include "src/__support/elf/load_module.h"
+#include "src/__support/elf/passive_abi.h"
+#include "test/UnitTest/Test.h"
+
+using LIBC_NAMESPACE::elf::DynamicThreadVector;
+using LIBC_NAMESPACE::elf::load_module;
+using LIBC_NAMESPACE::elf::LoadedModule;
+using LIBC_NAMESPACE::elf::MappedModule;
+using LIBC_NAMESPACE::elf::Module;
+using LIBC_NAMESPACE::elf::ModuleSet;
+using LIBC_NAMESPACE::elf::unmap_module;
+
+namespace {
+
+constexpr size_t PAGE = 4096;
+constexpr const char *LIBC_PATH = LIBC_TEST_SHARED_OBJECT;
+
+// The record is handed in rather than read from the process, which a test
+// binary has no loader to fill in. This stands a module in one past those
+// loaded at startup, the way an object opened later sits.
+class Record {
+public:
+ Record() {
+ set_.modules = modules_;
+ set_.mappings = mappings_;
+ set_.tls_offsets = offsets_;
+ set_.references = references_;
+ set_.capacity = CAPACITY;
+ set_.static_count = 1;
+ set_.count = 2;
+ set_.generation = 1;
+ set_.linked = true;
+ }
+
+ size_t lend(const Module &module) { return lend_at(1, module); }
+
+ // Puts a module at `index` the way dlopen does, stamping the index with the
+ // generation its opening reaches.
+ size_t lend_at(size_t index, const Module &module) {
+ modules_[index] = module;
+ if (index >= set_.count)
+ set_.count = index + 1;
+ ++set_.generation;
+ mappings_[index].generation = set_.generation;
+ return index;
+ }
+
+ // Something opened or closed that has nothing to do with the modules here.
+ void open_elsewhere() { ++set_.generation; }
+
+ const ModuleSet &get() const { return set_; }
+
+private:
+ static constexpr size_t CAPACITY = 4;
+ ModuleSet set_{};
+ Module modules_[CAPACITY];
+ MappedModule mappings_[CAPACITY];
+ intptr_t offsets_[CAPACITY] = {};
+ size_t references_[CAPACITY] = {};
+};
+
+} // anonymous namespace
+
+TEST(LlvmLibcElfDtvTest, NothingForAModuleWithoutTls) {
+ Record record;
+ Module bare;
+ const size_t index = record.lend(bare);
+
+ DynamicThreadVector vector;
+ EXPECT_EQ(vector.block_for(record.get(), index),
+ static_cast<void *>(nullptr));
+ vector.release();
+}
+
+TEST(LlvmLibcElfDtvTest, NothingForAModuleThatIsNotThere) {
+ Record record;
+ Module bare;
+ record.lend(bare);
+
+ DynamicThreadVector vector;
+ EXPECT_EQ(vector.block_for(record.get(), 99), static_cast<void *>(nullptr));
+ vector.release();
+}
+
+TEST(LlvmLibcElfDtvTest, KeepsABlockWhileItsModuleIsLoaded) {
+ auto loaded = load_module(LIBC_PATH, PAGE);
+ ASSERT_TRUE(loaded.has_value());
+ LoadedModule c = loaded.value();
+ ASSERT_TRUE(c.module.tls() != nullptr);
+
+ Record record;
+ const size_t index = record.lend(c.module);
+
+ DynamicThreadVector vector;
+ auto *first =
+ static_cast<unsigned char *>(vector.block_for(record.get(), index));
+ ASSERT_TRUE(first != nullptr);
+ // Asking again is the same block: a thread local keeps its value.
+ EXPECT_EQ(static_cast<void *>(vector.block_for(record.get(), index)),
+ static_cast<void *>(first));
+ first[0] = static_cast<unsigned char>(first[0] + 1);
+ const unsigned char now = first[0];
+
+ // Opening another module and reaching for its block leaves this one where
+ // it is, as does anything else being opened or closed. The address of a
+ // thread local may be kept anywhere, a destructor registered to run as the
+ // thread ends among them.
+ const size_t other = record.lend_at(index + 1, c.module);
+ ASSERT_TRUE(vector.block_for(record.get(), other) != nullptr);
+ record.open_elsewhere();
+ EXPECT_EQ(static_cast<void *>(vector.block_for(record.get(), index)),
+ static_cast<void *>(first));
+ EXPECT_EQ(first[0], now);
+
+ vector.release();
+ unmap_module(c.mapping);
+}
+
+TEST(LlvmLibcElfDtvTest, MakesTheBlockAgainWhenItsIndexIsReused) {
+ auto loaded = load_module(LIBC_PATH, PAGE);
+ ASSERT_TRUE(loaded.has_value());
+ LoadedModule c = loaded.value();
+ ASSERT_TRUE(c.module.tls() != nullptr);
+
+ Record record;
+ const size_t index = record.lend(c.module);
+
+ DynamicThreadVector vector;
+ auto *first =
+ static_cast<unsigned char *>(vector.block_for(record.get(), index));
+ ASSERT_TRUE(first != nullptr);
+ const unsigned char was = first[0];
+ first[0] = static_cast<unsigned char>(was + 1);
+
+ // The module at the index was closed and the index given to one opened
+ // since, so what was kept belonged to the one before and is made again from
+ // the new module's image. The kernel may hand back the same memory, so what
+ // says this happened is the contents rather than the address.
+ record.lend_at(index, c.module);
+ auto *second =
+ static_cast<unsigned char *>(vector.block_for(record.get(), index));
+ ASSERT_TRUE(second != nullptr);
+ EXPECT_EQ(second[0], was);
+
+ vector.release();
+ unmap_module(c.mapping);
+}
+
+TEST(LlvmLibcElfDtvTest, BlockStartsFromTheModulesInitialImage) {
+ auto loaded = load_module(LIBC_PATH, PAGE);
+ ASSERT_TRUE(loaded.has_value());
+ LoadedModule c = loaded.value();
+ const ElfW(Phdr) *tls = c.module.tls();
+ ASSERT_TRUE(tls != nullptr);
+
+ Record record;
+ const size_t index = record.lend(c.module);
+
+ DynamicThreadVector vector;
+ auto *block =
+ static_cast<const unsigned char *>(vector.block_for(record.get(), index));
+ ASSERT_TRUE(block != nullptr);
+
+ // The alignment the segment asks for has to be met.
+ if (tls->p_align > 1)
+ EXPECT_EQ(reinterpret_cast<uintptr_t>(block) % tls->p_align, uintptr_t(0));
+
+ // What was initialised in the module is copied in, and the rest is zero.
+ const auto *image = reinterpret_cast<const unsigned char *>(
+ c.module.load_bias() + tls->p_vaddr);
+ for (size_t i = 0; i < tls->p_filesz; ++i)
+ ASSERT_EQ(block[i], image[i]);
+ for (size_t i = tls->p_filesz; i < tls->p_memsz; ++i)
+ ASSERT_EQ(block[i], static_cast<unsigned char>(0));
+
+ vector.release();
+ unmap_module(c.mapping);
+}
+
+TEST(LlvmLibcElfDtvTest, EachVectorHasItsOwnBlocks) {
+ auto loaded = load_module(LIBC_PATH, PAGE);
+ ASSERT_TRUE(loaded.has_value());
+ LoadedModule c = loaded.value();
+ ASSERT_TRUE(c.module.tls() != nullptr);
+
+ Record record;
+ const size_t index = record.lend(c.module);
+
+ // One of these belongs to each thread, so no two hand out the same block.
+ DynamicThreadVector one;
+ DynamicThreadVector two;
+ void *first = one.block_for(record.get(), index);
+ void *second = two.block_for(record.get(), index);
+ ASSERT_TRUE(first != nullptr);
+ ASSERT_TRUE(second != nullptr);
+ EXPECT_NE(first, second);
+
+ one.release();
+ two.release();
+ unmap_module(c.mapping);
+}
>From 7c51f5c64cc42add44299429f94890cab40c7150 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Sat, 5 Sep 2026 17:19:48 +0100
Subject: [PATCH 42/46] [libc] Look for a shared object the way the loader does
in dlopen
dlopen took what it was given as a path and nothing else, so a program
asking for a library by its SONAME, which is what a program does when it
opens something optional at run time, was told the object could not be
opened even where it was installed. It also loaded only the named object
and nothing that object was itself linked against, so anything with a
dependency failed to bind.
The search the loader performs at startup now lives beside the rest of
the loader support and is used by both.
The loader took DT_RUNPATH and the older DT_RPATH for the same thing and
looked at both before LD_LIBRARY_PATH. They are not the same thing, and
the difference is the only reason the newer tag exists: DT_RPATH cannot
be overridden and DT_RUNPATH can.
What that costs is the ordinary way a library is built and tested.
ncurses links its own tic against the copy it has just built, records
where that copy will be installed, and runs it with LD_LIBRARY_PATH
naming where the copy actually is. Here the recorded path won, so tic
ran against the ncurses already installed, which does not have the
symbols the new tic was linked against, and the load failed. Nothing had
noticed because the package had never been built by the package manager
before.
A module carrying DT_RUNPATH is not using DT_RPATH even where it has
both, so the newer tag turns the older one off.
Assisted-by: Claude
---
.../linux/syscall_wrappers/CMakeLists.txt | 13 +
.../linux/syscall_wrappers/newfstatat.h | 35 +++
libc/src/__support/elf/CMakeLists.txt | 17 ++
libc/src/__support/elf/module.h | 14 ++
libc/src/__support/elf/search.h | 185 ++++++++++++++
libc/src/__support/elf/startup_linker.h | 118 +--------
libc/src/dlfcn/CMakeLists.txt | 3 +
libc/src/dlfcn/dlopen.cpp | 225 ++++++++++++++++--
libc/test/src/__support/elf/CMakeLists.txt | 10 +
.../src/__support/elf/search_paths_test.cpp | 76 ++++++
10 files changed, 563 insertions(+), 133 deletions(-)
create mode 100644 libc/src/__support/OSUtil/linux/syscall_wrappers/newfstatat.h
create mode 100644 libc/src/__support/elf/search.h
create mode 100644 libc/test/src/__support/elf/search_paths_test.cpp
diff --git a/libc/src/__support/OSUtil/linux/syscall_wrappers/CMakeLists.txt b/libc/src/__support/OSUtil/linux/syscall_wrappers/CMakeLists.txt
index e682daff23fb5e..e904b3a98659d6 100644
--- a/libc/src/__support/OSUtil/linux/syscall_wrappers/CMakeLists.txt
+++ b/libc/src/__support/OSUtil/linux/syscall_wrappers/CMakeLists.txt
@@ -1955,3 +1955,16 @@ add_header_library(
libc.src.__support.error_or
libc.src.__support.macros.config
)
+
+add_header_library(
+ newfstatat
+ HDRS
+ newfstatat.h
+ DEPENDS
+ libc.src.__support.OSUtil.osutil
+ libc.src.__support.common
+ libc.src.__support.error_or
+ libc.src.__support.macros.config
+ libc.hdr.types.struct_stat
+ libc.include.sys_syscall
+)
diff --git a/libc/src/__support/OSUtil/linux/syscall_wrappers/newfstatat.h b/libc/src/__support/OSUtil/linux/syscall_wrappers/newfstatat.h
new file mode 100644
index 00000000000000..4afab466da7d7b
--- /dev/null
+++ b/libc/src/__support/OSUtil/linux/syscall_wrappers/newfstatat.h
@@ -0,0 +1,35 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Syscall wrapper for newfstatat.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_OSUTIL_SYSCALL_WRAPPERS_NEWFSTATAT_H
+#define LLVM_LIBC_SRC___SUPPORT_OSUTIL_SYSCALL_WRAPPERS_NEWFSTATAT_H
+
+#include "hdr/types/struct_stat.h"
+#include "src/__support/OSUtil/linux/syscall.h" // syscall_checked
+#include "src/__support/common.h"
+#include "src/__support/error_or.h"
+#include "src/__support/macros/config.h"
+#include <sys/syscall.h> // For syscall numbers
+
+namespace LIBC_NAMESPACE_DECL {
+namespace linux_syscalls {
+
+LIBC_INLINE ErrorOr<int> newfstatat(int dfd, const char *path, struct stat *buf,
+ int flags) {
+ return syscall_checked<int>(SYS_newfstatat, dfd, path, buf, flags);
+}
+
+} // namespace linux_syscalls
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_OSUTIL_SYSCALL_WRAPPERS_NEWFSTATAT_H
diff --git a/libc/src/__support/elf/CMakeLists.txt b/libc/src/__support/elf/CMakeLists.txt
index 26263bc3edb16f..214705693c7998 100644
--- a/libc/src/__support/elf/CMakeLists.txt
+++ b/libc/src/__support/elf/CMakeLists.txt
@@ -162,6 +162,23 @@ add_header_library(
libc.src.__support.macros.config
)
+add_header_library(
+ search
+ HDRS
+ search.h
+ DEPENDS
+ libc.hdr.elf_macros
+ libc.hdr.link_macros
+ libc.hdr.types.size_t
+ libc.src.__support.CPP.optional
+ libc.src.__support.elf.load_module
+ libc.src.__support.elf.module
+ libc.src.__support.elf.run_path
+ libc.src.__support.macros.attributes
+ libc.src.__support.macros.config
+)
+
+
add_header_library(
bind
HDRS
diff --git a/libc/src/__support/elf/module.h b/libc/src/__support/elf/module.h
index ac6c3ab1d4e19d..3a792eca334c7d 100644
--- a/libc/src/__support/elf/module.h
+++ b/libc/src/__support/elf/module.h
@@ -60,6 +60,20 @@ class Module {
}
}
+ // Whether the given address falls in one of the module's mapped segments,
+ // which is how the module some code belongs to is found from a return
+ // address.
+ LIBC_INLINE bool contains(ElfW(Addr) address) const {
+ for (ElfW(Half) i = 0; i < phnum_; ++i) {
+ if (phdrs_[i].p_type != PT_LOAD)
+ continue;
+ const ElfW(Addr) start = load_bias_ + phdrs_[i].p_vaddr;
+ if (address >= start && address < start + phdrs_[i].p_memsz)
+ return true;
+ }
+ return false;
+ }
+
LIBC_INLINE constexpr const char *name() const { return name_; }
LIBC_INLINE constexpr ElfW(Addr) load_bias() const { return load_bias_; }
LIBC_INLINE constexpr const ElfW(Phdr) * phdrs() const { return phdrs_; }
diff --git a/libc/src/__support/elf/search.h b/libc/src/__support/elf/search.h
new file mode 100644
index 00000000000000..28a5f38d456747
--- /dev/null
+++ b/libc/src/__support/elf/search.h
@@ -0,0 +1,185 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Finding a shared object by name
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_SRC___SUPPORT_ELF_SEARCH_H
+#define LLVM_LIBC_SRC___SUPPORT_ELF_SEARCH_H
+
+#include "hdr/elf_macros.h"
+#include "hdr/link_macros.h"
+#include "hdr/types/size_t.h"
+#include "src/__support/CPP/optional.h"
+#include "src/__support/elf/load_module.h"
+#include "src/__support/elf/module.h"
+#include "src/__support/elf/run_path.h"
+#include "src/__support/macros/attributes.h"
+#include "src/__support/macros/config.h"
+
+namespace LIBC_NAMESPACE_DECL {
+namespace elf {
+
+// Where a bare soname is looked for. There is no cache of what is installed:
+// musl manages without one and a format compatible with glibc's would buy
+// nothing here.
+constexpr const char *DEFAULT_SEARCH_PATHS[] = {"/lib", "/usr/lib"};
+
+// The longest path this will build. A name longer than this is one nothing
+// can open anyway.
+constexpr size_t MAX_SEARCH_PATH = 256;
+
+// Joins a directory and a name with a separator between them. Returns false
+// if the result would not fit.
+LIBC_INLINE bool join_path(const char *dir, const char *name, char *out,
+ size_t capacity) {
+ size_t n = 0;
+ for (const char *p = dir; *p != '\0'; ++p) {
+ if (n + 2 >= capacity)
+ return false;
+ out[n++] = *p;
+ }
+ out[n++] = '/';
+ for (const char *p = name; *p != '\0'; ++p) {
+ if (n + 1 >= capacity)
+ return false;
+ out[n++] = *p;
+ }
+ out[n] = '\0';
+ return true;
+}
+
+// The older DT_RPATH, which is looked at before LD_LIBRARY_PATH and so cannot
+// be overridden. A module carrying DT_RUNPATH is not using DT_RPATH at all,
+// even where both are present, so the newer one turns the older one off.
+LIBC_INLINE const char *rpath_of(const Module &module) {
+ const char *strings = module.strtab();
+ if (strings == nullptr)
+ return nullptr;
+ if (module.dynamic().value(DT_RUNPATH))
+ return nullptr;
+ if (auto offset = module.dynamic().value(DT_RPATH))
+ return strings + *offset;
+ return nullptr;
+}
+
+// DT_RUNPATH, which is looked at after LD_LIBRARY_PATH. Being overridable is
+// the whole reason it was added: a program built against libraries that are
+// not installed yet is run with LD_LIBRARY_PATH naming where they actually
+// are, and that has to win over the path recorded for where they will end up.
+LIBC_INLINE const char *runpath_of(const Module &module) {
+ const char *strings = module.strtab();
+ if (strings == nullptr)
+ return nullptr;
+ if (auto offset = module.dynamic().value(DT_RUNPATH))
+ return strings + *offset;
+ return nullptr;
+}
+
+// Tries `name` under each colon separated directory in `list`. `origin` is
+// what $ORIGIN stands for in that list, and is null where the list may not
+// use it.
+LIBC_INLINE cpp::optional<LoadedModule>
+load_from_path_list(const char *list, const char *name, size_t page_size,
+ const char *origin = nullptr) {
+ if (list == nullptr)
+ return cpp::nullopt;
+ for (const char *segment = list; segment != nullptr;) {
+ const char *end = segment;
+ while (*end != '\0' && *end != ':')
+ ++end;
+ char dir[MAX_SEARCH_PATH];
+ const size_t length = static_cast<size_t>(end - segment);
+ if (length > 0 &&
+ expand_run_path(segment, length, origin, dir, sizeof(dir))) {
+ char path[MAX_SEARCH_PATH];
+ if (join_path(dir, name, path, sizeof(path))) {
+ auto loaded = load_module(path, page_size);
+ if (loaded.has_value())
+ return loaded.value();
+ }
+ }
+ segment = (*end == '\0') ? nullptr : end + 1;
+ }
+ return cpp::nullopt;
+}
+
+// Loads the object `name` stands for, in the order a loader is expected to
+// look: a name with a directory in it is taken as it is, and a bare one is
+// looked for under the asking object's DT_RPATH, then under `library_path`,
+// then under its DT_RUNPATH, then under the system directories. The two run
+// paths sit on either side of LD_LIBRARY_PATH, which is what tells them
+// apart.
+//
+// `requester` is the module whose run path applies, or null where none does.
+// `origin` is what $ORIGIN in that run path stands for, or null for the
+// directory of the requester's own name.
+LIBC_INLINE cpp::optional<LoadedModule>
+find_and_load(const char *name, const Module *requester, const char *origin,
+ const char *library_path, size_t page_size) {
+ if (name == nullptr)
+ return cpp::nullopt;
+
+ for (const char *p = name; *p != '\0'; ++p)
+ if (*p == '/') {
+ auto loaded = load_module(name, page_size);
+ if (loaded.has_value())
+ return loaded.value();
+ return cpp::nullopt;
+ }
+
+ if (requester != nullptr && origin == nullptr)
+ origin = requester->name();
+ if (requester != nullptr) {
+ auto loaded =
+ load_from_path_list(rpath_of(*requester), name, page_size, origin);
+ if (loaded.has_value())
+ return loaded;
+ }
+ if (auto loaded = load_from_path_list(library_path, name, page_size))
+ return loaded;
+ if (requester != nullptr) {
+ auto loaded =
+ load_from_path_list(runpath_of(*requester), name, page_size, origin);
+ if (loaded.has_value())
+ return loaded;
+ }
+ for (const char *dir : DEFAULT_SEARCH_PATHS) {
+ char path[MAX_SEARCH_PATH];
+ if (!join_path(dir, name, path, sizeof(path)))
+ continue;
+ auto loaded = load_module(path, page_size);
+ if (loaded.has_value())
+ return loaded.value();
+ }
+ return cpp::nullopt;
+}
+
+// LD_LIBRARY_PATH, without the value being copied anywhere: the environment
+// outlives whoever looks at it.
+LIBC_INLINE const char *library_path_from(char **envp) {
+ if (envp == nullptr)
+ return nullptr;
+ const char key[] = "LD_LIBRARY_PATH=";
+ for (char **e = envp; *e != nullptr; ++e) {
+ const char *p = *e;
+ size_t i = 0;
+ for (; key[i] != '\0' && p[i] == key[i]; ++i)
+ ;
+ if (key[i] == '\0')
+ return p + i;
+ }
+ return nullptr;
+}
+
+} // namespace elf
+} // namespace LIBC_NAMESPACE_DECL
+
+#endif // LLVM_LIBC_SRC___SUPPORT_ELF_SEARCH_H
diff --git a/libc/src/__support/elf/startup_linker.h b/libc/src/__support/elf/startup_linker.h
index d852ba596fd2f6..f83752ee6ba492 100644
--- a/libc/src/__support/elf/startup_linker.h
+++ b/libc/src/__support/elf/startup_linker.h
@@ -24,6 +24,7 @@
#include "src/__support/elf/module.h"
#include "src/__support/elf/passive_abi.h"
#include "src/__support/elf/run_path.h"
+#include "src/__support/elf/search.h"
#include "src/__support/elf/startup_stack.h"
#include "src/__support/elf/thread_pointer.h"
#include "src/__support/elf/tls_block.h"
@@ -35,14 +36,11 @@ namespace elf {
// How many objects one process may load at startup.
constexpr size_t MAX_STARTUP_MODULES = MAX_PROCESS_MODULES;
-// Where a bare SONAME is looked for. There is no ld.so.cache: musl manages
-// without one and a glibc compatible cache format would buy nothing here.
-constexpr const char *DEFAULT_SEARCH_PATHS[] = {"/lib", "/usr/lib"};
class StartupLinker {
public:
LIBC_INLINE StartupLinker(size_t page_size, char **envp)
- : page_size_(page_size), library_path_(find_library_path(envp)) {}
+ : page_size_(page_size), library_path_(library_path_from(envp)) {}
// Loads the executable's dependency graph, binds it, sets up thread local
// storage and runs the initialisers. Returns false with a message already
@@ -156,99 +154,19 @@ class StartupLinker {
return false;
}
- // LD_LIBRARY_PATH, without the value being copied anywhere: the environment
- // outlives the loader.
- LIBC_INLINE static const char *find_library_path(char **envp) {
- if (envp == nullptr)
- return nullptr;
- const char key[] = "LD_LIBRARY_PATH=";
- for (char **e = envp; *e != nullptr; ++e) {
- const char *p = *e;
- size_t i = 0;
- for (; key[i] != '\0' && p[i] == key[i]; ++i)
- ;
- if (key[i] == '\0')
- return p + i;
- }
- return nullptr;
- }
-
- // Tries `name` under each colon separated directory in `list`. `origin` is
- // what $ORIGIN stands for in that list, and is null where the list may not
- // use it.
- LIBC_INLINE bool try_path_list(const char *list, const char *name,
- LoadedModule &out,
- const char *origin = nullptr) {
- if (list == nullptr)
- return false;
- for (const char *segment = list; segment != nullptr;) {
- const char *end = segment;
- while (*end != '\0' && *end != ':')
- ++end;
- char dir[256];
- const size_t length = static_cast<size_t>(end - segment);
- if (length > 0 &&
- expand_run_path(segment, length, origin, dir, sizeof(dir))) {
- char path[256];
- if (join(dir, name, path, sizeof(path))) {
- auto loaded = load_module(path, page_size_);
- if (loaded.has_value()) {
- out = loaded.value();
- return true;
- }
- }
- }
- segment = (*end == '\0') ? nullptr : end + 1;
- }
- return false;
- }
-
- // The search order a loader is expected to use: the requesting object's own
- // run path, then LD_LIBRARY_PATH, then the system directories. `origin` is
- // what $ORIGIN in `from`'s run path stands for, or null for the directory of
- // its own name.
+ // The search order a loader is expected to use, which dlopen uses too.
+ // `origin` is what $ORIGIN in `from`'s run path stands for, or null for the
+ // directory of its own name.
LIBC_INLINE bool load_dependency(const Module &from, const char *origin,
const char *name) {
if (count_ >= MAX_STARTUP_MODULES) {
report("too many shared objects\n");
return false;
}
- for (const char *p = name; *p != '\0'; ++p) {
- if (*p == '/') {
- auto loaded = load_module(name, page_size_);
- if (!loaded.has_value())
- return missing(name);
- return remember(loaded.value());
- }
- }
-
- LoadedModule loaded;
- if (try_path_list(run_path(from), name, loaded,
- origin != nullptr ? origin : from.name()))
- return remember(loaded);
- if (try_path_list(library_path_, name, loaded))
- return remember(loaded);
- for (const char *dir : DEFAULT_SEARCH_PATHS) {
- char path[256];
- if (!join(dir, name, path, sizeof(path)))
- continue;
- auto found = load_module(path, page_size_);
- if (found.has_value())
- return remember(found.value());
- }
- return missing(name);
- }
-
- // DT_RUNPATH, or the older DT_RPATH when a module still carries one.
- LIBC_INLINE static const char *run_path(const Module &module) {
- const char *strings = module.strtab();
- if (strings == nullptr)
- return nullptr;
- if (auto offset = module.dynamic().value(DT_RUNPATH))
- return strings + *offset;
- if (auto offset = module.dynamic().value(DT_RPATH))
- return strings + *offset;
- return nullptr;
+ auto loaded = find_and_load(name, &from, origin, library_path_, page_size_);
+ if (!loaded.has_value())
+ return missing(name);
+ return remember(loaded.value());
}
LIBC_INLINE bool remember(const LoadedModule &loaded) {
@@ -265,24 +183,6 @@ class StartupLinker {
return false;
}
- LIBC_INLINE static bool join(const char *dir, const char *name, char *out,
- size_t capacity) {
- size_t n = 0;
- for (const char *p = dir; *p != '\0'; ++p) {
- if (n + 2 >= capacity)
- return false;
- out[n++] = *p;
- }
- out[n++] = '/';
- for (const char *p = name; *p != '\0'; ++p) {
- if (n + 1 >= capacity)
- return false;
- out[n++] = *p;
- }
- out[n] = '\0';
- return true;
- }
-
LIBC_INLINE bool setup_tls() {
auto block = allocate_tls_block(modules_, count_, tls_offsets_,
/*copy_images=*/false);
diff --git a/libc/src/dlfcn/CMakeLists.txt b/libc/src/dlfcn/CMakeLists.txt
index f97f5905947349..37aec0a604e61a 100644
--- a/libc/src/dlfcn/CMakeLists.txt
+++ b/libc/src/dlfcn/CMakeLists.txt
@@ -51,6 +51,9 @@ add_entrypoint_object(
HDRS
dlopen.h
DEPENDS
+ libc.src.__support.OSUtil.linux.syscall_wrappers.newfstatat
+ libc.src.__support.elf.search
+ libc.src.unistd.environ
libc.src.__support.CPP.mutex
libc.src.__support.elf.passive_abi
libc.src.__support.threads.mutex
diff --git a/libc/src/dlfcn/dlopen.cpp b/libc/src/dlfcn/dlopen.cpp
index 56a347e258bc96..6754d22a4508a0 100644
--- a/libc/src/dlfcn/dlopen.cpp
+++ b/libc/src/dlfcn/dlopen.cpp
@@ -9,13 +9,18 @@
#include "dlopen.h"
#include "src/__support/CPP/mutex.h"
+#include "src/__support/OSUtil/linux/syscall_wrappers/newfstatat.h"
#include "src/__support/common.h"
#include "src/__support/elf/bind.h"
#include "src/__support/elf/load_module.h"
#include "src/__support/elf/passive_abi.h"
+#include "src/__support/elf/search.h"
#include "src/__support/macros/config.h"
#include "src/dlfcn/dl_internal.h"
+#include "src/unistd/environ.h"
+#include <fcntl.h>
+#include <sys/stat.h>
namespace LIBC_NAMESPACE_DECL {
namespace {
@@ -29,6 +34,101 @@ bool same_name(const char *a, const char *b) {
return false;
}
+constexpr size_t PATH_BUFFER = 4096;
+
+// The path with the "." and ".." components and the doubled slashes taken
+// out, without asking the filesystem. Returns false when it does not fit.
+bool normalise(const char *path, char *out) {
+ size_t length = 0;
+ const bool absolute = path[0] == '/';
+ if (absolute)
+ out[length++] = '/';
+ const size_t root = length;
+ for (const char *p = path; *p != '\0';) {
+ while (*p == '/')
+ ++p;
+ const char *start = p;
+ while (*p != '\0' && *p != '/')
+ ++p;
+ const size_t size = static_cast<size_t>(p - start);
+ if (size == 0 || (size == 1 && start[0] == '.'))
+ continue;
+ if (size == 2 && start[0] == '.' && start[1] == '.') {
+ // Back over the last component, unless it is one that stays.
+ size_t end = length;
+ if (end > root && out[end - 1] == '/')
+ --end;
+ size_t begin = end;
+ while (begin > root && out[begin - 1] != '/')
+ --begin;
+ if (begin < end &&
+ !(end - begin == 2 && out[begin] == '.' && out[begin + 1] == '.')) {
+ length = begin;
+ continue;
+ }
+ if (absolute)
+ continue;
+ }
+ if (length + size + 2 >= PATH_BUFFER)
+ return false;
+ if (length > root && out[length - 1] != '/')
+ out[length++] = '/';
+ for (size_t i = 0; i < size; ++i)
+ out[length++] = start[i];
+ }
+ out[length] = '\0';
+ return true;
+}
+
+const char *last_component(const char *path) {
+ const char *last = path;
+ for (const char *p = path; *p != '\0'; ++p)
+ if (*p == '/')
+ last = p + 1;
+ return last;
+}
+
+// Whether two paths name the same file: the same once their "." and ".."
+// are taken out, or, when they end in the same name, the same device and
+// inode, which is what a symbolic link comes to. An object reached by two
+// paths has to be one object, as it holds the program's state of whatever it
+// defines.
+bool same_file(const char *known, const char *wanted) {
+ if (known == nullptr || wanted == nullptr)
+ return false;
+ bool slash = false;
+ for (const char *p = wanted; *p != '\0'; ++p)
+ slash = slash || *p == '/';
+ if (!slash || known[0] != '/')
+ return false;
+ char a[PATH_BUFFER];
+ char b[PATH_BUFFER];
+ if (!normalise(known, a) || !normalise(wanted, b))
+ return false;
+ if (same_name(a, b))
+ return true;
+ if (!same_name(last_component(a), last_component(b)))
+ return false;
+ struct stat one;
+ struct stat two;
+ if (!linux_syscalls::newfstatat(AT_FDCWD, a, &one, 0) ||
+ !linux_syscalls::newfstatat(AT_FDCWD, b, &two, 0))
+ return false;
+ return one.st_dev == two.st_dev && one.st_ino == two.st_ino;
+}
+
+// Whether the set already holds something of this name, by either the path
+// it was opened under, the file that path comes to, or the name it says it
+// goes by.
+bool already_loaded(const elf::ModuleSet &set, const char *name) {
+ for (size_t i = 0; i < set.count; ++i)
+ if (same_name(set.modules[i].name(), name) ||
+ same_file(set.modules[i].name(), name) ||
+ same_name(set.modules[i].soname(), name))
+ return true;
+ return false;
+}
+
void run_init_array(const elf::Module &module) {
auto array = module.dynamic().address(DT_INIT_ARRAY);
auto size = module.dynamic().value(DT_INIT_ARRAYSZ);
@@ -43,6 +143,12 @@ void run_init_array(const elf::Module &module) {
} // anonymous namespace
LLVM_LIBC_FUNCTION(void *, dlopen, (const char *path, int)) {
+ // The run path that applies is the one belonging to whatever called, which
+ // is found from where the call is returning to. Taken before anything else,
+ // so nothing stands between this and the caller.
+ const ElfW(Addr) caller =
+ reinterpret_cast<ElfW(Addr)>(__builtin_return_address(0));
+
elf::ModuleSet &set = elf::loaded_modules();
cpp::lock_guard lock(dl::dl_mutex);
@@ -58,6 +164,7 @@ LLVM_LIBC_FUNCTION(void *, dlopen, (const char *path, int)) {
for (size_t i = 0; i < set.count; ++i) {
if (same_name(set.modules[i].name(), path) ||
+ same_file(set.modules[i].name(), path) ||
same_name(set.modules[i].soname(), path)) {
++set.references[i];
return dl::handle_for(i);
@@ -69,41 +176,111 @@ LLVM_LIBC_FUNCTION(void *, dlopen, (const char *path, int)) {
return nullptr;
}
- auto loaded = elf::load_module(path, set.page_size);
+ const char *library_path =
+ elf::library_path_from(reinterpret_cast<char **>(environ));
+
+ const elf::Module *caller_module = nullptr;
+ size_t caller_index = 0;
+ for (size_t i = 0; i < set.count; ++i) {
+ if (set.modules[i].contains(caller)) {
+ caller_module = &set.modules[i];
+ caller_index = i;
+ break;
+ }
+ }
+
+ // The executable, which is always the first module, takes its run path
+ // against the file it really is, as it did when the process started.
+ char executable_path[elf::MAX_EXECUTABLE_PATH];
+ const char *caller_origin = nullptr;
+ if (caller_module != nullptr && caller_index == 0)
+ caller_origin = elf::executable_origin(
+ caller_module->name(), executable_path, sizeof(executable_path));
+
+ auto loaded = elf::find_and_load(path, caller_module, caller_origin,
+ library_path, set.page_size);
if (!loaded.has_value()) {
dl::set_error(path, "cannot open shared object file");
return nullptr;
}
- const size_t index = set.count;
- set.modules[index] = loaded.value().module;
- set.mappings[index] = loaded.value().mapping;
+ const size_t first = set.count;
+ set.modules[first] = loaded.value().module;
+ set.mappings[first] = loaded.value().mapping;
// Nothing loaded now has a place in the block each thread already has, so
// there is no offset from a thread pointer to record. Each thread takes a
// block of its own for it, the first time it asks.
- set.tls_offsets[index] = 0;
- set.references[index] = 1;
- set.count = index + 1;
- // A thread may still hold a block for a module that used to be at this
- // index. Stamping it with the generation this reaches is what tells it that
- // block is not for what is there now.
- ++set.generation;
- set.mappings[index].generation = set.generation;
-
- // Bind against everything now loaded, the new object included, so it can
- // refer to itself as well as to what was already there.
- elf::SearchOrder order(set.modules, set.count);
- order.set_tls_offsets(set.tls_offsets);
- elf::BindResult result = elf::bind_module(set.modules[index], index, order);
- if (result.unresolved != 0) {
- set.count = index;
- elf::unmap_module(loaded.value().mapping);
- dl::set_error("shared object has undefined symbols");
+ set.tls_offsets[first] = 0;
+ set.references[first] = 1;
+ elf::set_module_count(set, first + 1);
+
+ // What the object was linked against has to be there too, and so does what
+ // that was linked against. Anything already loaded is left where it is and
+ // simply referred to again.
+ bool failed = false;
+ const char *missing = nullptr;
+ for (size_t i = first; i < set.count && !failed; ++i) {
+ const elf::Module requester = set.modules[i];
+ requester.for_each_needed([&](const char *name) {
+ if (failed || already_loaded(set, name))
+ return;
+ if (set.count >= set.capacity) {
+ failed = true;
+ missing = name;
+ return;
+ }
+ auto dependency = elf::find_and_load(name, &requester, nullptr,
+ library_path, set.page_size);
+ if (!dependency.has_value()) {
+ failed = true;
+ missing = name;
+ return;
+ }
+ const size_t at = set.count;
+ set.modules[at] = dependency.value().module;
+ set.mappings[at] = dependency.value().mapping;
+ set.tls_offsets[at] = 0;
+ set.references[at] = 1;
+ elf::set_module_count(set, at + 1);
+ });
+ }
+
+ // Bind against everything now loaded, the new objects included, so they can
+ // refer to themselves and to each other as well as to what was there.
+ elf::BindResult result;
+ if (!failed) {
+ elf::SearchOrder order(set.modules, set.count);
+ order.set_tls_offsets(set.tls_offsets);
+ for (size_t i = first; i < set.count; ++i) {
+ const elf::BindResult one = elf::bind_module(set.modules[i], i, order);
+ result.bound += one.bound;
+ result.unresolved += one.unresolved;
+ }
+ }
+
+ if (failed || result.unresolved != 0) {
+ for (size_t i = set.count; i > first; --i)
+ elf::unmap_module(set.mappings[i - 1]);
+ elf::set_module_count(set, first);
+ ++set.generation;
+ dl::set_error(failed ? "cannot open a shared object it was linked against"
+ : "shared object has undefined symbols");
+ (void)missing;
return nullptr;
}
- run_init_array(set.modules[index]);
- return dl::handle_for(index);
+ // A thread may still hold a block for a module that used to be at one of
+ // these indices. Stamping them with the generation this reaches is what
+ // tells it that block is not for what is there now.
+ ++set.generation;
+ for (size_t i = first; i < set.count; ++i)
+ set.mappings[i].generation = set.generation;
+
+ // Dependencies are initialised before what needs them, so the new modules
+ // are run backwards.
+ for (size_t i = set.count; i > first; --i)
+ run_init_array(set.modules[i - 1]);
+ return dl::handle_for(first);
}
} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/test/src/__support/elf/CMakeLists.txt b/libc/test/src/__support/elf/CMakeLists.txt
index 0a31a5aa726ec8..b6fa2d2f237253 100644
--- a/libc/test/src/__support/elf/CMakeLists.txt
+++ b/libc/test/src/__support/elf/CMakeLists.txt
@@ -169,6 +169,16 @@ add_libc_test(
libc.src.__support.elf.run_path
)
+add_libc_test(
+ search_paths_test
+ SUITE
+ libc-support-tests
+ SRCS
+ search_paths_test.cpp
+ DEPENDS
+ libc.src.__support.elf.search
+)
+
if(LIBC_ENABLE_SHARED)
add_libc_test(
dtv_test
diff --git a/libc/test/src/__support/elf/search_paths_test.cpp b/libc/test/src/__support/elf/search_paths_test.cpp
new file mode 100644
index 00000000000000..daec71feb04e47
--- /dev/null
+++ b/libc/test/src/__support/elf/search_paths_test.cpp
@@ -0,0 +1,76 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// Unittests for which run path a module offers
+///
+//===----------------------------------------------------------------------===//
+
+#include "src/__support/elf/search.h"
+#include "test/UnitTest/Test.h"
+
+#include <link.h>
+
+using LIBC_NAMESPACE::elf::Module;
+using LIBC_NAMESPACE::elf::rpath_of;
+using LIBC_NAMESPACE::elf::runpath_of;
+
+namespace {
+
+const char STRTAB[] = "\0/where/it/will/live\0/where/it/is/now";
+constexpr ElfW(Word) LIVE = 1;
+constexpr ElfW(Word) NOW = 21;
+
+ElfW(Dyn) DYNAMIC[4];
+ElfW(Phdr) PHDRS[1];
+
+// A module carrying whichever of the two tags the caller asks for.
+Module with(bool rpath, bool runpath) {
+ size_t n = 0;
+ if (rpath)
+ DYNAMIC[n++] = {DT_RPATH, {NOW}};
+ if (runpath)
+ DYNAMIC[n++] = {DT_RUNPATH, {LIVE}};
+ DYNAMIC[n++] = {DT_STRTAB, {reinterpret_cast<ElfW(Addr)>(STRTAB)}};
+ DYNAMIC[n++] = {DT_NULL, {0}};
+
+ PHDRS[0].p_type = PT_DYNAMIC;
+ PHDRS[0].p_vaddr = reinterpret_cast<ElfW(Addr)>(DYNAMIC);
+ PHDRS[0].p_memsz = n * sizeof(ElfW(Dyn));
+ return Module(PHDRS, 1, /*load_bias=*/0, "libtest.so.1");
+}
+
+} // anonymous namespace
+
+TEST(LlvmLibcElfSearchPathsTest, NeitherTagOffersNothing) {
+ Module m = with(/*rpath=*/false, /*runpath=*/false);
+ ASSERT_TRUE(rpath_of(m) == nullptr);
+ ASSERT_TRUE(runpath_of(m) == nullptr);
+}
+
+TEST(LlvmLibcElfSearchPathsTest, TheOlderTagOnItsOwn) {
+ Module m = with(/*rpath=*/true, /*runpath=*/false);
+ ASSERT_STREQ(rpath_of(m), STRTAB + NOW);
+ ASSERT_TRUE(runpath_of(m) == nullptr);
+}
+
+TEST(LlvmLibcElfSearchPathsTest, TheNewerTagOnItsOwn) {
+ Module m = with(/*rpath=*/false, /*runpath=*/true);
+ ASSERT_TRUE(rpath_of(m) == nullptr);
+ ASSERT_STREQ(runpath_of(m), STRTAB + LIVE);
+}
+
+// A module carrying both is not using the older one at all. Only the newer
+// one is offered, and it is looked at after LD_LIBRARY_PATH rather than
+// before it, so what the two are called apart decides whether the caller can
+// override the path.
+TEST(LlvmLibcElfSearchPathsTest, TheNewerTagTurnsTheOlderOneOff) {
+ Module m = with(/*rpath=*/true, /*runpath=*/true);
+ ASSERT_TRUE(rpath_of(m) == nullptr);
+ ASSERT_STREQ(runpath_of(m), STRTAB + LIVE);
+}
>From a0de333c4a62971f36cc8dc4e01723f14fa47616 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Sat, 5 Sep 2026 17:40:56 +0100
Subject: [PATCH 43/46] [libc] Report every loaded module from dl_iterate_phdr
dl_iterate_phdr reported the main program and the vDSO only, so a caller
walking the modules to find which one an address belongs to was told
about none of the shared libraries. An unwinder is such a caller: it
locates frame information for a return address this way, so a C++
exception thrown across a library boundary went uncaught and the program
was terminated.
Where a loader linked the process it leaves a record of everything it
loaded, and that record is now walked.
Assisted-by: Claude
---
libc/src/link/CMakeLists.txt | 3 ++
libc/src/link/dl_iterate_phdr.cpp | 66 +++++++++++++++++++++++++++----
2 files changed, 61 insertions(+), 8 deletions(-)
diff --git a/libc/src/link/CMakeLists.txt b/libc/src/link/CMakeLists.txt
index 19c1ba645840ff..ee4aa6c8ba70ea 100644
--- a/libc/src/link/CMakeLists.txt
+++ b/libc/src/link/CMakeLists.txt
@@ -17,6 +17,9 @@ add_entrypoint_object(
HDRS
dl_iterate_phdr.h
DEPENDS
+ libc.src.__support.CPP.mutex
+ libc.src.__support.elf.passive_abi
+ libc.src.dlfcn.dl_internal
libc.hdr.stdint_proxy
libc.include.elf
libc.include.llvm-libc-macros.link_macros
diff --git a/libc/src/link/dl_iterate_phdr.cpp b/libc/src/link/dl_iterate_phdr.cpp
index 01d11de3673e9b..e98e225a6e066f 100644
--- a/libc/src/link/dl_iterate_phdr.cpp
+++ b/libc/src/link/dl_iterate_phdr.cpp
@@ -17,7 +17,9 @@
#include "src/__support/CPP/mutex.h"
#include "src/__support/OSUtil/linux/auxv.h"
#include "src/__support/common.h"
+#include "src/__support/elf/passive_abi.h"
#include "src/__support/macros/config.h"
+#include "src/dlfcn/dl_internal.h"
#include <elf.h>
@@ -52,16 +54,64 @@ struct dl_phdr_info create_module_info(ElfW(Ehdr) * header, const char *name) {
return to_return;
}
+// What a module's own headers say about it, for one the loader recorded. It
+// already knows where the headers are and where the module was put, so
+// nothing has to be worked out again.
+static struct dl_phdr_info info_for(const elf::Module &module,
+ const elf::ModuleSet &set) {
+ struct dl_phdr_info to_return;
+ to_return.dlpi_phdr = module.phdrs();
+ to_return.dlpi_phnum = module.phnum();
+ to_return.dlpi_addr = module.load_bias();
+ // The path it was loaded from, as glibc and musl report it. A caller such
+ // as a JVM finds its own library's directory from this.
+ const char *soname = module.soname();
+ to_return.dlpi_name = module.name() != nullptr ? module.name() : soname;
+ to_return.dlpi_adds = set.generation;
+ to_return.dlpi_subs = 0;
+ to_return.dlpi_tls_modid = 0;
+ to_return.dlpi_tls_data = nullptr;
+ return to_return;
+}
+
LLVM_LIBC_FUNCTION(int, dl_iterate_phdr,
(__dl_iterate_phdr_callback_t callback, void *arg)) {
- ElfW(Ehdr) *executable_header = reinterpret_cast<ElfW(Ehdr) *>(&__ehdr_start);
- // The main executable is always reported with an empty name.
- struct dl_phdr_info executable_info =
- create_module_info(executable_header, "");
- int executable_return_code =
- callback(&executable_info, sizeof(executable_info), arg);
- if (executable_return_code != 0)
- return executable_return_code;
+ // Where a loader linked the process, what it loaded is what there is to
+ // report, and it left a record of all of it. An unwinder walks this to find
+ // the frame information of whatever module a return address is in, so a
+ // library left out of it is one nothing can unwind through.
+ const elf::ModuleSet *modules = elf::process_modules();
+ if (modules != nullptr && modules->linked) {
+ // dlopen and dlclose change the set under this lock, and a library that
+ // is unmapped or an array that is replaced part way through the walk
+ // would be read after it was gone. The load lock is not used, as it is
+ // held while the initialisers of a new library run, and one of them can
+ // be waiting for a thread that is here to unwind.
+ cpp::lock_guard lock(dl::dl_write_lock);
+ for (size_t i = 0; i < modules->count; ++i) {
+ struct dl_phdr_info info = info_for(modules->modules[i], *modules);
+ // The main executable is reported with an empty name, which is how a
+ // caller tells it from everything else. Its own headers cannot be
+ // reached from here: the linker defined symbol naming them means this
+ // library's, not the program's, so the record is what says where they
+ // are.
+ if (i == 0)
+ info.dlpi_name = "";
+ const int code = callback(&info, sizeof(info), arg);
+ if (code != 0)
+ return code;
+ }
+ } else {
+ // Nothing linked this, so the executable is all there is besides what the
+ // kernel mapped.
+ ElfW(Ehdr) *executable_header =
+ reinterpret_cast<ElfW(Ehdr) *>(&__ehdr_start);
+ struct dl_phdr_info executable_info =
+ create_module_info(executable_header, "");
+ const int code = callback(&executable_info, sizeof(executable_info), arg);
+ if (code != 0)
+ return code;
+ }
cpp::optional<unsigned long> vdso_start_address = auxv::get(AT_SYSINFO_EHDR);
if (!vdso_start_address)
>From 32ab803e9085edf37220ffd6f2df3d0af51afa29 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Sun, 6 Sep 2026 06:31:24 +0100
Subject: [PATCH 44/46] [libc] Let dlopen grow the module set
The set of loaded modules was a fixed block of 64 entries and dlopen
failed with "too many shared objects are open" once it was full. A
Python interpreter loads about sixty extension modules and runs out
partway through importing them.
The startup linker's storage has to be fixed because it runs before
there is an allocator to ask, but dlopen does not: it moves the set onto
the heap the first time it would overflow and doubles it from there. The
set is reached through pointers already, so the arrays can be replaced;
a flag records whether they are the block libc starts with or one a
previous growth allocated, so only the latter is released.
Sixty-four was not enough to start a Qt program: libQt6Quick alone brings
fifty-eight before the program's own are counted.
Assisted-by: Claude
---
libc/src/__support/elf/passive_abi.h | 12 +++--
libc/src/dlfcn/dlopen.cpp | 71 ++++++++++++++++++++++++++--
2 files changed, 77 insertions(+), 6 deletions(-)
diff --git a/libc/src/__support/elf/passive_abi.h b/libc/src/__support/elf/passive_abi.h
index c82b17e02c7080..f7cd2e6e06ee01 100644
--- a/libc/src/__support/elf/passive_abi.h
+++ b/libc/src/__support/elf/passive_abi.h
@@ -24,9 +24,15 @@
namespace LIBC_NAMESPACE_DECL {
namespace elf {
-// How many objects a process may have loaded at once, startup and dlopen
-// together.
-constexpr size_t MAX_PROCESS_MODULES = 64;
+// How many objects the startup linker can load. It runs before there is an
+// allocator to ask, so its storage is fixed and lives on the stack the
+// kernel handed the process. dlopen grows the set past this onto the heap.
+//
+// A Qt program is already past sixty libraries before any of its own are
+// counted, and a desktop built on it goes further (Spectacle links nearly
+// three hundred), so the room here is for that rather than for the handful a
+// small program needs.
+constexpr size_t MAX_PROCESS_MODULES = 512;
// The description of the loaded module set.
//
diff --git a/libc/src/dlfcn/dlopen.cpp b/libc/src/dlfcn/dlopen.cpp
index 6754d22a4508a0..c7072de88b4e0a 100644
--- a/libc/src/dlfcn/dlopen.cpp
+++ b/libc/src/dlfcn/dlopen.cpp
@@ -8,6 +8,8 @@
#include "dlopen.h"
+#include "hdr/func/free.h"
+#include "hdr/func/malloc.h"
#include "src/__support/CPP/mutex.h"
#include "src/__support/OSUtil/linux/syscall_wrappers/newfstatat.h"
#include "src/__support/common.h"
@@ -129,6 +131,68 @@ bool already_loaded(const elf::ModuleSet &set, const char *name) {
return false;
}
+// Makes room for one more module, moving the set onto the heap once the fixed
+// block the startup linker filled in is full. The arrays are reached through
+// pointers precisely so that they can be replaced here; nothing that holds
+// the load lock may hold a pointer into them across this call.
+//
+// The arrays that were replaced are not freed. __tls_get_addr and the
+// creation of a thread read them without a lock, and one of those may be
+// part way through when this runs. They are left as they were, which is
+// still right for what such a reader asks, and cost no more than the arrays
+// that replaced them, since each is half the size of the next.
+bool make_room(elf::ModuleSet &set) {
+ if (set.count < set.capacity)
+ return true;
+
+ const size_t capacity = set.capacity * 2;
+ auto *modules =
+ static_cast<elf::Module *>(malloc(capacity * sizeof(elf::Module)));
+ auto *mappings = static_cast<elf::MappedModule *>(
+ malloc(capacity * sizeof(elf::MappedModule)));
+ auto *tls_offsets =
+ static_cast<intptr_t *>(malloc(capacity * sizeof(intptr_t)));
+ auto *references = static_cast<size_t *>(malloc(capacity * sizeof(size_t)));
+ if (modules == nullptr || mappings == nullptr || tls_offsets == nullptr ||
+ references == nullptr) {
+ free(modules);
+ free(mappings);
+ free(tls_offsets);
+ free(references);
+ return false;
+ }
+
+ for (size_t i = 0; i < set.count; ++i) {
+ modules[i] = set.modules[i];
+ mappings[i] = set.mappings[i];
+ tls_offsets[i] = set.tls_offsets[i];
+ references[i] = set.references[i];
+ }
+
+ __atomic_store_n(&set.modules, modules, __ATOMIC_RELEASE);
+ __atomic_store_n(&set.mappings, mappings, __ATOMIC_RELEASE);
+ __atomic_store_n(&set.tls_offsets, tls_offsets, __ATOMIC_RELEASE);
+ set.references = references;
+ set.capacity = capacity;
+ return true;
+}
+
+// Holds the write lock for as long as the module set is being changed, and no
+// longer: the initialisers of what was opened run after it is let go.
+class WriteGuard {
+public:
+ WriteGuard() { dl::dl_write_lock.lock(); }
+ ~WriteGuard() { release(); }
+ void release() {
+ if (held_)
+ dl::dl_write_lock.unlock();
+ held_ = false;
+ }
+
+private:
+ bool held_ = true;
+};
+
void run_init_array(const elf::Module &module) {
auto array = module.dynamic().address(DT_INIT_ARRAY);
auto size = module.dynamic().value(DT_INIT_ARRAYSZ);
@@ -171,8 +235,9 @@ LLVM_LIBC_FUNCTION(void *, dlopen, (const char *path, int)) {
}
}
- if (set.count >= set.capacity) {
- dl::set_error("too many shared objects are open");
+ WriteGuard write;
+ if (!make_room(set)) {
+ dl::set_error("cannot make room for another shared object");
return nullptr;
}
@@ -224,7 +289,7 @@ LLVM_LIBC_FUNCTION(void *, dlopen, (const char *path, int)) {
requester.for_each_needed([&](const char *name) {
if (failed || already_loaded(set, name))
return;
- if (set.count >= set.capacity) {
+ if (!make_room(set)) {
failed = true;
missing = name;
return;
>From f864a8c28edabb6f753990fd5e3e6dc9c51cd751 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Sun, 6 Sep 2026 11:59:08 +0100
Subject: [PATCH 45/46] [libc] Implement dladdr
It was a stub which filled nothing in and answered -1, which a caller
reads as success, so anything asking what an address belongs to was
handed an uninitialised Dl_info.
Everything it needs is already recorded: the module set says which
object an address falls in, and that module's symbol table says which
symbol is defined closest below it. The count of symbols is worked out
from the GNU hash table, since the section header is not mapped.
Failure is reported with zero and dlerror is left alone, which is what
every other implementation does with this one.
The module set the loader keeps is a weak symbol, so that a program
built against the static library links without one. Where nothing
defines it the reference resolves to nothing, and the four calls that
read it took the address of nothing and dereferenced it: any statically
linked program calling dlopen stopped on a fault rather than being told
there is no loader.
The record already has an accessor that hands back a pointer for exactly
this reason, and the callers now go through it and check.
backtrace calls dlopen looking for an unwinder, which is how this was
found: a program with no unwinder and no loader crashed inside the call
that was meant to report that there is none.
Assisted-by: Claude
---
libc/src/__support/elf/symbol.h | 29 +++++++++++++
libc/src/dlfcn/CMakeLists.txt | 3 ++
libc/src/dlfcn/dladdr.cpp | 75 +++++++++++++++++++++++++++++++--
libc/src/dlfcn/dlclose.cpp | 5 ++-
libc/src/dlfcn/dlopen.cpp | 8 +++-
libc/src/dlfcn/dlsym.cpp | 5 ++-
6 files changed, 115 insertions(+), 10 deletions(-)
diff --git a/libc/src/__support/elf/symbol.h b/libc/src/__support/elf/symbol.h
index 3be6d15cf6525e..1a946ac19271c3 100644
--- a/libc/src/__support/elf/symbol.h
+++ b/libc/src/__support/elf/symbol.h
@@ -81,6 +81,35 @@ class GnuSymbolTable {
}
}
+ // How many symbols .dynsym holds. The section header is not mapped, so the
+ // count is worked out from the hash table: every symbol at or above
+ // symoffset is on exactly one chain, and the chains end at the highest
+ // index there is.
+ LIBC_INLINE uint32_t count() const {
+ if (header_ == nullptr)
+ return 0;
+ uint32_t last = header_->symoffset;
+ for (uint32_t i = 0; i < header_->nbuckets; ++i)
+ if (buckets_[i] > last)
+ last = buckets_[i];
+ if (last < header_->symoffset)
+ return header_->symoffset;
+ // Walk to the end of the chain the highest bucket starts.
+ while ((chain_[last - header_->symoffset] & 1) == 0)
+ ++last;
+ return last + 1;
+ }
+
+ LIBC_INLINE const ElfW(Sym) * symbol(uint32_t index) const {
+ return symtab_ == nullptr ? nullptr : &symtab_[index];
+ }
+
+ LIBC_INLINE const char *name(uint32_t index) const {
+ if (symtab_ == nullptr || strtab_ == nullptr)
+ return nullptr;
+ return strtab_ + symtab_[index].st_name;
+ }
+
private:
LIBC_INLINE bool bloom_may_contain(uint32_t hash) const {
if (header_->bloom_size == 0)
diff --git a/libc/src/dlfcn/CMakeLists.txt b/libc/src/dlfcn/CMakeLists.txt
index 37aec0a604e61a..a91a3becfcfe42 100644
--- a/libc/src/dlfcn/CMakeLists.txt
+++ b/libc/src/dlfcn/CMakeLists.txt
@@ -95,6 +95,9 @@ add_entrypoint_object(
HDRS
dladdr.h
DEPENDS
+ .dl_internal
libc.include.dlfcn
libc.hdr.types.dl_info
+ libc.src.__support.CPP.mutex
+ libc.src.__support.elf.passive_abi
)
diff --git a/libc/src/dlfcn/dladdr.cpp b/libc/src/dlfcn/dladdr.cpp
index e61f72c3e86403..43ede115619657 100644
--- a/libc/src/dlfcn/dladdr.cpp
+++ b/libc/src/dlfcn/dladdr.cpp
@@ -13,16 +13,83 @@
#include "dladdr.h"
+#include "src/__support/CPP/mutex.h"
#include "src/__support/common.h"
+#include "src/__support/elf/passive_abi.h"
#include "src/__support/macros/config.h"
+#include "src/dlfcn/dl_internal.h"
namespace LIBC_NAMESPACE_DECL {
-// TODO: https:// github.com/llvm/llvm-project/issues/97929
+namespace {
+
+// The symbol an address belongs to is the one defined closest below it. A
+// symbol with a size covers only that much; one with none, which is what an
+// assembly routine often has, covers up to whatever is defined next.
+const ElfW(Sym) * enclosing_symbol(const elf::Module &module,
+ ElfW(Addr) address, const char **name) {
+ const elf::GnuSymbolTable table = module.symbols();
+ const uint32_t count = table.count();
+ const ElfW(Sym) *best = nullptr;
+ ElfW(Addr) best_value = 0;
+
+ for (uint32_t i = 0; i < count; ++i) {
+ const ElfW(Sym) *symbol = table.symbol(i);
+ if (symbol->st_shndx == SHN_UNDEF || symbol->st_name == 0)
+ continue;
+ const ElfW(Addr) value = module.load_bias() + symbol->st_value;
+ if (value > address)
+ continue;
+ if (symbol->st_size != 0 && address >= value + symbol->st_size)
+ continue;
+ if (best == nullptr || value > best_value) {
+ best = symbol;
+ best_value = value;
+ *name = table.name(i);
+ }
+ }
+ return best;
+}
+
+} // anonymous namespace
+
+// Says which object an address is in, and which symbol of it. Unlike the rest
+// of dlfcn this reports failure with zero rather than with a message, and it
+// leaves dlerror alone, which is what every other implementation does.
LLVM_LIBC_FUNCTION(int, dladdr,
- ([[maybe_unused]] const void *__restrict addr,
- [[maybe_unused]] Dl_info *__restrict info)) {
- return 0; // Returns 0 on fail, nonzero on success.
+ (const void *__restrict addr, Dl_info *__restrict info)) {
+ if (info == nullptr)
+ return 0;
+
+ elf::ModuleSet *modules = elf::process_modules();
+ cpp::lock_guard lock(dl::dl_mutex);
+ if (modules == nullptr || !modules->linked)
+ return 0;
+ elf::ModuleSet &set = *modules;
+
+ const ElfW(Addr) address = reinterpret_cast<ElfW(Addr)>(addr);
+ const elf::Module *module = nullptr;
+ for (size_t i = 0; i < set.count; ++i) {
+ if (set.modules[i].contains(address)) {
+ module = &set.modules[i];
+ break;
+ }
+ }
+ if (module == nullptr)
+ return 0;
+
+ info->dli_fname = module->name();
+ info->dli_fbase = reinterpret_cast<void *>(module->load_bias());
+ info->dli_sname = nullptr;
+ info->dli_saddr = nullptr;
+
+ const char *name = nullptr;
+ if (const ElfW(Sym) *symbol = enclosing_symbol(*module, address, &name)) {
+ info->dli_sname = name;
+ info->dli_saddr =
+ reinterpret_cast<void *>(module->load_bias() + symbol->st_value);
+ }
+ return 1;
}
} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/src/dlfcn/dlclose.cpp b/libc/src/dlfcn/dlclose.cpp
index 4999e16280b531..b429f8c2c767f8 100644
--- a/libc/src/dlfcn/dlclose.cpp
+++ b/libc/src/dlfcn/dlclose.cpp
@@ -18,14 +18,15 @@
namespace LIBC_NAMESPACE_DECL {
LLVM_LIBC_FUNCTION(int, dlclose, (void *handle)) {
- elf::ModuleSet &set = elf::loaded_modules();
+ elf::ModuleSet *modules = elf::process_modules();
cpp::lock_guard lock(dl::dl_mutex);
size_t index = 0;
- if (!set.linked || !dl::index_for(handle, index)) {
+ if (modules == nullptr || !modules->linked || !dl::index_for(handle, index)) {
dl::set_error("dlclose was given a handle that dlopen did not return");
return -1;
}
+ elf::ModuleSet &set = *modules;
if (set.references[index] == 0) {
dl::set_error("shared object is already closed");
return -1;
diff --git a/libc/src/dlfcn/dlopen.cpp b/libc/src/dlfcn/dlopen.cpp
index c7072de88b4e0a..c26d61eb7cd84e 100644
--- a/libc/src/dlfcn/dlopen.cpp
+++ b/libc/src/dlfcn/dlopen.cpp
@@ -213,15 +213,19 @@ LLVM_LIBC_FUNCTION(void *, dlopen, (const char *path, int)) {
const ElfW(Addr) caller =
reinterpret_cast<ElfW(Addr)>(__builtin_return_address(0));
- elf::ModuleSet &set = elf::loaded_modules();
+ // A statically linked program has no module set at all: the record is
+ // weak and nothing defines it, so the pointer is null rather than a set
+ // that says it is not linked.
+ elf::ModuleSet *modules = elf::process_modules();
cpp::lock_guard lock(dl::dl_mutex);
// Without a loader there is no module set to add to, and nothing the caller
// opened could be linked against the program.
- if (!set.linked) {
+ if (modules == nullptr || !modules->linked) {
dl::set_error("dlopen is only available in a dynamically linked process");
return nullptr;
}
+ elf::ModuleSet &set = *modules;
// A null path asks for a handle to the main program.
if (path == nullptr)
return dl::handle_for(0);
diff --git a/libc/src/dlfcn/dlsym.cpp b/libc/src/dlfcn/dlsym.cpp
index 74487f08dbe06c..49e7bced1672ea 100644
--- a/libc/src/dlfcn/dlsym.cpp
+++ b/libc/src/dlfcn/dlsym.cpp
@@ -19,13 +19,14 @@ namespace LIBC_NAMESPACE_DECL {
LLVM_LIBC_FUNCTION(void *, dlsym,
(void *__restrict handle, const char *__restrict name)) {
- elf::ModuleSet &set = elf::loaded_modules();
+ elf::ModuleSet *modules = elf::process_modules();
cpp::lock_guard lock(dl::dl_mutex);
- if (!set.linked) {
+ if (modules == nullptr || !modules->linked) {
dl::set_error("dlsym is only available in a dynamically linked process");
return nullptr;
}
+ elf::ModuleSet &set = *modules;
if (name == nullptr) {
dl::set_error("dlsym was given no symbol name");
return nullptr;
>From a3f66656798e0d012191e1932bce1fd8dc247ff9 Mon Sep 17 00:00:00 2001
From: Mike Lothian <mike at fireburn.co.uk>
Date: Sun, 6 Sep 2026 12:24:46 +0100
Subject: [PATCH 46/46] [libc] Make dlsym understand RTLD_DEFAULT and RTLD_NEXT
Both were rejected as handles dlopen had not returned, so a lookup
through either always came back with nothing. That is how a program asks
whether an optional symbol is there at all, and how a wrapper reaches
the definition it stands in front of, and neither worked.
RTLD_DEFAULT searches every module in the order they were loaded.
RTLD_NEXT searches the ones after whatever called, which is found from
the address the call returns to, the same way dlopen finds the run path
that applies.
Rust's standard library probes for posix_spawn_file_actions_addchdir_np
this way. Never finding it, it fell back to fork and exec for any
command with a working directory set, and cargo deadlocked in the child.
Assisted-by: Claude
---
libc/src/dlfcn/dlsym.cpp | 54 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 54 insertions(+)
diff --git a/libc/src/dlfcn/dlsym.cpp b/libc/src/dlfcn/dlsym.cpp
index 49e7bced1672ea..d508fab9c7180d 100644
--- a/libc/src/dlfcn/dlsym.cpp
+++ b/libc/src/dlfcn/dlsym.cpp
@@ -8,6 +8,7 @@
#include "dlsym.h"
+#include "hdr/stdint_proxy.h"
#include "src/__support/CPP/mutex.h"
#include "src/__support/common.h"
#include "src/__support/elf/bind.h"
@@ -17,8 +18,36 @@
namespace LIBC_NAMESPACE_DECL {
+namespace {
+
+// <dlfcn.h> states the two of these as values rather than in a header this
+// can include, so they are named here rather than spelled out at the tests
+// below.
+constexpr void *DEFAULT_HANDLE = nullptr;
+constexpr uintptr_t NEXT_HANDLE = static_cast<uintptr_t>(-1l);
+
+// The first module at or after `from` which defines `name`. Returns nothing
+// when none does.
+void *search_from(const elf::ModuleSet &set, size_t from, const char *name) {
+ for (size_t i = from; i < set.count; ++i) {
+ const ElfW(Sym) *symbol = set.modules[i].symbols().lookup(name);
+ if (symbol == nullptr || symbol->st_shndx == SHN_UNDEF)
+ continue;
+ return reinterpret_cast<void *>(set.modules[i].load_bias() +
+ symbol->st_value);
+ }
+ return nullptr;
+}
+
+} // anonymous namespace
+
LLVM_LIBC_FUNCTION(void *, dlsym,
(void *__restrict handle, const char *__restrict name)) {
+ // Which module the call is returning to, for RTLD_NEXT. Taken before
+ // anything else, so nothing stands between this and the caller.
+ const ElfW(Addr) caller =
+ reinterpret_cast<ElfW(Addr)>(__builtin_return_address(0));
+
elf::ModuleSet *modules = elf::process_modules();
cpp::lock_guard lock(dl::dl_mutex);
@@ -32,6 +61,31 @@ LLVM_LIBC_FUNCTION(void *, dlsym,
return nullptr;
}
+ // Every module, in the order they were loaded, which is the order a lookup
+ // through the program's own scope goes in.
+ if (handle == DEFAULT_HANDLE) {
+ if (void *found = search_from(set, 0, name))
+ return found;
+ dl::set_error("symbol not found");
+ return nullptr;
+ }
+
+ // Everything after whatever called, which is how a wrapper reaches the
+ // definition it is standing in front of.
+ if (reinterpret_cast<uintptr_t>(handle) == NEXT_HANDLE) {
+ size_t after = set.count;
+ for (size_t i = 0; i < set.count; ++i) {
+ if (set.modules[i].contains(caller)) {
+ after = i + 1;
+ break;
+ }
+ }
+ if (void *found = search_from(set, after, name))
+ return found;
+ dl::set_error("symbol not found");
+ return nullptr;
+ }
+
size_t index = 0;
if (!dl::index_for(handle, index)) {
dl::set_error("dlsym was given a handle that dlopen did not return");
More information about the libcxx-commits
mailing list