[libc-commits] [libc] [libc] Add UTF-16 support in mbrtowc, wcrtomb, etc. (PR #230917)
via libc-commits
libc-commits at lists.llvm.org
Sun Oct 11 08:22:50 PDT 2026
llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-libc
Author: yangbowen
<details>
<summary>Changes</summary>
This is a follow-up. This PR is based on #<!-- -->230434.
This pull request adds support for UTF-16 `wchar_t` in the conversion functions `mbrtowc` `mbtowc` `mbsnrtowcs` `mbsrtowcs` `mbstowcs` `wcrtomb` `wctomb` `wcsnrtombs` `wcsrtombs` `wcstombs` and also related functions `mbrlen` `mblen`.
It also generalizes the implementation of `mbrtowc` and `wcrtomb`, in a way that makes implementation of `mbrtoc16`, `mbrtoc32`, `c16rtomb`, `c32rtomb` trivial in the future.
Note that, due to UTF-16 not actually conforming to the "wide char" semantics in the standard, we have to introduce some deliberate non-conformances and inconsistencies. They are all related to surrogate pairs.
- `mbrtowc` `mbtowc` `mbrlen` `mblen` These functions write one `wchar_t` each call, and have no return value to indicate that "one more `wchar_t` needs to be output to encode this multi-byte character". Thus, these functions fail with `EILSEQ` upon any non-BMP character (>U+FFFF, 4 byte UTF-8).
- `wcrtomb` `wctomb` These functions read one `wchar_t` each call, and have no return value to indicate that "one more `wchar_t` is needed before this character is complete". Also, we keep them symmetric to `mbrtowc` `mbtowc`. Thus, these functions also fail with `EILSEQ` upon any surrogate (0xD800-0xDFFF).
- `mbsnrtowcs` `mbsrtowcs` `mbstowcs` `wcsnrtombs` `wcsrtombs` `wcstombs` These functions also cannot indicate "a UTF-16 surrogate in the state". When a complete surrogate pair is encountered in the string, these functions proceed to convert them.
However, when a lone surrogate appears at the end of the string, these functions fail with `EILSEQ`, as if the surrogate is an invalid character.
We believe that this behavior is mostly identical to that of Windows UCRT, except that:
- For `mbrtowc` `mbtowc`, UCRT accepts a non-BMP character but writes `U+FFFD`, silently replacing the character. We fail with `EILSEQ`.
- For `mbrlen` `mblen`, UCRT accepts a non-BMP character and returns `4`. We fail with `EILSEQ`.
AI Tool Use: Chatbots are used for web searching. LLM-based agents are also used extensively for testing and checking bugs. All code is hand-written by a human. This PR description is also hand-written by a human.
---
Patch is 133.69 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/230917.diff
34 Files Affected:
- (modified) libc/hdr/types/CMakeLists.txt (+11)
- (added) libc/hdr/types/char16_t.h (+27)
- (modified) libc/src/__support/wchar/CMakeLists.txt (+11-15)
- (modified) libc/src/__support/wchar/character_converter.h (+292-4)
- (removed) libc/src/__support/wchar/mbrtowc.cpp (-52)
- (modified) libc/src/__support/wchar/mbrtowc.h (+10-2)
- (modified) libc/src/__support/wchar/mbsnrtowcs.h (+3-2)
- (modified) libc/src/__support/wchar/string_converter.h (-2)
- (modified) libc/src/__support/wchar/wcrtomb.h (+6-29)
- (modified) libc/src/__support/wchar/wcsnrtombs.h (+21-7)
- (modified) libc/src/stdlib/wctomb.cpp (+2-2)
- (modified) libc/src/wchar/wcrtomb.cpp (+1-7)
- (modified) libc/test/src/__support/wchar/CMakeLists.txt (+43-2)
- (added) libc/test/src/__support/wchar/string_converter_test_utf16.cpp (+546)
- (renamed) libc/test/src/__support/wchar/string_converter_test_utf32.cpp ()
- (added) libc/test/src/__support/wchar/utf16_to_8_test.cpp (+235)
- (modified) libc/test/src/__support/wchar/utf32_to_8_test.cpp (+14-12)
- (added) libc/test/src/__support/wchar/utf8_to_16_test.cpp (+183)
- (modified) libc/test/src/__support/wchar/utf8_to_32_test.cpp (+139)
- (modified) libc/test/src/__support/wchar/wcsnrtombs_test.cpp (+218-7)
- (modified) libc/test/src/stdlib/mblen_test.cpp (+12)
- (modified) libc/test/src/stdlib/mbstowcs_test.cpp (+170)
- (modified) libc/test/src/stdlib/mbtowc_test.cpp (+13)
- (modified) libc/test/src/stdlib/wcstombs_test.cpp (+74-1)
- (modified) libc/test/src/stdlib/wctomb_test.cpp (+27)
- (modified) libc/test/src/wchar/CMakeLists.txt (+2-1)
- (modified) libc/test/src/wchar/mbrlen_test.cpp (+14)
- (modified) libc/test/src/wchar/mbrtowc_test.cpp (+29)
- (modified) libc/test/src/wchar/mbsinit_test.cpp (+16)
- (modified) libc/test/src/wchar/mbsnrtowcs_test.cpp (+207)
- (modified) libc/test/src/wchar/mbsrtowcs_test.cpp (+181)
- (modified) libc/test/src/wchar/wcrtomb_test.cpp (+36)
- (modified) libc/test/src/wchar/wcsnrtombs_test.cpp (+181-3)
- (modified) libc/test/src/wchar/wcsrtombs_test.cpp (+137-1)
``````````diff
diff --git a/libc/hdr/types/CMakeLists.txt b/libc/hdr/types/CMakeLists.txt
index 92a2985b74be7..6ca38970729cf 100644
--- a/libc/hdr/types/CMakeLists.txt
+++ b/libc/hdr/types/CMakeLists.txt
@@ -9,6 +9,17 @@ add_proxy_header_library(
libc.include.uchar
)
+add_proxy_header_library(
+ char16_t
+ HDRS
+ char16_t.h
+ DEPENDS
+ libc.hdr.uchar_overlay
+ FULL_BUILD_DEPENDS
+ libc.include.llvm-libc-types.char16_t
+ libc.include.uchar
+)
+
add_proxy_header_library(
char32_t
HDRS
diff --git a/libc/hdr/types/char16_t.h b/libc/hdr/types/char16_t.h
new file mode 100644
index 0000000000000..a59d698243b09
--- /dev/null
+++ b/libc/hdr/types/char16_t.h
@@ -0,0 +1,27 @@
+//===-- Definition of char16_t.h ------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_LIBC_HDR_TYPES_CHAR16_T_H
+#define LLVM_LIBC_HDR_TYPES_CHAR16_T_H
+
+#ifdef LIBC_FULL_BUILD
+
+#include "include/llvm-libc-types/char16_t.h"
+
+#else // overlay mode
+
+// MacOS doesn't provide uchar.h so we use the types provided by LLVM-libc.
+#ifdef __APPLE__
+#include "include/llvm-libc-types/char16_t.h"
+#else
+#include "hdr/uchar_overlay.h"
+#endif
+
+#endif // LIBC_FULL_BUILD
+
+#endif // LLVM_LIBC_HDR_TYPES_CHAR16_T_H
diff --git a/libc/src/__support/wchar/CMakeLists.txt b/libc/src/__support/wchar/CMakeLists.txt
index c4e7f0dbaa178..83462b9f0fc93 100644
--- a/libc/src/__support/wchar/CMakeLists.txt
+++ b/libc/src/__support/wchar/CMakeLists.txt
@@ -12,8 +12,6 @@ add_header_library(
HDRS
string_converter.h
DEPENDS
- libc.hdr.types.char8_t
- libc.hdr.types.char32_t
libc.hdr.types.size_t
libc.src.__support.common
libc.src.__support.CPP.type_traits
@@ -29,11 +27,13 @@ add_header_library(
DEPENDS
libc.hdr.errno_macros
libc.hdr.types.char8_t
+ libc.hdr.types.char16_t
libc.hdr.types.char32_t
libc.hdr.types.size_t
libc.src.__support.common
libc.src.__support.CPP.type_traits
libc.src.__support.error_or
+ libc.src.__support.libc_assert
libc.src.__support.macros.config
libc.src.__support.macros.properties.types
libc.src.__support.math_extras
@@ -45,33 +45,29 @@ add_header_library(
HDRS
wcrtomb.h
DEPENDS
- libc.hdr.errno_macros
- libc.hdr.types.char32_t
libc.hdr.types.size_t
libc.hdr.types.wchar_t
libc.src.__support.common
libc.src.__support.error_or
- libc.src.__support.libc_assert
+ libc.src.__support.macros.config
libc.src.__support.macros.null_check
- .character_converter
.mbstate
+ .character_converter
)
-add_object_library(
+add_header_library(
mbrtowc
HDRS
mbrtowc.h
- SRCS
- mbrtowc.cpp
DEPENDS
- libc.hdr.errno_macros
- libc.hdr.types.wchar_t
libc.hdr.types.size_t
+ libc.hdr.types.wchar_t
libc.src.__support.common
libc.src.__support.error_or
libc.src.__support.macros.config
- .character_converter
+ libc.src.__support.macros.null_check
.mbstate
+ .character_converter
)
add_header_library(
@@ -80,8 +76,9 @@ add_header_library(
mbsnrtowcs.h
DEPENDS
libc.hdr.errno_macros
- libc.hdr.types.wchar_t
+ libc.hdr.types.char8_t
libc.hdr.types.size_t
+ libc.hdr.types.wchar_t
libc.src.__support.common
libc.src.__support.error_or
libc.src.__support.macros.config
@@ -98,14 +95,13 @@ add_header_library(
DEPENDS
libc.hdr.errno_macros
libc.hdr.types.char8_t
- libc.hdr.types.char32_t
libc.hdr.types.size_t
libc.hdr.types.wchar_t
libc.src.__support.common
libc.src.__support.error_or
libc.src.__support.macros.config
libc.src.__support.macros.null_check
- .string_converter
.character_converter
.mbstate
+ .string_converter
)
diff --git a/libc/src/__support/wchar/character_converter.h b/libc/src/__support/wchar/character_converter.h
index f754a602d2cce..a34d3401a4b65 100644
--- a/libc/src/__support/wchar/character_converter.h
+++ b/libc/src/__support/wchar/character_converter.h
@@ -10,6 +10,7 @@
#define LLVM_LIBC_SRC___SUPPORT_CHARACTER_CONVERTER_H
#include "hdr/errno_macros.h"
+#include "hdr/types/char16_t.h"
#include "hdr/types/char32_t.h"
#include "hdr/types/char8_t.h"
#include "hdr/types/size_t.h"
@@ -18,6 +19,7 @@
#include "src/__support/CPP/type_traits.h"
#include "src/__support/common.h"
#include "src/__support/error_or.h"
+#include "src/__support/libc_assert.h"
#include "src/__support/macros/properties/types.h"
#include "src/__support/math_extras.h"
#include "src/__support/wchar/mbstate.h"
@@ -57,20 +59,82 @@ class CharacterConverter {
LIBC_INLINE bool isEmpty() { return state->bytes_stored == 0; }
bool isValidState();
+ /*
+ * When converting between UTF-16 and UTF-8, the converter might be in a state
+ * of either partially-pushing or partially-popping. For example consider the
+ * following sequence:
+ * * push((char8_t)0xF0);
+ * * push((char8_t)0xA4);
+ * * [A](partial=0x24,bytes_stored=2,total_bytes=4)
+ * * push((char8_t)0xAD);
+ * * push((char8_t)0xA2);
+ * * (partial=0x24B62,bytes_stored=4,total_bytes=4)
+ * * pop_utf16(); -> 0xD852
+ * * [B](partial=0x24B62,bytes_stored=2,total_bytes=4)
+ * * pop_utf16(); -> 0xDF62
+ * At point [A] and point [B], the stored bytes would both be 2. But at [A] we
+ * must continue reading input, while at [B] we must write the output low
+ * surrogate and clear the state.
+ * Thus we must have some means to distinguish between pushing and popping.
+ * Notice that: When pushing, we do left-shifts on state->partial before each
+ * push. When popping, we keep state->partial at the complete code point.
+ * Since we disallow overlong encodings, the complete UTF-8 code point is
+ * always greater than
+ * * MAX_VALUE_PER_UTF8_LEN[state->total_bytes - 2]
+ * but at most
+ * * MAX_VALUE_PER_UTF8_LEN[state->total_bytes - 1]
+ * If we are partially through pushing, state->partial is at most
+ * * char32 >> 6
+ * thus at most
+ * * MAX_VALUE_PER_UTF8_LEN[state->total_bytes - 1] >> 6
+ * Observe that, an item in MAX_VALUE_PER_UTF8_LEN right-shifted by 6 is
+ * always less than the preceding item. Based on these facts, we can determine
+ * if we are partially through pushing or popping, by comparing state->partial
+ * with MAX_VALUE_PER_UTF8_LEN[state->total_bytes - 2]
+ * Also when converting UTF-16 -> UTF-8, in a partial state,
+ * * bytes_stored=2
+ * * total_bytes=4
+ * so the threshold 0xFFFF is also correct.
+ */
+ LIBC_INLINE bool isPartiallyPopping() {
+ if (isEmpty() || isFull())
+ return false;
+ // When neither empty nor full, state->total_bytes is at least 2
+ char32_t threshold = MAX_VALUE_PER_UTF8_LEN[state->total_bytes - 2];
+ return state->partial > threshold;
+ }
+
template <typename CharType> size_t sizeAs();
int push(char8_t utf8_byte);
+ int push(char16_t utf16);
int push(char32_t utf32);
ErrorOr<char8_t> pop_utf8();
+ ErrorOr<char16_t> pop_utf16();
ErrorOr<char32_t> pop_utf32();
-#if defined(LIBC_TYPES_WCHAR_T_IS_UTF32)
+#if defined(LIBC_TYPES_WCHAR_T_IS_UTF32) || defined(LIBC_TYPES_WCHAR_T_IS_UTF16)
int push(wchar_t wchar);
ErrorOr<wchar_t> pop_wchar();
-#endif // LIBC_TYPES_WCHAR_T_IS_UTF32
+#endif // LIBC_TYPES_WCHAR_T_IS_UTF32 || LIBC_TYPES_WCHAR_T_IS_UTF16
template <typename CharType> ErrorOr<CharType> pop();
+
+ // mbrtowc, mbrtoc16, mbrtoc32
+ // mbrtoc16 may return a surrogate pair in 2 consecutive calls, and specify -3
+ // as a return value (for the second call). mbrtowc doesn't specify -3 as a
+ // return value. Set allowPartialPop to true for mbrtoc16.
+ template <typename T, bool allowPartialPop>
+ ErrorOr<size_t> mbrto_generic(T *__restrict dest_ptr,
+ const char *__restrict src_ptr,
+ size_t max_src_bytes);
+ // wcrtomb, c16rtomb, c32rtomb
+ // c16rtomb may consume a surrogate pair in 2 consecutive calls, and return 0
+ // on the first call. wcrtomb doesn't use that behavior. Set allowPartialPush
+ // to true for c16rtomb.
+ template <typename T, bool allowPartialPush>
+ ErrorOr<size_t> rtomb_generic(char *__restrict dest_ptr, T src);
};
LIBC_INLINE bool CharacterConverter::isValidState() {
@@ -112,6 +176,11 @@ LIBC_INLINE int CharacterConverter::push(char8_t utf8_byte) {
state->bytes_stored++;
return 0;
}
+ // Pushing when partially popping is a misuse in the library
+ if (isPartiallyPopping()) {
+ clear();
+ return -1;
+ }
// Any subsequent push
// Adding 6 more bits so need to left shift
if (num_ones == 1 && !isFull()) {
@@ -119,6 +188,19 @@ LIBC_INLINE int CharacterConverter::push(char8_t utf8_byte) {
state->partial = state->partial << ENCODED_BITS_PER_UTF8;
state->partial |= byte;
state->bytes_stored++;
+ if (isFull()) {
+ // Detect and reject overlong encodings
+ // Needed for isPartiallyPopping to work
+ if (state->partial <= MAX_VALUE_PER_UTF8_LEN[state->total_bytes - 2]) {
+ clear();
+ return EILSEQ;
+ }
+ // Also reject out-of-range code points
+ if (state->partial > MAX_VALUE_PER_UTF8_LEN[state->total_bytes - 1]) {
+ clear();
+ return EILSEQ;
+ }
+ }
return 0;
}
@@ -127,6 +209,57 @@ LIBC_INLINE int CharacterConverter::push(char8_t utf8_byte) {
return EILSEQ;
}
+LIBC_INLINE int CharacterConverter::push(char16_t utf16) {
+ // Checking for high surrogate if first push
+ if (isEmpty()) {
+ bool is_surrogate = (utf16 & 0xF800) == 0xD800;
+ if (is_surrogate) {
+ bool is_high_surrogate = (utf16 & 0x400) == 0;
+ if (!is_high_surrogate) {
+ // unpaired surrogate -> reset the state
+ // bytes_stored and total_bytes will always be 0 here
+ state->partial = static_cast<char32_t>(0);
+ return EILSEQ;
+ }
+ // high surrogate
+ // code point between 0x10000 and 0x10FFFF
+ state->total_bytes = 4;
+ state->bytes_stored = 2;
+ // 10 bits each from the high surrogate and the low surrogate, and the
+ // full code point is from 0x10000, so we add 0x40 to partial, which after
+ // shifting becomes 0x10000
+ state->partial = static_cast<char32_t>(utf16 & 0x3FF) + 0x40;
+ } else {
+ state->partial = static_cast<char32_t>(utf16);
+ // determine number of utf-8 bytes needed to represent this utf16 value
+ uint8_t i;
+ for (i = 0; i < 2; i++) {
+ if (state->partial <= MAX_VALUE_PER_UTF8_LEN[i])
+ break;
+ }
+ state->total_bytes = i + 1;
+ state->bytes_stored = i + 1;
+ }
+ return 0;
+ }
+ // Pushing when partially popping is a misuse in the library
+ if (isFull() || isPartiallyPopping()) {
+ clear();
+ return -1;
+ }
+ bool is_low_surrogate = (utf16 & 0xFC00) == 0xDC00;
+ if (!is_low_surrogate) {
+ // unpaired surrogate -> reset the state
+ clear();
+ return EILSEQ;
+ }
+ char32_t low = utf16 & 0x3FF;
+ state->partial <<= 10;
+ state->partial |= low;
+ state->bytes_stored += 2;
+ return 0;
+}
+
LIBC_INLINE int CharacterConverter::push(char32_t utf32) {
// we can't be partially through a conversion when pushing a utf32 value
if (!isEmpty())
@@ -160,8 +293,36 @@ LIBC_INLINE ErrorOr<char32_t> CharacterConverter::pop_utf32() {
return utf32;
}
+LIBC_INLINE ErrorOr<char16_t> CharacterConverter::pop_utf16() {
+ if (!isFull() && !isPartiallyPopping())
+ return Error(-1);
+ char32_t output;
+ if (state->total_bytes <= 3) {
+ // no surrogates needed
+ output = state->partial;
+ // disallow using UTF-8 or UTF-32 to encode unpaired surrogates
+ bool is_surrogate = (output & 0xF800) == 0xD800;
+ if (is_surrogate) {
+ clear();
+ return Error(EILSEQ);
+ }
+ clear();
+ } else {
+ if (isFull()) {
+ // high surrogate first
+ output = ((state->partial - 0x10000) >> 10) | 0xD800;
+ state->bytes_stored = 2;
+ } else {
+ // low surrogate
+ output = ((state->partial - 0x10000) & 0x3FF) | 0xDC00;
+ clear();
+ }
+ }
+ return static_cast<char16_t>(output);
+}
+
LIBC_INLINE ErrorOr<char8_t> CharacterConverter::pop_utf8() {
- if (isEmpty())
+ if (!isFull() && !isPartiallyPopping())
return Error(-1);
constexpr char8_t FIRST_BYTE_HEADERS[] = {0, 0xC0, 0xE0, 0xF0};
@@ -196,6 +357,10 @@ template <> LIBC_INLINE ErrorOr<char8_t> CharacterConverter::pop() {
return pop_utf8();
}
+template <> LIBC_INLINE ErrorOr<char16_t> CharacterConverter::pop() {
+ return pop_utf16();
+}
+
template <> LIBC_INLINE ErrorOr<char32_t> CharacterConverter::pop() {
return pop_utf32();
}
@@ -204,6 +369,10 @@ template <> LIBC_INLINE size_t CharacterConverter::sizeAs<char8_t>() {
return state->total_bytes;
}
+template <> LIBC_INLINE size_t CharacterConverter::sizeAs<char16_t>() {
+ return state->total_bytes > 3 ? 2 : 1;
+}
+
template <> LIBC_INLINE size_t CharacterConverter::sizeAs<char32_t>() {
return 1;
}
@@ -229,7 +398,126 @@ template <> LIBC_INLINE size_t CharacterConverter::sizeAs<wchar_t>() {
return sizeAs<char32_t>();
}
-#endif // LIBC_TYPES_WCHAR_T_IS_UTF32
+#elif defined(LIBC_TYPES_WCHAR_T_IS_UTF16)
+
+LIBC_INLINE int CharacterConverter::push(wchar_t wchar) {
+ return push(static_cast<char16_t>(wchar));
+}
+
+LIBC_INLINE ErrorOr<wchar_t> CharacterConverter::pop_wchar() {
+ ErrorOr<char16_t> Result = pop_utf16();
+ if (!Result)
+ return Error(Result.error());
+ return static_cast<wchar_t>(*Result);
+}
+
+template <> LIBC_INLINE ErrorOr<wchar_t> CharacterConverter::pop() {
+ return pop_wchar();
+}
+
+template <> LIBC_INLINE size_t CharacterConverter::sizeAs<wchar_t>() {
+ return sizeAs<char16_t>();
+}
+
+#endif // LIBC_TYPES_WCHAR_T_IS_UTF32 || LIBC_TYPES_WCHAR_T_IS_UTF16
+
+template <typename T, bool allowPartialPop>
+LIBC_INLINE ErrorOr<size_t>
+CharacterConverter::mbrto_generic(T *__restrict dest_ptr,
+ const char *__restrict src_ptr,
+ size_t max_src_bytes) {
+ if (!isValidState())
+ return Error(EINVAL);
+ if (!allowPartialPop && isPartiallyPopping()) {
+ // don't allow a complete char32 spanning over 2 consecutive calls of char16
+ clear();
+ return Error(EILSEQ);
+ }
+
+ char empty_src = '\0';
+ if (src_ptr == nullptr) {
+ dest_ptr = nullptr;
+ src_ptr = &empty_src;
+ max_src_bytes = 1;
+ }
+
+ size_t i = 0;
+ // If we still have the lower part to write, don't read yet (UTF-16)
+ if (!isPartiallyPopping()) {
+ // Reading in bytes until we have a complete char32 or error
+ for (; i < max_src_bytes && !isFull(); ++i) {
+ int err = push(static_cast<char8_t>(src_ptr[i]));
+ // Encoding error
+ if (err != 0)
+ return Error(err);
+ }
+ }
+ if (!isFull() && !isPartiallyPopping()) {
+ // We haven't read in a full char32 yet.
+ // Incomplete but potentially valid
+ return (size_t)-2;
+ }
+ auto result = pop<T>();
+ if (!result.has_value()) {
+ return Error(result.error());
+ }
+ if (!allowPartialPop && !isEmpty()) {
+ // don't allow a complete char32 spanning over 2 consecutive calls of char16
+ clear();
+ return Error(EILSEQ);
+ }
+ if (dest_ptr != nullptr)
+ *dest_ptr = result.value();
+ if (i == 0) {
+ // lower part written, no input processed
+ return (size_t)-3;
+ }
+ // null terminator -> return 0
+ if (result.value() == T{})
+ return 0;
+ return i;
+}
+
+template <typename T, bool allowPartialPush>
+LIBC_INLINE ErrorOr<size_t>
+CharacterConverter::rtomb_generic(char *__restrict dest_ptr, T src) {
+ if (dest_ptr == nullptr)
+ src = {};
+
+ if (!isValidState())
+ return Error(EINVAL);
+ if (!allowPartialPush && !isEmpty()) {
+ // don't allow a complete char32 spanning over 2 consecutive calls of char16
+ clear();
+ return Error(EILSEQ);
+ }
+
+ int status = push(static_cast<T>(src));
+ if (status != 0)
+ return Error(status);
+
+ if (!isFull()) {
+ if (!allowPartialPush) {
+ // don't allow a complete char32 spanning over 2 consecutive calls of
+ // char16
+ clear();
+ return Error(EILSEQ);
+ }
+ return 0;
+ }
+
+ size_t count = 0;
+ for (; !isEmpty(); ++count) {
+ auto utf8 = pop_utf8(); // can never fail as long as the push succeeded
+ LIBC_ASSERT(utf8.has_value());
+
+ if (dest_ptr != nullptr) {
+ *dest_ptr = static_cast<char>(utf8.value());
+ dest_ptr++;
+ }
+ }
+ return count;
+}
} // namespace internal
} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/src/__support/wchar/mbrtowc.cpp b/libc/src/__support/wchar/mbrtowc.cpp
deleted file mode 100644
index 33a42a4a31110..0000000000000
--- a/libc/src/__support/wchar/mbrtowc.cpp
+++ /dev/null
@@ -1,52 +0,0 @@
-//===-- Implementation for mbrtowc function ---------------------*- C++ -*-===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#include "src/__support/wchar/mbrtowc.h"
-#include "hdr/errno_macros.h"
-#include "hdr/types/size_t.h"
-#include "hdr/types/wchar_t.h"
-#include "src/__support/common.h"
-#include "src/__support/error_or.h"
-#include "src/__support/macros/config.h"
-#include "src/__support/wchar/character_converter.h"
-#include "src/__support/wchar/mbstate.h"
-
-namespace LIBC_NAMESPACE_DECL {
-namespace internal {
-
-ErrorOr<size_t> mbrtowc(wchar_t *__restrict pwc, const char *__restrict src_ptr,
- size_t max_src_bytes, mbstate *__restrict ps) {
- CharacterConverter char_conv(ps);
- if (!char_conv.isValidState())
- return Error(EINVAL);
- if (src_ptr == nullptr)
- return 0;
- size_t i = 0;
- // Reading in bytes until we have a complete wc or error
- for (; i < max_src_bytes && !char_conv.isFull(); ++i) {
- int err = char_conv.push(static_cast<char8_t>(src_ptr[i]));
- // Encoding error
- if (err == EILSEQ)
- return Error(err);
- }
- auto wc = char_conv.pop_utf32();
- if (wc.has_value()) {
- if (pwc != nullptr)
- *pwc = wc.value();
- // null terminator -> return 0
- if (wc.value() == L'\0')
- return 0;
- return i;
- }
- // Incomplete but potentially valid
- return -2;
-}
-
-} // namespace internal
-
-} // namespace LIBC_NAMESPACE_DECL
diff --git a/libc/src/__support/wchar/mbrtowc.h b/libc/src/__support/wchar/mbrtowc.h
index d15c4844499a6..1c802fa9f8dfd 100644
--- a/libc/src/__support/wchar/mbrtowc.h
+++ b/libc/src/__support/wchar/mbrtowc.h
@@ -14,13 +14,21 @@
#include "src/__support/common.h"
#include "src/__support/error_or.h"
#include "src/__support/macros/config.h"
+#include "src/__support/macros/null_check.h"
+#include "src/__support/wchar/character_converter.h"
#include "src/__support/wchar/mbstate.h"
namespace LIBC_NAMESPACE_DECL {
namespace internal {
-ErrorOr<size_t> mbrtowc(wchar_t *__restrict pwc, const char *__restrict src_ptr,
- size_t max_src_bytes, mbstate *__restrict ps);
+LIBC_INLINE ErrorOr<size_t> mbrtowc(wchar_t *__restrict pwc,
+ const char *__restrict src_ptr,
+ size_t max_src_bytes,
+ mbstate *__restrict ps) {
+ LIBC_CRASH_ON_NULLPTR(ps);
+ CharacterConverter char_conv(ps);
+ return char_conv.mbrto_generic<wchar_t, false>(pwc, src_ptr, max_src_bytes);
+}
} // namespace internal
diff --git a/libc/src/__support/wchar/mbsnrtowcs.h b/libc/src/__support/wchar/mbsnrtowcs.h
index 50c3fa23a7d58..ebddc9802574d 100644
--- a/libc/src/__support/wchar/mbsnrtowcs.h
+++ b/libc/src/__support/wchar/mbsnrtowcs.h
@@ -10,6 +10,7 @@
#define LLVM_LIBC_SRC___SUPPORT_WCHAR_MBSNRTOWCS_H
#include "hdr/errno_macros.h"
+#include "hdr/types/char8_t.h"
#include "hdr/types/size_t.h"
#include "hdr/types/wchar_t.h"
#include "src/__support/common.h"
@@ -3...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/230917
More information about the libc-commits
mailing list