[libc-commits] [libc] [libc] Add UTF-16 support in CharacterConverter [1/6] (PR #230434)

[Michael Jones via libc-commits]

================
@@ -112,13 +161,31 @@ LIBC_INLINE int CharacterConverter::push(char8_t utf8_byte) {
     state->bytes_stored++;
     return 0;
   }
+  // Pushing when partially popping is a misuse in the library
+  if (isPartiallyPopping()) {
+    clear();
+    return -1;
+  }
   // Any subsequent push
   // Adding 6 more bits so need to left shift
   if (num_ones == 1 && !isFull()) {
     char32_t byte = utf8_byte & MASK_ENCODED_BITS;
     state->partial = state->partial << ENCODED_BITS_PER_UTF8;
     state->partial |= byte;
     state->bytes_stored++;
+    if (isFull()) {
+      // Detect and reject overlong encodings
+      // Needed for isPartiallyPopping to work
+      if (state->partial <= MAX_VALUE_PER_UTF8_LEN[state->total_bytes - 2]) {
+        clear();
+        return EILSEQ;
+      }
+      // Also reject out-of-range code points
+      if (state->partial > MAX_VALUE_PER_UTF8_LEN[state->total_bytes - 1]) {
+        clear();
+        return EILSEQ;
+      }
+    }
----------------
michaelrj-google wrote:

this check shouldn't be necessary, because it should be impossible to put more bytes in than `MAX_VALUE_PER_UTF8_LEN` for a given multibyte length.

https://github.com/llvm/llvm-project/pull/230434