[libc-commits] [libc] [libc] Add UTF-16 support in CharacterConverter [1/6] (PR #230434)

[via libc-commits]

================
@@ -112,13 +161,31 @@ LIBC_INLINE int CharacterConverter::push(char8_t utf8_byte) {
     state->bytes_stored++;
     return 0;
   }
+  // Pushing when partially popping is a misuse in the library
+  if (isPartiallyPopping()) {
+    clear();
+    return -1;
+  }
   // Any subsequent push
   // Adding 6 more bits so need to left shift
   if (num_ones == 1 && !isFull()) {
     char32_t byte = utf8_byte & MASK_ENCODED_BITS;
     state->partial = state->partial << ENCODED_BITS_PER_UTF8;
     state->partial |= byte;
     state->bytes_stored++;
+    if (isFull()) {
+      // Detect and reject overlong encodings
+      // Needed for isPartiallyPopping to work
+      if (state->partial <= MAX_VALUE_PER_UTF8_LEN[state->total_bytes - 2]) {
+        clear();
+        return EILSEQ;
+      }
+      // Also reject out-of-range code points
+      if (state->partial > MAX_VALUE_PER_UTF8_LEN[state->total_bytes - 1]) {
+        clear();
+        return EILSEQ;
+      }
+    }
----------------
yangbowen wrote:

The out-of-range check is only necessary for the 4-byte case, where the 4 bytes can encode at most U+1FFFFF which is out of U+10FFFF maximum.
The overlong check is necessary because you can encode a code point that's actually encodable in less bytes. e.g. C1 BF encodes U+7F, E0 9F BF encodes U+7FF, F0 8F BF BF encodes U+FFFF.
Only the overlong check is needed for isPartiallyPopping to work correctly.

https://github.com/llvm/llvm-project/pull/230434