[libc-commits] [libc] [libc] utf8 to 32 CharacterConverter (PR #143973)

[Michael Jones via libc-commits]

================
@@ -22,13 +24,69 @@ bool CharacterConverter::isComplete() {
   return state->bytes_processed == state->total_bytes;
 }
 
-int CharacterConverter::push(char8_t utf8_byte) {}
-
-int CharacterConverter::push(char32_t utf32) {}
-
-utf_ret<char8_t> CharacterConverter::pop_utf8() {}
+int CharacterConverter::push(char8_t utf8_byte) {
+  // Checking the first byte if first push
+  if (state->bytes_processed == 0 && state->total_bytes == 0) {
+    state->partial = static_cast<char32_t>(0);
+    int numOnes = cpp::countl_one(utf8_byte);
+    switch (numOnes) {
+    // 1 byte total
+    case 0:
+      state->total_bytes = 1;
+      break;
+    // 2 bytes total
+    case 2:
+      state->total_bytes = 2;
+      utf8_byte &= 0x1F;
+      break;
+    // 3 bytes total
+    case 3:
+      state->total_bytes = 3;
+      utf8_byte &= 0x0F;
+      break;
+    // 4 bytes total
+    case 4:
+      state->total_bytes = 4;
+      utf8_byte &= 0x07;
+      break;
----------------
michaelrj-google wrote:

I think cases 2, 3, and 4 can be collapsed. In all 3 cases:
`state->total_bytes = numOnes;`
`utf8_byte &= (0x7f >> numOnes);`

https://github.com/llvm/llvm-project/pull/143973