[llvm] [llvm-strings] Add --encoding={s,S,utf8} option (PR #221794)

James Henderson via llvm-commits llvm-commits at lists.llvm.org
Mon Oct 5 01:33:41 PDT 2026


================
@@ -90,6 +91,118 @@ static void parseIntArg(const opt::InputArgList &Args, int ID, T &Value) {
   }
 }
 
+// Tries to read one character from the bytes P...E and store it in Ch.
+// Returns true if a (possibly invalid) character was read, false otherwise.
+//
+// If P...E starts with a valid complete character, P and MBState are updated
+// and true is returned.
+// If P...E is empty, or holds an incomplete character and AtEOF is false, P
+// and MBState are unchanged, Ch is set to 0, and false is returned. This is
+// intended to allow more bytes to be read and Read to be called again.
+// If P...E holds an incomplete character and AtEOF is true, or if P...E
+// starts with an invalid character, the first bytes are skipped and MBState is
+// reset to allow continuing from the next point, Ch is set to 0, and true is
+// returned.
+//
+// The number of skipped bytes for invalid characters follows the Unicode
+// definition of the maximal subpart of an ill-formed subsequence, applied to
+// arbitrary locales: it is the longest subsequence that could start a valid
+// multibyte character, or if the initial byte cannot start a valid multibyte
+// character, the initial byte. This allows consistent error recovery.
+static bool readChar(const char *&P, const char *E, std::mbstate_t &MBState,
+                     bool AtEOF, UTF32 &Ch) {
+  if (P == E)
+    return false;
+
+  switch (Encoding) {
+  case Encoding::Ascii:
+    Ch = *P++;
+    return true;
+
+  case Encoding::Locale: {
+    std::mbstate_t SaveMBState = MBState;
+    wchar_t WCh;
+    const size_t BytesRead = mbrtowc(&WCh, P, E - P, &MBState);
+    switch (BytesRead) {
+    default:
+      P += BytesRead;
+      Ch = WCh;
+      return true;
+    case 0:
+    case (size_t)-1: {
+      // We encountered a (possibly null) byte that cannot be part of a valid
+      // character from the previous state. Recover by retrying one byte at a
+      // time, processing as many bytes as possible. If this gives us a
+      // partial character, treat this partial character as a single invalid
+      // character, otherwise treat the first byte as a single invalid
+      // character.
----------------
jh7370 wrote:

If I'm following this correctly (I easily could not be), you first try to read a large chunk of bytes as a single character and if it isn't, you go through it one byte at a time. Is that correct? Is there a point in doing that first "max length" check? It feels like in the typical case, the character is only going to be a single byte long. (There's possibly an argument for a doubling approach too, i.e. try first 1 byte long, then 2, then 4 etc, using a binary search if we overshoot, but that's somewhat orthogonal and probably depends on the distribution of likely character widths across all encodings we might see, which gets into stats that I have no clue about).

https://github.com/llvm/llvm-project/pull/221794


More information about the llvm-commits mailing list