[llvm] [llvm-strings] Add --encoding={s,S,utf8} option (PR #221794)
James Henderson via llvm-commits
llvm-commits at lists.llvm.org
Mon Oct 5 01:33:41 PDT 2026
================
@@ -90,6 +91,118 @@ static void parseIntArg(const opt::InputArgList &Args, int ID, T &Value) {
}
}
+// Tries to read one character from the bytes P...E and store it in Ch.
+// Returns true if a (possibly invalid) character was read, false otherwise.
+//
+// If P...E starts with a valid complete character, P and MBState are updated
+// and true is returned.
+// If P...E is empty, or holds an incomplete character and AtEOF is false, P
+// and MBState are unchanged, Ch is set to 0, and false is returned. This is
+// intended to allow more bytes to be read and Read to be called again.
+// If P...E holds an incomplete character and AtEOF is true, or if P...E
+// starts with an invalid character, the first bytes are skipped and MBState is
+// reset to allow continuing from the next point, Ch is set to 0, and true is
+// returned.
+//
+// The number of skipped bytes for invalid characters follows the Unicode
+// definition of the maximal subpart of an ill-formed subsequence, applied to
+// arbitrary locales: it is the longest subsequence that could start a valid
+// multibyte character, or if the initial byte cannot start a valid multibyte
+// character, the initial byte. This allows consistent error recovery.
+static bool readChar(const char *&P, const char *E, std::mbstate_t &MBState,
+ bool AtEOF, UTF32 &Ch) {
+ if (P == E)
+ return false;
+
+ switch (Encoding) {
+ case Encoding::Ascii:
+ Ch = *P++;
+ return true;
+
+ case Encoding::Locale: {
+ std::mbstate_t SaveMBState = MBState;
+ wchar_t WCh;
+ const size_t BytesRead = mbrtowc(&WCh, P, E - P, &MBState);
+ switch (BytesRead) {
+ default:
+ P += BytesRead;
+ Ch = WCh;
+ return true;
+ case 0:
+ case (size_t)-1: {
+ // We encountered a (possibly null) byte that cannot be part of a valid
+ // character from the previous state. Recover by retrying one byte at a
+ // time, processing as many bytes as possible. If this gives us a
+ // partial character, treat this partial character as a single invalid
+ // character, otherwise treat the first byte as a single invalid
+ // character.
----------------
jh7370 wrote:
If I'm following this correctly (I easily could not be), you first try to read a large chunk of bytes as a single character and if it isn't, you go through it one byte at a time. Is that correct? Is there a point in doing that first "max length" check? It feels like in the typical case, the character is only going to be a single byte long. (There's possibly an argument for a doubling approach too, i.e. try first 1 byte long, then 2, then 4 etc, using a binary search if we overshoot, but that's somewhat orthogonal and probably depends on the distribution of likely character widths across all encodings we might see, which gets into stats that I have no clue about).
https://github.com/llvm/llvm-project/pull/221794
More information about the llvm-commits
mailing list