[clang] [llvm] [llvm-strings] Add --encoding={s, S, utf8} option (PR #221794)
Harald van Dijk via cfe-commits
cfe-commits at lists.llvm.org
Fri Nov 6 04:48:54 PST 2026
https://github.com/hvdijk updated https://github.com/llvm/llvm-project/pull/221794
>From b2334d9f932e6c6f8f7e106e4678c673a19ac944 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Mon, 7 Sep 2026 18:33:02 +0100
Subject: [PATCH 01/41] [llvm-strings] Add --encoding & --unicode options
The --encoding and --unicode options come from GNU strings, and are
mostly compatible with it, but intentionally not fully.
The --encoding option allows specifying the following encodings:
s (single 7-bit byte characters)
S (8-bit byte characters)
u (UTF-8 characters)
b/l (UTF-16 characters, big/little endian)
B/L (UTF-32 characters, big/little endian)
--encoding=s matches GNU strings and interprets 7-bit characters as
ASCII, rejecting any bytes with the high bit set.
--encoding=S interprets characters according to the current locale's
character set. In GNU strings, its meaning depends on --unicode and
either means all characters with the high bit set are considered
printable, or means characters are interpreted as UTF-8.
--encoding=u is an extension to GNU strings, which implicitly uses UTF-8
when --unicode options are used, but does not have a dedicated encoding
value for this and puts it under --encoding=S.
--encoding=b/l/B/L match GNU strings's documentation but deviate
slightly from their implementation: they interpret characters as
UTF-16/UTF-32. Although GNU strings's manual suggests it does the same,
it actually only accepts ASCII characters stored as 16-bit/32-bit units,
whereas this implementation accepts any printable Unicode character. To
reduce the number of false positives, unlike GNU strings, this
implementation requires strings to be aligned.
--unicode=default prints characters as UTF-8. It is the default output
for all UTF encodings.
--unicode=invalid rejects any non-ASCII characters. It is the default
output for --encoding=s.
--unicode=locale prints characters in the encoding specified by the
current locale. It is the default output for --encoding=S.
--unicode=escape/highlight print non-ASCII characters as \u0000 escape
sequences, optionally highlighted if output is to a terminal.
--unicode=hex prints non-ASCII characters as their UTF-8 byte
representation in hex.
The default encoding is --encoding=S, as specified by POSIX. Although
this deviates from GNU strings, this default is meant to be a good fit
for the common case where the user's character encoding (or, in Windows
terms, the code page) matches the binary expects. --encoding=s can be
used to restore the prior behaviour.
The fact that the -n option is aliased to --bytes is confusing, but GNU
strings already uses --bytes to specify the minimum character length,
not the minimum byte length, as well.
---
llvm/test/tools/llvm-strings/length.test | 57 +++-
.../tools/llvm-strings/negative-char.test | 7 +-
llvm/tools/llvm-strings/Opts.td | 16 +
llvm/tools/llvm-strings/llvm-strings.cpp | 297 +++++++++++++++++-
4 files changed, 357 insertions(+), 20 deletions(-)
diff --git a/llvm/test/tools/llvm-strings/length.test b/llvm/test/tools/llvm-strings/length.test
index 896917d8db90dc..aaa3df198f6333 100644
--- a/llvm/test/tools/llvm-strings/length.test
+++ b/llvm/test/tools/llvm-strings/length.test
@@ -1,5 +1,5 @@
## Show that llvm-strings prints only strings with length of at least the
-## requested number of bytes.
+## requested number of characters.
RUN: echo a > %t
RUN: echo ab >> %t
@@ -44,3 +44,58 @@ CHECK-5: abcde
## Show that a non-numeric argument is rejected.
RUN: not llvm-strings -n foo %t 2>&1 | FileCheck %s --check-prefix=ERR
ERR: error: expected a positive integer, but got 'foo'
+
+## Show that the count is number of characters, not number of bytes.
+RUN: printf '\302\241\n' > %t
+RUN: printf '\302\241\302\242\n' >> %t
+RUN: printf '\302\241\302\242\302\243\n' >> %t
+RUN: not llvm-strings --encoding u -n 0 2>&1 %t | FileCheck --check-prefix CHECK-0 %s
+RUN: llvm-strings --encoding u -n 1 %t | FileCheck --check-prefix UTF-8-1 %s --implicit-check-not={{.}}
+RUN: llvm-strings --encoding u -n 2 %t | FileCheck --check-prefix UTF-8-2 %s --implicit-check-not={{.}}
+RUN: llvm-strings --encoding u -n 3 %t | FileCheck --check-prefix UTF-8-3 %s --implicit-check-not={{.}}
+RUN: llvm-strings --encoding u -n 4 %t | FileCheck %s --implicit-check-not={{.}} --allow-empty
+
+RUN: printf '\241\0\n\0' > %t
+RUN: printf '\241\0\242\0\n\0' >> %t
+RUN: printf '\241\0\242\0\243\0\n\0' >> %t
+RUN: not llvm-strings --encoding l -n 0 2>&1 %t | FileCheck --check-prefix CHECK-0 %s
+RUN: llvm-strings --encoding l -n 1 %t | FileCheck --check-prefix UTF-8-1 %s --implicit-check-not={{.}}
+RUN: llvm-strings --encoding l -n 2 %t | FileCheck --check-prefix UTF-8-2 %s --implicit-check-not={{.}}
+RUN: llvm-strings --encoding l -n 3 %t | FileCheck --check-prefix UTF-8-3 %s --implicit-check-not={{.}}
+RUN: llvm-strings --encoding l -n 4 %t | FileCheck %s --implicit-check-not={{.}} --allow-empty
+
+RUN: printf '\0\241\0\n' > %t
+RUN: printf '\0\241\0\242\0\n' >> %t
+RUN: printf '\0\241\0\242\0\243\0\n' >> %t
+RUN: not llvm-strings --encoding b -n 0 2>&1 %t | FileCheck --check-prefix CHECK-0 %s
+RUN: llvm-strings --encoding b -n 1 %t | FileCheck --check-prefix UTF-8-1 %s --implicit-check-not={{.}}
+RUN: llvm-strings --encoding b -n 2 %t | FileCheck --check-prefix UTF-8-2 %s --implicit-check-not={{.}}
+RUN: llvm-strings --encoding b -n 3 %t | FileCheck --check-prefix UTF-8-3 %s --implicit-check-not={{.}}
+RUN: llvm-strings --encoding b -n 4 %t | FileCheck %s --implicit-check-not={{.}} --allow-empty
+
+RUN: printf '\241\0\0\0\n\0\0\0' > %t
+RUN: printf '\241\0\0\0\242\0\0\0\n\0\0\0' >> %t
+RUN: printf '\241\0\0\0\242\0\0\0\243\0\0\0\n\0\0\0' >> %t
+RUN: not llvm-strings --encoding L -n 0 2>&1 %t | FileCheck --check-prefix CHECK-0 %s
+RUN: llvm-strings --encoding L -n 1 %t | FileCheck --check-prefix UTF-8-1 %s --implicit-check-not={{.}}
+RUN: llvm-strings --encoding L -n 2 %t | FileCheck --check-prefix UTF-8-2 %s --implicit-check-not={{.}}
+RUN: llvm-strings --encoding L -n 3 %t | FileCheck --check-prefix UTF-8-3 %s --implicit-check-not={{.}}
+RUN: llvm-strings --encoding L -n 4 %t | FileCheck %s --implicit-check-not={{.}} --allow-empty
+
+RUN: printf '\0\0\0\241\0\0\0\n' > %t
+RUN: printf '\0\0\0\241\0\0\0\242\0\0\0\n' >> %t
+RUN: printf '\0\0\0\241\0\0\0\242\0\0\0\243\0\0\0\n' >> %t
+RUN: not llvm-strings --encoding B -n 0 2>&1 %t | FileCheck --check-prefix CHECK-0 %s
+RUN: llvm-strings --encoding B -n 1 %t | FileCheck --check-prefix UTF-8-1 %s --implicit-check-not={{.}}
+RUN: llvm-strings --encoding B -n 2 %t | FileCheck --check-prefix UTF-8-2 %s --implicit-check-not={{.}}
+RUN: llvm-strings --encoding B -n 3 %t | FileCheck --check-prefix UTF-8-3 %s --implicit-check-not={{.}}
+RUN: llvm-strings --encoding B -n 4 %t | FileCheck %s --implicit-check-not={{.}} --allow-empty
+
+UTF-8-1: ¡
+UTF-8-1-NEXT: ¡¢
+UTF-8-1-NEXT: ¡¢£
+
+UTF-8-2: ¡¢
+UTF-8-2-NEXT: ¡¢£
+
+UTF-8-3: ¡¢£
diff --git a/llvm/test/tools/llvm-strings/negative-char.test b/llvm/test/tools/llvm-strings/negative-char.test
index 5154886f9200f0..3e661b04724706 100644
--- a/llvm/test/tools/llvm-strings/negative-char.test
+++ b/llvm/test/tools/llvm-strings/negative-char.test
@@ -1,6 +1,9 @@
## Show that llvm-strings can handle a negative signed char value (i.e. > 0x7f).
-## Such characters should form string delimiters like other unprintable ones.
+## Such characters should form string delimiters like other unprintable ones,
+## unless using an encoding where they form valid characters.
-# RUN: printf "z\0\200\0a\0" | llvm-strings --bytes 1 - | FileCheck %s
+# RUN: printf "z\0\200\0\302\241\0a\0" | llvm-strings --encoding=s --bytes 1 - | FileCheck %s
+# RUN: printf "z\0\200\0\302\241\0a\0" | llvm-strings --unicode=default --bytes 1 - | FileCheck %s --check-prefixes=CHECK,CHECK-UTF8
# CHECK: z{{$}}
+# CHECK-UTF8-NEXT: {{^}}¡{{$}}
# CHECK-NEXT: {{^}}a
diff --git a/llvm/tools/llvm-strings/Opts.td b/llvm/tools/llvm-strings/Opts.td
index 2ad77fae6c15ff..cf7a406fcec1f9 100644
--- a/llvm/tools/llvm-strings/Opts.td
+++ b/llvm/tools/llvm-strings/Opts.td
@@ -11,13 +11,29 @@ multiclass Eq<string name, string help> {
def all : FF<"all", "Silently ignored. Present for GNU strings compatibility">;
defm bytes : Eq<"bytes", "Print sequences of the specified length">;
+defm encoding : Eq<"encoding", [{Select the character encoding of the strings to find:
+s (7-bit characters)
+S (8-bit characters in the current locale)
+u (UTF-8)
+b (UTF-16 big endian)
+l (UTF-16 little endian)
+B (UTF-32 big endian)
+L (UTF-32 little endian)}]>;
def help : FF<"help", "Display this help">;
def print_file_name : Flag<["--"], "print-file-name">, HelpText<"Print the name of the file before each string">;
defm radix : Eq<"radix", "Print the offset within the file with the specified radix: o (octal), d (decimal), x (hexadecimal)">, MetaVarName<"<radix>">;
+defm unicode : Eq<"unicode", [{Control the display of UTF-8 multibyte characters:
+default (print UTF-8 characters)
+invalid (reject multibyte characters)
+locale (print characters in the current locale)
+escape (\u0000 escape sequences)
+hex (print UTF-8 encoding in hex)
+highlight (like escape, but highlighted)}]>;
def version : FF<"version", "Display the version">;
def : F<"a", "Alias for --all">, Alias<all>;
def : F<"f", "Alias for --print-file-name">, Alias<print_file_name>;
def : F<"h", "Alias for --help">, Alias<help>;
def : JoinedOrSeparate<["-"], "n">, Alias<bytes_EQ>, HelpText<"Alias for --bytes">;
+def : JoinedOrSeparate<["-"], "e">, Alias<encoding_EQ>, HelpText<"Alias for --encoding">;
def : JoinedOrSeparate<["-"], "t">, Alias<radix_EQ>, HelpText<"Alias for --radix">, MetaVarName<"<radix>">;
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 9979b93de84270..655cca584437c2 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -18,13 +18,17 @@
#include "llvm/Option/ArgList.h"
#include "llvm/Option/Option.h"
#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/ConvertUTF.h"
#include "llvm/Support/Error.h"
#include "llvm/Support/Format.h"
#include "llvm/Support/InitLLVM.h"
#include "llvm/Support/MemoryBuffer.h"
#include "llvm/Support/Program.h"
+#include "llvm/Support/SwapByteOrder.h"
+#include "llvm/Support/Unicode.h"
#include "llvm/Support/WithColor.h"
#include <cctype>
+#include <locale>
#include <string>
using namespace llvm;
@@ -71,9 +75,15 @@ static cl::list<std::string> InputFileNames(cl::Positional,
static int MinLength = 4;
static bool PrintFileName;
-enum radix { none, octal, hexadecimal, decimal };
+enum class encoding { s, S, u, b, l, B, L };
+static encoding Encoding;
+
+enum class radix { none, octal, hexadecimal, decimal };
static radix Radix;
+enum class unicode { default_, invalid, locale, escape, hex, highlight };
+static unicode Unicode;
+
[[noreturn]] static void reportCmdLineError(const Twine &Message) {
WithColor::error(errs(), ToolName) << Message << "\n";
exit(1);
@@ -89,40 +99,224 @@ static void parseIntArg(const opt::InputArgList &Args, int ID, T &Value) {
}
static void strings(raw_ostream &OS, StringRef FileName, StringRef Contents) {
- auto print = [&OS, FileName](unsigned Offset, StringRef L) {
- if (L.size() < static_cast<size_t>(MinLength))
+ std::locale loc("");
+ auto &cvt = std::use_facet<std::codecvt<wchar_t, char, std::mbstate_t>>(loc);
+ auto &ctype = std::use_facet<std::ctype<wchar_t>>(loc);
+ std::mbstate_t mbs{};
+
+ const bool PrintRawBytes =
+ Encoding == encoding::s ||
+ (Encoding == encoding::S &&
+ (Unicode == unicode::invalid || Unicode == unicode::locale)) ||
+ (Encoding == encoding::u &&
+ (Unicode == unicode::default_ || Unicode == unicode::invalid));
+
+ auto read = [&cvt, &mbs](const char *&P, const char *E) -> UTF32 {
+ UTF32 Ch;
+
+ switch (Encoding) {
+ case encoding::s:
+ Ch = *P++;
+ break;
+
+ case encoding::S: {
+ const char *N;
+ wchar_t WCh;
+ wchar_t *WNext;
+ [[maybe_unused]] const auto res =
+ cvt.in(mbs, P, E, N, &WCh, &WCh + 1, WNext);
+ assert(res != std::codecvt_base::noconv);
+
+ // Only treat a non-null character as a successful conversion, as a null
+ // character may be the result of an incomplete multibyte character
+ // followed by a null byte. A null byte is safe to treat as an error, as
+ // a null byte is never printable in any locale.
+ if (WNext != &WCh && WCh) {
+ // Note: this assumes wchar_t is UCS2 or UTF32.
+ Ch = WCh;
+ P = N;
+ } else {
+ // If there was any error, skip the current byte and reset the state to
+ // allow the next byte to start a character.
+ Ch = 0;
+ mbs = {};
+ ++P;
+ }
+ break;
+ }
+
+ case encoding::u: {
+ const UTF8 *UP = reinterpret_cast<const UTF8 *>(P);
+ const UTF8 *UE = reinterpret_cast<const UTF8 *>(E);
+ UTF32 *Next = &Ch;
+ ConvertUTF8toUTF32(&UP, UE, &Next, &Ch + 1, strictConversion);
+ if (Next == &Ch || !Ch) {
+ // If there was any error, skip the current byte to allow the next to
+ // start a character.
+ Ch = 0;
+ UP = std::next(reinterpret_cast<const UTF8 *>(P));
+ }
+ P = reinterpret_cast<const char *>(UP);
+ break;
+ }
+
+ case encoding::b:
+ case encoding::l: {
+ const UTF16 *UP = reinterpret_cast<const UTF16 *>(P);
+ const UTF16 *UE = reinterpret_cast<const UTF16 *>(E);
+ const bool DoByteSwap =
+ (Encoding == encoding::b &&
+ endianness::native == endianness::little) ||
+ (Encoding == encoding::l && endianness::native == endianness::big);
+ UTF32 *Next = &Ch;
+ if (!DoByteSwap) {
+ ConvertUTF16toUTF32(&UP, UE, &Next, &Ch + 1, strictConversion);
+ } else {
+ // We never need more than two UTF16 words to make up one character.
+ const UTF16 Buf[2] = {byteswap(UP[0]),
+ UP + 1 == UE ? UTF16(0) : byteswap(UP[1])};
+ const UTF16 *BufP = Buf;
+ ConvertUTF16toUTF32(&BufP, &Buf[2], &Next, &Ch + 1, strictConversion);
+ UP += BufP - Buf;
+ }
+ if (Next == &Ch || !Ch) {
+ // If there was any error, skip the current word to allow the next to
+ // start a character.
+ Ch = 0;
+ UP = std::next(reinterpret_cast<const UTF16 *>(P));
+ }
+ P = reinterpret_cast<const char *>(UP);
+ break;
+ }
+
+ case encoding::B:
+ case encoding::L: {
+ const UTF32 *UP = reinterpret_cast<const UTF32 *>(P);
+ const bool DoByteSwap =
+ (Encoding == encoding::B &&
+ endianness::native == endianness::little) ||
+ (Encoding == encoding::L && endianness::native == endianness::big);
+ Ch = DoByteSwap ? byteswap(*UP) : *UP;
+ ++UP;
+ P = reinterpret_cast<const char *>(UP);
+ break;
+ }
+
+ default:
+ llvm_unreachable("unhandled encoding");
+ }
+
+ return Ch;
+ };
+
+ auto print = [&OS, FileName, PrintRawBytes, &read,
+ &cvt](unsigned Offset, StringRef L, size_t N) {
+ if (N < static_cast<size_t>(MinLength))
return;
if (PrintFileName)
OS << FileName << ": ";
switch (Radix) {
- case none:
+ case radix::none:
break;
- case octal:
+ case radix::octal:
OS << format("%7o ", Offset);
break;
- case hexadecimal:
+ case radix::hexadecimal:
OS << format("%7x ", Offset);
break;
- case decimal:
+ case radix::decimal:
OS << format("%7u ", Offset);
break;
}
- OS << L << '\n';
+
+ if (PrintRawBytes) {
+ OS << L << '\n';
+ } else {
+ mbstate_t mbs = {};
+
+ const char *P = L.begin();
+ const char *E = L.end();
+ while (P < E) {
+ const UTF32 Ch = read(P, E);
+ if (Unicode == unicode::invalid || Ch <= 0x7F) {
+ OS << (char)Ch;
+ continue;
+ }
+
+ if (Unicode == unicode::locale) {
+ // If we have a 16-bit wchar_t and the character does not fit, replace
+ // it with U+FFFD.
+ wchar_t WCh = Ch;
+ const wchar_t *WNext;
+ if ((UTF32)WCh != Ch)
+ WCh = 0xFFFD;
+ char mbstring[MB_LEN_MAX];
+ char *mbend;
+ [[maybe_unused]] const auto res =
+ cvt.out(mbs, &WCh, &WCh + 1, WNext, mbstring,
+ &mbstring[MB_LEN_MAX], mbend);
+ assert(res == std::codecvt_base::ok);
+ OS << StringRef(mbstring, mbend - mbstring);
+ continue;
+ }
+
+ if (Unicode == unicode::escape || Unicode == unicode::highlight) {
+ WithColor COS(OS, raw_ostream::RED, false, false,
+ Unicode == unicode::highlight ? ColorMode::Auto
+ : ColorMode::Disable);
+ if (Ch <= 0xFFFF)
+ COS << "\\u" << utohexstr(Ch, false, 4);
+ else
+ COS << "\\U" << utohexstr(Ch, false, 8);
+ continue;
+ }
+
+ char UTF8[4];
+ char *UTF8end = UTF8;
+ ConvertCodePointToUTF8(Ch, UTF8end);
+ if (Unicode == unicode::hex) {
+ OS << "<0x";
+ for (unsigned char Byte : StringRef(UTF8, UTF8end - UTF8))
+ OS << utohexstr(Byte, true, 2);
+ OS << '>';
+ continue;
+ }
+
+ assert(Unicode == unicode::default_);
+ OS << StringRef(UTF8, UTF8end - UTF8);
+ }
+ OS << '\n';
+ }
};
+ std::size_t NumPrintable = 0;
+
const char *B = Contents.begin();
- const char *P = nullptr, *E = nullptr, *S = nullptr;
- for (P = Contents.begin(), E = Contents.end(); P < E; ++P) {
- if (isPrint(*P) || *P == '\t') {
+ const char *P = Contents.begin(), *E = Contents.end(), *S = nullptr;
+ for (P = Contents.begin(), E = Contents.end(); P < E;) {
+ const char *N = P;
+ UTF32 Ch = read(N, E);
+
+ const bool Printable =
+ Ch == '\t' ||
+ (Unicode == unicode::invalid ? Ch <= 0x7f && isPrint(Ch)
+ : Encoding == encoding::S ? ctype.is(std::ctype_base::print, Ch)
+ : sys::unicode::isPrintable(Ch));
+
+ if (Printable) {
if (S == nullptr)
S = P;
+ ++NumPrintable;
} else if (S) {
- print(S - B, StringRef(S, P - S));
+ print(S - B, StringRef(S, P - S), NumPrintable);
S = nullptr;
+ NumPrintable = 0;
}
+
+ P = N;
}
if (S)
- print(S - B, StringRef(S, E - S));
+ print(S - B, StringRef(S, E - S), NumPrintable);
}
int main(int argc, char **argv) {
@@ -149,20 +343,89 @@ int main(int argc, char **argv) {
return 0;
}
+ const auto *EncodingArg = Args.getLastArg(OPT_encoding_EQ);
+ const auto *UnicodeArg = Args.getLastArg(OPT_unicode_EQ);
+
+ Encoding =
+ EncodingArg
+ ? llvm::StringSwitch<encoding>(EncodingArg->getValue())
+ .Case("s", encoding::s)
+ .Case("S", encoding::S)
+ .Case("u", encoding::u)
+ .Case("b", encoding::b)
+ .Case("l", encoding::l)
+ .Case("B", encoding::B)
+ .Case("L", encoding::L)
+ .Predicate(
+ [](StringRef) -> bool {
+ reportCmdLineError(
+ "--encoding value should be one of: "
+ "'s' (7-bit characters), "
+ "'S' (8-bit characters), "
+ "'u' (UTF-8 characters), "
+ "'b', 'l' (16-bit big/little endian characters), "
+ "'B', 'L' (32-bit big/little endian characters)");
+ },
+ encoding{})
+ .DefaultUnreachable()
+ : encoding::S;
parseIntArg(Args, OPT_bytes_EQ, MinLength);
PrintFileName = Args.hasArg(OPT_print_file_name);
StringRef R = Args.getLastArgValue(OPT_radix_EQ);
if (R.empty())
- Radix = none;
+ Radix = radix::none;
else if (R == "o")
- Radix = octal;
+ Radix = radix::octal;
else if (R == "d")
- Radix = decimal;
+ Radix = radix::decimal;
else if (R == "x")
- Radix = hexadecimal;
+ Radix = radix::hexadecimal;
else
reportCmdLineError("--radix value should be one of: '' (no offset), 'o' "
"(octal), 'd' (decimal), 'x' (hexadecimal)");
+ Unicode = UnicodeArg ? llvm::StringSwitch<unicode>(UnicodeArg->getValue())
+ .Case("default", unicode::default_)
+ .Case("invalid", unicode::invalid)
+ .Case("locale", unicode::locale)
+ .Case("escape", unicode::escape)
+ .Case("hex", unicode::hex)
+ .Case("highlight", unicode::highlight)
+ .Predicate(
+ [](StringRef) -> bool {
+ reportCmdLineError(
+ "--unicode value should be one of: "
+ "default, "
+ "invalid, "
+ "locale, "
+ "escape, "
+ "hex, "
+ "highlight");
+ },
+ unicode::default_)
+ .DefaultUnreachable()
+ : unicode::locale;
+
+ // The defaults are ugly to maintain compatibility with GNU strings in the
+ // common cases. The general idea is that --encoding specifies the input
+ // encoding, --unicode specifies the output encoding, but when one is
+ // specified, the other defaults to the best match with the limitation that
+ // the output encoding is never UTF-16 or UTF-32.
+ if (!EncodingArg && Unicode != unicode::invalid && Unicode != unicode::locale)
+ Encoding = encoding::u;
+
+ if (!UnicodeArg && Encoding != encoding::S)
+ Unicode = unicode::default_;
+
+ // With --encoding=s, all output encodings would give the same results. Use
+ // the simplest one.
+ if (Encoding == encoding::s)
+ Unicode = unicode::invalid;
+
+ // With --encoding=u --unicode=invalid, we do not need to spend time decoding
+ // UTF-8 only to reject whatever results we get, we can reject multibyte
+ // characters right away.
+ if (Encoding == encoding::u && Unicode == unicode::invalid)
+ Encoding = encoding::s;
if (MinLength == 0) {
errs() << "invalid minimum string length 0\n";
>From 11d4e557a1824d7cd71f720cbed1e9a162e69f50 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Mon, 7 Sep 2026 19:46:55 +0100
Subject: [PATCH 02/41] Remove llvm_unreachable to suppress compiler warning
---
llvm/tools/llvm-strings/llvm-strings.cpp | 3 ---
1 file changed, 3 deletions(-)
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 655cca584437c2..79baa08cc6801f 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -201,9 +201,6 @@ static void strings(raw_ostream &OS, StringRef FileName, StringRef Contents) {
P = reinterpret_cast<const char *>(UP);
break;
}
-
- default:
- llvm_unreachable("unhandled encoding");
}
return Ch;
>From d6cb511d4d1f18486f1221f7b57ea0cca7d7d026 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Tue, 8 Sep 2026 14:47:58 +0100
Subject: [PATCH 03/41] Update manpage, remove --encoding=b/l/B/L and
--unicode, add test
---
llvm/docs/CommandGuide/llvm-strings.md | 15 +-
llvm/test/tools/llvm-strings/encoding.test | 28 +++
llvm/test/tools/llvm-strings/length.test | 36 ---
.../tools/llvm-strings/negative-char.test | 2 +-
llvm/tools/llvm-strings/Opts.td | 17 +-
llvm/tools/llvm-strings/llvm-strings.cpp | 208 ++----------------
6 files changed, 67 insertions(+), 239 deletions(-)
create mode 100644 llvm/test/tools/llvm-strings/encoding.test
diff --git a/llvm/docs/CommandGuide/llvm-strings.md b/llvm/docs/CommandGuide/llvm-strings.md
index da9d0a03d9fd55..6789a316a66849 100644
--- a/llvm/docs/CommandGuide/llvm-strings.md
+++ b/llvm/docs/CommandGuide/llvm-strings.md
@@ -12,8 +12,8 @@
{program}`llvm-strings` is a tool intended as a drop-in replacement for GNU's
{program}`strings`, which looks for printable strings in files and writes them
to the standard output stream. A printable string is any sequence of four (by
-default) or more printable ASCII characters. The end of the file, or any other
-byte, terminates the current sequence.
+default) or more printable characters. The end of the file, or any other byte,
+terminates the current sequence.
{program}`llvm-strings` looks for strings in each `input` file specified.
Unlike GNU {program}`strings` it looks in the entire input file, regardless of
@@ -40,8 +40,15 @@ Silently ignored. Present for GNU {program}`strings` compatibility.
:::
:::{option} --bytes=<length>, -n
-Set the minimum number of printable ASCII characters required for a sequence of
-bytes to be considered a string. The default value is 4.
+Set the minimum number of printable characters required for a sequence to be
+considered a string. The default value is 4.
+:::
+
+:::{option} --encoding=<encoding>, -e
+Specifies the encoding of the input file. Valid arguments are:
+- s (ASCII characters)
+- S (characters in the system's or user's selected character set)
+- u (UTF-8 characters)
:::
:::{option} --help, -h
diff --git a/llvm/test/tools/llvm-strings/encoding.test b/llvm/test/tools/llvm-strings/encoding.test
new file mode 100644
index 00000000000000..7355516e693cfa
--- /dev/null
+++ b/llvm/test/tools/llvm-strings/encoding.test
@@ -0,0 +1,28 @@
+## Show that llvm-strings uses the specified encoding.
+
+RUN: echo a > %t
+RUN: echo ab >> %t
+RUN: echo abc >> %t
+RUN: echo abcd >> %t
+RUN: echo abcd€ >> %t
+
+# Check that long form options work.
+RUN: llvm-strings --encoding s 2>&1 %t | FileCheck --check-prefixes CHECK,CHECK-ASCII %s
+RUN: llvm-strings --encoding S 2>&1 %t | FileCheck %s
+RUN: llvm-strings --encoding u 2>&1 %t | FileCheck --check-prefixes CHECK,CHECK-UTF8 %s
+RUN: not llvm-strings --encoding x 2>&1 %t | FileCheck --check-prefixes CHECK-ERROR %s
+
+# Check that short form options work.
+RUN: llvm-strings -e s 2>&1 %t | FileCheck --check-prefixes CHECK,CHECK-ASCII %s
+RUN: llvm-strings -e S 2>&1 %t | FileCheck %s
+RUN: llvm-strings -e u 2>&1 %t | FileCheck --check-prefixes CHECK,CHECK-UTF8 %s
+
+CHECK: abcd
+CHECK-ASCII-NOT: abcd€
+CHECK-UTF8: abcd€
+
+# Check that invalid encodings are rejected.
+RUN: not llvm-strings --encoding x 2>&1 %t | FileCheck --check-prefix CHECK-ERROR %s
+RUN: not llvm-strings -e x 2>&1 %t | FileCheck --check-prefix CHECK-ERROR %s
+
+CHECK-ERROR: error: --encoding value
diff --git a/llvm/test/tools/llvm-strings/length.test b/llvm/test/tools/llvm-strings/length.test
index aaa3df198f6333..5ec95baea317eb 100644
--- a/llvm/test/tools/llvm-strings/length.test
+++ b/llvm/test/tools/llvm-strings/length.test
@@ -55,42 +55,6 @@ RUN: llvm-strings --encoding u -n 2 %t | FileCheck --check-prefix UTF-8-2 %s --i
RUN: llvm-strings --encoding u -n 3 %t | FileCheck --check-prefix UTF-8-3 %s --implicit-check-not={{.}}
RUN: llvm-strings --encoding u -n 4 %t | FileCheck %s --implicit-check-not={{.}} --allow-empty
-RUN: printf '\241\0\n\0' > %t
-RUN: printf '\241\0\242\0\n\0' >> %t
-RUN: printf '\241\0\242\0\243\0\n\0' >> %t
-RUN: not llvm-strings --encoding l -n 0 2>&1 %t | FileCheck --check-prefix CHECK-0 %s
-RUN: llvm-strings --encoding l -n 1 %t | FileCheck --check-prefix UTF-8-1 %s --implicit-check-not={{.}}
-RUN: llvm-strings --encoding l -n 2 %t | FileCheck --check-prefix UTF-8-2 %s --implicit-check-not={{.}}
-RUN: llvm-strings --encoding l -n 3 %t | FileCheck --check-prefix UTF-8-3 %s --implicit-check-not={{.}}
-RUN: llvm-strings --encoding l -n 4 %t | FileCheck %s --implicit-check-not={{.}} --allow-empty
-
-RUN: printf '\0\241\0\n' > %t
-RUN: printf '\0\241\0\242\0\n' >> %t
-RUN: printf '\0\241\0\242\0\243\0\n' >> %t
-RUN: not llvm-strings --encoding b -n 0 2>&1 %t | FileCheck --check-prefix CHECK-0 %s
-RUN: llvm-strings --encoding b -n 1 %t | FileCheck --check-prefix UTF-8-1 %s --implicit-check-not={{.}}
-RUN: llvm-strings --encoding b -n 2 %t | FileCheck --check-prefix UTF-8-2 %s --implicit-check-not={{.}}
-RUN: llvm-strings --encoding b -n 3 %t | FileCheck --check-prefix UTF-8-3 %s --implicit-check-not={{.}}
-RUN: llvm-strings --encoding b -n 4 %t | FileCheck %s --implicit-check-not={{.}} --allow-empty
-
-RUN: printf '\241\0\0\0\n\0\0\0' > %t
-RUN: printf '\241\0\0\0\242\0\0\0\n\0\0\0' >> %t
-RUN: printf '\241\0\0\0\242\0\0\0\243\0\0\0\n\0\0\0' >> %t
-RUN: not llvm-strings --encoding L -n 0 2>&1 %t | FileCheck --check-prefix CHECK-0 %s
-RUN: llvm-strings --encoding L -n 1 %t | FileCheck --check-prefix UTF-8-1 %s --implicit-check-not={{.}}
-RUN: llvm-strings --encoding L -n 2 %t | FileCheck --check-prefix UTF-8-2 %s --implicit-check-not={{.}}
-RUN: llvm-strings --encoding L -n 3 %t | FileCheck --check-prefix UTF-8-3 %s --implicit-check-not={{.}}
-RUN: llvm-strings --encoding L -n 4 %t | FileCheck %s --implicit-check-not={{.}} --allow-empty
-
-RUN: printf '\0\0\0\241\0\0\0\n' > %t
-RUN: printf '\0\0\0\241\0\0\0\242\0\0\0\n' >> %t
-RUN: printf '\0\0\0\241\0\0\0\242\0\0\0\243\0\0\0\n' >> %t
-RUN: not llvm-strings --encoding B -n 0 2>&1 %t | FileCheck --check-prefix CHECK-0 %s
-RUN: llvm-strings --encoding B -n 1 %t | FileCheck --check-prefix UTF-8-1 %s --implicit-check-not={{.}}
-RUN: llvm-strings --encoding B -n 2 %t | FileCheck --check-prefix UTF-8-2 %s --implicit-check-not={{.}}
-RUN: llvm-strings --encoding B -n 3 %t | FileCheck --check-prefix UTF-8-3 %s --implicit-check-not={{.}}
-RUN: llvm-strings --encoding B -n 4 %t | FileCheck %s --implicit-check-not={{.}} --allow-empty
-
UTF-8-1: ¡
UTF-8-1-NEXT: ¡¢
UTF-8-1-NEXT: ¡¢£
diff --git a/llvm/test/tools/llvm-strings/negative-char.test b/llvm/test/tools/llvm-strings/negative-char.test
index 3e661b04724706..8e61700e07e8af 100644
--- a/llvm/test/tools/llvm-strings/negative-char.test
+++ b/llvm/test/tools/llvm-strings/negative-char.test
@@ -3,7 +3,7 @@
## unless using an encoding where they form valid characters.
# RUN: printf "z\0\200\0\302\241\0a\0" | llvm-strings --encoding=s --bytes 1 - | FileCheck %s
-# RUN: printf "z\0\200\0\302\241\0a\0" | llvm-strings --unicode=default --bytes 1 - | FileCheck %s --check-prefixes=CHECK,CHECK-UTF8
+# RUN: printf "z\0\200\0\302\241\0a\0" | llvm-strings --encoding=u --bytes 1 - | FileCheck %s --check-prefixes=CHECK,CHECK-UTF8
# CHECK: z{{$}}
# CHECK-UTF8-NEXT: {{^}}¡{{$}}
# CHECK-NEXT: {{^}}a
diff --git a/llvm/tools/llvm-strings/Opts.td b/llvm/tools/llvm-strings/Opts.td
index cf7a406fcec1f9..dd4f4349d7eed9 100644
--- a/llvm/tools/llvm-strings/Opts.td
+++ b/llvm/tools/llvm-strings/Opts.td
@@ -12,23 +12,12 @@ multiclass Eq<string name, string help> {
def all : FF<"all", "Silently ignored. Present for GNU strings compatibility">;
defm bytes : Eq<"bytes", "Print sequences of the specified length">;
defm encoding : Eq<"encoding", [{Select the character encoding of the strings to find:
-s (7-bit characters)
-S (8-bit characters in the current locale)
-u (UTF-8)
-b (UTF-16 big endian)
-l (UTF-16 little endian)
-B (UTF-32 big endian)
-L (UTF-32 little endian)}]>;
+s (ASCII characters)
+S (characters in the system's or user's character set)
+u (UTF-8 characters)}]>;
def help : FF<"help", "Display this help">;
def print_file_name : Flag<["--"], "print-file-name">, HelpText<"Print the name of the file before each string">;
defm radix : Eq<"radix", "Print the offset within the file with the specified radix: o (octal), d (decimal), x (hexadecimal)">, MetaVarName<"<radix>">;
-defm unicode : Eq<"unicode", [{Control the display of UTF-8 multibyte characters:
-default (print UTF-8 characters)
-invalid (reject multibyte characters)
-locale (print characters in the current locale)
-escape (\u0000 escape sequences)
-hex (print UTF-8 encoding in hex)
-highlight (like escape, but highlighted)}]>;
def version : FF<"version", "Display the version">;
def : F<"a", "Alias for --all">, Alias<all>;
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 79baa08cc6801f..bb61425f790d34 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -75,15 +75,12 @@ static cl::list<std::string> InputFileNames(cl::Positional,
static int MinLength = 4;
static bool PrintFileName;
-enum class encoding { s, S, u, b, l, B, L };
+enum class encoding { s, S, u };
static encoding Encoding;
enum class radix { none, octal, hexadecimal, decimal };
static radix Radix;
-enum class unicode { default_, invalid, locale, escape, hex, highlight };
-static unicode Unicode;
-
[[noreturn]] static void reportCmdLineError(const Twine &Message) {
WithColor::error(errs(), ToolName) << Message << "\n";
exit(1);
@@ -104,13 +101,6 @@ static void strings(raw_ostream &OS, StringRef FileName, StringRef Contents) {
auto &ctype = std::use_facet<std::ctype<wchar_t>>(loc);
std::mbstate_t mbs{};
- const bool PrintRawBytes =
- Encoding == encoding::s ||
- (Encoding == encoding::S &&
- (Unicode == unicode::invalid || Unicode == unicode::locale)) ||
- (Encoding == encoding::u &&
- (Unicode == unicode::default_ || Unicode == unicode::invalid));
-
auto read = [&cvt, &mbs](const char *&P, const char *E) -> UTF32 {
UTF32 Ch;
@@ -159,55 +149,13 @@ static void strings(raw_ostream &OS, StringRef FileName, StringRef Contents) {
P = reinterpret_cast<const char *>(UP);
break;
}
-
- case encoding::b:
- case encoding::l: {
- const UTF16 *UP = reinterpret_cast<const UTF16 *>(P);
- const UTF16 *UE = reinterpret_cast<const UTF16 *>(E);
- const bool DoByteSwap =
- (Encoding == encoding::b &&
- endianness::native == endianness::little) ||
- (Encoding == encoding::l && endianness::native == endianness::big);
- UTF32 *Next = &Ch;
- if (!DoByteSwap) {
- ConvertUTF16toUTF32(&UP, UE, &Next, &Ch + 1, strictConversion);
- } else {
- // We never need more than two UTF16 words to make up one character.
- const UTF16 Buf[2] = {byteswap(UP[0]),
- UP + 1 == UE ? UTF16(0) : byteswap(UP[1])};
- const UTF16 *BufP = Buf;
- ConvertUTF16toUTF32(&BufP, &Buf[2], &Next, &Ch + 1, strictConversion);
- UP += BufP - Buf;
- }
- if (Next == &Ch || !Ch) {
- // If there was any error, skip the current word to allow the next to
- // start a character.
- Ch = 0;
- UP = std::next(reinterpret_cast<const UTF16 *>(P));
- }
- P = reinterpret_cast<const char *>(UP);
- break;
- }
-
- case encoding::B:
- case encoding::L: {
- const UTF32 *UP = reinterpret_cast<const UTF32 *>(P);
- const bool DoByteSwap =
- (Encoding == encoding::B &&
- endianness::native == endianness::little) ||
- (Encoding == encoding::L && endianness::native == endianness::big);
- Ch = DoByteSwap ? byteswap(*UP) : *UP;
- ++UP;
- P = reinterpret_cast<const char *>(UP);
- break;
- }
}
return Ch;
};
- auto print = [&OS, FileName, PrintRawBytes, &read,
- &cvt](unsigned Offset, StringRef L, size_t N) {
+ auto print = [&OS, FileName, &read, &cvt](unsigned Offset, StringRef L,
+ size_t N) {
if (N < static_cast<size_t>(MinLength))
return;
if (PrintFileName)
@@ -226,64 +174,7 @@ static void strings(raw_ostream &OS, StringRef FileName, StringRef Contents) {
break;
}
- if (PrintRawBytes) {
- OS << L << '\n';
- } else {
- mbstate_t mbs = {};
-
- const char *P = L.begin();
- const char *E = L.end();
- while (P < E) {
- const UTF32 Ch = read(P, E);
- if (Unicode == unicode::invalid || Ch <= 0x7F) {
- OS << (char)Ch;
- continue;
- }
-
- if (Unicode == unicode::locale) {
- // If we have a 16-bit wchar_t and the character does not fit, replace
- // it with U+FFFD.
- wchar_t WCh = Ch;
- const wchar_t *WNext;
- if ((UTF32)WCh != Ch)
- WCh = 0xFFFD;
- char mbstring[MB_LEN_MAX];
- char *mbend;
- [[maybe_unused]] const auto res =
- cvt.out(mbs, &WCh, &WCh + 1, WNext, mbstring,
- &mbstring[MB_LEN_MAX], mbend);
- assert(res == std::codecvt_base::ok);
- OS << StringRef(mbstring, mbend - mbstring);
- continue;
- }
-
- if (Unicode == unicode::escape || Unicode == unicode::highlight) {
- WithColor COS(OS, raw_ostream::RED, false, false,
- Unicode == unicode::highlight ? ColorMode::Auto
- : ColorMode::Disable);
- if (Ch <= 0xFFFF)
- COS << "\\u" << utohexstr(Ch, false, 4);
- else
- COS << "\\U" << utohexstr(Ch, false, 8);
- continue;
- }
-
- char UTF8[4];
- char *UTF8end = UTF8;
- ConvertCodePointToUTF8(Ch, UTF8end);
- if (Unicode == unicode::hex) {
- OS << "<0x";
- for (unsigned char Byte : StringRef(UTF8, UTF8end - UTF8))
- OS << utohexstr(Byte, true, 2);
- OS << '>';
- continue;
- }
-
- assert(Unicode == unicode::default_);
- OS << StringRef(UTF8, UTF8end - UTF8);
- }
- OS << '\n';
- }
+ OS << L << '\n';
};
std::size_t NumPrintable = 0;
@@ -296,9 +187,9 @@ static void strings(raw_ostream &OS, StringRef FileName, StringRef Contents) {
const bool Printable =
Ch == '\t' ||
- (Unicode == unicode::invalid ? Ch <= 0x7f && isPrint(Ch)
- : Encoding == encoding::S ? ctype.is(std::ctype_base::print, Ch)
- : sys::unicode::isPrintable(Ch));
+ (Encoding == encoding::s ? Ch <= 0x7f && isPrint(Ch)
+ : Encoding == encoding::S ? ctype.is(std::ctype_base::print, Ch)
+ : sys::unicode::isPrintable(Ch));
if (Printable) {
if (S == nullptr)
@@ -341,31 +232,23 @@ int main(int argc, char **argv) {
}
const auto *EncodingArg = Args.getLastArg(OPT_encoding_EQ);
- const auto *UnicodeArg = Args.getLastArg(OPT_unicode_EQ);
-
- Encoding =
- EncodingArg
- ? llvm::StringSwitch<encoding>(EncodingArg->getValue())
- .Case("s", encoding::s)
- .Case("S", encoding::S)
- .Case("u", encoding::u)
- .Case("b", encoding::b)
- .Case("l", encoding::l)
- .Case("B", encoding::B)
- .Case("L", encoding::L)
- .Predicate(
- [](StringRef) -> bool {
- reportCmdLineError(
- "--encoding value should be one of: "
- "'s' (7-bit characters), "
- "'S' (8-bit characters), "
- "'u' (UTF-8 characters), "
- "'b', 'l' (16-bit big/little endian characters), "
- "'B', 'L' (32-bit big/little endian characters)");
- },
- encoding{})
- .DefaultUnreachable()
- : encoding::S;
+
+ Encoding = EncodingArg ? llvm::StringSwitch<encoding>(EncodingArg->getValue())
+ .Case("s", encoding::s)
+ .Case("S", encoding::S)
+ .Case("u", encoding::u)
+ .Predicate(
+ [](StringRef) -> bool {
+ reportCmdLineError(
+ "--encoding value should be one of: "
+ "'s' (ASCII characters), "
+ "'S' (characters in the system's or "
+ "user's selected character set), "
+ "'u' (UTF-8 characters)");
+ },
+ encoding{})
+ .DefaultUnreachable()
+ : encoding::S;
parseIntArg(Args, OPT_bytes_EQ, MinLength);
PrintFileName = Args.hasArg(OPT_print_file_name);
StringRef R = Args.getLastArgValue(OPT_radix_EQ);
@@ -380,49 +263,6 @@ int main(int argc, char **argv) {
else
reportCmdLineError("--radix value should be one of: '' (no offset), 'o' "
"(octal), 'd' (decimal), 'x' (hexadecimal)");
- Unicode = UnicodeArg ? llvm::StringSwitch<unicode>(UnicodeArg->getValue())
- .Case("default", unicode::default_)
- .Case("invalid", unicode::invalid)
- .Case("locale", unicode::locale)
- .Case("escape", unicode::escape)
- .Case("hex", unicode::hex)
- .Case("highlight", unicode::highlight)
- .Predicate(
- [](StringRef) -> bool {
- reportCmdLineError(
- "--unicode value should be one of: "
- "default, "
- "invalid, "
- "locale, "
- "escape, "
- "hex, "
- "highlight");
- },
- unicode::default_)
- .DefaultUnreachable()
- : unicode::locale;
-
- // The defaults are ugly to maintain compatibility with GNU strings in the
- // common cases. The general idea is that --encoding specifies the input
- // encoding, --unicode specifies the output encoding, but when one is
- // specified, the other defaults to the best match with the limitation that
- // the output encoding is never UTF-16 or UTF-32.
- if (!EncodingArg && Unicode != unicode::invalid && Unicode != unicode::locale)
- Encoding = encoding::u;
-
- if (!UnicodeArg && Encoding != encoding::S)
- Unicode = unicode::default_;
-
- // With --encoding=s, all output encodings would give the same results. Use
- // the simplest one.
- if (Encoding == encoding::s)
- Unicode = unicode::invalid;
-
- // With --encoding=u --unicode=invalid, we do not need to spend time decoding
- // UTF-8 only to reject whatever results we get, we can reject multibyte
- // characters right away.
- if (Encoding == encoding::u && Unicode == unicode::invalid)
- Encoding = encoding::s;
if (MinLength == 0) {
errs() << "invalid minimum string length 0\n";
>From 16213a08dc2fb154d0d3c105b611093f194c84ba Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Tue, 8 Sep 2026 16:04:04 +0100
Subject: [PATCH 04/41] Remove unused captures
---
llvm/tools/llvm-strings/llvm-strings.cpp | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index bb61425f790d34..2444847ef812ca 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -154,8 +154,7 @@ static void strings(raw_ostream &OS, StringRef FileName, StringRef Contents) {
return Ch;
};
- auto print = [&OS, FileName, &read, &cvt](unsigned Offset, StringRef L,
- size_t N) {
+ auto print = [&OS, FileName](unsigned Offset, StringRef L, size_t N) {
if (N < static_cast<size_t>(MinLength))
return;
if (PrintFileName)
>From 02654008d5f94523fa2fc8e8ac521af576b3653d Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Fri, 11 Sep 2026 14:17:34 +0100
Subject: [PATCH 05/41] Apply review comments to docs, encoding.test
---
llvm/docs/CommandGuide/llvm-strings.md | 6 +++---
llvm/test/tools/llvm-strings/encoding.test | 17 +++++++++++------
2 files changed, 14 insertions(+), 9 deletions(-)
diff --git a/llvm/docs/CommandGuide/llvm-strings.md b/llvm/docs/CommandGuide/llvm-strings.md
index e72cea195d6e5e..5a45dcd426c058 100644
--- a/llvm/docs/CommandGuide/llvm-strings.md
+++ b/llvm/docs/CommandGuide/llvm-strings.md
@@ -12,8 +12,8 @@
{program}`llvm-strings` is a tool intended as a drop-in replacement for GNU's
{program}`strings`, which looks for printable strings in files and writes them
to the standard output stream. A printable string is any sequence of four (by
-default) or more printable characters. The end of the file, or any other byte,
-terminates the current sequence.
+default) or more printable characters. The end of the file, or any unprintable
+byte sequence, terminates the current sequence.
{program}`llvm-strings` looks for strings in each `input` file specified.
Unlike GNU {program}`strings` it looks in the entire input file, regardless of
@@ -51,7 +51,7 @@ characters.
:::{option} --encoding=<encoding>, -e
Specifies the encoding of the input file. Valid arguments are:
- `s` (ASCII)
-- `S` (default character set)
+- `S` (current locale's character set)
- `utf8` (UTF-8)
:::
diff --git a/llvm/test/tools/llvm-strings/encoding.test b/llvm/test/tools/llvm-strings/encoding.test
index 46a58789c5bb27..b07df9209d5852 100644
--- a/llvm/test/tools/llvm-strings/encoding.test
+++ b/llvm/test/tools/llvm-strings/encoding.test
@@ -1,31 +1,36 @@
-## Show that llvm-strings uses the specified encoding.
+# Show that llvm-strings uses the specified encoding.
RUN: echo a > %t
RUN: echo ab >> %t
RUN: echo abc >> %t
RUN: echo abcd >> %t
RUN: echo abcd€ >> %t
-## Check that long form options work.
+# Check that long form options work.
RUN: llvm-strings --encoding s %t | FileCheck --check-prefixes CHECK,CHECK-ASCII --strict-whitespace --match-full-lines %s
RUN: llvm-strings --encoding S %t | FileCheck --check-prefixes CHECK,CHECK-LOCALE --strict-whitespace --match-full-lines %s
RUN: llvm-strings --encoding utf8 %t | FileCheck --check-prefixes CHECK,CHECK-UTF8 --strict-whitespace --match-full-lines %s
-## Check that short form options work.
+# Check that short form options work.
RUN: llvm-strings -e s %t | FileCheck --check-prefixes CHECK,CHECK-ASCII --strict-whitespace --match-full-lines %s
RUN: llvm-strings -e S %t | FileCheck --check-prefixes CHECK,CHECK-LOCALE --strict-whitespace --match-full-lines %s
RUN: llvm-strings -e utf8 %t | FileCheck --check-prefixes CHECK,CHECK-UTF8 --strict-whitespace --match-full-lines %s
-## Check that the default output matches that of -e S.
+# Check that the default output matches that of -e S.
RUN: llvm-strings -e S %t > %t.1
RUN: llvm-strings %t > %t.2
RUN: cmp %t.1 %t.2
+# The first line containing just abcd should always be matched.
CHECK:abcd
-CHECK-ASCII-NOT:abcd€
+# In -e s, the bytes of € should not be included.
+CHECK-ASCII:abcd
+# In -e S, any bytes of € may or may not be included depending on the locale
+# in effect. The locale in effect may also be altered by llvm-lit.
CHECK-LOCALE:abcd{{.*}}
+# In -e utf8, all bytes of € should be included.
CHECK-UTF8:abcd€
-## Check that invalid --encoding values are rejected.
+# Check that invalid --encoding values are rejected.
RUN: not llvm-strings --encoding x 2>&1 %t | FileCheck --check-prefix CHECK-ERROR %s
RUN: not llvm-strings -e x 2>&1 %t | FileCheck --check-prefix CHECK-ERROR %s
>From 1210ded87928d89b63865cb4d22ee3ad07adc965 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Fri, 25 Sep 2026 17:42:42 +0100
Subject: [PATCH 06/41] Avoid raw UTF-8 in test
---
llvm/test/tools/llvm-strings/encoding.test | 22 +++++++++++++++-------
1 file changed, 15 insertions(+), 7 deletions(-)
diff --git a/llvm/test/tools/llvm-strings/encoding.test b/llvm/test/tools/llvm-strings/encoding.test
index b07df9209d5852..b0b1b8f3164584 100644
--- a/llvm/test/tools/llvm-strings/encoding.test
+++ b/llvm/test/tools/llvm-strings/encoding.test
@@ -3,7 +3,10 @@ RUN: echo a > %t
RUN: echo ab >> %t
RUN: echo abc >> %t
RUN: echo abcd >> %t
-RUN: echo abcd€ >> %t
+# UTF-8 encoding of U+200A HAIR SPACE (printable)
+RUN: printf 'abcd\342\200\212\n' >> %t
+# UTF-8 encoding of U+200B ZERO WIDTH SPACE (non-printable)
+RUN: printf 'abcd\342\200\212\342\200\213\n' >> %t
# Check that long form options work.
RUN: llvm-strings --encoding s %t | FileCheck --check-prefixes CHECK,CHECK-ASCII --strict-whitespace --match-full-lines %s
@@ -22,13 +25,18 @@ RUN: cmp %t.1 %t.2
# The first line containing just abcd should always be matched.
CHECK:abcd
-# In -e s, the bytes of € should not be included.
+# In -e s, the bytes of U+200A and U+200B should not be included.
CHECK-ASCII:abcd
-# In -e S, any bytes of € may or may not be included depending on the locale
-# in effect. The locale in effect may also be altered by llvm-lit.
-CHECK-LOCALE:abcd{{.*}}
-# In -e utf8, all bytes of € should be included.
-CHECK-UTF8:abcd€
+CHECK-ASCII:abcd
+# In -e S, any bytes of U+200A and U+200B may or may not be included depending
+# on the locale in effect. The locale in effect may also be altered by llvm-lit.
+# Make sure if changing the test that the byte representations should not allow
+# this to be split into multiple strings in non-UTF-8 locales.
+CHECK-LOCALE:abcd{{(\xe2(\x80\x8a?)?)?}}
+CHECK-LOCALE:abcd{{(\xe2(\x80(\x8a(\xe2(\x80\x8b?)?)?)?)?)?}}
+# In -e utf8, all bytes of U+200A should be included, but none of U+200B.
+CHECK-UTF8:abcd{{\xe2\x80\x8a}}
+CHECK-UTF8:abcd{{\xe2\x80\x8a}}
# Check that invalid --encoding values are rejected.
RUN: not llvm-strings --encoding x 2>&1 %t | FileCheck --check-prefix CHECK-ERROR %s
>From 61021570fd59da32bbf3df18d35a7b5b5147d215 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Fri, 25 Sep 2026 17:55:13 +0100
Subject: [PATCH 07/41] Change other tests too
---
llvm/test/tools/llvm-strings/length.test | 12 ++++++------
llvm/test/tools/llvm-strings/negative-char.test | 2 +-
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/llvm/test/tools/llvm-strings/length.test b/llvm/test/tools/llvm-strings/length.test
index 01d64ee9cc71b2..79b7a324df6e9a 100644
--- a/llvm/test/tools/llvm-strings/length.test
+++ b/llvm/test/tools/llvm-strings/length.test
@@ -55,11 +55,11 @@ RUN: llvm-strings --encoding utf8 -n 2 %t | FileCheck --check-prefix UTF-8-2 %s
RUN: llvm-strings --encoding utf8 -n 3 %t | FileCheck --check-prefix UTF-8-3 %s --implicit-check-not={{.}}
RUN: llvm-strings --encoding utf8 -n 4 %t | FileCheck %s --implicit-check-not={{.}} --allow-empty
-UTF-8-1: ¡
-UTF-8-1-NEXT: ¡¢
-UTF-8-1-NEXT: ¡¢£
+UTF-8-1: {{\xc2\xa1}}
+UTF-8-1-NEXT: {{\xc2\xa1\xc2\xa2}}
+UTF-8-1-NEXT: {{\xc2\xa1\xc2\xa2\xc2\xa3}}
-UTF-8-2: ¡¢
-UTF-8-2-NEXT: ¡¢£
+UTF-8-2: {{\xc2\xa1\xc2\xa2}}
+UTF-8-2-NEXT: {{\xc2\xa1\xc2\xa2\xc2\xa3}}
-UTF-8-3: ¡¢£
+UTF-8-3: {{\xc2\xa1\xc2\xa2\xc2\xa3}}
diff --git a/llvm/test/tools/llvm-strings/negative-char.test b/llvm/test/tools/llvm-strings/negative-char.test
index b85dd133335e0a..d738f2fcb0b999 100644
--- a/llvm/test/tools/llvm-strings/negative-char.test
+++ b/llvm/test/tools/llvm-strings/negative-char.test
@@ -5,5 +5,5 @@
# RUN: printf "z\0\200\0\302\241\0a\0" | llvm-strings --encoding=s --bytes 1 - | FileCheck %s
# RUN: printf "z\0\200\0\302\241\0a\0" | llvm-strings --encoding=utf8 --bytes 1 - | FileCheck %s --check-prefixes=CHECK,CHECK-UTF8
# CHECK: z{{$}}
-# CHECK-UTF8-NEXT: {{^}}¡{{$}}
+# CHECK-UTF8-NEXT: {{^\xc2\xa1$}}
# CHECK-NEXT: {{^}}a
>From 85c6cf66e754250c85ae02e4517133414933565d Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Tue, 29 Sep 2026 15:19:03 +0100
Subject: [PATCH 08/41] Handle ok for partial multibyte characters, check EOF
---
llvm/test/tools/llvm-strings/encoding.test | 2 +-
llvm/tools/llvm-strings/llvm-strings.cpp | 6 ++++--
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/llvm/test/tools/llvm-strings/encoding.test b/llvm/test/tools/llvm-strings/encoding.test
index 7b5931d774d32b..c2f014345e98c0 100644
--- a/llvm/test/tools/llvm-strings/encoding.test
+++ b/llvm/test/tools/llvm-strings/encoding.test
@@ -65,7 +65,7 @@ RUN: printf "%16377se\342\200\212e\342\200\213e\342\200\212e\342\200\212e" | \
RUN: tr " " "\0" | llvm-strings -e utf8 | \
RUN: FileCheck %s --check-prefix=CHECK-UTF8-CHUNK-B --match-full-lines --strict-whitespace --implicit-check-not={{.}}
# v chunk boundary
-RUN: printf "%16373se\342\200\213e\342\200\212e\342\200\212e" | \
+RUN: printf "%16373se\342\200\213e\342\200\212e\342\200\212e\342" | \
RUN: tr " " "\0" | llvm-strings -e utf8 | \
RUN: FileCheck %s --check-prefix=CHECK-UTF8-CHUNK-B --match-full-lines --strict-whitespace --implicit-check-not={{.}}
CHECK-UTF8-CHUNK-B:{{e\xe2\x80\x8ae\xe2\x80\x8ae}}
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index b9661f54084578..9a56175a7b85c6 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -175,7 +175,9 @@ static void strings(raw_ostream &OS, StringRef FileName,
}
// Otherwise treat it as an error. A null byte is safe to treat as an
// error, as a null byte is never printable in any locale.
- } else if (Res == std::codecvt_base::partial && !AtEOF) {
+ } else if ((Res == std::codecvt_base::ok ||
+ Res == std::codecvt_base::partial) &&
+ !AtEOF) {
// If we got a partial result but no character was written, we have an
// incomplete multibyte character. Do not treat this as an error,
// instead reset the conversion state so that we can try again if/when
@@ -203,7 +205,7 @@ static void strings(raw_ostream &OS, StringRef FileName,
P = reinterpret_cast<const char *>(UP);
return true;
}
- } else if (Res == sourceExhausted) {
+ } else if (Res == sourceExhausted && !AtEOF) {
Ch = 0;
return false;
}
>From 8ef0e08d724d30fed701771073b125b240da210b Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Sun, 4 Oct 2026 22:18:26 +0100
Subject: [PATCH 09/41] Drop C++ locale API, use C locale API directly. Be
consistent in error handling.
---
llvm/test/tools/llvm-strings/encoding.test | 56 +++--
llvm/tools/llvm-strings/llvm-strings.cpp | 255 ++++++++++-----------
2 files changed, 161 insertions(+), 150 deletions(-)
diff --git a/llvm/test/tools/llvm-strings/encoding.test b/llvm/test/tools/llvm-strings/encoding.test
index c2f014345e98c0..00d1d7f7d93f5f 100644
--- a/llvm/test/tools/llvm-strings/encoding.test
+++ b/llvm/test/tools/llvm-strings/encoding.test
@@ -3,10 +3,15 @@ RUN: echo a > %t
RUN: echo ab >> %t
RUN: echo abc >> %t
RUN: echo abcd >> %t
-# UTF-8 encoding of U+200A HAIR SPACE (printable)
-RUN: printf 'abcd\342\200\212\n' >> %t
+# UTF-8 encoding of U+2027 HYPHENATION POINT (printable)
+RUN: printf 'bcde\342\200\247\n' >> %t
# UTF-8 encoding of U+200B ZERO WIDTH SPACE (non-printable)
-RUN: printf 'abcd\342\200\212\342\200\213\n' >> %t
+RUN: printf 'cdef\342\200\247\342\200\250\n' >> %t
+
+# Check that partial and invalid UTF-8 characters do not interfere with
+# preceding and subsequent strings.
+RUN: printf '\342defg\342\n' >> %t
+RUN: printf '\342\200efgh\342\200\n' >> %t
# Check that long form options work.
RUN: llvm-strings --encoding s %t | FileCheck --check-prefixes CHECK,CHECK-ASCII --strict-whitespace --match-full-lines %s
@@ -18,25 +23,38 @@ RUN: llvm-strings -e s %t | FileCheck --check-prefixes CHECK,CHECK-ASCII --stric
RUN: llvm-strings -e S %t | FileCheck --check-prefixes CHECK,CHECK-LOCALE --strict-whitespace --match-full-lines %s
RUN: llvm-strings -e utf8 %t | FileCheck --check-prefixes CHECK,CHECK-UTF8 --strict-whitespace --match-full-lines %s
-# Check that the default output matches that of -e S.
+# Check that the output of no options, -e S, and --encoding S all match.
RUN: llvm-strings -e S %t > %t.1
-RUN: llvm-strings %t > %t.2
+RUN: llvm-strings --encoding S %t > %t.2
+RUN: llvm-strings %t > %t.3
RUN: cmp %t.1 %t.2
+RUN: cmp %t.1 %t.3
# The first line containing just abcd should always be matched.
CHECK:abcd
-# In -e s, the bytes of U+200A and U+200B should not be included.
-CHECK-ASCII:abcd
-CHECK-ASCII:abcd
+
+# In -e s, non-ASCII bytes should not be included.
+CHECK-ASCII:bcde
+CHECK-ASCII:cdef
+CHECK-ASCII:defg
+CHECK-ASCII:efgh
+
# In -e S, any bytes of U+200A and U+200B may or may not be included depending
# on the locale in effect. The locale in effect may also be altered by llvm-lit.
# Make sure if changing the test that the byte representations should not allow
# this to be split into multiple strings in non-UTF-8 locales.
-CHECK-LOCALE:abcd{{(\xe2(\x80\x8a?)?)?}}
-CHECK-LOCALE:abcd{{(\xe2(\x80(\x8a(\xe2(\x80\x8b?)?)?)?)?)?}}
-# In -e utf8, all bytes of U+200A should be included, but none of U+200B.
-CHECK-UTF8:abcd{{\xe2\x80\x8a}}
-CHECK-UTF8:abcd{{\xe2\x80\x8a}}
+CHECK-LOCALE:bcde{{(\xe2(\x80\xa7?)?)?}}
+CHECK-LOCALE:cdef{{(\xe2(\x80(\xa7(\xe2(\x80\xa8?)?)?)?)?)?}}
+CHECK-LOCALE:{{\xe2?}}defg{{\xe2?}}
+CHECK-LOCALE:{{(\xe2?\x80)?}}efgh{{(\xe2\x80?)?}}
+
+# In -e utf8, all bytes of U+200A should be included, but none of U+200B. The
+# partial multibyte characters should not prevent abcd from being detected as a
+# string.
+CHECK-UTF8:bcde{{\xe2\x80\xa7}}
+CHECK-UTF8:cdef{{\xe2\x80\xa7}}
+CHECK-UTF8:defg
+CHECK-UTF8:efgh
# Check that invalid --encoding values are rejected.
RUN: not llvm-strings --encoding x 2>&1 %t | FileCheck --check-prefix CHECK-ERROR %s
@@ -51,21 +69,21 @@ CHECK-ERROR: error: 'x' is not a valid value for '--encoding='
# the minimum length may or may not already be reached before the chunk
# boundary, and the character that crosses the boundary may or may not be
# printable. v chunk boundary
-RUN: printf "%16378s\342\200\212e\342\200\212e\342\200\212e\342\200\212" | \
+RUN: printf "%16378s\342\200\247e\342\200\247e\342\200\247e\342\200\247" | \
RUN: tr " " "\0" | llvm-strings -e utf8 | \
RUN: FileCheck %s --check-prefix=CHECK-UTF8-CHUNK-A --match-full-lines --strict-whitespace --implicit-check-not={{.}}
# v chunk boundary
-RUN: printf "%16374s\342\200\212e\342\200\212e\342\200\212e\342\200\212" | \
+RUN: printf "%16374s\342\200\247e\342\200\247e\342\200\247e\342\200\247" | \
RUN: tr " " "\0" | llvm-strings -e utf8 | \
RUN: FileCheck %s --check-prefix=CHECK-UTF8-CHUNK-A --match-full-lines --strict-whitespace --implicit-check-not={{.}}
-CHECK-UTF8-CHUNK-A:{{\xe2\x80\x8ae\xe2\x80\x8ae\xe2\x80\x8ae\xe2\x80\x8a}}
+CHECK-UTF8-CHUNK-A:{{\xe2\x80\xa7e\xe2\x80\xa7e\xe2\x80\xa7e\xe2\x80\xa7}}
# v chunk boundary
-RUN: printf "%16377se\342\200\212e\342\200\213e\342\200\212e\342\200\212e" | \
+RUN: printf "%16377se\342\200\247e\342\200\250e\342\200\247e\342\200\247e" | \
RUN: tr " " "\0" | llvm-strings -e utf8 | \
RUN: FileCheck %s --check-prefix=CHECK-UTF8-CHUNK-B --match-full-lines --strict-whitespace --implicit-check-not={{.}}
# v chunk boundary
-RUN: printf "%16373se\342\200\213e\342\200\212e\342\200\212e\342" | \
+RUN: printf "%16373se\342\200\250e\342\200\247e\342\200\247e\342" | \
RUN: tr " " "\0" | llvm-strings -e utf8 | \
RUN: FileCheck %s --check-prefix=CHECK-UTF8-CHUNK-B --match-full-lines --strict-whitespace --implicit-check-not={{.}}
-CHECK-UTF8-CHUNK-B:{{e\xe2\x80\x8ae\xe2\x80\x8ae}}
+CHECK-UTF8-CHUNK-B:{{e\xe2\x80\xa7e\xe2\x80\xa7e}}
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 9a56175a7b85c6..13eaa0a359e31a 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -29,7 +29,7 @@
#include "llvm/Support/Unicode.h"
#include "llvm/Support/WithColor.h"
#include <cctype>
-#include <locale>
+#include <cwctype>
#include <string>
using namespace llvm;
@@ -90,6 +90,118 @@ static void parseIntArg(const opt::InputArgList &Args, int ID, T &Value) {
}
}
+// Tries to read one character from the bytes P...E and store it in Ch.
+// Returns true if a (possibly invalid) character was read, false otherwise.
+//
+// If P...E starts with a valid complete character, P and MBState are updated
+// and true is returned.
+// If P...E is empty, or holds an incomplete character and AtEOF is false, P
+// and MBState are unchanged, Ch is set to 0, and false is returned. This is
+// intended to allow more bytes to be read and Read to be called again.
+// If P...E holds an incomplete character and AtEOF is true, or if P...E
+// starts with an invalid character, the first bytes are skipped and MBState is
+// reset to allow continuing from the next point, Ch is set to 0, and true is
+// returned.
+//
+// The number of skipped bytes for invalid characters follows the Unicode
+// definition of the maximal subpart of an ill-formed subsequence, applied to
+// arbitrary locales: it is the longest subsequence that could start a valid
+// multibyte character, or if the initial byte cannot start a valid multibyte
+// character, the initial byte. This allows consistent error recovery.
+static bool readChar(const char *&P, const char *E, std::mbstate_t &MBState,
+ bool AtEOF, UTF32 &Ch) {
+ if (P == E)
+ return false;
+
+ switch (Encoding) {
+ case Encoding::Ascii:
+ Ch = *P++;
+ return true;
+
+ case Encoding::Locale: {
+ std::mbstate_t SaveMBState = MBState;
+ wchar_t WCh;
+ const size_t BytesRead = mbrtowc(&WCh, P, E - P, &MBState);
+ switch (BytesRead) {
+ default:
+ P += BytesRead;
+ Ch = WCh;
+ return true;
+ case 0:
+ case (size_t)-1: {
+ // We encountered a (possibly null) byte that cannot be part of a valid
+ // character from the previous state. Recover by retrying one byte at a
+ // time, processing as many bytes as possible. If this gives us a
+ // partial character, treat this partial character as a single invalid
+ // character, otherwise treat the first byte as a single invalid
+ // character.
+ const bool PartialChar = mbrtowc(&WCh, P, 1, &SaveMBState) == (size_t)-2;
+ do {
+ ++P;
+ assert(!PartialChar || P != E);
+ } while (PartialChar && mbrtowc(&WCh, P, 1, &SaveMBState) == (size_t)-2);
+ Ch = 0;
+ MBState = {};
+ return true;
+ }
+ case (size_t)-2:
+ MBState = SaveMBState;
+ return false;
+ }
+ }
+
+ case Encoding::Utf8: {
+ const UTF8 *UP = reinterpret_cast<const UTF8 *>(P);
+ const UTF8 *UE = reinterpret_cast<const UTF8 *>(E);
+ UTF32 *Next = &Ch;
+ const auto Res =
+ ConvertUTF8toUTF32(&UP, UE, &Next, &Ch + 1, strictConversion);
+ if (UP != reinterpret_cast<const UTF8 *>(P)) {
+ assert(Next != &Ch);
+ } else if (Res == sourceExhausted && !AtEOF) {
+ return false;
+ } else {
+ assert(Next == &Ch);
+ UP += findMaximalSubpartOfIllFormedUTF8Sequence(UP, UE);
+ *Next++ = 0;
+ }
+ P = reinterpret_cast<const char *>(UP);
+ return true;
+ }
+ }
+
+ llvm_unreachable("unhandled encoding");
+}
+
+static bool isStringChar(UTF32 Ch) {
+ if (Ch == '\t')
+ return true;
+
+ switch (Encoding) {
+ case Encoding::Ascii:
+ return isPrint(Ch);
+
+ case Encoding::Locale:
+ return iswprint(Ch);
+
+ case Encoding::Utf8:
+ return sys::unicode::isPrintable(Ch);
+ }
+
+ llvm_unreachable("unhandled encoding");
+}
+
+static void endString(raw_ostream &OS, std::mbstate_t &MBState) {
+ if (Encoding == Encoding::Locale) {
+ char Buf[MB_LEN_MAX];
+ // Note: This is only required for stateful encodings such as the
+ // ISO-2022 ones.
+ const size_t BytesWritten = wcrtomb(Buf, L'\0', &MBState);
+ OS << StringRef(Buf, BytesWritten - 1);
+ };
+ OS << '\n';
+}
+
static void strings(raw_ostream &OS, StringRef FileName,
sys::fs::file_t Handle) {
SmallString<sys::fs::DefaultReadChunkSize> Buffer;
@@ -111,127 +223,6 @@ static void strings(raw_ostream &OS, StringRef FileName,
}
};
- std::locale Loc("");
- auto &Cvt = std::use_facet<std::codecvt<wchar_t, char, std::mbstate_t>>(Loc);
- auto &Ctype = std::use_facet<std::ctype<wchar_t>>(Loc);
-
- auto IsStringChar = [&Ctype](UTF32 Ch) {
- if (Ch == '\t')
- return true;
-
- switch (Encoding) {
- case Encoding::Ascii:
- return isPrint(Ch);
-
- case Encoding::Locale:
- return Ctype.is(std::ctype_base::print, Ch);
-
- case Encoding::Utf8:
- return sys::unicode::isPrintable(Ch);
- }
-
- llvm_unreachable("unhandled encoding");
- };
-
- // Tries to read one character from the bytes P...E and store it in Ch.
- // Returns true if a (possibly invalid) character was read, false otherwise.
- //
- // If P...E starts with a valid complete character, P and MBState are updated
- // and true is returned.
- // If P...E is empty, or holds an incomplete character and AtEOF is false, P
- // and MBState are unchanged, Ch is set to 0, and false is returned. This is
- // intended to allow more bytes to be read and Read to be called again.
- // If P...E holds an incomplete character and AtEOF is true, or if P...E
- // starts with an invalid character, the first byte is skipped and MBState is
- // reset to allow continuing from the next point, Ch is set to 0, and true is
- // returned.
- auto Read = [&Cvt](const char *&P, const char *E, std::mbstate_t &MBState,
- bool AtEOF, UTF32 &Ch) -> bool {
- if (P == E)
- return false;
-
- switch (Encoding) {
- case Encoding::Ascii:
- Ch = *P++;
- return true;
-
- case Encoding::Locale: {
- const char *N;
- wchar_t WCh;
- wchar_t *WNext;
- std::mbstate_t SaveMBState = MBState;
- const auto Res = Cvt.in(MBState, P, E, N, &WCh, &WCh + 1, WNext);
- assert(Res != std::codecvt_base::noconv);
-
- if (WNext != &WCh) {
- // Only treat a non-null character as a successful conversion, as a
- // null character may be the result of an incomplete multibyte
- // character followed by a null byte.
- if (WCh) {
- // Note: this assumes wchar_t is UCS2 or UTF32.
- Ch = WCh;
- P = N;
- return true;
- }
- // Otherwise treat it as an error. A null byte is safe to treat as an
- // error, as a null byte is never printable in any locale.
- } else if ((Res == std::codecvt_base::ok ||
- Res == std::codecvt_base::partial) &&
- !AtEOF) {
- // If we got a partial result but no character was written, we have an
- // incomplete multibyte character. Do not treat this as an error,
- // instead reset the conversion state so that we can try again if/when
- // we have more characters, unless we know there are no more characters.
- Ch = 0;
- MBState = SaveMBState;
- return false;
- }
- // If there was any error, reset the state to allow the next byte to
- // start a character.
- Ch = 0;
- MBState = {};
- ++P;
- return true;
- }
-
- case Encoding::Utf8: {
- const UTF8 *UP = reinterpret_cast<const UTF8 *>(P);
- const UTF8 *UE = reinterpret_cast<const UTF8 *>(E);
- UTF32 *Next = &Ch;
- const auto Res =
- ConvertUTF8toUTF32Partial(&UP, UE, &Next, &Ch + 1, strictConversion);
- if (Next != &Ch) {
- if (Ch) {
- P = reinterpret_cast<const char *>(UP);
- return true;
- }
- } else if (Res == sourceExhausted && !AtEOF) {
- Ch = 0;
- return false;
- }
- Ch = 0;
- ++P;
- return true;
- }
- }
-
- llvm_unreachable("unhandled encoding");
- };
-
- auto EndString = [&OS, &Cvt](std::mbstate_t &MBState) {
- if (Encoding == Encoding::Locale) {
- char Buf[MB_LEN_MAX];
- char *End;
- auto Res = Cvt.unshift(MBState, Buf, &Buf[MB_LEN_MAX], End);
- if (Res == std::codecvt_base::ok) {
- // Note: This is only required for stateful encodings such as the
- // ISO-2022 ones.
- OS << StringRef(Buf, End - Buf);
- }
- };
- OS << '\n';
- };
-
// To handle very large files without consuming excessive memory, we read the
// file in a little at a time and process it then rather than reading the
// entire file at once.
@@ -297,8 +288,8 @@ static void strings(raw_ostream &OS, StringRef FileName,
for (;;) {
StringEnd = Cur;
PrevMBState = MBState;
- EndOfChunk = !Read(Cur, End, MBState, AtEOF, Ch);
- if (EndOfChunk || !IsStringChar(Ch))
+ EndOfChunk = !readChar(Cur, End, MBState, AtEOF, Ch);
+ if (EndOfChunk || !isStringChar(Ch))
break;
++Len;
}
@@ -333,7 +324,7 @@ static void strings(raw_ostream &OS, StringRef FileName,
CandidateLength = 0;
}
- if (EndOfChunk || Cur == End) {
+ if (EndOfChunk) {
// Finish handling the current chunk and update ChunkOffset.
ChunkOffset += Cur - Begin;
Buffer.erase(Buffer.begin(), Cur);
@@ -342,8 +333,8 @@ static void strings(raw_ostream &OS, StringRef FileName,
if (InString) {
// We haven't reached the end of the chunk, which means the string is
- // terminated. Add a '\n' to start printing a new string.
- EndString(PrevMBState);
+ // terminated.
+ endString(OS, PrevMBState);
InString = false;
}
}
@@ -355,9 +346,9 @@ static void strings(raw_ostream &OS, StringRef FileName,
for (;;) {
const char *Prev = Cur;
std::mbstate_t PrevMBState = MBState;
- if (!Read(Cur, End, MBState, AtEOF, Ch))
+ if (!readChar(Cur, End, MBState, AtEOF, Ch))
break;
- if (IsStringChar(Ch)) {
+ if (isStringChar(Ch)) {
// Find the start of the next string.
if (!StrHead)
StrHead = Prev;
@@ -368,7 +359,7 @@ static void strings(raw_ostream &OS, StringRef FileName,
if (Len >= Min) {
PrintHeader(ChunkOffset + (StrHead - Begin));
OS << StringRef(StrHead, Prev - StrHead);
- EndString(PrevMBState);
+ endString(OS, PrevMBState);
}
StrHead = nullptr;
Len = 0;
@@ -398,10 +389,12 @@ static void strings(raw_ostream &OS, StringRef FileName,
}
if (InString)
- EndString(MBState);
+ endString(OS, MBState);
}
int main(int argc, char **argv) {
+ setlocale(LC_ALL, "");
+
InitLLVM X(argc, argv);
BumpPtrAllocator A;
StringSaver Saver(A);
>From 6fea204af11a1917d0cd202bfb91712d76ff930d Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Sun, 4 Oct 2026 23:25:53 +0100
Subject: [PATCH 10/41] Add missing #include
---
llvm/tools/llvm-strings/llvm-strings.cpp | 1 +
1 file changed, 1 insertion(+)
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 13eaa0a359e31a..002c6f9b93a0a9 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -29,6 +29,7 @@
#include "llvm/Support/Unicode.h"
#include "llvm/Support/WithColor.h"
#include <cctype>
+#include <clocale>
#include <cwctype>
#include <string>
>From cbb8e13345199dc50509a5eb085e0a229f47d236 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Tue, 6 Oct 2026 02:07:35 +0100
Subject: [PATCH 11/41] Optimise mbrtowc() for invalid characters, rename
variables for clarity
---
llvm/tools/llvm-strings/llvm-strings.cpp | 147 +++++++++++++----------
1 file changed, 84 insertions(+), 63 deletions(-)
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 002c6f9b93a0a9..3dd25be37fdf81 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -98,7 +98,7 @@ static void parseIntArg(const opt::InputArgList &Args, int ID, T &Value) {
// and true is returned.
// If P...E is empty, or holds an incomplete character and AtEOF is false, P
// and MBState are unchanged, Ch is set to 0, and false is returned. This is
-// intended to allow more bytes to be read and Read to be called again.
+// intended to allow more bytes to be read and readChar to be called again.
// If P...E holds an incomplete character and AtEOF is true, or if P...E
// starts with an invalid character, the first bytes are skipped and MBState is
// reset to allow continuing from the next point, Ch is set to 0, and true is
@@ -109,64 +109,85 @@ static void parseIntArg(const opt::InputArgList &Args, int ID, T &Value) {
// arbitrary locales: it is the longest subsequence that could start a valid
// multibyte character, or if the initial byte cannot start a valid multibyte
// character, the initial byte. This allows consistent error recovery.
-static bool readChar(const char *&P, const char *E, std::mbstate_t &MBState,
+static bool readChar(const char *&Cur, const char *End, std::mbstate_t &MBState,
bool AtEOF, UTF32 &Ch) {
- if (P == E)
+ if (Cur == End)
return false;
switch (Encoding) {
case Encoding::Ascii:
- Ch = *P++;
+ Ch = *Cur++;
return true;
case Encoding::Locale: {
- std::mbstate_t SaveMBState = MBState;
+ const char *Next = Cur;
+ std::mbstate_t NextMBState = MBState;
wchar_t WCh;
- const size_t BytesRead = mbrtowc(&WCh, P, E - P, &MBState);
- switch (BytesRead) {
- default:
- P += BytesRead;
- Ch = WCh;
- return true;
- case 0:
- case (size_t)-1: {
- // We encountered a (possibly null) byte that cannot be part of a valid
- // character from the previous state. Recover by retrying one byte at a
- // time, processing as many bytes as possible. If this gives us a
- // partial character, treat this partial character as a single invalid
- // character, otherwise treat the first byte as a single invalid
- // character.
- const bool PartialChar = mbrtowc(&WCh, P, 1, &SaveMBState) == (size_t)-2;
- do {
- ++P;
- assert(!PartialChar || P != E);
- } while (PartialChar && mbrtowc(&WCh, P, 1, &SaveMBState) == (size_t)-2);
- Ch = 0;
- MBState = {};
- return true;
- }
- case (size_t)-2:
- MBState = SaveMBState;
- return false;
+ for (;;) {
+ // Read one byte at a time. This is usually not the best way to use
+ // mbrtowc(), usually it would make more sense to pass the size of the
+ // buffer, but the strings utility is unusual in that it is expected to
+ // encounter many bytes that do not form valid characters and it is more
+ // useful to optimise for this case.
+ const size_t BytesRead = mbrtowc(&WCh, Next, 1, &NextMBState);
+ switch (BytesRead) {
+ case 1:
+ ++Next;
+ Cur = Next;
+ MBState = NextMBState;
+ Ch = WCh;
+ return true;
+
+ case (size_t)-2:
+ // We encountered a byte that is a valid start or continuation of a
+ // multibyte character. If we have more bytes, carry on. If we don't
+ // have more bytes yet, but we are not at the end of file, return false.
+ // If we don't have more bytes and we are at the end of file, fall
+ // through to treat it as an error.
+ ++Next;
+ if (Next != End)
+ continue;
+ if (AtEOF)
+ return false;
+ LLVM_FALLTHROUGH;
+
+ case 0:
+ case (size_t)-1:
+ // We cannot form a valid non-null character.
+ //
+ // If we processed any bytes already that formed an incomplete multibyte
+ // character, treat those bytes as a single null character, otherwise
+ // treat the current byte as a single null character.
+ if (Next == Cur)
+ ++Next;
+ Cur = Next;
+ MBState = {};
+ Ch = 0;
+ return true;
+
+ default:
+ llvm_unreachable("unexpected result from mbrtowc");
+ }
}
}
case Encoding::Utf8: {
- const UTF8 *UP = reinterpret_cast<const UTF8 *>(P);
- const UTF8 *UE = reinterpret_cast<const UTF8 *>(E);
- UTF32 *Next = &Ch;
- const auto Res =
- ConvertUTF8toUTF32(&UP, UE, &Next, &Ch + 1, strictConversion);
- if (UP != reinterpret_cast<const UTF8 *>(P)) {
- assert(Next != &Ch);
+ const UTF8 *UTF8Cur = reinterpret_cast<const UTF8 *>(Cur);
+ const UTF8 *UTF8Next = UTF8Cur;
+ const UTF8 *UTF8End = reinterpret_cast<const UTF8 *>(End);
+ UTF32 *UTF32Next = &Ch;
+ const auto Res = ConvertUTF8toUTF32(&UTF8Next, UTF8End, &UTF32Next, &Ch + 1,
+ strictConversion);
+ if (UTF8Next != UTF8Cur) {
+ assert(UTF32Next != &Ch);
} else if (Res == sourceExhausted && !AtEOF) {
return false;
} else {
- assert(Next == &Ch);
- UP += findMaximalSubpartOfIllFormedUTF8Sequence(UP, UE);
- *Next++ = 0;
+ assert(UTF32Next == &Ch);
+ UTF8Next += findMaximalSubpartOfIllFormedUTF8Sequence(UTF8Next, UTF8End);
+ Ch = 0;
}
- P = reinterpret_cast<const char *>(UP);
+ Cur = reinterpret_cast<const char *>(UTF8Next);
return true;
}
}
@@ -233,9 +254,9 @@ static void strings(raw_ostream &OS, StringRef FileName,
// never needs an arbitrarily large buffer. Candidate therefore only ever
// holds a run that is shorter than MinLength and that was cut off by the end
// of a chunk.
- const size_t Min = MinLength;
+ const size_t MinLengthChars = MinLength;
SmallString<DefaultMinLength> Candidate;
- size_t CandidateLength = 0;
+ size_t CandidateLengthChars = 0;
bool InString = false;
std::mbstate_t MBState{};
// Offset of the start of the current chunk within the file.
@@ -243,20 +264,20 @@ static void strings(raw_ostream &OS, StringRef FileName,
while (true) {
// Size of any partial character left over from the previous chunk.
- size_t PartialCharSize = Buffer.size();
- Buffer.resize_for_overwrite(PartialCharSize +
+ const size_t PendingCharSizeBytes = Buffer.size();
+ Buffer.resize_for_overwrite(PendingCharSizeBytes +
sys::fs::DefaultReadChunkSize);
Expected<size_t> ReadBytesOrErr = sys::fs::readNativeFile(
Handle,
- MutableArrayRef(Buffer.begin() + PartialCharSize, Buffer.end()));
+ MutableArrayRef(Buffer.begin() + PendingCharSizeBytes, Buffer.end()));
if (!ReadBytesOrErr) {
errs() << FileName << ": "
<< errorToErrorCode(ReadBytesOrErr.takeError()).message() << '\n';
return;
}
const bool AtEOF = *ReadBytesOrErr == 0;
- const size_t ChunkSize = PartialCharSize + *ReadBytesOrErr;
+ const size_t ChunkSize = PendingCharSizeBytes + *ReadBytesOrErr;
if (ChunkSize == 0)
break;
@@ -282,7 +303,7 @@ static void strings(raw_ostream &OS, StringRef FileName,
if (InString || !Candidate.empty()) {
const char *StringEnd;
std::mbstate_t PrevMBState;
- size_t Len = 0;
+ size_t LengthChars = 0;
bool EndOfChunk;
// Find the end of the current string.
@@ -292,37 +313,37 @@ static void strings(raw_ostream &OS, StringRef FileName,
EndOfChunk = !readChar(Cur, End, MBState, AtEOF, Ch);
if (EndOfChunk || !isStringChar(Ch))
break;
- ++Len;
+ ++LengthChars;
}
- size_t Size = StringEnd - Begin;
+ size_t SizeBytes = StringEnd - Begin;
if (InString) {
// Print the remaining part if the previous chunk has already printed
// the header. E.g. header: aaaaa | bbbbb, where | is the chunk
// boundary.
// Output: Header: aaaaabbbbb, where bbbbb is printed in here.
- OS << StringRef(Begin, Size);
- } else if (CandidateLength + Len >= Min) {
+ OS << StringRef(Begin, SizeBytes);
+ } else if (CandidateLengthChars + LengthChars >= MinLengthChars) {
// If the header hasn't been printed yet (i.e. the previous candidate
// was smaller than Min), but we can print it now, print the header
// first, followed by the candidate from the previous chunk and the
// current string. E.g. aa | bbbbbb
// Output Header: aabbbbbb, where aabbbbbb is printed in here.
PrintHeader(ChunkOffset - Candidate.size());
- OS << Candidate << StringRef(Begin, Size);
+ OS << Candidate << StringRef(Begin, SizeBytes);
Candidate.clear();
- CandidateLength = 0;
+ CandidateLengthChars = 0;
InString = true;
} else if (EndOfChunk) {
// If the current chunk + previous candidate is still smaller than Min,
// append it to Candidate.
Candidate.append(Begin, StringEnd);
- CandidateLength += Len;
+ CandidateLengthChars += LengthChars;
} else {
// If the string has terminated but is still smaller than Min, clear the
// buffer since it is too short to print.
Candidate.clear();
- CandidateLength = 0;
+ CandidateLengthChars = 0;
}
if (EndOfChunk) {
@@ -343,7 +364,7 @@ static void strings(raw_ostream &OS, StringRef FileName,
// At this point, we are always at the start of a new string because the
// remaining part of the previous string has already been handled.
const char *StrHead = nullptr;
- size_t Len = 0;
+ size_t LengthChars = 0;
for (;;) {
const char *Prev = Cur;
std::mbstate_t PrevMBState = MBState;
@@ -353,17 +374,17 @@ static void strings(raw_ostream &OS, StringRef FileName,
// Find the start of the next string.
if (!StrHead)
StrHead = Prev;
- ++Len;
+ ++LengthChars;
} else if (StrHead) {
// If it is not a printable character, we have reached the end of the
// current string. Print it if long enough.
- if (Len >= Min) {
+ if (LengthChars >= MinLengthChars) {
PrintHeader(ChunkOffset + (StrHead - Begin));
OS << StringRef(StrHead, Prev - StrHead);
endString(OS, PrevMBState);
}
StrHead = nullptr;
- Len = 0;
+ LengthChars = 0;
}
}
@@ -372,13 +393,13 @@ static void strings(raw_ostream &OS, StringRef FileName,
// it again.
if (StrHead) {
// Print it, or append it to Candidate if it is too short.
- if (Len >= Min) {
+ if (LengthChars >= MinLengthChars) {
PrintHeader(ChunkOffset + (StrHead - Begin));
OS << StringRef(StrHead, Cur - StrHead);
InString = true;
} else {
Candidate.append(StrHead, Cur);
- CandidateLength += Len;
+ CandidateLengthChars += LengthChars;
}
}
>From 03a6e3d753c1f54e777bbf75313b6847c945f9e8 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Wed, 7 Oct 2026 14:35:12 +0100
Subject: [PATCH 12/41] Move to a Struct class template to minimise runtime
checks on Encoding
---
llvm/tools/llvm-strings/llvm-strings.cpp | 274 +++++++++++++----------
1 file changed, 155 insertions(+), 119 deletions(-)
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 3dd25be37fdf81..6abb571ad825e5 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -70,7 +70,6 @@ static Encoding Encoding;
enum class Radix { None, Octal, Hexadecimal, Decimal };
static Radix Radix;
-} // namespace
[[noreturn]] static void reportCmdLineError(const Twine &Message) {
WithColor::error(errs(), ToolName) << Message << "\n";
@@ -91,141 +90,160 @@ static void parseIntArg(const opt::InputArgList &Args, int ID, T &Value) {
}
}
-// Tries to read one character from the bytes P...E and store it in Ch.
-// Returns true if a (possibly invalid) character was read, false otherwise.
-//
-// If P...E starts with a valid complete character, P and MBState are updated
-// and true is returned.
-// If P...E is empty, or holds an incomplete character and AtEOF is false, P
-// and MBState are unchanged, Ch is set to 0, and false is returned. This is
-// intended to allow more bytes to be read and readChar to be called again.
-// If P...E holds an incomplete character and AtEOF is true, or if P...E
-// starts with an invalid character, the first bytes are skipped and MBState is
-// reset to allow continuing from the next point, Ch is set to 0, and true is
-// returned.
-//
-// The number of skipped bytes for invalid characters follows the Unicode
-// definition of the maximal subpart of an ill-formed subsequence, applied to
-// arbitrary locales: it is the longest subsequence that could start a valid
-// multibyte character, or if the initial byte cannot start a valid multibyte
-// character, the initial byte. This allows consistent error recovery.
-static bool readChar(const char *&Cur, const char *End, std::mbstate_t &MBState,
- bool AtEOF, UTF32 &Ch) {
+template <enum Encoding> struct Strings {
+ Strings() = delete;
+
+ // Tries to read one character from the bytes P...E and store it in Ch.
+ // Returns true if a (possibly invalid) character was read, false otherwise.
+ //
+ // If P...E starts with a valid complete character, P and MBState are updated
+ // and true is returned.
+ // If P...E is empty, or holds an incomplete character and AtEOF is false, P
+ // and MBState are unchanged, Ch is set to 0, and false is returned. This is
+ // intended to allow more bytes to be read and readChar to be called again.
+ // If P...E holds an incomplete character and AtEOF is true, or if P...E
+ // starts with an invalid character, the first bytes are skipped and MBState
+ // is reset to allow continuing from the next point, Ch is set to 0, and true
+ // is returned.
+ //
+ // The number of skipped bytes for invalid characters follows the Unicode
+ // definition of the maximal subpart of an ill-formed subsequence, applied to
+ // arbitrary locales: it is the longest subsequence that could start a valid
+ // multibyte character, or if the initial byte cannot start a valid multibyte
+ // character, the initial byte. This allows consistent error recovery.
+ static bool readChar(const char *&Cur, const char *End,
+ std::mbstate_t &MBState, bool AtEOF, UTF32 &Ch);
+
+ static bool isStringChar(UTF32 Ch);
+
+ static void endString(raw_ostream &OS, std::mbstate_t &MBState);
+
+ static void run(raw_ostream &OS, StringRef FileName, sys::fs::file_t Handle);
+};
+
+template <>
+bool Strings<Encoding::Ascii>::readChar(const char *&Cur, const char *End,
+ std::mbstate_t &MBState, bool AtEOF,
+ UTF32 &Ch) {
if (Cur == End)
return false;
- switch (Encoding) {
- case Encoding::Ascii:
- Ch = *Cur++;
- return true;
+ Ch = *Cur++;
+ return true;
+}
- case Encoding::Locale: {
- const char *Next = Cur;
- std::mbstate_t NextMBState = MBState;
- wchar_t WCh;
- for (;;) {
- // Read one byte at a time. This is usually not the best way to use
- // mbrtowc(), usually it would make more sense to pass the size of the
- // buffer, but the strings utility is unusual in that it is expected to
- // encounter many bytes that do not form valid characters and it is more
- // useful to optimise for this case.
- const size_t BytesRead = mbrtowc(&WCh, Next, 1, &NextMBState);
- switch (BytesRead) {
- case 1:
- ++Next;
- Cur = Next;
- MBState = NextMBState;
- Ch = WCh;
- return true;
-
- case (size_t)-2:
- // We encountered a byte that is a valid start or continuation of a
- // multibyte character. If we have more bytes, carry on. If we don't
- // have more bytes yet, but we are not at the end of file, return false.
- // If we don't have more bytes and we are at the end of file, fall
- // through to treat it as an error.
- ++Next;
- if (Next != End)
- continue;
- if (AtEOF)
- return false;
- LLVM_FALLTHROUGH;
-
- case 0:
- case (size_t)-1:
- // We cannot form a valid non-null character.
- //
- // If we processed any bytes already that formed an incomplete multibyte
- // character, treat those bytes as a single null character, otherwise
- // treat the current byte as a single null character.
- if (Next == Cur)
- ++Next;
- Cur = Next;
- MBState = {};
- Ch = 0;
- return true;
-
- default:
- llvm_unreachable("unexpected result from mbrtowc");
- }
- }
- }
+template <>
+bool Strings<Encoding::Locale>::readChar(const char *&Cur, const char *End,
+ std::mbstate_t &MBState, bool AtEOF,
+ UTF32 &Ch) {
+ if (Cur == End)
+ return false;
- case Encoding::Utf8: {
- const UTF8 *UTF8Cur = reinterpret_cast<const UTF8 *>(Cur);
- const UTF8 *UTF8Next = UTF8Cur;
- const UTF8 *UTF8End = reinterpret_cast<const UTF8 *>(End);
- UTF32 *UTF32Next = &Ch;
- const auto Res = ConvertUTF8toUTF32(&UTF8Next, UTF8End, &UTF32Next, &Ch + 1,
- strictConversion);
- if (UTF8Next != UTF8Cur) {
- assert(UTF32Next != &Ch);
- } else if (Res == sourceExhausted && !AtEOF) {
- return false;
- } else {
- assert(UTF32Next == &Ch);
- UTF8Next += findMaximalSubpartOfIllFormedUTF8Sequence(UTF8Next, UTF8End);
+ const char *Next = Cur;
+ std::mbstate_t NextMBState = MBState;
+ wchar_t WCh;
+ for (;;) {
+ // Read one byte at a time. This is usually not the best way to use
+ // mbrtowc(), usually it would make more sense to pass the size of the
+ // buffer, but the strings utility is unusual in that it is expected to
+ // encounter many bytes that do not form valid characters and it is more
+ // useful to optimise for this case.
+ const size_t BytesRead = mbrtowc(&WCh, Next, 1, &NextMBState);
+ switch (BytesRead) {
+ case 1:
+ ++Next;
+ Cur = Next;
+ MBState = NextMBState;
+ Ch = WCh;
+ return true;
+
+ case (size_t)-2:
+ // We encountered a byte that is a valid start or continuation of a
+ // multibyte character. If we have more bytes, carry on. If we don't
+ // have more bytes yet, but we are not at the end of file, return false.
+ // If we don't have more bytes and we are at the end of file, fall
+ // through to treat it as an error.
+ ++Next;
+ if (Next != End)
+ continue;
+ if (AtEOF)
+ return false;
+ LLVM_FALLTHROUGH;
+
+ case 0:
+ case (size_t)-1:
+ // We cannot form a valid non-null character.
+ //
+ // If we processed any bytes already that formed an incomplete multibyte
+ // character, treat those bytes as a single null character, otherwise
+ // treat the current byte as a single null character.
+ if (Next == Cur)
+ ++Next;
+ Cur = Next;
+ MBState = {};
Ch = 0;
+ return true;
}
- Cur = reinterpret_cast<const char *>(UTF8Next);
- return true;
- }
}
-
- llvm_unreachable("unhandled encoding");
}
-static bool isStringChar(UTF32 Ch) {
- if (Ch == '\t')
- return true;
+template <>
+bool Strings<Encoding::Utf8>::readChar(const char *&Cur, const char *End,
+ std::mbstate_t &MBState, bool AtEOF,
+ UTF32 &Ch) {
+ if (Cur == End)
+ return false;
- switch (Encoding) {
- case Encoding::Ascii:
- return isPrint(Ch);
+ const UTF8 *UTF8Cur = reinterpret_cast<const UTF8 *>(Cur);
+ const UTF8 *UTF8Next = UTF8Cur;
+ const UTF8 *UTF8End = reinterpret_cast<const UTF8 *>(End);
+ UTF32 *UTF32Next = &Ch;
+ const auto Res = ConvertUTF8toUTF32(&UTF8Next, UTF8End, &UTF32Next, &Ch + 1,
+ strictConversion);
+ if (UTF8Next != UTF8Cur) {
+ assert(UTF32Next != &Ch);
+ } else if (Res == sourceExhausted && !AtEOF) {
+ return false;
+ } else {
+ assert(UTF32Next == &Ch);
+ UTF8Next += findMaximalSubpartOfIllFormedUTF8Sequence(UTF8Next, UTF8End);
+ Ch = 0;
+ }
+ Cur = reinterpret_cast<const char *>(UTF8Next);
+ return true;
+}
- case Encoding::Locale:
- return iswprint(Ch);
+template <> bool Strings<Encoding::Ascii>::isStringChar(UTF32 Ch) {
+ return Ch == '\t' || isPrint(Ch);
+}
- case Encoding::Utf8:
- return sys::unicode::isPrintable(Ch);
- }
+template <> bool Strings<Encoding::Locale>::isStringChar(UTF32 Ch) {
+ return Ch == L'\t' || iswprint(Ch);
+}
- llvm_unreachable("unhandled encoding");
+template <> bool Strings<Encoding::Utf8>::isStringChar(UTF32 Ch) {
+ return Ch == u'\t' || sys::unicode::isPrintable(Ch);
}
-static void endString(raw_ostream &OS, std::mbstate_t &MBState) {
- if (Encoding == Encoding::Locale) {
- char Buf[MB_LEN_MAX];
- // Note: This is only required for stateful encodings such as the
- // ISO-2022 ones.
- const size_t BytesWritten = wcrtomb(Buf, L'\0', &MBState);
- OS << StringRef(Buf, BytesWritten - 1);
- };
+template <enum Encoding Encoding>
+void Strings<Encoding>::endString(raw_ostream &OS, std::mbstate_t &MBState) {
OS << '\n';
}
-static void strings(raw_ostream &OS, StringRef FileName,
- sys::fs::file_t Handle) {
+template <>
+void Strings<Encoding::Locale>::endString(raw_ostream &OS,
+ std::mbstate_t &MBState) {
+ char Buf[MB_LEN_MAX];
+ // Note: This is only required for stateful encodings such as the
+ // ISO-2022 ones.
+ const size_t BytesWritten = wcrtomb(Buf, L'\0', &MBState);
+ Buf[BytesWritten - 1] = '\n';
+ OS << StringRef(Buf, BytesWritten);
+}
+
+template <enum Encoding Encoding>
+void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
+ sys::fs::file_t Handle) {
SmallString<sys::fs::DefaultReadChunkSize> Buffer;
auto PrintHeader = [&OS, FileName](size_t StringStart) {
if (PrintFileName)
@@ -413,6 +431,7 @@ static void strings(raw_ostream &OS, StringRef FileName,
if (InString)
endString(OS, MBState);
}
+} // namespace
int main(int argc, char **argv) {
setlocale(LC_ALL, "");
@@ -474,13 +493,30 @@ int main(int argc, char **argv) {
return EXIT_FAILURE;
}
+ void (*const StringsImpl)(raw_ostream &OS, StringRef FileName,
+ sys::fs::file_t Handle) = [] {
+ switch (Encoding) {
+ case Encoding::Ascii:
+ return Strings<Encoding::Ascii>::run;
+
+ case Encoding::Locale:
+ return Strings<Encoding::Locale>::run;
+
+ case Encoding::Utf8:
+ return Strings<Encoding::Utf8>::run;
+
+ default:
+ llvm_unreachable("unhandled encoding");
+ }
+ }();
+
std::vector<std::string> InputFileNames = Args.getAllArgValues(OPT_INPUT);
if (InputFileNames.empty())
InputFileNames.push_back("-");
for (const auto &File : InputFileNames) {
if (File == "-") {
- strings(llvm::outs(), "{standard input}", sys::fs::getStdinHandle());
+ StringsImpl(llvm::outs(), "{standard input}", sys::fs::getStdinHandle());
} else {
Expected<sys::fs::file_t> FDOrErr =
sys::fs::openNativeFileForRead(File, sys::fs::OF_TextWithCRLF);
@@ -490,7 +526,7 @@ int main(int argc, char **argv) {
<< '\n';
continue;
}
- strings(llvm::outs(), File, *FDOrErr);
+ StringsImpl(llvm::outs(), File, *FDOrErr);
}
}
>From 6be72df4d23b83d7c55bcd3ff731a97c42cc37e6 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Wed, 7 Oct 2026 18:17:58 +0100
Subject: [PATCH 13/41] Avoid compiler warning
---
llvm/tools/llvm-strings/llvm-strings.cpp | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 6abb571ad825e5..94ea5cb9986c9b 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -504,10 +504,9 @@ int main(int argc, char **argv) {
case Encoding::Utf8:
return Strings<Encoding::Utf8>::run;
-
- default:
- llvm_unreachable("unhandled encoding");
}
+
+ llvm_unreachable("unhandled encoding");
}();
std::vector<std::string> InputFileNames = Args.getAllArgValues(OPT_INPUT);
>From 2c19f00af54bb5f7257bb022e32d55d62c6a07a6 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Fri, 9 Oct 2026 15:00:22 +0100
Subject: [PATCH 14/41] Use std:: where appropriate, expand on comments
---
llvm/test/tools/llvm-strings/encoding.test | 16 ++++---
llvm/tools/llvm-strings/llvm-strings.cpp | 55 ++++++++++++----------
2 files changed, 38 insertions(+), 33 deletions(-)
diff --git a/llvm/test/tools/llvm-strings/encoding.test b/llvm/test/tools/llvm-strings/encoding.test
index 00d1d7f7d93f5f..4d15387814f562 100644
--- a/llvm/test/tools/llvm-strings/encoding.test
+++ b/llvm/test/tools/llvm-strings/encoding.test
@@ -62,13 +62,15 @@ RUN: not llvm-strings -e x 2>&1 %t | FileCheck --check-prefix CHECK-ERROR %s
CHECK-ERROR: error: 'x' is not a valid value for '--encoding='
-# Encodings with multibyte characters may have characters cross chunk
-# boundaries. The string contains a full multibyte character before the
-# boundary, a multibyte character that crosses the boundary, and a full
-# multibyte character after the boundary. This test has four variants:
-# the minimum length may or may not already be reached before the chunk
-# boundary, and the character that crosses the boundary may or may not be
-# printable. v chunk boundary
+# llvm-strings reads the input in chunks of 16384 bytes. Encodings with
+# multibyte characters may have characters cross chunk boundaries. The string
+# contains a full multibyte character before the boundary, a multibyte
+# character that crosses the boundary, and a full multibyte character after the
+# boundary. This test has four variants: the minimum length may or may not
+# already be reached before the chunk boundary, and the character that crosses
+# the boundary may or may not be printable.
+#
+# v chunk boundary
RUN: printf "%16378s\342\200\247e\342\200\247e\342\200\247e\342\200\247" | \
RUN: tr " " "\0" | llvm-strings -e utf8 | \
RUN: FileCheck %s --check-prefix=CHECK-UTF8-CHUNK-A --match-full-lines --strict-whitespace --implicit-check-not={{.}}
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 94ea5cb9986c9b..7f27012b4d8323 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -93,18 +93,21 @@ static void parseIntArg(const opt::InputArgList &Args, int ID, T &Value) {
template <enum Encoding> struct Strings {
Strings() = delete;
- // Tries to read one character from the bytes P...E and store it in Ch.
+ // Tries to read one character from the bytes Cur...End and store it in Ch.
// Returns true if a (possibly invalid) character was read, false otherwise.
//
- // If P...E starts with a valid complete character, P and MBState are updated
- // and true is returned.
- // If P...E is empty, or holds an incomplete character and AtEOF is false, P
- // and MBState are unchanged, Ch is set to 0, and false is returned. This is
- // intended to allow more bytes to be read and readChar to be called again.
- // If P...E holds an incomplete character and AtEOF is true, or if P...E
- // starts with an invalid character, the first bytes are skipped and MBState
- // is reset to allow continuing from the next point, Ch is set to 0, and true
- // is returned.
+ // If Cur...End starts with a valid complete character, Cur and MBState are
+ // updated and true is returned.
+ // If Cur...End is empty, or holds an incomplete character and AtEOF is false,
+ // Cur and MBState are unchanged, Ch is set to 0, and false is returned. This
+ // is intended to allow more bytes to be read and readChar to be called again.
+ // If Cur...End holds an incomplete character and AtEOF is true, or if
+ // Cur...End starts with an invalid character, the first bytes are skipped and
+ // MBState is reset to allow continuing from the next point, Ch is set to 0,
+ // and true is returned. This case is indistinguishable to the caller from a
+ // null byte being successfully converted to a null character, but a null
+ // character is never printable under any settings and that is how invalid
+ // bytes are treated as well.
//
// The number of skipped bytes for invalid characters follows the Unicode
// definition of the maximal subpart of an ill-formed subsequence, applied to
@@ -148,7 +151,7 @@ bool Strings<Encoding::Locale>::readChar(const char *&Cur, const char *End,
// buffer, but the strings utility is unusual in that it is expected to
// encounter many bytes that do not form valid characters and it is more
// useful to optimise for this case.
- const size_t BytesRead = mbrtowc(&WCh, Next, 1, &NextMBState);
+ const std::size_t BytesRead = std::mbrtowc(&WCh, Next, 1, &NextMBState);
switch (BytesRead) {
case 1:
++Next;
@@ -157,7 +160,7 @@ bool Strings<Encoding::Locale>::readChar(const char *&Cur, const char *End,
Ch = WCh;
return true;
- case (size_t)-2:
+ case (std::size_t)-2:
// We encountered a byte that is a valid start or continuation of a
// multibyte character. If we have more bytes, carry on. If we don't
// have more bytes yet, but we are not at the end of file, return false.
@@ -171,7 +174,7 @@ bool Strings<Encoding::Locale>::readChar(const char *&Cur, const char *End,
LLVM_FALLTHROUGH;
case 0:
- case (size_t)-1:
+ case (std::size_t)-1:
// We cannot form a valid non-null character.
//
// If we processed any bytes already that formed an incomplete multibyte
@@ -218,7 +221,7 @@ template <> bool Strings<Encoding::Ascii>::isStringChar(UTF32 Ch) {
}
template <> bool Strings<Encoding::Locale>::isStringChar(UTF32 Ch) {
- return Ch == L'\t' || iswprint(Ch);
+ return Ch == L'\t' || std::iswprint(Ch);
}
template <> bool Strings<Encoding::Utf8>::isStringChar(UTF32 Ch) {
@@ -236,7 +239,7 @@ void Strings<Encoding::Locale>::endString(raw_ostream &OS,
char Buf[MB_LEN_MAX];
// Note: This is only required for stateful encodings such as the
// ISO-2022 ones.
- const size_t BytesWritten = wcrtomb(Buf, L'\0', &MBState);
+ const std::size_t BytesWritten = std::wcrtomb(Buf, L'\0', &MBState);
Buf[BytesWritten - 1] = '\n';
OS << StringRef(Buf, BytesWritten);
}
@@ -245,7 +248,7 @@ template <enum Encoding Encoding>
void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
sys::fs::file_t Handle) {
SmallString<sys::fs::DefaultReadChunkSize> Buffer;
- auto PrintHeader = [&OS, FileName](size_t StringStart) {
+ auto PrintHeader = [&OS, FileName](std::size_t StringStart) {
if (PrintFileName)
OS << FileName << ": ";
switch (Radix) {
@@ -272,21 +275,21 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
// never needs an arbitrarily large buffer. Candidate therefore only ever
// holds a run that is shorter than MinLength and that was cut off by the end
// of a chunk.
- const size_t MinLengthChars = MinLength;
+ const std::size_t MinLengthChars = MinLength;
SmallString<DefaultMinLength> Candidate;
- size_t CandidateLengthChars = 0;
+ std::size_t CandidateLengthChars = 0;
bool InString = false;
std::mbstate_t MBState{};
// Offset of the start of the current chunk within the file.
- size_t ChunkOffset = 0;
+ std::size_t ChunkOffset = 0;
while (true) {
// Size of any partial character left over from the previous chunk.
- const size_t PendingCharSizeBytes = Buffer.size();
+ const std::size_t PendingCharSizeBytes = Buffer.size();
Buffer.resize_for_overwrite(PendingCharSizeBytes +
sys::fs::DefaultReadChunkSize);
- Expected<size_t> ReadBytesOrErr = sys::fs::readNativeFile(
+ Expected<std::size_t> ReadBytesOrErr = sys::fs::readNativeFile(
Handle,
MutableArrayRef(Buffer.begin() + PendingCharSizeBytes, Buffer.end()));
if (!ReadBytesOrErr) {
@@ -295,7 +298,7 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
return;
}
const bool AtEOF = *ReadBytesOrErr == 0;
- const size_t ChunkSize = PendingCharSizeBytes + *ReadBytesOrErr;
+ const std::size_t ChunkSize = PendingCharSizeBytes + *ReadBytesOrErr;
if (ChunkSize == 0)
break;
@@ -321,7 +324,7 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
if (InString || !Candidate.empty()) {
const char *StringEnd;
std::mbstate_t PrevMBState;
- size_t LengthChars = 0;
+ std::size_t LengthChars = 0;
bool EndOfChunk;
// Find the end of the current string.
@@ -334,7 +337,7 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
++LengthChars;
}
- size_t SizeBytes = StringEnd - Begin;
+ std::size_t SizeBytes = StringEnd - Begin;
if (InString) {
// Print the remaining part if the previous chunk has already printed
// the header. E.g. header: aaaaa | bbbbb, where | is the chunk
@@ -382,7 +385,7 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
// At this point, we are always at the start of a new string because the
// remaining part of the previous string has already been handled.
const char *StrHead = nullptr;
- size_t LengthChars = 0;
+ std::size_t LengthChars = 0;
for (;;) {
const char *Prev = Cur;
std::mbstate_t PrevMBState = MBState;
@@ -434,7 +437,7 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
} // namespace
int main(int argc, char **argv) {
- setlocale(LC_ALL, "");
+ std::setlocale(LC_ALL, "");
InitLLVM X(argc, argv);
BumpPtrAllocator A;
>From d427e670738304efca81c8eeaf8d1ad88811a8de Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Fri, 16 Oct 2026 19:47:52 +0100
Subject: [PATCH 15/41] Use consistent definition of printable across platforms
---
llvm/docs/CommandGuide/llvm-strings.md | 5 +++--
llvm/tools/llvm-strings/llvm-strings.cpp | 19 ++++++++++---------
2 files changed, 13 insertions(+), 11 deletions(-)
diff --git a/llvm/docs/CommandGuide/llvm-strings.md b/llvm/docs/CommandGuide/llvm-strings.md
index 5a45dcd426c058..06be9b0cf6e13f 100644
--- a/llvm/docs/CommandGuide/llvm-strings.md
+++ b/llvm/docs/CommandGuide/llvm-strings.md
@@ -12,8 +12,9 @@
{program}`llvm-strings` is a tool intended as a drop-in replacement for GNU's
{program}`strings`, which looks for printable strings in files and writes them
to the standard output stream. A printable string is any sequence of four (by
-default) or more printable characters. The end of the file, or any unprintable
-byte sequence, terminates the current sequence.
+default) or more printable characters. A printable character is tab, or any
+character that is not a control character. The end of the file, or any
+unprintable byte sequence, terminates the current sequence.
{program}`llvm-strings` looks for strings in each `input` file specified.
Unlike GNU {program}`strings` it looks in the entire input file, regardless of
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 7f27012b4d8323..cb32fbef1aff88 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -28,9 +28,8 @@
#include "llvm/Support/SwapByteOrder.h"
#include "llvm/Support/Unicode.h"
#include "llvm/Support/WithColor.h"
-#include <cctype>
#include <clocale>
-#include <cwctype>
+#include <cuchar>
#include <string>
using namespace llvm;
@@ -144,20 +143,20 @@ bool Strings<Encoding::Locale>::readChar(const char *&Cur, const char *End,
const char *Next = Cur;
std::mbstate_t NextMBState = MBState;
- wchar_t WCh;
+ char32_t UCh;
for (;;) {
// Read one byte at a time. This is usually not the best way to use
- // mbrtowc(), usually it would make more sense to pass the size of the
+ // mbrtoc32(), usually it would make more sense to pass the size of the
// buffer, but the strings utility is unusual in that it is expected to
// encounter many bytes that do not form valid characters and it is more
// useful to optimise for this case.
- const std::size_t BytesRead = std::mbrtowc(&WCh, Next, 1, &NextMBState);
+ const std::size_t BytesRead = std::mbrtoc32(&UCh, Next, 1, &NextMBState);
switch (BytesRead) {
case 1:
++Next;
Cur = Next;
MBState = NextMBState;
- Ch = WCh;
+ Ch = UCh;
return true;
case (std::size_t)-2:
@@ -221,11 +220,13 @@ template <> bool Strings<Encoding::Ascii>::isStringChar(UTF32 Ch) {
}
template <> bool Strings<Encoding::Locale>::isStringChar(UTF32 Ch) {
- return Ch == L'\t' || std::iswprint(Ch);
+ return llvm::sys::unicode::isCharacter(Ch) &&
+ (Ch == U'\t' || !llvm::sys::unicode::isControl(Ch));
}
template <> bool Strings<Encoding::Utf8>::isStringChar(UTF32 Ch) {
- return Ch == u'\t' || sys::unicode::isPrintable(Ch);
+ return llvm::sys::unicode::isCharacter(Ch) &&
+ (Ch == U'\t' || !llvm::sys::unicode::isControl(Ch));
}
template <enum Encoding Encoding>
@@ -239,7 +240,7 @@ void Strings<Encoding::Locale>::endString(raw_ostream &OS,
char Buf[MB_LEN_MAX];
// Note: This is only required for stateful encodings such as the
// ISO-2022 ones.
- const std::size_t BytesWritten = std::wcrtomb(Buf, L'\0', &MBState);
+ const std::size_t BytesWritten = std::c32rtomb(Buf, U'\0', &MBState);
Buf[BytesWritten - 1] = '\n';
OS << StringRef(Buf, BytesWritten);
}
>From 82bc20fcba94c61ad30106a8583970fa58f5c178 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Sat, 17 Oct 2026 00:09:07 +0100
Subject: [PATCH 16/41] Update test
---
llvm/test/tools/llvm-strings/encoding.test | 40 +++++++++++-----------
1 file changed, 20 insertions(+), 20 deletions(-)
diff --git a/llvm/test/tools/llvm-strings/encoding.test b/llvm/test/tools/llvm-strings/encoding.test
index 4d15387814f562..20b7d6d8a30133 100644
--- a/llvm/test/tools/llvm-strings/encoding.test
+++ b/llvm/test/tools/llvm-strings/encoding.test
@@ -3,10 +3,10 @@ RUN: echo a > %t
RUN: echo ab >> %t
RUN: echo abc >> %t
RUN: echo abcd >> %t
-# UTF-8 encoding of U+2027 HYPHENATION POINT (printable)
-RUN: printf 'bcde\342\200\247\n' >> %t
-# UTF-8 encoding of U+200B ZERO WIDTH SPACE (non-printable)
-RUN: printf 'cdef\342\200\247\342\200\250\n' >> %t
+# UTF-8 encoding of U+00A0 NON-BREAKING SPACE (which we treat as printable)
+RUN: printf 'bcde\302\240\n' >> %t
+# UTF-8 encoding of U+009F APPLICATION PROGRAM COMMAND (non-printable)
+RUN: printf 'cdef\302\240\302\237\n' >> %t
# Check that partial and invalid UTF-8 characters do not interfere with
# preceding and subsequent strings.
@@ -39,20 +39,20 @@ CHECK-ASCII:cdef
CHECK-ASCII:defg
CHECK-ASCII:efgh
-# In -e S, any bytes of U+200A and U+200B may or may not be included depending
+# In -e S, any bytes of U+00A0 and U+009F may or may not be included depending
# on the locale in effect. The locale in effect may also be altered by llvm-lit.
# Make sure if changing the test that the byte representations should not allow
# this to be split into multiple strings in non-UTF-8 locales.
-CHECK-LOCALE:bcde{{(\xe2(\x80\xa7?)?)?}}
-CHECK-LOCALE:cdef{{(\xe2(\x80(\xa7(\xe2(\x80\xa8?)?)?)?)?)?}}
+CHECK-LOCALE:bcde{{(\xc2\xa0?)?}}
+CHECK-LOCALE:cdef{{(\xc2(\xa0(\xc2\x9f?)?)?)?}}
CHECK-LOCALE:{{\xe2?}}defg{{\xe2?}}
CHECK-LOCALE:{{(\xe2?\x80)?}}efgh{{(\xe2\x80?)?}}
-# In -e utf8, all bytes of U+200A should be included, but none of U+200B. The
+# In -e utf8, all bytes of U+00A0 should be included, but none of U+009F. The
# partial multibyte characters should not prevent abcd from being detected as a
# string.
-CHECK-UTF8:bcde{{\xe2\x80\xa7}}
-CHECK-UTF8:cdef{{\xe2\x80\xa7}}
+CHECK-UTF8:bcde{{\xc2\xa0}}
+CHECK-UTF8:cdef{{\xc2\xa0}}
CHECK-UTF8:defg
CHECK-UTF8:efgh
@@ -70,22 +70,22 @@ CHECK-ERROR: error: 'x' is not a valid value for '--encoding='
# already be reached before the chunk boundary, and the character that crosses
# the boundary may or may not be printable.
#
-# v chunk boundary
-RUN: printf "%16378s\342\200\247e\342\200\247e\342\200\247e\342\200\247" | \
+# v chunk boundary
+RUN: printf "%16380s\302\240e\302\240e\302\240e\302\240" | \
RUN: tr " " "\0" | llvm-strings -e utf8 | \
RUN: FileCheck %s --check-prefix=CHECK-UTF8-CHUNK-A --match-full-lines --strict-whitespace --implicit-check-not={{.}}
-# v chunk boundary
-RUN: printf "%16374s\342\200\247e\342\200\247e\342\200\247e\342\200\247" | \
+# v chunk boundary
+RUN: printf "%16383s\302\240e\302\240e\302\240e\302\240" | \
RUN: tr " " "\0" | llvm-strings -e utf8 | \
RUN: FileCheck %s --check-prefix=CHECK-UTF8-CHUNK-A --match-full-lines --strict-whitespace --implicit-check-not={{.}}
-CHECK-UTF8-CHUNK-A:{{\xe2\x80\xa7e\xe2\x80\xa7e\xe2\x80\xa7e\xe2\x80\xa7}}
+CHECK-UTF8-CHUNK-A:{{\xc2\xa0e\xc2\xa0e\xc2\xa0e\xc2\xa0}}
-# v chunk boundary
-RUN: printf "%16377se\342\200\247e\342\200\250e\342\200\247e\342\200\247e" | \
+# v chunk boundary
+RUN: printf "%16379se\302\240e\302\237e\302\240e\302\240e" | \
RUN: tr " " "\0" | llvm-strings -e utf8 | \
RUN: FileCheck %s --check-prefix=CHECK-UTF8-CHUNK-B --match-full-lines --strict-whitespace --implicit-check-not={{.}}
-# v chunk boundary
-RUN: printf "%16373se\342\200\250e\342\200\247e\342\200\247e\342" | \
+# v chunk boundary
+RUN: printf "%16382se\302\237e\302\240e\302\240e\342" | \
RUN: tr " " "\0" | llvm-strings -e utf8 | \
RUN: FileCheck %s --check-prefix=CHECK-UTF8-CHUNK-B --match-full-lines --strict-whitespace --implicit-check-not={{.}}
-CHECK-UTF8-CHUNK-B:{{e\xe2\x80\xa7e\xe2\x80\xa7e}}
+CHECK-UTF8-CHUNK-B:{{e\xc2\xa0e\xc2\xa0e}}
>From b46f1d1206cfbc1f5e6e7effb64b9504c38456d4 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Tue, 20 Oct 2026 23:01:52 +0100
Subject: [PATCH 17/41] Fix format specifiers
---
llvm/tools/llvm-strings/llvm-strings.cpp | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index cb32fbef1aff88..4695ab01e9c4ba 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -256,13 +256,13 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
case Radix::None:
break;
case Radix::Octal:
- OS << format("%7o ", StringStart);
+ OS << format("%7zo ", StringStart);
break;
case Radix::Hexadecimal:
- OS << format("%7x ", StringStart);
+ OS << format("%7zx ", StringStart);
break;
case Radix::Decimal:
- OS << format("%7u ", StringStart);
+ OS << format("%7zu ", StringStart);
break;
}
};
>From 4327bd5d21f6435291afe727b2b560c5fbf543ff Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Wed, 21 Oct 2026 00:51:06 +0100
Subject: [PATCH 18/41] Address review feedback.
* Document default --encoding=S in docs.
* Handle the fact that mbrtoc32() is non-conforming on all the major platforms.
* On Windows, it ignores locale and always converts from UTF-8.
* On macOS, it is not present.
* On glibc, it returns the (size_t)-3 case as 0.
On Windows and macOS, wchar_t is used instead, except that on Windows,
if the active code page is UTF-8, the built-in UTF-8 decoder is used
instead to handle characters outside the BMP.
* Do not try to read again after EOF.
---
llvm/docs/CommandGuide/llvm-strings.md | 1 +
llvm/tools/llvm-strings/llvm-strings.cpp | 44 +++++++++++++++++++++++-
2 files changed, 44 insertions(+), 1 deletion(-)
diff --git a/llvm/docs/CommandGuide/llvm-strings.md b/llvm/docs/CommandGuide/llvm-strings.md
index 06be9b0cf6e13f..1de4326d0fd00c 100644
--- a/llvm/docs/CommandGuide/llvm-strings.md
+++ b/llvm/docs/CommandGuide/llvm-strings.md
@@ -54,6 +54,7 @@ Specifies the encoding of the input file. Valid arguments are:
- `s` (ASCII)
- `S` (current locale's character set)
- `utf8` (UTF-8)
+The default is `--encoding=S`.
:::
:::{option} --help, -h
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 4695ab01e9c4ba..067e37057cc700 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -32,6 +32,10 @@
#include <cuchar>
#include <string>
+#ifdef _WIN32
+#include "llvm/Support/Windows/WindowsSupport.h"
+#endif
+
using namespace llvm;
using namespace llvm::object;
@@ -143,17 +147,42 @@ bool Strings<Encoding::Locale>::readChar(const char *&Cur, const char *End,
const char *Next = Cur;
std::mbstate_t NextMBState = MBState;
+#if defined(_WIN32) || defined(__APPLE__)
+ wchar_t UCh;
+#else
char32_t UCh;
+#endif
for (;;) {
// Read one byte at a time. This is usually not the best way to use
// mbrtoc32(), usually it would make more sense to pass the size of the
// buffer, but the strings utility is unusual in that it is expected to
// encounter many bytes that do not form valid characters and it is more
// useful to optimise for this case.
- const std::size_t BytesRead = std::mbrtoc32(&UCh, Next, 1, &NextMBState);
+#if defined(_WIN32) || defined(__APPLE__)
+ // Workarounds:
+ // On Windows, mbrtoc32() ignores the locale and only converts UTF-8, but
+ // wchar_t is UTF-16 and is close enough, as long as we make sure to avoid
+ // it for UTF-8 locales.
+ // On Apple, <uchar.h> is not available, but wchar_t is UTF-32 and is a full
+ // substitute.
+ std::size_t BytesRead = std::mbrtowc(&UCh, Next, 1, &NextMBState);
+#else
+ std::size_t BytesRead = std::mbrtoc32(&UCh, Next, 1, &NextMBState);
+#endif
+ // Workaround:
+ // BytesRead == 0 should mean we found a null byte and should always have
+ // UCh == 0, but on glibc, it also gets returned for the case that should
+ // be indicated by (std::size_t)-3, as indicating that 0 bytes were
+ // converted. It can be detected by UCh != 0.
+ if (BytesRead == 0 && UCh) {
+ BytesRead = (std::size_t)-3;
+ }
switch (BytesRead) {
case 1:
++Next;
+ LLVM_FALLTHROUGH;
+
+ case (std::size_t)-3:
Cur = Next;
MBState = NextMBState;
Ch = UCh;
@@ -240,7 +269,13 @@ void Strings<Encoding::Locale>::endString(raw_ostream &OS,
char Buf[MB_LEN_MAX];
// Note: This is only required for stateful encodings such as the
// ISO-2022 ones.
+#if defined(_WIN32) || defined(__APPLE__)
+ const std::size_t BytesWritten = std::wcrtomb(Buf, L'\0', &MBState);
+#else
const std::size_t BytesWritten = std::c32rtomb(Buf, U'\0', &MBState);
+#endif
+ assert(BytesWritten < (std::size_t)-1 &&
+ "A null character must always be valid");
Buf[BytesWritten - 1] = '\n';
OS << StringRef(Buf, BytesWritten);
}
@@ -369,6 +404,9 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
}
if (EndOfChunk) {
+ if (AtEOF)
+ break;
+
// Finish handling the current chunk and update ChunkOffset.
ChunkOffset += Cur - Begin;
Buffer.erase(Buffer.begin(), Cur);
@@ -477,6 +515,10 @@ int main(int argc, char **argv) {
invalidArgValue(EncodingArg);
Encoding = *EncodingVal;
}
+#ifdef _WIN32
+ if (Encoding == Encoding::Locale && GetACP() == CP_UTF8)
+ Encoding = Encoding::Utf8;
+#endif
parseIntArg(Args, OPT_bytes_EQ, MinLength);
PrintFileName = Args.hasArg(OPT_print_file_name);
Arg *RadixArg = Args.getLastArg(OPT_radix_EQ);
>From b577caa8345adfa6f4277f3190bc0b5f8231e274 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Wed, 21 Oct 2026 13:25:36 +0100
Subject: [PATCH 19/41] Use wrapper functions for mbrto(wc/c32) and
(wc/c32)rtomb
---
llvm/tools/llvm-strings/llvm-strings.cpp | 67 ++++++++++++++----------
1 file changed, 38 insertions(+), 29 deletions(-)
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 067e37057cc700..2f6a21015da7fd 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -93,6 +93,41 @@ static void parseIntArg(const opt::InputArgList &Args, int ID, T &Value) {
}
}
+// Ideally we would use mbrtoc32() and c32rtomb() unconditionally but there are
+// problems on most platforms, thus we introduce wrappers. On Windows,
+// mbrtoc32() ignores the locale and only converts UTF-8, but wchar_t is UTF-16
+// and is close enough, as long as we make sure to avoid it for UTF-8 locales.
+// On Apple, <uchar.h> is not available, but wchar_t is UTF-32 and is a full
+// substitute.
+// On glibc, mbrtoc32() is a wrapper for mbrtowc() and never returns (size_t)-3,
+// instead it handles this case by returning 0 but setting Ch to a non-null
+// character.
+#if defined(_WIN32) || defined(__APPLE__)
+using char_t = wchar_t;
+static std::size_t mbrtoc(char_t *Ch, const char *S, size_t N,
+ std::mbstate_t *MBState) {
+ const std::size_t Result = std::mbrtowc(Ch, S, N, MBState);
+ if (Result == 0 && *Ch)
+ Result = (std::size_t)-3;
+ return Result;
+}
+static std::size_t crtomb(char *S, char_t Ch, mbstate_t *MBState) {
+ return std::wcrtomb(S, Ch, MBState);
+}
+#else
+using char_t = char32_t;
+static std::size_t mbrtoc(char_t *Ch, const char *S, size_t N,
+ std::mbstate_t *MBState) {
+ std::size_t Result = std::mbrtoc32(Ch, S, N, MBState);
+ if (Result == 0 && *Ch)
+ Result = (std::size_t)-3;
+ return Result;
+}
+static std::size_t crtomb(char *S, char_t Ch, mbstate_t *MBState) {
+ return std::c32rtomb(S, Ch, MBState);
+}
+#endif
+
template <enum Encoding> struct Strings {
Strings() = delete;
@@ -147,36 +182,14 @@ bool Strings<Encoding::Locale>::readChar(const char *&Cur, const char *End,
const char *Next = Cur;
std::mbstate_t NextMBState = MBState;
-#if defined(_WIN32) || defined(__APPLE__)
- wchar_t UCh;
-#else
- char32_t UCh;
-#endif
+ char_t UCh;
for (;;) {
// Read one byte at a time. This is usually not the best way to use
// mbrtoc32(), usually it would make more sense to pass the size of the
// buffer, but the strings utility is unusual in that it is expected to
// encounter many bytes that do not form valid characters and it is more
// useful to optimise for this case.
-#if defined(_WIN32) || defined(__APPLE__)
- // Workarounds:
- // On Windows, mbrtoc32() ignores the locale and only converts UTF-8, but
- // wchar_t is UTF-16 and is close enough, as long as we make sure to avoid
- // it for UTF-8 locales.
- // On Apple, <uchar.h> is not available, but wchar_t is UTF-32 and is a full
- // substitute.
- std::size_t BytesRead = std::mbrtowc(&UCh, Next, 1, &NextMBState);
-#else
- std::size_t BytesRead = std::mbrtoc32(&UCh, Next, 1, &NextMBState);
-#endif
- // Workaround:
- // BytesRead == 0 should mean we found a null byte and should always have
- // UCh == 0, but on glibc, it also gets returned for the case that should
- // be indicated by (std::size_t)-3, as indicating that 0 bytes were
- // converted. It can be detected by UCh != 0.
- if (BytesRead == 0 && UCh) {
- BytesRead = (std::size_t)-3;
- }
+ const std::size_t BytesRead = mbrtoc(&UCh, Next, 1, &NextMBState);
switch (BytesRead) {
case 1:
++Next;
@@ -269,11 +282,7 @@ void Strings<Encoding::Locale>::endString(raw_ostream &OS,
char Buf[MB_LEN_MAX];
// Note: This is only required for stateful encodings such as the
// ISO-2022 ones.
-#if defined(_WIN32) || defined(__APPLE__)
- const std::size_t BytesWritten = std::wcrtomb(Buf, L'\0', &MBState);
-#else
- const std::size_t BytesWritten = std::c32rtomb(Buf, U'\0', &MBState);
-#endif
+ const std::size_t BytesWritten = crtomb(Buf, 0, &MBState);
assert(BytesWritten < (std::size_t)-1 &&
"A null character must always be valid");
Buf[BytesWritten - 1] = '\n';
>From b428a50393d1d533f34592888be5a34b052a6d8c Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Wed, 21 Oct 2026 22:12:40 +0100
Subject: [PATCH 20/41] Remove leftover const, use size_t without std::
---
llvm/tools/llvm-strings/llvm-strings.cpp | 54 ++++++++++++------------
1 file changed, 27 insertions(+), 27 deletions(-)
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 2f6a21015da7fd..0b697f44e2e400 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -30,6 +30,7 @@
#include "llvm/Support/WithColor.h"
#include <clocale>
#include <cuchar>
+#include <stddef.h>
#include <string>
#ifdef _WIN32
@@ -104,26 +105,26 @@ static void parseIntArg(const opt::InputArgList &Args, int ID, T &Value) {
// character.
#if defined(_WIN32) || defined(__APPLE__)
using char_t = wchar_t;
-static std::size_t mbrtoc(char_t *Ch, const char *S, size_t N,
- std::mbstate_t *MBState) {
- const std::size_t Result = std::mbrtowc(Ch, S, N, MBState);
+static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
+ std::mbstate_t *MBState) {
+ size_t Result = std::mbrtowc(Ch, S, N, MBState);
if (Result == 0 && *Ch)
- Result = (std::size_t)-3;
+ Result = (size_t)-3;
return Result;
}
-static std::size_t crtomb(char *S, char_t Ch, mbstate_t *MBState) {
+static size_t crtomb(char *S, char_t Ch, mbstate_t *MBState) {
return std::wcrtomb(S, Ch, MBState);
}
#else
using char_t = char32_t;
-static std::size_t mbrtoc(char_t *Ch, const char *S, size_t N,
- std::mbstate_t *MBState) {
- std::size_t Result = std::mbrtoc32(Ch, S, N, MBState);
+static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
+ std::mbstate_t *MBState) {
+ size_t Result = std::mbrtoc32(Ch, S, N, MBState);
if (Result == 0 && *Ch)
- Result = (std::size_t)-3;
+ Result = (size_t)-3;
return Result;
}
-static std::size_t crtomb(char *S, char_t Ch, mbstate_t *MBState) {
+static size_t crtomb(char *S, char_t Ch, mbstate_t *MBState) {
return std::c32rtomb(S, Ch, MBState);
}
#endif
@@ -189,19 +190,19 @@ bool Strings<Encoding::Locale>::readChar(const char *&Cur, const char *End,
// buffer, but the strings utility is unusual in that it is expected to
// encounter many bytes that do not form valid characters and it is more
// useful to optimise for this case.
- const std::size_t BytesRead = mbrtoc(&UCh, Next, 1, &NextMBState);
+ const size_t BytesRead = mbrtoc(&UCh, Next, 1, &NextMBState);
switch (BytesRead) {
case 1:
++Next;
LLVM_FALLTHROUGH;
- case (std::size_t)-3:
+ case (size_t)-3:
Cur = Next;
MBState = NextMBState;
Ch = UCh;
return true;
- case (std::size_t)-2:
+ case (size_t)-2:
// We encountered a byte that is a valid start or continuation of a
// multibyte character. If we have more bytes, carry on. If we don't
// have more bytes yet, but we are not at the end of file, return false.
@@ -215,7 +216,7 @@ bool Strings<Encoding::Locale>::readChar(const char *&Cur, const char *End,
LLVM_FALLTHROUGH;
case 0:
- case (std::size_t)-1:
+ case (size_t)-1:
// We cannot form a valid non-null character.
//
// If we processed any bytes already that formed an incomplete multibyte
@@ -282,9 +283,8 @@ void Strings<Encoding::Locale>::endString(raw_ostream &OS,
char Buf[MB_LEN_MAX];
// Note: This is only required for stateful encodings such as the
// ISO-2022 ones.
- const std::size_t BytesWritten = crtomb(Buf, 0, &MBState);
- assert(BytesWritten < (std::size_t)-1 &&
- "A null character must always be valid");
+ const size_t BytesWritten = crtomb(Buf, 0, &MBState);
+ assert(BytesWritten < (size_t)-1 && "A null character must always be valid");
Buf[BytesWritten - 1] = '\n';
OS << StringRef(Buf, BytesWritten);
}
@@ -293,7 +293,7 @@ template <enum Encoding Encoding>
void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
sys::fs::file_t Handle) {
SmallString<sys::fs::DefaultReadChunkSize> Buffer;
- auto PrintHeader = [&OS, FileName](std::size_t StringStart) {
+ auto PrintHeader = [&OS, FileName](size_t StringStart) {
if (PrintFileName)
OS << FileName << ": ";
switch (Radix) {
@@ -320,21 +320,21 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
// never needs an arbitrarily large buffer. Candidate therefore only ever
// holds a run that is shorter than MinLength and that was cut off by the end
// of a chunk.
- const std::size_t MinLengthChars = MinLength;
+ const size_t MinLengthChars = MinLength;
SmallString<DefaultMinLength> Candidate;
- std::size_t CandidateLengthChars = 0;
+ size_t CandidateLengthChars = 0;
bool InString = false;
std::mbstate_t MBState{};
// Offset of the start of the current chunk within the file.
- std::size_t ChunkOffset = 0;
+ size_t ChunkOffset = 0;
while (true) {
// Size of any partial character left over from the previous chunk.
- const std::size_t PendingCharSizeBytes = Buffer.size();
+ const size_t PendingCharSizeBytes = Buffer.size();
Buffer.resize_for_overwrite(PendingCharSizeBytes +
sys::fs::DefaultReadChunkSize);
- Expected<std::size_t> ReadBytesOrErr = sys::fs::readNativeFile(
+ Expected<size_t> ReadBytesOrErr = sys::fs::readNativeFile(
Handle,
MutableArrayRef(Buffer.begin() + PendingCharSizeBytes, Buffer.end()));
if (!ReadBytesOrErr) {
@@ -343,7 +343,7 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
return;
}
const bool AtEOF = *ReadBytesOrErr == 0;
- const std::size_t ChunkSize = PendingCharSizeBytes + *ReadBytesOrErr;
+ const size_t ChunkSize = PendingCharSizeBytes + *ReadBytesOrErr;
if (ChunkSize == 0)
break;
@@ -369,7 +369,7 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
if (InString || !Candidate.empty()) {
const char *StringEnd;
std::mbstate_t PrevMBState;
- std::size_t LengthChars = 0;
+ size_t LengthChars = 0;
bool EndOfChunk;
// Find the end of the current string.
@@ -382,7 +382,7 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
++LengthChars;
}
- std::size_t SizeBytes = StringEnd - Begin;
+ size_t SizeBytes = StringEnd - Begin;
if (InString) {
// Print the remaining part if the previous chunk has already printed
// the header. E.g. header: aaaaa | bbbbb, where | is the chunk
@@ -433,7 +433,7 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
// At this point, we are always at the start of a new string because the
// remaining part of the previous string has already been handled.
const char *StrHead = nullptr;
- std::size_t LengthChars = 0;
+ size_t LengthChars = 0;
for (;;) {
const char *Prev = Cur;
std::mbstate_t PrevMBState = MBState;
>From 1b9ef79f804738e27e269e1622538a6fd108b0ad Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Thu, 22 Oct 2026 14:50:03 +0100
Subject: [PATCH 21/41] Add release note, remove include of <stddef.h> already
indirectly included
---
llvm/docs/ReleaseNotes.md | 9 +++++++++
llvm/tools/llvm-strings/llvm-strings.cpp | 1 -
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md
index 302aa57396105e..825f346621f03d 100644
--- a/llvm/docs/ReleaseNotes.md
+++ b/llvm/docs/ReleaseNotes.md
@@ -428,6 +428,15 @@ Makes programs 10x faster by doing Special New Thing.
* llvm-rc now supports `/showIncludes` to report header and resource-file
dependencies in a format compatible with Ninja's `deps = msvc` mode.
+* llvm-strings now supports the `-e` (`--encoding`) option and by default prints
+ non-ASCII strings it finds, using the system's configured character set. The
+ prior behavior can be restored by specifying `-es` (`--encoding=s`), which is
+ mostly useful on large files when the searched-for strings are known ahead of
+ time to be ASCII, as it searches faster and gives the user fewer results to
+ sift through. The `-eutf8` is also available to search for UTF-8 strings,
+ which is mostly useful on Windows where configuring the system to use UTF-8 is
+ more involved than setting an environment variable.
+
### Changes to LLDB
* `platform.plugin.wasm.runtime-args` now precede the port argument on the Wasm
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 0b697f44e2e400..6d06b2f49959a3 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -30,7 +30,6 @@
#include "llvm/Support/WithColor.h"
#include <clocale>
#include <cuchar>
-#include <stddef.h>
#include <string>
#ifdef _WIN32
>From b465e8a3e15f29b37d4da1120ade6a411dfd468b Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Thu, 22 Oct 2026 15:09:39 +0100
Subject: [PATCH 22/41] Avoid isPrint
Strings<Encoding::Ascii>::isStringChar would implicitly convert its
argument to char in calling isPrint. It would not produce incorrect
results for how it is used, but it would produce incorrect results for
how a subsequent change would potentially use it, so just fix that now.
---
llvm/tools/llvm-strings/llvm-strings.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 6d06b2f49959a3..c174335508c48b 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -258,7 +258,7 @@ bool Strings<Encoding::Utf8>::readChar(const char *&Cur, const char *End,
}
template <> bool Strings<Encoding::Ascii>::isStringChar(UTF32 Ch) {
- return Ch == '\t' || isPrint(Ch);
+ return Ch == U'\t' || (Ch >= U' ' && Ch <= U'~');
}
template <> bool Strings<Encoding::Locale>::isStringChar(UTF32 Ch) {
>From 4fb09c36a5c9d3753105ea0264bdfd02fa36f271 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Fri, 23 Oct 2026 23:30:29 +0100
Subject: [PATCH 23/41] Optimise llvm-strings further
* For single-byte character sets (MB_CUR_MAX == 1), build a
256-element lookup table rather than converting each character in the
input to UTF-32.
* For multi-byte character sets, including UTF-8, build an
0x110000-element lookup table rather than calling isCharacter and
isControl for each converted character in the input.
* For UTF-8 locales, also switch to the internal decoder on Unix where
we know it is safe.
---
llvm/tools/llvm-strings/llvm-strings.cpp | 393 ++++++++++++++---------
1 file changed, 233 insertions(+), 160 deletions(-)
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index c174335508c48b..43025b5adf99c5 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -27,11 +27,17 @@
#include "llvm/Support/Program.h"
#include "llvm/Support/SwapByteOrder.h"
#include "llvm/Support/Unicode.h"
+#include "llvm/Support/UnicodeCharRanges.h"
#include "llvm/Support/WithColor.h"
+#include <bitset>
#include <clocale>
#include <cuchar>
#include <string>
+#ifdef LLVM_ON_UNIX
+#include <langinfo.h>
+#endif
+
#ifdef _WIN32
#include "llvm/Support/Windows/WindowsSupport.h"
#endif
@@ -39,6 +45,11 @@
using namespace llvm;
using namespace llvm::object;
+namespace llvm::sys::unicode {
+extern const UnicodeCharSet::CharRanges GeneratedCharacterRanges;
+extern const UnicodeCharSet::CharRanges GeneratedControlCharacterRanges;
+} // namespace llvm::sys::unicode
+
namespace {
enum ID {
OPT_INVALID = 0, // This is not an option ID.
@@ -128,169 +139,216 @@ static size_t crtomb(char *S, char_t Ch, mbstate_t *MBState) {
}
#endif
-template <enum Encoding> struct Strings {
- Strings() = delete;
+struct CharSet {
+ template <typename T> struct Base {
+ Base() = delete;
+
+ using CharT = T;
+
+ // Tries to read one character from the bytes Cur...End and store it in Ch.
+ // Returns true if a (possibly invalid) character was read, false otherwise.
+ //
+ // If Cur...End starts with a valid complete character, Cur and MBState are
+ // updated and true is returned.
+ // If Cur...End is empty, or holds an incomplete character and AtEOF is
+ // false, Cur and MBState are unchanged, Ch is set to 0, and false is
+ // returned. This is intended to allow more bytes to be read and readChar to
+ // be called again. If Cur...End holds an incomplete character and AtEOF is
+ // true, or if Cur...End starts with an invalid character, the first bytes
+ // are skipped and MBState is reset to allow continuing from the next point,
+ // Ch is set to 0, and true is returned. This case is indistinguishable to
+ // the caller from a null byte being successfully converted to a null
+ // character, but a null character is never printable under any settings and
+ // that is how invalid bytes are treated as well.
+ //
+ // The number of skipped bytes for invalid characters follows the Unicode
+ // definition of the maximal subpart of an ill-formed subsequence, applied
+ // to arbitrary locales: it is the longest subsequence that could start a
+ // valid multibyte character, or if the initial byte cannot start a valid
+ // multibyte character, the initial byte. This allows consistent error
+ // recovery.
+ static bool readChar(const char *&Cur, const char *End,
+ std::mbstate_t &MBState, bool AtEOF, CharT &Ch);
+
+ static bool isStringChar(CharT Ch);
+
+ static void endString(raw_ostream &OS, std::mbstate_t &MBState) {
+ OS << '\n';
+ }
+ };
- // Tries to read one character from the bytes Cur...End and store it in Ch.
- // Returns true if a (possibly invalid) character was read, false otherwise.
- //
- // If Cur...End starts with a valid complete character, Cur and MBState are
- // updated and true is returned.
- // If Cur...End is empty, or holds an incomplete character and AtEOF is false,
- // Cur and MBState are unchanged, Ch is set to 0, and false is returned. This
- // is intended to allow more bytes to be read and readChar to be called again.
- // If Cur...End holds an incomplete character and AtEOF is true, or if
- // Cur...End starts with an invalid character, the first bytes are skipped and
- // MBState is reset to allow continuing from the next point, Ch is set to 0,
- // and true is returned. This case is indistinguishable to the caller from a
- // null byte being successfully converted to a null character, but a null
- // character is never printable under any settings and that is how invalid
- // bytes are treated as well.
- //
- // The number of skipped bytes for invalid characters follows the Unicode
- // definition of the maximal subpart of an ill-formed subsequence, applied to
- // arbitrary locales: it is the longest subsequence that could start a valid
- // multibyte character, or if the initial byte cannot start a valid multibyte
- // character, the initial byte. This allows consistent error recovery.
- static bool readChar(const char *&Cur, const char *End,
- std::mbstate_t &MBState, bool AtEOF, UTF32 &Ch);
+ struct MultiByte : Base<UTF32> {
+ static bool readChar(const char *&Cur, const char *End,
+ std::mbstate_t &MBState, bool AtEOF, CharT &Ch) {
+ if (Cur == End)
+ return false;
- static bool isStringChar(UTF32 Ch);
+ const char *Next = Cur;
+ std::mbstate_t NextMBState = MBState;
+ char_t UCh;
+ for (;;) {
+ // Read one byte at a time. This is usually not the best way to use
+ // mbrtoc32(), usually it would make more sense to pass the size of the
+ // buffer, but the strings utility is unusual in that it is expected to
+ // encounter many bytes that do not form valid characters and it is more
+ // useful to optimise for this case.
+ const size_t BytesRead = mbrtoc(&UCh, Next, 1, &NextMBState);
+ switch (BytesRead) {
+ case 1:
+ ++Next;
+ LLVM_FALLTHROUGH;
+
+ case (size_t)-3:
+ Cur = Next;
+ MBState = NextMBState;
+ Ch = std::min<UTF32>(UCh, 0x10FFFF);
+ return true;
+
+ case (size_t)-2:
+ // We encountered a byte that is a valid start or continuation of a
+ // multibyte character. If we have more bytes, carry on. If we don't
+ // have more bytes yet, but we are not at the end of file, return
+ // false. If we don't have more bytes and we are at the end of file,
+ // fall through to treat it as an error.
+ ++Next;
+ if (Next != End)
+ continue;
+ if (!AtEOF)
+ return false;
+ LLVM_FALLTHROUGH;
+
+ case 0:
+ case (size_t)-1:
+ // We cannot form a valid non-null character.
+ //
+ // If we processed any bytes already that formed an incomplete
+ // multibyte character, treat those bytes as a single null character,
+ // otherwise treat the current byte as a single null character.
+ if (Next == Cur)
+ ++Next;
+ Cur = Next;
+ MBState = {};
+ Ch = 0;
+ return true;
+ }
+ }
+ }
- static void endString(raw_ostream &OS, std::mbstate_t &MBState);
+ static bool isStringChar(CharT Ch) {
+ using StringCharSet = std::bitset<0x110000>;
+ static const StringCharSet &StringChars = []() -> const StringCharSet & {
+ static StringCharSet StringChars;
- static void run(raw_ostream &OS, StringRef FileName, sys::fs::file_t Handle);
-};
+ using llvm::sys::unicode::GeneratedCharacterRanges;
+ using llvm::sys::unicode::GeneratedControlCharacterRanges;
+ using llvm::sys::UnicodeCharRange;
-template <>
-bool Strings<Encoding::Ascii>::readChar(const char *&Cur, const char *End,
- std::mbstate_t &MBState, bool AtEOF,
- UTF32 &Ch) {
- if (Cur == End)
- return false;
+ for (const UnicodeCharRange &Range : GeneratedCharacterRanges)
+ for (UTF32 Ch = Range.Lower; Ch <= Range.Upper; ++Ch)
+ StringChars[Ch] = true;
+ for (const UnicodeCharRange &Range : GeneratedControlCharacterRanges)
+ for (UTF32 Ch = Range.Lower; Ch <= Range.Upper; ++Ch)
+ StringChars[Ch] = false;
+ StringChars[U'\t'] = true;
- Ch = *Cur++;
- return true;
-}
+ return StringChars;
+ }();
-template <>
-bool Strings<Encoding::Locale>::readChar(const char *&Cur, const char *End,
- std::mbstate_t &MBState, bool AtEOF,
- UTF32 &Ch) {
- if (Cur == End)
- return false;
-
- const char *Next = Cur;
- std::mbstate_t NextMBState = MBState;
- char_t UCh;
- for (;;) {
- // Read one byte at a time. This is usually not the best way to use
- // mbrtoc32(), usually it would make more sense to pass the size of the
- // buffer, but the strings utility is unusual in that it is expected to
- // encounter many bytes that do not form valid characters and it is more
- // useful to optimise for this case.
- const size_t BytesRead = mbrtoc(&UCh, Next, 1, &NextMBState);
- switch (BytesRead) {
- case 1:
- ++Next;
- LLVM_FALLTHROUGH;
-
- case (size_t)-3:
- Cur = Next;
- MBState = NextMBState;
- Ch = UCh;
- return true;
+ return StringChars[Ch];
+ }
- case (size_t)-2:
- // We encountered a byte that is a valid start or continuation of a
- // multibyte character. If we have more bytes, carry on. If we don't
- // have more bytes yet, but we are not at the end of file, return false.
- // If we don't have more bytes and we are at the end of file, fall
- // through to treat it as an error.
- ++Next;
- if (Next != End)
- continue;
- if (AtEOF)
+ static void endString(raw_ostream &OS, std::mbstate_t &MBState) {
+ char Buf[MB_LEN_MAX];
+ // Note: This is only required for stateful encodings such as the
+ // ISO-2022 ones.
+ const size_t BytesWritten = crtomb(Buf, 0, &MBState);
+ assert(BytesWritten < (size_t)-1 &&
+ "A null character must always be valid");
+ Buf[BytesWritten - 1] = '\n';
+ OS << StringRef(Buf, BytesWritten);
+ }
+ };
+
+ struct Utf8 : MultiByte {
+ static bool readChar(const char *&Cur, const char *End,
+ std::mbstate_t &MBState, bool AtEOF, CharT &Ch) {
+ if (Cur == End)
return false;
- LLVM_FALLTHROUGH;
-
- case 0:
- case (size_t)-1:
- // We cannot form a valid non-null character.
- //
- // If we processed any bytes already that formed an incomplete multibyte
- // character, treat those bytes as a single null character, otherwise
- // treat the current byte as a single null character.
- if (Next == Cur)
- ++Next;
- Cur = Next;
- MBState = {};
- Ch = 0;
+
+ const UTF8 *UTF8Cur = reinterpret_cast<const UTF8 *>(Cur);
+ const UTF8 *UTF8Next = UTF8Cur;
+ const UTF8 *UTF8End = reinterpret_cast<const UTF8 *>(End);
+ UTF32 *UTF32Next = &Ch;
+ const auto Res = ConvertUTF8toUTF32(&UTF8Next, UTF8End, &UTF32Next,
+ &Ch + 1, strictConversion);
+ if (UTF8Next != UTF8Cur) {
+ assert(UTF32Next != &Ch);
+ assert(Ch <= 0x10FFFF &&
+ "ConvertUTF8toUTF32 produced an out-of-range character");
+ } else if (Res == sourceExhausted && !AtEOF) {
+ return false;
+ } else {
+ assert(UTF32Next == &Ch);
+ UTF8Next +=
+ findMaximalSubpartOfIllFormedUTF8Sequence(UTF8Next, UTF8End);
+ Ch = 0;
+ }
+ Cur = reinterpret_cast<const char *>(UTF8Next);
return true;
}
- }
-}
-template <>
-bool Strings<Encoding::Utf8>::readChar(const char *&Cur, const char *End,
- std::mbstate_t &MBState, bool AtEOF,
- UTF32 &Ch) {
- if (Cur == End)
- return false;
-
- const UTF8 *UTF8Cur = reinterpret_cast<const UTF8 *>(Cur);
- const UTF8 *UTF8Next = UTF8Cur;
- const UTF8 *UTF8End = reinterpret_cast<const UTF8 *>(End);
- UTF32 *UTF32Next = &Ch;
- const auto Res = ConvertUTF8toUTF32(&UTF8Next, UTF8End, &UTF32Next, &Ch + 1,
- strictConversion);
- if (UTF8Next != UTF8Cur) {
- assert(UTF32Next != &Ch);
- } else if (Res == sourceExhausted && !AtEOF) {
- return false;
- } else {
- assert(UTF32Next == &Ch);
- UTF8Next += findMaximalSubpartOfIllFormedUTF8Sequence(UTF8Next, UTF8End);
- Ch = 0;
- }
- Cur = reinterpret_cast<const char *>(UTF8Next);
- return true;
-}
+ static void endString(raw_ostream &OS, std::mbstate_t &MBState) {
+ Base::endString(OS, MBState);
+ }
+ };
-template <> bool Strings<Encoding::Ascii>::isStringChar(UTF32 Ch) {
- return Ch == U'\t' || (Ch >= U' ' && Ch <= U'~');
-}
+ struct SingleByte : Base<unsigned char> {
+ static bool readChar(const char *&Cur, const char *End,
+ std::mbstate_t &MBState, bool AtEOF, CharT &Ch) {
+ if (Cur == End)
+ return false;
-template <> bool Strings<Encoding::Locale>::isStringChar(UTF32 Ch) {
- return llvm::sys::unicode::isCharacter(Ch) &&
- (Ch == U'\t' || !llvm::sys::unicode::isControl(Ch));
-}
+ Ch = *reinterpret_cast<const unsigned char *>(Cur++);
+ return true;
+ }
-template <> bool Strings<Encoding::Utf8>::isStringChar(UTF32 Ch) {
- return llvm::sys::unicode::isCharacter(Ch) &&
- (Ch == U'\t' || !llvm::sys::unicode::isControl(Ch));
-}
+ static bool isStringChar(CharT Ch) {
+ using StringCharSet = std::bitset<0x100>;
+ static const StringCharSet &StringChars = []() -> const StringCharSet & {
+ static StringCharSet StringChars;
+
+ using llvm::sys::unicode::isCharacter;
+ using llvm::sys::unicode::isControl;
+
+ for (int I = 0; I < 0x100; ++I) {
+ const unsigned char Ch = I;
+ const char *Cur = reinterpret_cast<const char *>(&Ch), *End = Cur + 1;
+ std::mbstate_t MBState = {};
+ UTF32 UCh;
+ if (MultiByte::readChar(Cur, End, MBState, /*AtEOF=*/true, UCh)) {
+ if (UCh == U'\t' || (isCharacter(UCh) && !isControl(UCh)))
+ StringChars[I] = true;
+ }
+ }
-template <enum Encoding Encoding>
-void Strings<Encoding>::endString(raw_ostream &OS, std::mbstate_t &MBState) {
- OS << '\n';
-}
+ return StringChars;
+ }();
-template <>
-void Strings<Encoding::Locale>::endString(raw_ostream &OS,
- std::mbstate_t &MBState) {
- char Buf[MB_LEN_MAX];
- // Note: This is only required for stateful encodings such as the
- // ISO-2022 ones.
- const size_t BytesWritten = crtomb(Buf, 0, &MBState);
- assert(BytesWritten < (size_t)-1 && "A null character must always be valid");
- Buf[BytesWritten - 1] = '\n';
- OS << StringRef(Buf, BytesWritten);
-}
+ return StringChars[Ch];
+ }
+ };
+
+ struct Ascii : SingleByte {
+ static bool isStringChar(CharT Ch) {
+ return Ch == '\t' || (Ch >= ' ' && Ch <= '~');
+ }
+ };
+};
+
+template <typename CharSet>
+void strings(raw_ostream &OS, StringRef FileName, sys::fs::file_t Handle) {
+ using CharT = typename CharSet::CharT;
-template <enum Encoding Encoding>
-void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
- sys::fs::file_t Handle) {
SmallString<sys::fs::DefaultReadChunkSize> Buffer;
auto PrintHeader = [&OS, FileName](size_t StringStart) {
if (PrintFileName)
@@ -355,7 +413,7 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
const char *End = Begin + ChunkSize;
const char *Cur = Begin;
- UTF32 Ch;
+ CharT Ch;
// Handle the remaining part from the previous chunk.
// The previous chunk can be either shorter than MinSize or part of the
@@ -375,8 +433,8 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
for (;;) {
StringEnd = Cur;
PrevMBState = MBState;
- EndOfChunk = !readChar(Cur, End, MBState, AtEOF, Ch);
- if (EndOfChunk || !isStringChar(Ch))
+ EndOfChunk = !CharSet::readChar(Cur, End, MBState, AtEOF, Ch);
+ if (EndOfChunk || !CharSet::isStringChar(Ch))
break;
++LengthChars;
}
@@ -424,7 +482,7 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
if (InString) {
// We haven't reached the end of the chunk, which means the string is
// terminated.
- endString(OS, PrevMBState);
+ CharSet::endString(OS, PrevMBState);
InString = false;
}
}
@@ -436,9 +494,9 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
for (;;) {
const char *Prev = Cur;
std::mbstate_t PrevMBState = MBState;
- if (!readChar(Cur, End, MBState, AtEOF, Ch))
+ if (!CharSet::readChar(Cur, End, MBState, AtEOF, Ch))
break;
- if (isStringChar(Ch)) {
+ if (CharSet::isStringChar(Ch)) {
// Find the start of the next string.
if (!StrHead)
StrHead = Prev;
@@ -449,7 +507,7 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
if (LengthChars >= MinLengthChars) {
PrintHeader(ChunkOffset + (StrHead - Begin));
OS << StringRef(StrHead, Prev - StrHead);
- endString(OS, PrevMBState);
+ CharSet::endString(OS, PrevMBState);
}
StrHead = nullptr;
LengthChars = 0;
@@ -479,7 +537,7 @@ void Strings<Encoding>::run(raw_ostream &OS, StringRef FileName,
}
if (InString)
- endString(OS, MBState);
+ CharSet::endString(OS, MBState);
}
} // namespace
@@ -523,10 +581,6 @@ int main(int argc, char **argv) {
invalidArgValue(EncodingArg);
Encoding = *EncodingVal;
}
-#ifdef _WIN32
- if (Encoding == Encoding::Locale && GetACP() == CP_UTF8)
- Encoding = Encoding::Utf8;
-#endif
parseIntArg(Args, OPT_bytes_EQ, MinLength);
PrintFileName = Args.hasArg(OPT_print_file_name);
Arg *RadixArg = Args.getLastArg(OPT_radix_EQ);
@@ -551,13 +605,32 @@ int main(int argc, char **argv) {
sys::fs::file_t Handle) = [] {
switch (Encoding) {
case Encoding::Ascii:
- return Strings<Encoding::Ascii>::run;
+ return strings<CharSet::Ascii>;
case Encoding::Locale:
- return Strings<Encoding::Locale>::run;
+ if (MB_CUR_MAX == 1)
+ return strings<CharSet::SingleByte>;
+
+#ifdef LLVM_ON_UNIX
+ // Using LLVM's built-in decoder is faster than going through libc, but
+ // this doesn't affect correctness (assuming 32-bit wchar_t), so it's fine
+ // if this check has false negatives (e.g. if some OS returns "utf8"
+ // instead).
+ if (StringRef(nl_langinfo(CODESET)) == "UTF-8")
+ return strings<CharSet::Utf8>;
+#endif
+#ifdef _WIN32
+ // Using LLVM's built-in decoder is necessary for correctness as
+ // Strings<CharSet::MultiByte> on WIndows is implemented on top of 16-bit
+ // wchar_t which cannot represent all code points.
+ if (GetACP() == CP_UTF8)
+ return strings<CharSet::Utf8>;
+#endif
+
+ return strings<CharSet::MultiByte>;
case Encoding::Utf8:
- return Strings<Encoding::Utf8>::run;
+ return strings<CharSet::Utf8>;
}
llvm_unreachable("unhandled encoding");
>From fe46d2ee4d34ddba09d861f96fd4fca76a989c0c Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Mon, 26 Oct 2026 20:46:02 +0000
Subject: [PATCH 24/41] Fix up stateful encoding handling, avoid the complexity
where not needed
---
llvm/tools/llvm-strings/llvm-strings.cpp | 135 +++++++++++++----------
1 file changed, 79 insertions(+), 56 deletions(-)
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 43025b5adf99c5..8c26e5c06fb624 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -141,8 +141,6 @@ static size_t crtomb(char *S, char_t Ch, mbstate_t *MBState) {
struct CharSet {
template <typename T> struct Base {
- Base() = delete;
-
using CharT = T;
// Tries to read one character from the bytes Cur...End and store it in Ch.
@@ -167,24 +165,27 @@ struct CharSet {
// valid multibyte character, or if the initial byte cannot start a valid
// multibyte character, the initial byte. This allows consistent error
// recovery.
- static bool readChar(const char *&Cur, const char *End,
- std::mbstate_t &MBState, bool AtEOF, CharT &Ch);
+ static bool readChar(const char *&Cur, const char *End, bool AtEOF,
+ CharT &Ch);
static bool isStringChar(CharT Ch);
- static void endString(raw_ostream &OS, std::mbstate_t &MBState) {
- OS << '\n';
- }
+ static void startString() {}
+
+ static void writeString(raw_ostream &OS, StringRef Str) { OS << Str; }
+
+ static void endString(raw_ostream &OS) { OS << '\n'; }
};
- struct MultiByte : Base<UTF32> {
- static bool readChar(const char *&Cur, const char *End,
- std::mbstate_t &MBState, bool AtEOF, CharT &Ch) {
+ struct MultiByteStateless : Base<UTF32> {
+ std::mbstate_t InMBState{};
+
+ bool readChar(const char *&Cur, const char *End, bool AtEOF, CharT &Ch) {
if (Cur == End)
return false;
const char *Next = Cur;
- std::mbstate_t NextMBState = MBState;
+ std::mbstate_t NextMBState = InMBState;
char_t UCh;
for (;;) {
// Read one byte at a time. This is usually not the best way to use
@@ -200,7 +201,7 @@ struct CharSet {
case (size_t)-3:
Cur = Next;
- MBState = NextMBState;
+ InMBState = NextMBState;
Ch = std::min<UTF32>(UCh, 0x10FFFF);
return true;
@@ -227,7 +228,7 @@ struct CharSet {
if (Next == Cur)
++Next;
Cur = Next;
- MBState = {};
+ InMBState = {};
Ch = 0;
return true;
}
@@ -256,22 +257,49 @@ struct CharSet {
return StringChars[Ch];
}
+ };
+
+ // Note: The complexity in writing is only required for stateful encodings
+ // such as the ISO-2022 ones.
+ struct MultiByteStateful : MultiByteStateless {
+ std::mbstate_t StrMBState{}, OutMBState{};
+
+ void startString() { StrMBState = InMBState; }
+
+ void writeString(raw_ostream &OS, StringRef Str) {
+ const char *Cur = Str.begin(), *End = Str.end();
+ while (Cur != End) {
+ char_t UCh;
+ const size_t BytesRead = mbrtoc(&UCh, Cur, End - Cur, &StrMBState);
+ assert(BytesRead != 0 && BytesRead != (size_t)-1 &&
+ BytesRead != (size_t)-2 &&
+ "If a string was initially valid, it cannot to become invalid "
+ "when re-reading");
+ if (BytesRead != (size_t)-3)
+ Cur += BytesRead;
+
+ char Buf[MB_LEN_MAX];
+ const size_t BytesWritten = crtomb(Buf, UCh, &OutMBState);
+ assert(BytesWritten != (size_t)-1 &&
+ "If we successfully read a character, we must also be able to "
+ "write it");
+ OS << StringRef(Buf, BytesWritten);
+ }
+ }
- static void endString(raw_ostream &OS, std::mbstate_t &MBState) {
+ void endString(raw_ostream &OS) {
char Buf[MB_LEN_MAX];
- // Note: This is only required for stateful encodings such as the
- // ISO-2022 ones.
- const size_t BytesWritten = crtomb(Buf, 0, &MBState);
- assert(BytesWritten < (size_t)-1 &&
+ const size_t BytesWritten = crtomb(Buf, 0, &OutMBState);
+ assert(BytesWritten != (size_t)-1 &&
"A null character must always be valid");
Buf[BytesWritten - 1] = '\n';
OS << StringRef(Buf, BytesWritten);
}
};
- struct Utf8 : MultiByte {
- static bool readChar(const char *&Cur, const char *End,
- std::mbstate_t &MBState, bool AtEOF, CharT &Ch) {
+ struct Utf8 : MultiByteStateless {
+ static bool readChar(const char *&Cur, const char *End, bool AtEOF,
+ CharT &Ch) {
if (Cur == End)
return false;
@@ -296,15 +324,11 @@ struct CharSet {
Cur = reinterpret_cast<const char *>(UTF8Next);
return true;
}
-
- static void endString(raw_ostream &OS, std::mbstate_t &MBState) {
- Base::endString(OS, MBState);
- }
};
struct SingleByte : Base<unsigned char> {
- static bool readChar(const char *&Cur, const char *End,
- std::mbstate_t &MBState, bool AtEOF, CharT &Ch) {
+ static bool readChar(const char *&Cur, const char *End, bool AtEOF,
+ CharT &Ch) {
if (Cur == End)
return false;
@@ -317,18 +341,14 @@ struct CharSet {
static const StringCharSet &StringChars = []() -> const StringCharSet & {
static StringCharSet StringChars;
- using llvm::sys::unicode::isCharacter;
- using llvm::sys::unicode::isControl;
-
+ MultiByteStateless MB;
for (int I = 0; I < 0x100; ++I) {
const unsigned char Ch = I;
const char *Cur = reinterpret_cast<const char *>(&Ch), *End = Cur + 1;
- std::mbstate_t MBState = {};
- UTF32 UCh;
- if (MultiByte::readChar(Cur, End, MBState, /*AtEOF=*/true, UCh)) {
- if (UCh == U'\t' || (isCharacter(UCh) && !isControl(UCh)))
- StringChars[I] = true;
- }
+ MultiByteStateless::CharT UCh;
+ if (MB.readChar(Cur, End, /*AtEOF=*/true, UCh) &&
+ MB.isStringChar(UCh))
+ StringChars[I] = true;
}
return StringChars;
@@ -345,9 +365,10 @@ struct CharSet {
};
};
-template <typename CharSet>
+template <typename CharSetT>
void strings(raw_ostream &OS, StringRef FileName, sys::fs::file_t Handle) {
- using CharT = typename CharSet::CharT;
+ using CharT = typename CharSetT::CharT;
+ CharSetT CharSet;
SmallString<sys::fs::DefaultReadChunkSize> Buffer;
auto PrintHeader = [&OS, FileName](size_t StringStart) {
@@ -381,7 +402,6 @@ void strings(raw_ostream &OS, StringRef FileName, sys::fs::file_t Handle) {
SmallString<DefaultMinLength> Candidate;
size_t CandidateLengthChars = 0;
bool InString = false;
- std::mbstate_t MBState{};
// Offset of the start of the current chunk within the file.
size_t ChunkOffset = 0;
@@ -425,16 +445,14 @@ void strings(raw_ostream &OS, StringRef FileName, sys::fs::file_t Handle) {
// enough data to decide whether to print it.
if (InString || !Candidate.empty()) {
const char *StringEnd;
- std::mbstate_t PrevMBState;
size_t LengthChars = 0;
bool EndOfChunk;
// Find the end of the current string.
for (;;) {
StringEnd = Cur;
- PrevMBState = MBState;
- EndOfChunk = !CharSet::readChar(Cur, End, MBState, AtEOF, Ch);
- if (EndOfChunk || !CharSet::isStringChar(Ch))
+ EndOfChunk = !CharSet.readChar(Cur, End, AtEOF, Ch);
+ if (EndOfChunk || !CharSet.isStringChar(Ch))
break;
++LengthChars;
}
@@ -445,7 +463,7 @@ void strings(raw_ostream &OS, StringRef FileName, sys::fs::file_t Handle) {
// the header. E.g. header: aaaaa | bbbbb, where | is the chunk
// boundary.
// Output: Header: aaaaabbbbb, where bbbbb is printed in here.
- OS << StringRef(Begin, SizeBytes);
+ CharSet.writeString(OS, StringRef(Begin, SizeBytes));
} else if (CandidateLengthChars + LengthChars >= MinLengthChars) {
// If the header hasn't been printed yet (i.e. the previous candidate
// was smaller than Min), but we can print it now, print the header
@@ -453,7 +471,8 @@ void strings(raw_ostream &OS, StringRef FileName, sys::fs::file_t Handle) {
// current string. E.g. aa | bbbbbb
// Output Header: aabbbbbb, where aabbbbbb is printed in here.
PrintHeader(ChunkOffset - Candidate.size());
- OS << Candidate << StringRef(Begin, SizeBytes);
+ CharSet.writeString(OS, Candidate);
+ CharSet.writeString(OS, StringRef(Begin, SizeBytes));
Candidate.clear();
CandidateLengthChars = 0;
InString = true;
@@ -482,7 +501,7 @@ void strings(raw_ostream &OS, StringRef FileName, sys::fs::file_t Handle) {
if (InString) {
// We haven't reached the end of the chunk, which means the string is
// terminated.
- CharSet::endString(OS, PrevMBState);
+ CharSet.endString(OS);
InString = false;
}
}
@@ -493,10 +512,11 @@ void strings(raw_ostream &OS, StringRef FileName, sys::fs::file_t Handle) {
size_t LengthChars = 0;
for (;;) {
const char *Prev = Cur;
- std::mbstate_t PrevMBState = MBState;
- if (!CharSet::readChar(Cur, End, MBState, AtEOF, Ch))
+ if (!StrHead)
+ CharSet.startString();
+ if (!CharSet.readChar(Cur, End, AtEOF, Ch))
break;
- if (CharSet::isStringChar(Ch)) {
+ if (CharSet.isStringChar(Ch)) {
// Find the start of the next string.
if (!StrHead)
StrHead = Prev;
@@ -506,8 +526,8 @@ void strings(raw_ostream &OS, StringRef FileName, sys::fs::file_t Handle) {
// current string. Print it if long enough.
if (LengthChars >= MinLengthChars) {
PrintHeader(ChunkOffset + (StrHead - Begin));
- OS << StringRef(StrHead, Prev - StrHead);
- CharSet::endString(OS, PrevMBState);
+ CharSet.writeString(OS, StringRef(StrHead, Prev - StrHead));
+ CharSet.endString(OS);
}
StrHead = nullptr;
LengthChars = 0;
@@ -521,7 +541,7 @@ void strings(raw_ostream &OS, StringRef FileName, sys::fs::file_t Handle) {
// Print it, or append it to Candidate if it is too short.
if (LengthChars >= MinLengthChars) {
PrintHeader(ChunkOffset + (StrHead - Begin));
- OS << StringRef(StrHead, Cur - StrHead);
+ CharSet.writeString(OS, StringRef(StrHead, Cur - StrHead));
InString = true;
} else {
Candidate.append(StrHead, Cur);
@@ -537,7 +557,7 @@ void strings(raw_ostream &OS, StringRef FileName, sys::fs::file_t Handle) {
}
if (InString)
- CharSet::endString(OS, MBState);
+ CharSet.endString(OS);
}
} // namespace
@@ -621,13 +641,16 @@ int main(int argc, char **argv) {
#endif
#ifdef _WIN32
// Using LLVM's built-in decoder is necessary for correctness as
- // Strings<CharSet::MultiByte> on WIndows is implemented on top of 16-bit
- // wchar_t which cannot represent all code points.
+ // Strings<CharSet::MultiByteStateless> on WIndows is implemented on top
+ // of 16-bit wchar_t which cannot represent all code points.
if (GetACP() == CP_UTF8)
return strings<CharSet::Utf8>;
#endif
- return strings<CharSet::MultiByte>;
+ if (!mblen(NULL, 0))
+ return strings<CharSet::MultiByteStateless>;
+ else
+ return strings<CharSet::MultiByteStateful>;
case Encoding::Utf8:
return strings<CharSet::Utf8>;
>From 8e435d45978aead5b0cdd0ce5df4b4d1ff941abc Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Thu, 29 Oct 2026 01:09:00 +0000
Subject: [PATCH 25/41] Fix locale handling on macOS
Despite what I had read and assumed, wchar_t on macOS is not always
UTF-32. For some locales it is, for others it isn't. To convert to
UTF-32 reliably, we need to instead go through iconv(). This commit does
that, and adds a test for it.
---
.../tools/llvm-strings/encoding-locale.test | 14 ++++
llvm/tools/llvm-strings/CMakeLists.txt | 4 +
llvm/tools/llvm-strings/llvm-strings.cpp | 75 +++++++++++++++++--
3 files changed, 88 insertions(+), 5 deletions(-)
create mode 100644 llvm/test/tools/llvm-strings/encoding-locale.test
diff --git a/llvm/test/tools/llvm-strings/encoding-locale.test b/llvm/test/tools/llvm-strings/encoding-locale.test
new file mode 100644
index 00000000000000..cb5af1edfd6b7a
--- /dev/null
+++ b/llvm/test/tools/llvm-strings/encoding-locale.test
@@ -0,0 +1,14 @@
+# REQUIRES: system-darwin
+
+RUN: printf 'Hello\342\202\254\202\261\202\361\202\311\202\277\202\315' > %t
+
+RUN: env LANG=en_US.US-ASCII llvm-strings %t | FileCheck --check-prefixes CHECK-US-ASCII %s
+RUN: env LANG=en_US.ISO8859-1 llvm-strings %t | FileCheck --check-prefixes CHECK-ISO8859-1 %s
+RUN: env LANG=en_US.UTF-8 llvm-strings %t | FileCheck --check-prefixes CHECK-UTF-8 %s
+RUN: env LANG=ja_JP.SJIS llvm-strings %t | FileCheck --check-prefixes CHECK-SJIS %s
+
+CHECK-US-ASCII:Hello
+CHECK-ISO8859-1:Hello{{\xe2}}
+CHECK-UTF-8:Hello{{\xe2\x82\xac}}
+CHECK-SJIS:Hello
+CHECK-SJIS:{{\x82\xb1\x82\xf1\x82\xc9\x82\xbf\x82\xcd}}
diff --git a/llvm/tools/llvm-strings/CMakeLists.txt b/llvm/tools/llvm-strings/CMakeLists.txt
index 95dd69c778e477..f1725efd355d49 100644
--- a/llvm/tools/llvm-strings/CMakeLists.txt
+++ b/llvm/tools/llvm-strings/CMakeLists.txt
@@ -15,6 +15,10 @@ add_llvm_tool(llvm-strings
StringsOptsTableGen
)
+if(APPLE)
+ target_link_libraries(llvm-strings PRIVATE iconv)
+endif()
+
if(LLVM_INSTALL_BINUTILS_SYMLINKS)
add_llvm_tool_symlink(strings llvm-strings)
endif()
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 8c26e5c06fb624..7855f13f928e2f 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -42,6 +42,10 @@
#include "llvm/Support/Windows/WindowsSupport.h"
#endif
+#ifdef __APPLE__
+#include <iconv.h>
+#endif
+
using namespace llvm;
using namespace llvm::object;
@@ -108,12 +112,12 @@ static void parseIntArg(const opt::InputArgList &Args, int ID, T &Value) {
// problems on most platforms, thus we introduce wrappers. On Windows,
// mbrtoc32() ignores the locale and only converts UTF-8, but wchar_t is UTF-16
// and is close enough, as long as we make sure to avoid it for UTF-8 locales.
-// On Apple, <uchar.h> is not available, but wchar_t is UTF-32 and is a full
-// substitute.
+// On Apple, <uchar.h> is not available, and wchar_t is not always UTF-32. We
+// implement mbrtoc() and crtomb() through iconv().
// On glibc, mbrtoc32() is a wrapper for mbrtowc() and never returns (size_t)-3,
// instead it handles this case by returning 0 but setting Ch to a non-null
// character.
-#if defined(_WIN32) || defined(__APPLE__)
+#if defined(_WIN32)
using char_t = wchar_t;
static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
std::mbstate_t *MBState) {
@@ -122,9 +126,70 @@ static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
Result = (size_t)-3;
return Result;
}
-static size_t crtomb(char *S, char_t Ch, mbstate_t *MBState) {
+static size_t crtomb(char *S, char_t Ch, std::mbstate_t *MBState) {
return std::wcrtomb(S, Ch, MBState);
}
+#elif defined(__APPLE__)
+using char_t = char32_t;
+static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
+ std::mbstate_t *MBState) {
+ static struct IconvHandle {
+ IconvHandle() : Value(iconv_open("UCS-4-INTERNAL", nl_langinfo(CODESET))) {
+ if (Value == (iconv_t)-1)
+ report_fatal_error("failed to open iconv");
+ }
+ ~IconvHandle() { iconv_close(Value); }
+
+ iconv_t Value;
+ } CD;
+
+ wchar_t WCh;
+ size_t BytesRead = mbrtowc(&WCh, S, N, MBState);
+ if (BytesRead != (size_t)-2 && BytesRead != (size_t)-1) {
+ char Cvt[MB_LEN_MAX];
+ std::mbstate_t CvtMBState{};
+ const size_t BytesWritten = wcrtomb(Cvt, WCh, &CvtMBState);
+ char *PIn = Cvt;
+ size_t NIn = BytesWritten;
+ char *POut = reinterpret_cast<char *>(Ch);
+ size_t NOut = sizeof(char_t);
+ const size_t NumErrors = iconv(CD.Value, &PIn, &NIn, &POut, &NOut);
+ // mbrtowc() may be lenient and accept invalid characters that iconv()
+ // rejects.
+ if (NIn == BytesWritten)
+ return (size_t)-1;
+ if (NumErrors || NIn || NOut)
+ llvm_unreachable("internal error: iconv failed on valid character");
+ }
+ return BytesRead;
+}
+static size_t crtomb(char *S, char_t Ch, std::mbstate_t *MBState) {
+ static struct IconvHandle {
+ IconvHandle() : Value(iconv_open(nl_langinfo(CODESET), "UCS-4-INTERNAL")) {
+ if (Value == (iconv_t)-1)
+ report_fatal_error("failed to open iconv");
+ }
+ ~IconvHandle() { iconv_close(Value); }
+
+ iconv_t Value;
+ } CD;
+
+ char Cvt[MB_LEN_MAX];
+ std::mbstate_t CvtMBState{};
+ char *PIn = reinterpret_cast<char *>(Ch);
+ size_t NIn = sizeof(char_t);
+ char *POut = Cvt;
+ size_t NOut = MB_LEN_MAX;
+ const size_t NumErrors = iconv(CD.Value, &PIn, &NIn, &POut, &NOut);
+ if (NumErrors == (size_t)-1 || NIn)
+ llvm_unreachable("internal error: iconv failed on valid character");
+ if (NumErrors)
+ return (size_t)-1;
+ wchar_t WCh;
+ if (mbrtowc(&WCh, Cvt, PIn - Cvt, &CvtMBState) != size_t(PIn - Cvt))
+ llvm_unreachable("internal error: mbrtowc failed on valid character");
+ return wcrtomb(S, WCh, MBState);
+}
#else
using char_t = char32_t;
static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
@@ -134,7 +199,7 @@ static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
Result = (size_t)-3;
return Result;
}
-static size_t crtomb(char *S, char_t Ch, mbstate_t *MBState) {
+static size_t crtomb(char *S, char_t Ch, std::mbstate_t *MBState) {
return std::c32rtomb(S, Ch, MBState);
}
#endif
>From db9481632558e78865dc32d41f6013922ec49c8b Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Fri, 30 Oct 2026 00:40:12 +0000
Subject: [PATCH 26/41] Further restrict tests, prepare for later update
---
llvm/test/tools/llvm-strings/encoding-locale.test | 14 --------------
llvm/test/tools/llvm-strings/locale.test | 14 ++++++++++++++
2 files changed, 14 insertions(+), 14 deletions(-)
delete mode 100644 llvm/test/tools/llvm-strings/encoding-locale.test
create mode 100644 llvm/test/tools/llvm-strings/locale.test
diff --git a/llvm/test/tools/llvm-strings/encoding-locale.test b/llvm/test/tools/llvm-strings/encoding-locale.test
deleted file mode 100644
index cb5af1edfd6b7a..00000000000000
--- a/llvm/test/tools/llvm-strings/encoding-locale.test
+++ /dev/null
@@ -1,14 +0,0 @@
-# REQUIRES: system-darwin
-
-RUN: printf 'Hello\342\202\254\202\261\202\361\202\311\202\277\202\315' > %t
-
-RUN: env LANG=en_US.US-ASCII llvm-strings %t | FileCheck --check-prefixes CHECK-US-ASCII %s
-RUN: env LANG=en_US.ISO8859-1 llvm-strings %t | FileCheck --check-prefixes CHECK-ISO8859-1 %s
-RUN: env LANG=en_US.UTF-8 llvm-strings %t | FileCheck --check-prefixes CHECK-UTF-8 %s
-RUN: env LANG=ja_JP.SJIS llvm-strings %t | FileCheck --check-prefixes CHECK-SJIS %s
-
-CHECK-US-ASCII:Hello
-CHECK-ISO8859-1:Hello{{\xe2}}
-CHECK-UTF-8:Hello{{\xe2\x82\xac}}
-CHECK-SJIS:Hello
-CHECK-SJIS:{{\x82\xb1\x82\xf1\x82\xc9\x82\xbf\x82\xcd}}
diff --git a/llvm/test/tools/llvm-strings/locale.test b/llvm/test/tools/llvm-strings/locale.test
new file mode 100644
index 00000000000000..3930576e54026c
--- /dev/null
+++ b/llvm/test/tools/llvm-strings/locale.test
@@ -0,0 +1,14 @@
+REQUIRES: system-darwin
+
+RUN: printf 'Hello\360\237\230\203\n\202\261\202\361\202\311\202\277\202\315\n' > %t
+
+RUN: env LANG=en_US.US-ASCII llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-US-ASCII %s
+RUN: env LANG=en_US.ISO8859-1 llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-ISO8859-1 %s
+RUN: env LANG=en_US.UTF-8 llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-UTF-8 %s
+RUN: env LANG=ja_JP.SJIS llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-SJIS %s
+
+CHECK-IN-US-ASCII:Hello
+CHECK-IN-ISO8859-1:Hello{{\xF0}}
+CHECK-IN-UTF-8:Hello{{\xF0\x9F\x98\x83}}
+CHECK-IN-SJIS:Hello
+CHECK-IN-SJIS:{{\x82\xB1\x82\xF1\x82\xC9\x82\xBF\x82\xCD}}
>From 7467f12e481ce57ce7e410fc533757e0ef96d13a Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Fri, 30 Oct 2026 18:33:50 +0000
Subject: [PATCH 27/41] Fix crtomb
---
llvm/tools/llvm-strings/llvm-strings.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 7855f13f928e2f..12be17b88f456a 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -176,7 +176,7 @@ static size_t crtomb(char *S, char_t Ch, std::mbstate_t *MBState) {
char Cvt[MB_LEN_MAX];
std::mbstate_t CvtMBState{};
- char *PIn = reinterpret_cast<char *>(Ch);
+ char *PIn = reinterpret_cast<char *>(&Ch);
size_t NIn = sizeof(char_t);
char *POut = Cvt;
size_t NOut = MB_LEN_MAX;
>From 61b4265815be8cb3da319a15f4654916dff450e4 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Mon, 2 Nov 2026 14:28:32 +0000
Subject: [PATCH 28/41] Apply fix on FreeBSD too and enable tests there
---
llvm/test/tools/llvm-strings/locale.test | 2 +-
llvm/tools/llvm-strings/llvm-strings.cpp | 6 ++++--
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/llvm/test/tools/llvm-strings/locale.test b/llvm/test/tools/llvm-strings/locale.test
index 3930576e54026c..4eac28dc727fd0 100644
--- a/llvm/test/tools/llvm-strings/locale.test
+++ b/llvm/test/tools/llvm-strings/locale.test
@@ -1,4 +1,4 @@
-REQUIRES: system-darwin
+REQUIRES: system-darwin || system-freebsd
RUN: printf 'Hello\360\237\230\203\n\202\261\202\361\202\311\202\277\202\315\n' > %t
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 12be17b88f456a..f7fe6845353e49 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -42,7 +42,7 @@
#include "llvm/Support/Windows/WindowsSupport.h"
#endif
-#ifdef __APPLE__
+#if defined(__APPLE__) || defined(__FreeBSD__)
#include <iconv.h>
#endif
@@ -114,6 +114,8 @@ static void parseIntArg(const opt::InputArgList &Args, int ID, T &Value) {
// and is close enough, as long as we make sure to avoid it for UTF-8 locales.
// On Apple, <uchar.h> is not available, and wchar_t is not always UTF-32. We
// implement mbrtoc() and crtomb() through iconv().
+// On FreeBSD, <uchar.h> is available but assumes wchar_t is always UTF-32, yet
+// on FreeBSD, wchar_t is not always UTF-32, so likewise use iconv().
// On glibc, mbrtoc32() is a wrapper for mbrtowc() and never returns (size_t)-3,
// instead it handles this case by returning 0 but setting Ch to a non-null
// character.
@@ -129,7 +131,7 @@ static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
static size_t crtomb(char *S, char_t Ch, std::mbstate_t *MBState) {
return std::wcrtomb(S, Ch, MBState);
}
-#elif defined(__APPLE__)
+#elif defined(__APPLE__) || defined(__FreeBSD__)
using char_t = char32_t;
static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
std::mbstate_t *MBState) {
>From 7011c5233831383e4b007fcab8aaef241e679664 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Mon, 2 Nov 2026 16:00:19 +0000
Subject: [PATCH 29/41] Correct FreeBSD description
---
llvm/tools/llvm-strings/llvm-strings.cpp | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index f7fe6845353e49..c6ee9c2e68d174 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -114,8 +114,9 @@ static void parseIntArg(const opt::InputArgList &Args, int ID, T &Value) {
// and is close enough, as long as we make sure to avoid it for UTF-8 locales.
// On Apple, <uchar.h> is not available, and wchar_t is not always UTF-32. We
// implement mbrtoc() and crtomb() through iconv().
-// On FreeBSD, <uchar.h> is available but assumes wchar_t is always UTF-32, yet
-// on FreeBSD, wchar_t is not always UTF-32, so likewise use iconv().
+// On FreeBSD, <uchar.h> is available but the result for encoding errors is
+// (size_t)-2 rather than (size_t) which we cannot recover from, so likewise use
+// iconv().
// On glibc, mbrtoc32() is a wrapper for mbrtowc() and never returns (size_t)-3,
// instead it handles this case by returning 0 but setting Ch to a non-null
// character.
>From 1c71494c011c26b99e9b3436eaf1f1e303a60723 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Mon, 2 Nov 2026 18:08:06 +0000
Subject: [PATCH 30/41] Apply review feedback
---
llvm/test/tools/llvm-strings/locale.test | 21 +++++++---
llvm/tools/llvm-strings/llvm-strings.cpp | 53 ++++++++++++------------
2 files changed, 42 insertions(+), 32 deletions(-)
diff --git a/llvm/test/tools/llvm-strings/locale.test b/llvm/test/tools/llvm-strings/locale.test
index 4eac28dc727fd0..f498e4b0a20646 100644
--- a/llvm/test/tools/llvm-strings/locale.test
+++ b/llvm/test/tools/llvm-strings/locale.test
@@ -1,11 +1,22 @@
-REQUIRES: system-darwin || system-freebsd
+UNSUPPORTED: system-windows
RUN: printf 'Hello\360\237\230\203\n\202\261\202\361\202\311\202\277\202\315\n' > %t
-RUN: env LANG=en_US.US-ASCII llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-US-ASCII %s
-RUN: env LANG=en_US.ISO8859-1 llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-ISO8859-1 %s
-RUN: env LANG=en_US.UTF-8 llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-UTF-8 %s
-RUN: env LANG=ja_JP.SJIS llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-SJIS %s
+RUN: %if locale-en_US.US-ASCII %{ env LANG=en_US.US-ASCII \
+RUN: llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-US-ASCII %s %} \
+RUN: %else %{ echo en_US.US-ASCII locale not supported, skipping test %}
+
+RUN: %if locale-en_US.ISO8859-1 %{ env LANG=en_US.ISO8859-1 \
+RUN: llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-ISO8859-1 %s %} \
+RUN: %else %{ echo en_US.ISO8859-1 locale not supported, skipping test %}
+
+RUN: %if locale-en_US.UTF-8 %{ env LANG=en_US.UTF-8 \
+RUN: llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-UTF-8 %s %} \
+RUN: %else %{ echo en_US.UTF-8 locale not supported, skipping test %}
+
+RUN: %if locale-ja_JP.SJIS %{ env LANG=ja_JP.SJIS \
+RUN: llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-SJIS %s %} \
+RUN: %else %{ echo ja_JP.SJIS locale not supported, skipping test %}
CHECK-IN-US-ASCII:Hello
CHECK-IN-ISO8859-1:Hello{{\xF0}}
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index c6ee9c2e68d174..3bcb3609218418 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -115,8 +115,8 @@ static void parseIntArg(const opt::InputArgList &Args, int ID, T &Value) {
// On Apple, <uchar.h> is not available, and wchar_t is not always UTF-32. We
// implement mbrtoc() and crtomb() through iconv().
// On FreeBSD, <uchar.h> is available but the result for encoding errors is
-// (size_t)-2 rather than (size_t) which we cannot recover from, so likewise use
-// iconv().
+// (size_t)-2 rather than (size_t)-1. We cannot recover from this, so likewise
+// use iconv().
// On glibc, mbrtoc32() is a wrapper for mbrtowc() and never returns (size_t)-3,
// instead it handles this case by returning 0 but setting Ch to a non-null
// character.
@@ -133,18 +133,24 @@ static size_t crtomb(char *S, char_t Ch, std::mbstate_t *MBState) {
return std::wcrtomb(S, Ch, MBState);
}
#elif defined(__APPLE__) || defined(__FreeBSD__)
+struct IconvHandle {
+ IconvHandle(const char *DstName, const char *SrcName)
+ : Value(iconv_open(DstName, SrcName)) {
+ if (Value == (iconv_t)-1)
+ reportCmdLineError("failed to open iconv");
+ }
+ IconvHandle(IconvHandle &&) = delete;
+ ~IconvHandle() { iconv_close(Value); }
+ iconv_t value() const { return Value; }
+
+private:
+ iconv_t Value;
+};
+
using char_t = char32_t;
static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
std::mbstate_t *MBState) {
- static struct IconvHandle {
- IconvHandle() : Value(iconv_open("UCS-4-INTERNAL", nl_langinfo(CODESET))) {
- if (Value == (iconv_t)-1)
- report_fatal_error("failed to open iconv");
- }
- ~IconvHandle() { iconv_close(Value); }
-
- iconv_t Value;
- } CD;
+ static IconvHandle CD("UCS-4-INTERNAL", nl_langinfo(CODESET));
wchar_t WCh;
size_t BytesRead = mbrtowc(&WCh, S, N, MBState);
@@ -156,7 +162,7 @@ static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
size_t NIn = BytesWritten;
char *POut = reinterpret_cast<char *>(Ch);
size_t NOut = sizeof(char_t);
- const size_t NumErrors = iconv(CD.Value, &PIn, &NIn, &POut, &NOut);
+ const size_t NumErrors = iconv(CD.value(), &PIn, &NIn, &POut, &NOut);
// mbrtowc() may be lenient and accept invalid characters that iconv()
// rejects.
if (NIn == BytesWritten)
@@ -167,15 +173,7 @@ static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
return BytesRead;
}
static size_t crtomb(char *S, char_t Ch, std::mbstate_t *MBState) {
- static struct IconvHandle {
- IconvHandle() : Value(iconv_open(nl_langinfo(CODESET), "UCS-4-INTERNAL")) {
- if (Value == (iconv_t)-1)
- report_fatal_error("failed to open iconv");
- }
- ~IconvHandle() { iconv_close(Value); }
-
- iconv_t Value;
- } CD;
+ static IconvHandle CD(nl_langinfo(CODESET), "UCS-4-INTERNAL");
char Cvt[MB_LEN_MAX];
std::mbstate_t CvtMBState{};
@@ -183,7 +181,7 @@ static size_t crtomb(char *S, char_t Ch, std::mbstate_t *MBState) {
size_t NIn = sizeof(char_t);
char *POut = Cvt;
size_t NOut = MB_LEN_MAX;
- const size_t NumErrors = iconv(CD.Value, &PIn, &NIn, &POut, &NOut);
+ const size_t NumErrors = iconv(CD.value(), &PIn, &NIn, &POut, &NOut);
if (NumErrors == (size_t)-1 || NIn)
llvm_unreachable("internal error: iconv failed on valid character");
if (NumErrors)
@@ -257,10 +255,11 @@ struct CharSet {
char_t UCh;
for (;;) {
// Read one byte at a time. This is usually not the best way to use
- // mbrtoc32(), usually it would make more sense to pass the size of the
- // buffer, but the strings utility is unusual in that it is expected to
- // encounter many bytes that do not form valid characters and it is more
- // useful to optimise for this case.
+ // mbrtoc32(): usually passing the size of the buffer reduces the
+ // number of times it needs to be called. The strings utility is unusual
+ // in that it is expected to encounter many bytes that do not form valid
+ // characters and to recover from the subsequent decoding errors,
+ // however, so it is more useful to optimise for this case.
const size_t BytesRead = mbrtoc(&UCh, Next, 1, &NextMBState);
switch (BytesRead) {
case 1:
@@ -709,7 +708,7 @@ int main(int argc, char **argv) {
#endif
#ifdef _WIN32
// Using LLVM's built-in decoder is necessary for correctness as
- // Strings<CharSet::MultiByteStateless> on WIndows is implemented on top
+ // Strings<CharSet::MultiByteStateless> on Windows is implemented on top
// of 16-bit wchar_t which cannot represent all code points.
if (GetACP() == CP_UTF8)
return strings<CharSet::Utf8>;
>From b817d09a820a5b37851961a8fe50131108b06f5f Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Mon, 2 Nov 2026 23:30:41 +0000
Subject: [PATCH 31/41] Optimise for single-byte characters
---
llvm/tools/llvm-strings/llvm-strings.cpp | 32 +++++++++++++++++++++++-
1 file changed, 31 insertions(+), 1 deletion(-)
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 3bcb3609218418..efbd201c579053 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -250,9 +250,34 @@ struct CharSet {
if (Cur == End)
return false;
+ static const UTF32(&LookupTable)[0x100] = [&]() -> const UTF32(&)[0x100] {
+ static UTF32 LookupTable[0x100];
+
+ for (int I = 0; I < 0x100; ++I) {
+ const unsigned char Ch = I;
+ const char *Cur = reinterpret_cast<const char *>(&Ch);
+ char_t UCh;
+ std::mbstate_t InitialMBState{};
+ if (mbrtoc(&UCh, Cur, 1, &InitialMBState) == 1)
+ LookupTable[Ch] = UCh;
+ }
+
+ return LookupTable;
+ }();
+
+ char_t UCh;
+
+ if (std::mbsinit(&InMBState)) {
+ UCh = LookupTable[static_cast<unsigned char>(*Cur)];
+ if (UCh || !*Cur) {
+ ++Cur;
+ Ch = UCh;
+ return true;
+ }
+ }
+
const char *Next = Cur;
std::mbstate_t NextMBState = InMBState;
- char_t UCh;
for (;;) {
// Read one byte at a time. This is usually not the best way to use
// mbrtoc32(): usually passing the size of the buffer reduces the
@@ -370,6 +395,11 @@ struct CharSet {
if (Cur == End)
return false;
+ if (static_cast<unsigned char>(*Cur) <= 0x7F) {
+ Ch = *Cur++;
+ return true;
+ }
+
const UTF8 *UTF8Cur = reinterpret_cast<const UTF8 *>(Cur);
const UTF8 *UTF8Next = UTF8Cur;
const UTF8 *UTF8End = reinterpret_cast<const UTF8 *>(End);
>From dc68790c2b5a80690d6a01c5bfc40de5cd4d20a7 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Tue, 3 Nov 2026 12:25:19 +0000
Subject: [PATCH 32/41] Add missing file
---
llvm/test/tools/llvm-strings/lit.local.cfg | 9 +++++++++
1 file changed, 9 insertions(+)
create mode 100644 llvm/test/tools/llvm-strings/lit.local.cfg
diff --git a/llvm/test/tools/llvm-strings/lit.local.cfg b/llvm/test/tools/llvm-strings/lit.local.cfg
new file mode 100644
index 00000000000000..62616b7347ce62
--- /dev/null
+++ b/llvm/test/tools/llvm-strings/lit.local.cfg
@@ -0,0 +1,9 @@
+import locale
+saved = locale.setlocale(locale.LC_ALL)
+for l in ('en_US.US-ASCII', 'en_US.ISO8859-1', 'en_US.UTF-8', 'ja_JP.SJIS'):
+ try:
+ old = locale.setlocale(locale.LC_ALL, l)
+ config.available_features.add('locale-' + l)
+ except locale.Error:
+ pass
+locale.setlocale(locale.LC_ALL, saved)
>From fc3ccccbd90965ba9ad3988a81df699dfd0b8945 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Tue, 3 Nov 2026 23:21:09 +0000
Subject: [PATCH 33/41] Add ISO2022-JP test, remove llvm_unreachable, fix
mbrtowc call
---
llvm/test/tools/llvm-strings/lit.local.cfg | 2 +-
llvm/test/tools/llvm-strings/locale.test | 10 +++++++++-
llvm/tools/llvm-strings/llvm-strings.cpp | 6 ++----
3 files changed, 12 insertions(+), 6 deletions(-)
diff --git a/llvm/test/tools/llvm-strings/lit.local.cfg b/llvm/test/tools/llvm-strings/lit.local.cfg
index 62616b7347ce62..50709f7c8e4005 100644
--- a/llvm/test/tools/llvm-strings/lit.local.cfg
+++ b/llvm/test/tools/llvm-strings/lit.local.cfg
@@ -1,6 +1,6 @@
import locale
saved = locale.setlocale(locale.LC_ALL)
-for l in ('en_US.US-ASCII', 'en_US.ISO8859-1', 'en_US.UTF-8', 'ja_JP.SJIS'):
+for l in ('en_US.US-ASCII', 'en_US.ISO8859-1', 'en_US.UTF-8', 'ja_JP.SJIS', 'ja_JP.ISO2022-JP'):
try:
old = locale.setlocale(locale.LC_ALL, l)
config.available_features.add('locale-' + l)
diff --git a/llvm/test/tools/llvm-strings/locale.test b/llvm/test/tools/llvm-strings/locale.test
index f498e4b0a20646..1439a5c8e9f5da 100644
--- a/llvm/test/tools/llvm-strings/locale.test
+++ b/llvm/test/tools/llvm-strings/locale.test
@@ -1,6 +1,8 @@
UNSUPPORTED: system-windows
-RUN: printf 'Hello\360\237\230\203\n\202\261\202\361\202\311\202\277\202\315\n' > %t
+RUN: printf 'Hello\360\237\230\203\n' > %t
+RUN: printf '\202\261\202\361\202\311\202\277\202\315\n' >> %t
+RUN: printf '\033$B$3$s$K$A$O\033(B\n' >> %t
RUN: %if locale-en_US.US-ASCII %{ env LANG=en_US.US-ASCII \
RUN: llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-US-ASCII %s %} \
@@ -18,8 +20,14 @@ RUN: %if locale-ja_JP.SJIS %{ env LANG=ja_JP.SJIS \
RUN: llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-SJIS %s %} \
RUN: %else %{ echo ja_JP.SJIS locale not supported, skipping test %}
+RUN: %if locale-ja_JP.ISO2022-JP %{ env LANG=ja_JP.ISO2022-JP \
+RUN: llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-ISO2022-JP %s %} \
+RUN: %else %{ echo ja_JP.ISO2022-JP locale not supported, skipping test %}
+
CHECK-IN-US-ASCII:Hello
CHECK-IN-ISO8859-1:Hello{{\xF0}}
CHECK-IN-UTF-8:Hello{{\xF0\x9F\x98\x83}}
CHECK-IN-SJIS:Hello
CHECK-IN-SJIS:{{\x82\xB1\x82\xF1\x82\xC9\x82\xBF\x82\xCD}}
+CHECK-IN-ISO2022-JP:Hello
+CHECK-IN-ISO2022-JP:{{\x1B}}$B$3$s$K$A$O{{\x1B}}(B
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index efbd201c579053..ecd61f27ea5e52 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -165,10 +165,8 @@ static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
const size_t NumErrors = iconv(CD.value(), &PIn, &NIn, &POut, &NOut);
// mbrtowc() may be lenient and accept invalid characters that iconv()
// rejects.
- if (NIn == BytesWritten)
- return (size_t)-1;
if (NumErrors || NIn || NOut)
- llvm_unreachable("internal error: iconv failed on valid character");
+ return (size_t)-1;
}
return BytesRead;
}
@@ -187,7 +185,7 @@ static size_t crtomb(char *S, char_t Ch, std::mbstate_t *MBState) {
if (NumErrors)
return (size_t)-1;
wchar_t WCh;
- if (mbrtowc(&WCh, Cvt, PIn - Cvt, &CvtMBState) != size_t(PIn - Cvt))
+ if (mbrtowc(&WCh, Cvt, POut - Cvt, &CvtMBState) != size_t(POut - Cvt))
llvm_unreachable("internal error: mbrtowc failed on valid character");
return wcrtomb(S, WCh, MBState);
}
>From 2adf6231123506485385aa3d73f86e01ab6e4c7f Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Wed, 4 Nov 2026 14:20:14 +0000
Subject: [PATCH 34/41] Comments, formatting
---
llvm/test/tools/llvm-strings/lit.local.cfg | 19 +++++++++++++------
llvm/tools/llvm-strings/llvm-strings.cpp | 14 ++++++++++++--
2 files changed, 25 insertions(+), 8 deletions(-)
diff --git a/llvm/test/tools/llvm-strings/lit.local.cfg b/llvm/test/tools/llvm-strings/lit.local.cfg
index 50709f7c8e4005..4bbad77f3c7e2d 100644
--- a/llvm/test/tools/llvm-strings/lit.local.cfg
+++ b/llvm/test/tools/llvm-strings/lit.local.cfg
@@ -1,9 +1,16 @@
import locale
+
saved = locale.setlocale(locale.LC_ALL)
-for l in ('en_US.US-ASCII', 'en_US.ISO8859-1', 'en_US.UTF-8', 'ja_JP.SJIS', 'ja_JP.ISO2022-JP'):
- try:
- old = locale.setlocale(locale.LC_ALL, l)
- config.available_features.add('locale-' + l)
- except locale.Error:
- pass
+for l in (
+ "en_US.US-ASCII",
+ "en_US.ISO8859-1",
+ "en_US.UTF-8",
+ "ja_JP.SJIS",
+ "ja_JP.ISO2022-JP",
+):
+ try:
+ old = locale.setlocale(locale.LC_ALL, l)
+ config.available_features.add("locale-" + l)
+ except locale.Error:
+ pass
locale.setlocale(locale.LC_ALL, saved)
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index ecd61f27ea5e52..65f23c3660b037 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -122,6 +122,7 @@ static void parseIntArg(const opt::InputArgList &Args, int ID, T &Value) {
// character.
#if defined(_WIN32)
using char_t = wchar_t;
+
static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
std::mbstate_t *MBState) {
size_t Result = std::mbrtowc(Ch, S, N, MBState);
@@ -129,6 +130,7 @@ static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
Result = (size_t)-3;
return Result;
}
+
static size_t crtomb(char *S, char_t Ch, std::mbstate_t *MBState) {
return std::wcrtomb(S, Ch, MBState);
}
@@ -148,6 +150,7 @@ struct IconvHandle {
};
using char_t = char32_t;
+
static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
std::mbstate_t *MBState) {
static IconvHandle CD("UCS-4-INTERNAL", nl_langinfo(CODESET));
@@ -162,6 +165,10 @@ static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
size_t NIn = BytesWritten;
char *POut = reinterpret_cast<char *>(Ch);
size_t NOut = sizeof(char_t);
+ // This is not generally correct for all systems, but this is limited to
+ // Apple and FreeBSD. Neither supports stateful encodings in its locales, so
+ // we do not need to worry about them in the call to iconv(). Beware if this
+ // needs to be adapted to run on other systems.
const size_t NumErrors = iconv(CD.value(), &PIn, &NIn, &POut, &NOut);
// mbrtowc() may be lenient and accept invalid characters that iconv()
// rejects.
@@ -170,6 +177,7 @@ static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
}
return BytesRead;
}
+
static size_t crtomb(char *S, char_t Ch, std::mbstate_t *MBState) {
static IconvHandle CD(nl_langinfo(CODESET), "UCS-4-INTERNAL");
@@ -191,6 +199,7 @@ static size_t crtomb(char *S, char_t Ch, std::mbstate_t *MBState) {
}
#else
using char_t = char32_t;
+
static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
std::mbstate_t *MBState) {
size_t Result = std::mbrtoc32(Ch, S, N, MBState);
@@ -198,6 +207,7 @@ static size_t mbrtoc(char_t *Ch, const char *S, size_t N,
Result = (size_t)-3;
return Result;
}
+
static size_t crtomb(char *S, char_t Ch, std::mbstate_t *MBState) {
return std::c32rtomb(S, Ch, MBState);
}
@@ -363,8 +373,8 @@ struct CharSet {
const size_t BytesRead = mbrtoc(&UCh, Cur, End - Cur, &StrMBState);
assert(BytesRead != 0 && BytesRead != (size_t)-1 &&
BytesRead != (size_t)-2 &&
- "If a string was initially valid, it cannot to become invalid "
- "when re-reading");
+ "If a string was initially valid, it cannot become invalid when "
+ "re-reading");
if (BytesRead != (size_t)-3)
Cur += BytesRead;
>From 77e017931f0e6d32934a9948906ec3e0184d7b12 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Wed, 4 Nov 2026 15:35:13 +0000
Subject: [PATCH 35/41] Mark Generated*Ranges as LLVM_ABI to allow direct
access
---
clang/lib/Lex/UnicodeCharSetsGenerated.cpp | 15 +++++++++------
llvm/lib/Support/UnicodeCharSetsGenerated.cpp | 14 ++++++++------
llvm/tools/llvm-strings/llvm-strings.cpp | 4 ++--
.../UnicodeData/UnicodeCharSetsGenerator.cpp | 3 ++-
4 files changed, 21 insertions(+), 15 deletions(-)
diff --git a/clang/lib/Lex/UnicodeCharSetsGenerated.cpp b/clang/lib/Lex/UnicodeCharSetsGenerated.cpp
index f1967d496d9da6..b8e06083eb7145 100644
--- a/clang/lib/Lex/UnicodeCharSetsGenerated.cpp
+++ b/clang/lib/Lex/UnicodeCharSetsGenerated.cpp
@@ -255,7 +255,8 @@ static constexpr UnicodeCharRange GeneratedXIDStartRangesData[] = {
{0x2CEB0, 0x2EBE0}, {0x2EBF0, 0x2EE5D}, {0x2F800, 0x2FA1D},
{0x30000, 0x3134A}, {0x31350, 0x33479}, {0x3D000, 0x3FC3F},
};
-extern const CharRanges GeneratedXIDStartRanges = GeneratedXIDStartRangesData;
+LLVM_ABI extern const CharRanges GeneratedXIDStartRanges =
+ GeneratedXIDStartRangesData;
// XID_Continue, excluding XID_Start
static constexpr UnicodeCharRange GeneratedXIDContinueRangesData[] = {
@@ -396,7 +397,7 @@ static constexpr UnicodeCharRange GeneratedXIDContinueRangesData[] = {
{0x1E8D0, 0x1E8D6}, {0x1E944, 0x1E94A}, {0x1E950, 0x1E959},
{0x1FBF0, 0x1FBF9}, {0xE0100, 0xE01EF},
};
-extern const CharRanges GeneratedXIDContinueRanges =
+LLVM_ABI extern const CharRanges GeneratedXIDContinueRanges =
GeneratedXIDContinueRangesData;
// ID_Compat_Math_Start
@@ -408,8 +409,9 @@ static constexpr UnicodeCharRange
{0x1D76F, 0x1D76F}, {0x1D789, 0x1D789}, {0x1D7A9, 0x1D7A9},
{0x1D7C3, 0x1D7C3},
};
-extern const CharRanges GeneratedMathematicalNotationProfileIDStartRanges =
- GeneratedMathematicalNotationProfileIDStartRangesData;
+LLVM_ABI extern const CharRanges
+ GeneratedMathematicalNotationProfileIDStartRanges =
+ GeneratedMathematicalNotationProfileIDStartRangesData;
// ID_Compat_Math_Continue, excluding ID_Compat_Math_Start
static constexpr UnicodeCharRange
@@ -417,7 +419,8 @@ static constexpr UnicodeCharRange
{0x00B2, 0x00B3}, {0x00B9, 0x00B9}, {0x2070, 0x2070},
{0x2074, 0x207E}, {0x2080, 0x208E},
};
-extern const CharRanges GeneratedMathematicalNotationProfileIDContinueRanges =
- GeneratedMathematicalNotationProfileIDContinueRangesData;
+LLVM_ABI extern const CharRanges
+ GeneratedMathematicalNotationProfileIDContinueRanges =
+ GeneratedMathematicalNotationProfileIDContinueRangesData;
} // namespace clang
diff --git a/llvm/lib/Support/UnicodeCharSetsGenerated.cpp b/llvm/lib/Support/UnicodeCharSetsGenerated.cpp
index cba25f21eacb93..51f58b5eda81ab 100644
--- a/llvm/lib/Support/UnicodeCharSetsGenerated.cpp
+++ b/llvm/lib/Support/UnicodeCharSetsGenerated.cpp
@@ -266,7 +266,8 @@ static constexpr UnicodeCharRange GeneratedCharacterRangesData[] = {
{0x3D000, 0x3FC3F}, {0xE0001, 0xE0001}, {0xE0020, 0xE007F},
{0xE0100, 0xE01EF}, {0xF0000, 0xFFFFD}, {0x100000, 0x10FFFD},
};
-extern const CharRanges GeneratedCharacterRanges = GeneratedCharacterRangesData;
+LLVM_ABI extern const CharRanges GeneratedCharacterRanges =
+ GeneratedCharacterRangesData;
// General_Category=L|M|N|P|S|Zs
static constexpr UnicodeCharRange GeneratedPrintableRangesData[] = {
@@ -518,14 +519,15 @@ static constexpr UnicodeCharRange GeneratedPrintableRangesData[] = {
{0x2F800, 0x2FA1D}, {0x30000, 0x3134A}, {0x31350, 0x33479},
{0x3D000, 0x3FC3F}, {0xE0100, 0xE01EF},
};
-extern const CharRanges GeneratedPrintableRanges = GeneratedPrintableRangesData;
+LLVM_ABI extern const CharRanges GeneratedPrintableRanges =
+ GeneratedPrintableRangesData;
// General_Category=Cc
static constexpr UnicodeCharRange GeneratedControlCharacterRangesData[] = {
{0x0000, 0x001F},
{0x007F, 0x009F},
};
-extern const CharRanges GeneratedControlCharacterRanges =
+LLVM_ABI extern const CharRanges GeneratedControlCharacterRanges =
GeneratedControlCharacterRangesData;
// General_Category=Cf
@@ -538,7 +540,7 @@ static constexpr UnicodeCharRange GeneratedFormatCharacterRangesData[] = {
{0x110CD, 0x110CD}, {0x13430, 0x1343F}, {0x1BCA0, 0x1BCA3},
{0x1D173, 0x1D17A}, {0xE0001, 0xE0001}, {0xE0020, 0xE007F},
};
-extern const CharRanges GeneratedFormatCharacterRanges =
+LLVM_ABI extern const CharRanges GeneratedFormatCharacterRanges =
GeneratedFormatCharacterRangesData;
// General_Category=Mn|Me
@@ -666,7 +668,7 @@ static constexpr UnicodeCharRange GeneratedCombiningCharacterRangesData[] = {
{0x1E6EE, 0x1E6EF}, {0x1E6F5, 0x1E6F5}, {0x1E8D0, 0x1E8D6},
{0x1E944, 0x1E94A}, {0xE0100, 0xE01EF},
};
-extern const CharRanges GeneratedCombiningCharacterRanges =
+LLVM_ABI extern const CharRanges GeneratedCombiningCharacterRanges =
GeneratedCombiningCharacterRangesData;
// East_Asian_Width=F|W
@@ -714,7 +716,7 @@ static constexpr UnicodeCharRange GeneratedDoubleWidthCharacterRangesData[] = {
{0x1FAC8, 0x1FAC8}, {0x1FACC, 0x1FADD}, {0x1FADF, 0x1FAEB},
{0x1FAEF, 0x1FAFA}, {0x20000, 0x2FFFD}, {0x30000, 0x3FFFD},
};
-extern const CharRanges GeneratedDoubleWidthCharacterRanges =
+LLVM_ABI extern const CharRanges GeneratedDoubleWidthCharacterRanges =
GeneratedDoubleWidthCharacterRangesData;
} // namespace llvm::sys::unicode
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 65f23c3660b037..75343c98ce40b5 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -50,8 +50,8 @@ using namespace llvm;
using namespace llvm::object;
namespace llvm::sys::unicode {
-extern const UnicodeCharSet::CharRanges GeneratedCharacterRanges;
-extern const UnicodeCharSet::CharRanges GeneratedControlCharacterRanges;
+LLVM_ABI extern const UnicodeCharSet::CharRanges GeneratedCharacterRanges;
+LLVM_ABI extern const UnicodeCharSet::CharRanges GeneratedControlCharacterRanges;
} // namespace llvm::sys::unicode
namespace {
diff --git a/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp b/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp
index 58b305a4fa32ad..6a8c859a739a48 100644
--- a/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp
+++ b/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp
@@ -201,7 +201,8 @@ static void emitTable(raw_ostream &OS, StringRef Name, StringRef Description,
<< "Data[] = {\n";
for (sys::UnicodeCharRange Range : Ranges)
OS << formatv(" {{{0:X4}, {1:X4}},\n", Range.Lower, Range.Upper);
- OS << "};\nextern const CharRanges " << Name << " = " << Name << "Data;\n";
+ OS << "};\nLLVM_ABI extern const CharRanges " << Name << " = " << Name
+ << "Data;\n";
}
static void writeFile(StringRef Path, StringRef Namespace, StringRef Version,
>From 3f86134c0c718268677c5bc525620df6c3515c9d Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Wed, 4 Nov 2026 15:39:22 +0000
Subject: [PATCH 36/41] clang-format a second time
---
llvm/tools/llvm-strings/llvm-strings.cpp | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/llvm/tools/llvm-strings/llvm-strings.cpp b/llvm/tools/llvm-strings/llvm-strings.cpp
index 75343c98ce40b5..95d66b503cde83 100644
--- a/llvm/tools/llvm-strings/llvm-strings.cpp
+++ b/llvm/tools/llvm-strings/llvm-strings.cpp
@@ -51,7 +51,8 @@ using namespace llvm::object;
namespace llvm::sys::unicode {
LLVM_ABI extern const UnicodeCharSet::CharRanges GeneratedCharacterRanges;
-LLVM_ABI extern const UnicodeCharSet::CharRanges GeneratedControlCharacterRanges;
+LLVM_ABI extern const UnicodeCharSet::CharRanges
+ GeneratedControlCharacterRanges;
} // namespace llvm::sys::unicode
namespace {
>From 1513c85fce4f493e1d5cef80c61e5242c7bba6f1 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Wed, 4 Nov 2026 22:14:40 +0000
Subject: [PATCH 37/41] Keep LLVM_ABI out of the Clang files
---
clang/lib/Lex/UnicodeCharSetsGenerated.cpp | 15 ++---
.../UnicodeData/UnicodeCharSetsGenerator.cpp | 67 ++++++++++---------
2 files changed, 43 insertions(+), 39 deletions(-)
diff --git a/clang/lib/Lex/UnicodeCharSetsGenerated.cpp b/clang/lib/Lex/UnicodeCharSetsGenerated.cpp
index b8e06083eb7145..f1967d496d9da6 100644
--- a/clang/lib/Lex/UnicodeCharSetsGenerated.cpp
+++ b/clang/lib/Lex/UnicodeCharSetsGenerated.cpp
@@ -255,8 +255,7 @@ static constexpr UnicodeCharRange GeneratedXIDStartRangesData[] = {
{0x2CEB0, 0x2EBE0}, {0x2EBF0, 0x2EE5D}, {0x2F800, 0x2FA1D},
{0x30000, 0x3134A}, {0x31350, 0x33479}, {0x3D000, 0x3FC3F},
};
-LLVM_ABI extern const CharRanges GeneratedXIDStartRanges =
- GeneratedXIDStartRangesData;
+extern const CharRanges GeneratedXIDStartRanges = GeneratedXIDStartRangesData;
// XID_Continue, excluding XID_Start
static constexpr UnicodeCharRange GeneratedXIDContinueRangesData[] = {
@@ -397,7 +396,7 @@ static constexpr UnicodeCharRange GeneratedXIDContinueRangesData[] = {
{0x1E8D0, 0x1E8D6}, {0x1E944, 0x1E94A}, {0x1E950, 0x1E959},
{0x1FBF0, 0x1FBF9}, {0xE0100, 0xE01EF},
};
-LLVM_ABI extern const CharRanges GeneratedXIDContinueRanges =
+extern const CharRanges GeneratedXIDContinueRanges =
GeneratedXIDContinueRangesData;
// ID_Compat_Math_Start
@@ -409,9 +408,8 @@ static constexpr UnicodeCharRange
{0x1D76F, 0x1D76F}, {0x1D789, 0x1D789}, {0x1D7A9, 0x1D7A9},
{0x1D7C3, 0x1D7C3},
};
-LLVM_ABI extern const CharRanges
- GeneratedMathematicalNotationProfileIDStartRanges =
- GeneratedMathematicalNotationProfileIDStartRangesData;
+extern const CharRanges GeneratedMathematicalNotationProfileIDStartRanges =
+ GeneratedMathematicalNotationProfileIDStartRangesData;
// ID_Compat_Math_Continue, excluding ID_Compat_Math_Start
static constexpr UnicodeCharRange
@@ -419,8 +417,7 @@ static constexpr UnicodeCharRange
{0x00B2, 0x00B3}, {0x00B9, 0x00B9}, {0x2070, 0x2070},
{0x2074, 0x207E}, {0x2080, 0x208E},
};
-LLVM_ABI extern const CharRanges
- GeneratedMathematicalNotationProfileIDContinueRanges =
- GeneratedMathematicalNotationProfileIDContinueRangesData;
+extern const CharRanges GeneratedMathematicalNotationProfileIDContinueRanges =
+ GeneratedMathematicalNotationProfileIDContinueRangesData;
} // namespace clang
diff --git a/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp b/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp
index 6a8c859a739a48..b5777ff06b6df7 100644
--- a/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp
+++ b/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp
@@ -186,7 +186,8 @@ static CharacterDatabase loadDatabase(StringRef Path) {
template <typename Predicate>
static void emitTable(raw_ostream &OS, StringRef Name, StringRef Description,
- ArrayRef<Properties> Chars, Predicate Pred) {
+ ArrayRef<Properties> Chars, Predicate Pred,
+ bool LlvmAbi) {
SmallVector<sys::UnicodeCharRange> Ranges;
for (unsigned CodePoint = 0, E = Chars.size(); CodePoint != E; ++CodePoint) {
if (!Pred(Chars[CodePoint]))
@@ -201,19 +202,25 @@ static void emitTable(raw_ostream &OS, StringRef Name, StringRef Description,
<< "Data[] = {\n";
for (sys::UnicodeCharRange Range : Ranges)
OS << formatv(" {{{0:X4}, {1:X4}},\n", Range.Lower, Range.Upper);
- OS << "};\nLLVM_ABI extern const CharRanges " << Name << " = " << Name
- << "Data;\n";
+ OS << "};\n";
+ if (LlvmAbi)
+ OS << "LLVM_ABI ";
+ OS << "extern const CharRanges " << Name << " = " << Name << "Data;\n";
}
+template <typename EmitTablesT>
static void writeFile(StringRef Path, StringRef Namespace, StringRef Version,
- function_ref<void(raw_ostream &)> EmitTables) {
+ ArrayRef<Properties> Chars, bool LlvmAbi,
+ EmitTablesT &&EmitTables) {
std::error_code EC;
ToolOutputFile Out(Path, EC, sys::fs::OF_Text);
if (EC)
error(Path + ": " + EC.message());
Out.os() << formatv(FileHeader.drop_front().data(), Version, Namespace);
- EmitTables(Out.os());
+ EmitTables([&](StringRef Name, StringRef Description, auto &&Pred) {
+ emitTable(Out.os(), Name, Description, Chars, Pred, LlvmAbi);
+ });
Out.os() << "\n} // namespace " << Namespace << '\n';
Out.keep();
}
@@ -232,35 +239,35 @@ int main(int argc, char **argv) {
CharacterDatabase UCD = loadDatabase(UCDFile);
ArrayRef<Properties> Chars = UCD.Chars;
- writeFile(ClangOutput, "clang", UCD.Version, [&](raw_ostream &OS) {
- emitTable(OS, "GeneratedXIDStartRanges", "XID_Start", Chars,
- [](Properties P) { return P.XIDStart; });
- emitTable(OS, "GeneratedXIDContinueRanges",
- "XID_Continue, excluding XID_Start", Chars,
- [](Properties P) { return P.XIDContinue && !P.XIDStart; });
- emitTable(OS, "GeneratedMathematicalNotationProfileIDStartRanges",
- "ID_Compat_Math_Start", Chars,
- [](Properties P) { return P.MathStart; });
- emitTable(OS, "GeneratedMathematicalNotationProfileIDContinueRanges",
- "ID_Compat_Math_Continue, excluding ID_Compat_Math_Start", Chars,
- [](Properties P) { return P.MathContinue && !P.MathStart; });
- });
writeFile(
- LLVMOutput, "llvm::sys::unicode", UCD.Version, [&](raw_ostream &OS) {
- emitTable(OS, "GeneratedCharacterRanges", "Characters", Chars,
+ ClangOutput, "clang", UCD.Version, Chars, /*LlvmAbi=*/false,
+ [&](auto &&EmitTable) {
+ EmitTable("GeneratedXIDStartRanges", "XID_Start",
+ [](Properties P) { return P.XIDStart; });
+ EmitTable("GeneratedXIDContinueRanges",
+ "XID_Continue, excluding XID_Start",
+ [](Properties P) { return P.XIDContinue && !P.XIDStart; });
+ EmitTable("GeneratedMathematicalNotationProfileIDStartRanges",
+ "ID_Compat_Math_Start",
+ [](Properties P) { return P.MathStart; });
+ EmitTable("GeneratedMathematicalNotationProfileIDContinueRanges",
+ "ID_Compat_Math_Continue, excluding ID_Compat_Math_Start",
+ [](Properties P) { return P.MathContinue && !P.MathStart; });
+ });
+ writeFile(
+ LLVMOutput, "llvm::sys::unicode", UCD.Version, Chars, /*LlvmAbi=*/true,
+ [&](auto &&EmitTable) {
+ EmitTable("GeneratedCharacterRanges", "Characters",
[](Properties P) { return P.Char; });
- emitTable(OS, "GeneratedPrintableRanges",
- "General_Category=L|M|N|P|S|Zs", Chars,
+ EmitTable("GeneratedPrintableRanges", "General_Category=L|M|N|P|S|Zs",
[](Properties P) { return P.Printable; });
- emitTable(OS, "GeneratedControlCharacterRanges", "General_Category=Cc",
- Chars, [](Properties P) { return P.Control; });
- emitTable(OS, "GeneratedFormatCharacterRanges", "General_Category=Cf",
- Chars, [](Properties P) { return P.Format; });
- emitTable(OS, "GeneratedCombiningCharacterRanges",
- "General_Category=Mn|Me", Chars,
+ EmitTable("GeneratedControlCharacterRanges", "General_Category=Cc",
+ [](Properties P) { return P.Control; });
+ EmitTable("GeneratedFormatCharacterRanges", "General_Category=Cf",
+ [](Properties P) { return P.Format; });
+ EmitTable("GeneratedCombiningCharacterRanges", "General_Category=Mn|Me",
[](Properties P) { return P.Combining; });
- emitTable(OS, "GeneratedDoubleWidthCharacterRanges",
- "East_Asian_Width=F|W", Chars,
+ EmitTable("GeneratedDoubleWidthCharacterRanges", "East_Asian_Width=F|W",
[](Properties P) { return P.DoubleWidth; });
});
}
>From 75f5c4bb461721d4d459a394ac6a1084e83a74e3 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Thu, 5 Nov 2026 20:34:34 +0000
Subject: [PATCH 38/41] More echo
---
llvm/test/tools/llvm-strings/negative-char.test | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/llvm/test/tools/llvm-strings/negative-char.test b/llvm/test/tools/llvm-strings/negative-char.test
index d738f2fcb0b999..71f9a860ca9e41 100644
--- a/llvm/test/tools/llvm-strings/negative-char.test
+++ b/llvm/test/tools/llvm-strings/negative-char.test
@@ -2,8 +2,9 @@
## Such characters should form string delimiters like other unprintable ones,
## unless using an encoding where they form valid characters.
-# RUN: printf "z\0\200\0\302\241\0a\0" | llvm-strings --encoding=s --bytes 1 - | FileCheck %s
-# RUN: printf "z\0\200\0\302\241\0a\0" | llvm-strings --encoding=utf8 --bytes 1 - | FileCheck %s --check-prefixes=CHECK,CHECK-UTF8
+# RUN: echo -n -e "z\0\200\0\302\241\0a\0" > %t
+# RUN: llvm-strings --encoding=s --bytes 1 %t | FileCheck %s
+# RUN: llvm-strings --encoding=utf8 --bytes 1 %t | FileCheck %s --check-prefixes=CHECK,CHECK-UTF8
# CHECK: z{{$}}
# CHECK-UTF8-NEXT: {{^\xc2\xa1$}}
# CHECK-NEXT: {{^}}a
>From e145f17c41b7a09b4cbabb34d31c167a9feb0dd3 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Thu, 5 Nov 2026 21:36:30 +0000
Subject: [PATCH 39/41] Even more echo
---
llvm/test/tools/llvm-strings/length.test | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/llvm/test/tools/llvm-strings/length.test b/llvm/test/tools/llvm-strings/length.test
index 79b7a324df6e9a..f17a304b89a0fc 100644
--- a/llvm/test/tools/llvm-strings/length.test
+++ b/llvm/test/tools/llvm-strings/length.test
@@ -46,9 +46,9 @@ RUN: not llvm-strings -n foo %t 2>&1 | FileCheck %s --check-prefix=ERR
ERR: error: expected a positive integer, but got 'foo'
## Show that the count is number of characters, not number of bytes.
-RUN: printf '\302\241\n' > %t
-RUN: printf '\302\241\302\242\n' >> %t
-RUN: printf '\302\241\302\242\302\243\n' >> %t
+RUN: echo -e '\302\241' > %t
+RUN: echo -e '\302\241\302\242' >> %t
+RUN: echo -e '\302\241\302\242\302\243' >> %t
RUN: not llvm-strings --encoding utf8 -n 0 2>&1 %t | FileCheck --check-prefix CHECK-0 %s
RUN: llvm-strings --encoding utf8 -n 1 %t | FileCheck --check-prefix UTF-8-1 %s --implicit-check-not={{.}}
RUN: llvm-strings --encoding utf8 -n 2 %t | FileCheck --check-prefix UTF-8-2 %s --implicit-check-not={{.}}
>From 74929ae079ca51b6f34acc298850f90c4cc258af Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Fri, 6 Nov 2026 12:05:34 +0000
Subject: [PATCH 40/41] Add AIX name for SJIS to test too
---
llvm/test/tools/llvm-strings/lit.local.cfg | 1 +
llvm/test/tools/llvm-strings/locale.test | 4 ++++
2 files changed, 5 insertions(+)
diff --git a/llvm/test/tools/llvm-strings/lit.local.cfg b/llvm/test/tools/llvm-strings/lit.local.cfg
index 4bbad77f3c7e2d..2f63209db7bb6b 100644
--- a/llvm/test/tools/llvm-strings/lit.local.cfg
+++ b/llvm/test/tools/llvm-strings/lit.local.cfg
@@ -6,6 +6,7 @@ for l in (
"en_US.ISO8859-1",
"en_US.UTF-8",
"ja_JP.SJIS",
+ "Ja_JP.IBM-943",
"ja_JP.ISO2022-JP",
):
try:
diff --git a/llvm/test/tools/llvm-strings/locale.test b/llvm/test/tools/llvm-strings/locale.test
index cc75b3d470abde..2f8c92857d3e60 100644
--- a/llvm/test/tools/llvm-strings/locale.test
+++ b/llvm/test/tools/llvm-strings/locale.test
@@ -20,6 +20,10 @@ RUN: %if locale-ja_JP.SJIS %{ env LANG=ja_JP.SJIS \
RUN: llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-SJIS %s %} \
RUN: %else %{ echo ja_JP.SJIS locale not supported, skipping test %}
+RUN: %if locale-Ja_JP.IBM-943 %{ env LANG=Ja_JP.IBM-943 \
+RUN: llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-SJIS %s %} \
+RUN: %else %{ echo Ja_JP.IBM-943 locale not supported, skipping test %}
+
RUN: %if locale-ja_JP.ISO2022-JP %{ env LANG=ja_JP.ISO2022-JP \
RUN: llvm-strings -eS %t | FileCheck --match-full-lines --strict-whitespace --check-prefixes CHECK-IN-ISO2022-JP %s %} \
RUN: %else %{ echo ja_JP.ISO2022-JP locale not supported, skipping test %}
>From b2b5c02ba7b094859e8c95d8ae4ba5dcc05200a9 Mon Sep 17 00:00:00 2001
From: Harald van Dijk <hdijk at accesssoftek.com>
Date: Fri, 6 Nov 2026 12:48:17 +0000
Subject: [PATCH 41/41] Add comment explaining IBM-943
---
llvm/test/tools/llvm-strings/lit.local.cfg | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/llvm/test/tools/llvm-strings/lit.local.cfg b/llvm/test/tools/llvm-strings/lit.local.cfg
index 2f63209db7bb6b..c37e131b9c4edc 100644
--- a/llvm/test/tools/llvm-strings/lit.local.cfg
+++ b/llvm/test/tools/llvm-strings/lit.local.cfg
@@ -6,7 +6,7 @@ for l in (
"en_US.ISO8859-1",
"en_US.UTF-8",
"ja_JP.SJIS",
- "Ja_JP.IBM-943",
+ "Ja_JP.IBM-943", # AIX name for SJIS
"ja_JP.ISO2022-JP",
):
try:
More information about the cfe-commits
mailing list