[clang] [llvm] [llvm-strings] Add --encoding={s, S, utf8} option (PR #221794)

Harald van Dijk via cfe-commits cfe-commits at lists.llvm.org
Thu Nov 5 12:49:34 PST 2026


================
@@ -0,0 +1,97 @@
+# Show that llvm-strings uses the specified encoding.
+RUN: echo a > %t
+RUN: echo ab >> %t
+RUN: echo abc >> %t
+RUN: echo abcd >> %t
+# UTF-8 encoding of U+00A0 NON-BREAKING SPACE (which we treat as printable)
+RUN: echo -e 'bcde\302\240' >> %t
+# UTF-8 encoding of U+009F APPLICATION PROGRAM COMMAND (non-printable)
+RUN: echo -e 'cdef\302\240\302\237' >> %t
+
+# Check that partial and invalid UTF-8 characters do not interfere with
+# preceding and subsequent strings.
+RUN: echo -e '\342defg\342' >> %t
+RUN: echo -e '\342\200efgh\342\200' >> %t
+
+# Check that long form options work.
+RUN: llvm-strings --encoding s %t | FileCheck --check-prefixes CHECK,CHECK-ASCII --strict-whitespace --match-full-lines %s
+RUN: llvm-strings --encoding S %t | FileCheck --check-prefixes CHECK,CHECK-LOCALE --strict-whitespace --match-full-lines %s
+RUN: llvm-strings --encoding utf8 %t | FileCheck --check-prefixes CHECK,CHECK-UTF8 --strict-whitespace --match-full-lines %s
+
+# Check that short form options work.
+RUN: llvm-strings -e s %t | FileCheck --check-prefixes CHECK,CHECK-ASCII --strict-whitespace --match-full-lines %s
+RUN: llvm-strings -e S %t | FileCheck --check-prefixes CHECK,CHECK-LOCALE --strict-whitespace --match-full-lines %s
+RUN: llvm-strings -e utf8 %t | FileCheck --check-prefixes CHECK,CHECK-UTF8 --strict-whitespace --match-full-lines %s
+
+# Check that the output of no options, -e S, and --encoding S all match.
+RUN: llvm-strings -e S %t > %t.1
+RUN: llvm-strings --encoding S %t > %t.2
+RUN: llvm-strings %t > %t.3
+RUN: cmp %t.1 %t.2
+RUN: cmp %t.1 %t.3
+
+# The first line containing just abcd should always be matched.
+CHECK:abcd
+
+# In -e s, non-ASCII bytes should not be included.
+CHECK-ASCII:bcde
+CHECK-ASCII:cdef
+CHECK-ASCII:defg
+CHECK-ASCII:efgh
+
+# In -e S, any bytes of U+00A0 and U+009F may or may not be included depending
+# on the locale in effect. The locale in effect may also be altered by llvm-lit.
+# Make sure if changing the test that the byte representations should not allow
+# this to be split into multiple strings in non-UTF-8 locales.
+CHECK-LOCALE:bcde{{(\xc2\xa0?)?}}
+CHECK-LOCALE:cdef{{(\xc2(\xa0(\xc2\x9f?)?)?)?}}
+CHECK-LOCALE:{{\xe2?}}defg{{\xe2?}}
+CHECK-LOCALE:{{(\xe2?\x80)?}}efgh{{(\xe2\x80?)?}}
+
+# In -e utf8, all bytes of U+00A0 should be included, but none of U+009F. The
+# partial multibyte characters should not prevent abcd from being detected as a
+# string.
+CHECK-UTF8:bcde{{\xc2\xa0}}
+CHECK-UTF8:cdef{{\xc2\xa0}}
+CHECK-UTF8:defg
+CHECK-UTF8:efgh
+
+# Check that invalid --encoding values are rejected.
+RUN: not llvm-strings --encoding x 2>&1 %t | FileCheck --check-prefix CHECK-ERROR %s
+RUN: not llvm-strings -e x 2>&1 %t | FileCheck --check-prefix CHECK-ERROR %s
+
+CHECK-ERROR: error: 'x' is not a valid value for '--encoding='
+
+# llvm-strings reads the input in chunks of 16384 bytes. Encodings with
+# multibyte characters may have characters cross chunk boundaries. The string
+# contains a full multibyte character before the boundary, a multibyte
+# character that crosses the boundary, and a full multibyte character after the
+# boundary. This test has four variants: the minimum length may or may not
+# already be reached before the chunk boundary, and the character that crosses
+# the boundary may or may not be printable.
+#
+RUN: printf "%16380s" | tr " " "\0" > %t
+#                             v chunk boundary
+RUN: echo -n -e "\302\240e\302\240e\302\240e\302\240" >> %t
----------------
hvdijk wrote:

Yes, I'm aware that this is not portable across external shells, this is relying on lit's built-in `echo` to avoid the problem @perry-ca had pointed out in 233878. `printf` is not available as a lit built-in and would use the host shell's version, which would break on z/OS.

https://github.com/llvm/llvm-project/pull/221794


More information about the cfe-commits mailing list