[clang] [llvm] Add a generator for Unicode properties. (PR #229072)
Corentin Jabot via cfe-commits
cfe-commits at lists.llvm.org
Wed Oct 7 03:53:59 PDT 2026
https://github.com/cor3ntin updated https://github.com/llvm/llvm-project/pull/229072
>From 46bdf639aaee2063b975acd15cbea3220373634e Mon Sep 17 00:00:00 2001
From: Corentin Jabot <corentinjabot at gmail.com>
Date: Mon, 5 Oct 2026 14:33:08 +0200
Subject: [PATCH 1/5] Add a generator for Unicode properties.
Over the past few years, I have been maintaining the
unicode tables mostly by hand with the help of an wnweidly
python script rthat only existed on my machine, and sed.
So I figured throwing an AI at the issue would be an improvement
over the status quo.
It took a lot of iteration and refinement but this is very much
AI-produced code.
This introduce a C++ program that parses the UCD data (in the xml format
which i find easier to read) and generate the tables.
I added some documentation.
As it stands, we have 3 different tools to update Unicode data,
and some that still needs manual work.
I plan to consolidate that a bit in future PRs, whenever I have the time.
I'll also need to update other generated tables subsequently as
the data changed between Unicode 18 beta and Unicode 18.
---
clang/docs/UnicodeSupport.md | 83 ++++
clang/docs/index.md | 2 +
clang/lib/Lex/CMakeLists.txt | 1 +
clang/lib/Lex/Lexer.cpp | 21 +-
clang/lib/Lex/UnicodeCharSets.h | 418 +---------------
clang/lib/Lex/UnicodeCharSetsGenerated.cpp | 423 ++++++++++++++++
llvm/lib/Support/CMakeLists.txt | 1 +
llvm/lib/Support/Unicode.cpp | 456 +----------------
llvm/lib/Support/UnicodeCharSetsGenerated.cpp | 462 ++++++++++++++++++
llvm/utils/UnicodeData/CMakeLists.txt | 16 +
.../UnicodeData/UnicodeCharSetsGenerator.cpp | 257 ++++++++++
11 files changed, 1276 insertions(+), 864 deletions(-)
create mode 100644 clang/docs/UnicodeSupport.md
create mode 100644 clang/lib/Lex/UnicodeCharSetsGenerated.cpp
create mode 100644 llvm/lib/Support/UnicodeCharSetsGenerated.cpp
create mode 100644 llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp
diff --git a/clang/docs/UnicodeSupport.md b/clang/docs/UnicodeSupport.md
new file mode 100644
index 0000000000000..7aae8ce73cb6b
--- /dev/null
+++ b/clang/docs/UnicodeSupport.md
@@ -0,0 +1,83 @@
+# Unicode Support
+
+Clang uses a subset of the [Unicode Character Database](https://www.unicode.org/ucd/)
+(UCD) for identifiers, `\N{...}` named characters, diagnostic column width,
+and simple case folding. The corresponding tables are generated from the UCD
+into Clang and LLVM Support.
+
+## Updating the Unicode Data
+
+Replace `N.M.0` with the Unicode version being imported. Build the generators
+from an LLVM build directory that has utilities enabled (`LLVM_BUILD_UTILS`,
+the default) and libxml2 (`LLVM_ENABLE_LIBXML2`, also the default):
+
+```bash
+ninja -C <build> UnicodeCharSetsGenerator UnicodeNameMappingGenerator
+```
+
+Then run them from the `llvm-project` root as shown below.
+
+### Character properties
+
+`UnicodeCharSetsGenerator` writes the identifier character sets used by the
+lexer (`XID_Start`, `XID_Continue`, and the mathematical compatibility notation
+profile) and the printable / formatting / combining / East-Asian-width sets
+used by LLVM Support.
+
+Download the [flat XML UCD](https://www.unicode.org/Public/UCD/latest/ucdxml/)
+(`ucd.nounihan.flat.zip` is enough):
+
+```text
+https://www.unicode.org/Public/N.M.0/ucdxml/ucd.nounihan.flat.zip
+```
+
+```bash
+<build>/bin/UnicodeCharSetsGenerator ucd.nounihan.flat.xml \
+ clang/lib/Lex/UnicodeCharSetsGenerated.cpp \
+ llvm/lib/Support/UnicodeCharSetsGenerated.cpp
+clang-format -i clang/lib/Lex/UnicodeCharSetsGenerated.cpp \
+ llvm/lib/Support/UnicodeCharSetsGenerated.cpp
+```
+
+The C99 and C11 identifier tables and the whitespace table in
+`clang/lib/Lex/UnicodeCharSets.h` are maintained by hand.
+
+### Character names
+
+`UnicodeNameMappingGenerator` writes the name-to-codepoint trie used by
+`\N{...}` and `llvm::sys::unicode::nameToCodepointStrict` /
+`nameToCodepointLooseMatching`.
+
+From the [UCD zip](https://www.unicode.org/Public/UCD/latest/ucd/UCD.zip), take
+`UnicodeData.txt` and `NameAliases.txt`:
+
+```text
+https://www.unicode.org/Public/N.M.0/ucd/UnicodeData.txt
+https://www.unicode.org/Public/N.M.0/ucd/NameAliases.txt
+```
+
+```bash
+<build>/bin/UnicodeNameMappingGenerator UnicodeData.txt NameAliases.txt \
+ llvm/lib/Support/UnicodeNameToCodepointGenerated.cpp
+clang-format -i llvm/lib/Support/UnicodeNameToCodepointGenerated.cpp
+```
+
+Algorithmically derived names (`CJK UNIFIED IDEOGRAPH-*`, and so on) are not
+in those files. Update `GeneratedNamesDataTable` in
+`llvm/lib/Support/UnicodeNameToCodepoint.cpp` from
+`extracted/DerivedName.txt` in the same UCD zip.
+
+### Case folding
+
+`llvm/utils/unicode-case-fold.py` fetches `CaseFolding.txt` and writes
+`llvm/lib/Support/UnicodeCaseFold.cpp`:
+
+```bash
+llvm/utils/unicode-case-fold.py \
+ https://www.unicode.org/Public/N.M.0/ucd/CaseFolding.txt \
+ > llvm/lib/Support/UnicodeCaseFold.cpp
+```
+
+After regenerating the tables, update
+`llvm/unittests/Support/UnicodeTest.cpp` and `clang/test/Lexer/unicode.c` for
+new characters and changed name ranges.
diff --git a/clang/docs/index.md b/clang/docs/index.md
index a05210b8e73a0..9c353317fd260 100644
--- a/clang/docs/index.md
+++ b/clang/docs/index.md
@@ -93,6 +93,7 @@ ClangSYCLLinker
Maintainers
InternalsManual
DriverInternals
+UnicodeSupport
Multilib
OffloadingDesign
PCHInternals
@@ -200,6 +201,7 @@ LLVMExceptionHandlingCodeGen
* {doc}`Maintainers`
* {doc}`InternalsManual`
* {doc}`DriverInternals`
+* {doc}`UnicodeSupport`
* {doc}`Multilib`
* {doc}`OffloadingDesign`
* {doc}`PCHInternals`
diff --git a/clang/lib/Lex/CMakeLists.txt b/clang/lib/Lex/CMakeLists.txt
index 7b0be7249cd99..f333605668abc 100644
--- a/clang/lib/Lex/CMakeLists.txt
+++ b/clang/lib/Lex/CMakeLists.txt
@@ -32,6 +32,7 @@ add_clang_library(clangLex
TextEncoding.cpp
TokenConcatenation.cpp
TokenLexer.cpp
+ UnicodeCharSetsGenerated.cpp
LINK_LIBS
clangBasic
diff --git a/clang/lib/Lex/Lexer.cpp b/clang/lib/Lex/Lexer.cpp
index 4762b38e67c00..367c7e2753d07 100644
--- a/clang/lib/Lex/Lexer.cpp
+++ b/clang/lib/Lex/Lexer.cpp
@@ -1614,9 +1614,9 @@ static bool isUnicodeWhitespace(uint32_t Codepoint) {
static bool isMathematicalExtensionID(uint32_t C, const LangOptions &LangOpts,
bool IsStart, bool &IsExtension) {
static const llvm::sys::UnicodeCharSet MathStartChars(
- MathematicalNotationProfileIDStartRanges);
+ GeneratedMathematicalNotationProfileIDStartRanges);
static const llvm::sys::UnicodeCharSet MathContinueChars(
- MathematicalNotationProfileIDContinueRanges);
+ GeneratedMathematicalNotationProfileIDContinueRanges);
if (MathStartChars.contains(C) ||
(!IsStart && MathContinueChars.contains(C))) {
IsExtension = true;
@@ -1633,11 +1633,13 @@ static bool isAllowedIDChar(uint32_t C, const LangOptions &LangOpts,
return true;
} else if (LangOpts.CPlusPlus || LangOpts.C23) {
// A non-leading codepoint must have the XID_Continue property.
- // XIDContinueRanges doesn't contains characters also in XIDStartRanges,
- // so we need to check both tables.
+ // GeneratedXIDContinueRanges doesn't contain characters also in
+ // GeneratedXIDStartRanges, so we need to check both tables.
// '_' doesn't have the XID_Continue property but is allowed in C and C++.
- static const llvm::sys::UnicodeCharSet XIDStartChars(XIDStartRanges);
- static const llvm::sys::UnicodeCharSet XIDContinueChars(XIDContinueRanges);
+ static const llvm::sys::UnicodeCharSet XIDStartChars(
+ GeneratedXIDStartRanges);
+ static const llvm::sys::UnicodeCharSet XIDContinueChars(
+ GeneratedXIDContinueRanges);
if (C == '_' || XIDStartChars.contains(C) || XIDContinueChars.contains(C))
return true;
return isMathematicalExtensionID(C, LangOpts, /*IsStart=*/false,
@@ -1661,7 +1663,8 @@ static bool isAllowedInitiallyIDChar(uint32_t C, const LangOptions &LangOpts,
return false;
}
if (LangOpts.CPlusPlus || LangOpts.C23) {
- static const llvm::sys::UnicodeCharSet XIDStartChars(XIDStartRanges);
+ static const llvm::sys::UnicodeCharSet XIDStartChars(
+ GeneratedXIDStartRanges);
if (XIDStartChars.contains(C))
return true;
return isMathematicalExtensionID(C, LangOpts, /*IsStart=*/true,
@@ -1685,9 +1688,9 @@ diagnoseMathematicalNotationInIdentifier(DiagnosticsEngine &Diags,
uint32_t C, CharSourceRange Range) {
static const llvm::sys::UnicodeCharSet MathStartChars(
- MathematicalNotationProfileIDStartRanges);
+ GeneratedMathematicalNotationProfileIDStartRanges);
static const llvm::sys::UnicodeCharSet MathContinueChars(
- MathematicalNotationProfileIDContinueRanges);
+ GeneratedMathematicalNotationProfileIDContinueRanges);
(void)MathStartChars;
(void)MathContinueChars;
diff --git a/clang/lib/Lex/UnicodeCharSets.h b/clang/lib/Lex/UnicodeCharSets.h
index eaee53172050f..186c983e2f6a5 100644
--- a/clang/lib/Lex/UnicodeCharSets.h
+++ b/clang/lib/Lex/UnicodeCharSets.h
@@ -10,416 +10,14 @@
#include "llvm/Support/UnicodeCharRanges.h"
-// Unicode 18.0 XID_Start
-static const llvm::sys::UnicodeCharRange XIDStartRanges[] = {
- {0x0041, 0x005A}, {0x0061, 0x007A}, {0x00AA, 0x00AA},
- {0x00B5, 0x00B5}, {0x00BA, 0x00BA}, {0x00C0, 0x00D6},
- {0x00D8, 0x00F6}, {0x00F8, 0x02C1}, {0x02C6, 0x02D1},
- {0x02E0, 0x02E4}, {0x02EC, 0x02EC}, {0x02EE, 0x02EE},
- {0x0370, 0x0374}, {0x0376, 0x0377}, {0x037B, 0x037D},
- {0x037F, 0x037F}, {0x0386, 0x0386}, {0x0388, 0x038A},
- {0x038C, 0x038C}, {0x038E, 0x03A1}, {0x03A3, 0x03F5},
- {0x03F7, 0x0481}, {0x048A, 0x052F}, {0x0531, 0x0556},
- {0x0558, 0x0559}, {0x0560, 0x0588}, {0x058B, 0x058C},
- {0x05D0, 0x05EA}, {0x05EF, 0x05F2}, {0x0620, 0x064A},
- {0x066E, 0x066F}, {0x0671, 0x06D3}, {0x06D5, 0x06D5},
- {0x06E5, 0x06E6}, {0x06EE, 0x06EF}, {0x06FA, 0x06FC},
- {0x06FF, 0x06FF}, {0x0710, 0x0710}, {0x0712, 0x072F},
- {0x074D, 0x07A5}, {0x07B1, 0x07B1}, {0x07CA, 0x07EA},
- {0x07F4, 0x07F5}, {0x07FA, 0x07FA}, {0x0800, 0x0815},
- {0x081A, 0x081A}, {0x0824, 0x0824}, {0x0828, 0x0828},
- {0x0840, 0x0858}, {0x0860, 0x086A}, {0x0870, 0x0887},
- {0x0889, 0x088F}, {0x08A0, 0x08C9}, {0x0904, 0x0939},
- {0x093D, 0x093D}, {0x0950, 0x0950}, {0x0958, 0x0961},
- {0x0971, 0x0980}, {0x0985, 0x098C}, {0x098F, 0x0990},
- {0x0993, 0x09A8}, {0x09AA, 0x09B0}, {0x09B2, 0x09B2},
- {0x09B6, 0x09B9}, {0x09BD, 0x09BD}, {0x09CE, 0x09CE},
- {0x09DC, 0x09DD}, {0x09DF, 0x09E1}, {0x09F0, 0x09F1},
- {0x09FC, 0x09FC}, {0x0A05, 0x0A0A}, {0x0A0F, 0x0A10},
- {0x0A13, 0x0A28}, {0x0A2A, 0x0A30}, {0x0A32, 0x0A33},
- {0x0A35, 0x0A36}, {0x0A38, 0x0A39}, {0x0A59, 0x0A5C},
- {0x0A5E, 0x0A5E}, {0x0A72, 0x0A74}, {0x0A85, 0x0A8D},
- {0x0A8F, 0x0A91}, {0x0A93, 0x0AA8}, {0x0AAA, 0x0AB0},
- {0x0AB2, 0x0AB3}, {0x0AB5, 0x0AB9}, {0x0ABD, 0x0ABD},
- {0x0AD0, 0x0AD0}, {0x0AE0, 0x0AE1}, {0x0AF9, 0x0AF9},
- {0x0B05, 0x0B0C}, {0x0B0F, 0x0B10}, {0x0B13, 0x0B28},
- {0x0B2A, 0x0B30}, {0x0B32, 0x0B33}, {0x0B35, 0x0B39},
- {0x0B3D, 0x0B3D}, {0x0B5C, 0x0B5D}, {0x0B5F, 0x0B61},
- {0x0B71, 0x0B71}, {0x0B83, 0x0B83}, {0x0B85, 0x0B8A},
- {0x0B8E, 0x0B90}, {0x0B92, 0x0B95}, {0x0B99, 0x0B9A},
- {0x0B9C, 0x0B9C}, {0x0B9E, 0x0B9F}, {0x0BA3, 0x0BA4},
- {0x0BA8, 0x0BAA}, {0x0BAE, 0x0BB9}, {0x0BD0, 0x0BD0},
- {0x0C05, 0x0C0C}, {0x0C0E, 0x0C10}, {0x0C12, 0x0C28},
- {0x0C2A, 0x0C39}, {0x0C3D, 0x0C3D}, {0x0C58, 0x0C5A},
- {0x0C5C, 0x0C5D}, {0x0C60, 0x0C61}, {0x0C80, 0x0C80},
- {0x0C85, 0x0C8C}, {0x0C8E, 0x0C90}, {0x0C92, 0x0CA8},
- {0x0CAA, 0x0CB3}, {0x0CB5, 0x0CB9}, {0x0CBD, 0x0CBD},
- {0x0CDC, 0x0CDE}, {0x0CE0, 0x0CE1}, {0x0CF1, 0x0CF2},
- {0x0D04, 0x0D0C}, {0x0D0E, 0x0D10}, {0x0D12, 0x0D3A},
- {0x0D3D, 0x0D3D}, {0x0D4E, 0x0D4E}, {0x0D54, 0x0D56},
- {0x0D5F, 0x0D61}, {0x0D7A, 0x0D7F}, {0x0D85, 0x0D96},
- {0x0D9A, 0x0DB1}, {0x0DB3, 0x0DBB}, {0x0DBD, 0x0DBD},
- {0x0DC0, 0x0DC6}, {0x0E01, 0x0E30}, {0x0E32, 0x0E32},
- {0x0E40, 0x0E46}, {0x0E81, 0x0E82}, {0x0E84, 0x0E84},
- {0x0E86, 0x0E8A}, {0x0E8C, 0x0EA3}, {0x0EA5, 0x0EA5},
- {0x0EA7, 0x0EB0}, {0x0EB2, 0x0EB2}, {0x0EBD, 0x0EBD},
- {0x0EC0, 0x0EC4}, {0x0EC6, 0x0EC6}, {0x0EDC, 0x0EDF},
- {0x0F00, 0x0F00}, {0x0F40, 0x0F47}, {0x0F49, 0x0F6C},
- {0x0F88, 0x0F8C}, {0x1000, 0x102A}, {0x103F, 0x103F},
- {0x1050, 0x1055}, {0x105A, 0x105D}, {0x1061, 0x1061},
- {0x1065, 0x1066}, {0x106E, 0x1070}, {0x1075, 0x1081},
- {0x108E, 0x108E}, {0x10A0, 0x10C5}, {0x10C7, 0x10C7},
- {0x10CD, 0x10CD}, {0x10D0, 0x10FA}, {0x10FC, 0x1248},
- {0x124A, 0x124D}, {0x1250, 0x1256}, {0x1258, 0x1258},
- {0x125A, 0x125D}, {0x1260, 0x1288}, {0x128A, 0x128D},
- {0x1290, 0x12B0}, {0x12B2, 0x12B5}, {0x12B8, 0x12BE},
- {0x12C0, 0x12C0}, {0x12C2, 0x12C5}, {0x12C8, 0x12D6},
- {0x12D8, 0x1310}, {0x1312, 0x1315}, {0x1318, 0x135A},
- {0x1380, 0x138F}, {0x13A0, 0x13F5}, {0x13F8, 0x13FD},
- {0x1401, 0x166C}, {0x166F, 0x167F}, {0x1681, 0x169A},
- {0x16A0, 0x16EA}, {0x16EE, 0x16F8}, {0x1700, 0x1711},
- {0x171F, 0x1731}, {0x1740, 0x1751}, {0x1760, 0x176C},
- {0x176E, 0x1770}, {0x1780, 0x17B3}, {0x17D7, 0x17D7},
- {0x17DC, 0x17DC}, {0x1820, 0x1878}, {0x1880, 0x18A8},
- {0x18AA, 0x18AA}, {0x18B0, 0x18F5}, {0x1900, 0x191E},
- {0x1950, 0x196D}, {0x1970, 0x1974}, {0x1980, 0x19AB},
- {0x19B0, 0x19C9}, {0x1A00, 0x1A16}, {0x1A20, 0x1A54},
- {0x1AA7, 0x1AA7}, {0x1B05, 0x1B33}, {0x1B45, 0x1B4C},
- {0x1B83, 0x1BA0}, {0x1BAE, 0x1BAF}, {0x1BBA, 0x1BE5},
- {0x1C00, 0x1C23}, {0x1C4D, 0x1C4F}, {0x1C5A, 0x1C7D},
- {0x1C80, 0x1C8A}, {0x1C90, 0x1CBA}, {0x1CBD, 0x1CBF},
- {0x1CE9, 0x1CEC}, {0x1CEE, 0x1CF3}, {0x1CF5, 0x1CF6},
- {0x1CFA, 0x1CFA}, {0x1D00, 0x1DBF}, {0x1E00, 0x1F15},
- {0x1F18, 0x1F1D}, {0x1F20, 0x1F45}, {0x1F48, 0x1F4D},
- {0x1F50, 0x1F57}, {0x1F59, 0x1F59}, {0x1F5B, 0x1F5B},
- {0x1F5D, 0x1F5D}, {0x1F5F, 0x1F7D}, {0x1F80, 0x1FB4},
- {0x1FB6, 0x1FBC}, {0x1FBE, 0x1FBE}, {0x1FC2, 0x1FC4},
- {0x1FC6, 0x1FCC}, {0x1FD0, 0x1FD3}, {0x1FD6, 0x1FDB},
- {0x1FE0, 0x1FEC}, {0x1FF2, 0x1FF4}, {0x1FF6, 0x1FFC},
- {0x2071, 0x2071}, {0x207F, 0x207F}, {0x208F, 0x209F},
- {0x2102, 0x2102}, {0x2107, 0x2107}, {0x210A, 0x2113},
- {0x2115, 0x2115}, {0x2118, 0x211D}, {0x2124, 0x2124},
- {0x2126, 0x2126}, {0x2128, 0x2128}, {0x212A, 0x2139},
- {0x213C, 0x213F}, {0x2145, 0x2149}, {0x214E, 0x214E},
- {0x2160, 0x2188}, {0x2C00, 0x2CE4}, {0x2CEB, 0x2CEE},
- {0x2CF2, 0x2CF3}, {0x2D00, 0x2D25}, {0x2D27, 0x2D27},
- {0x2D2D, 0x2D2D}, {0x2D30, 0x2D67}, {0x2D6F, 0x2D6F},
- {0x2D80, 0x2D96}, {0x2DA0, 0x2DA6}, {0x2DA8, 0x2DAE},
- {0x2DB0, 0x2DB6}, {0x2DB8, 0x2DBE}, {0x2DC0, 0x2DC6},
- {0x2DC8, 0x2DCE}, {0x2DD0, 0x2DD6}, {0x2DD8, 0x2DDE},
- {0x3005, 0x3007}, {0x3021, 0x3029}, {0x3031, 0x3035},
- {0x3038, 0x303C}, {0x3041, 0x3096}, {0x309D, 0x309F},
- {0x30A1, 0x30FA}, {0x30FC, 0x30FF}, {0x3105, 0x312F},
- {0x3131, 0x318E}, {0x31A0, 0x31BF}, {0x31F0, 0x31FF},
- {0x3400, 0x4DBF}, {0x4E00, 0xA48C}, {0xA4D0, 0xA4FD},
- {0xA500, 0xA60C}, {0xA610, 0xA61F}, {0xA62A, 0xA62B},
- {0xA640, 0xA66E}, {0xA67F, 0xA69D}, {0xA6A0, 0xA6EF},
- {0xA717, 0xA71F}, {0xA722, 0xA788}, {0xA78B, 0xA7DD},
- {0xA7E2, 0xA7E2}, {0xA7F1, 0xA801}, {0xA803, 0xA805},
- {0xA807, 0xA80A}, {0xA80C, 0xA822}, {0xA840, 0xA873},
- {0xA882, 0xA8B3}, {0xA8F2, 0xA8F7}, {0xA8FB, 0xA8FB},
- {0xA8FD, 0xA8FE}, {0xA90A, 0xA925}, {0xA930, 0xA946},
- {0xA960, 0xA97C}, {0xA984, 0xA9B2}, {0xA9CF, 0xA9CF},
- {0xA9E0, 0xA9E4}, {0xA9E6, 0xA9EF}, {0xA9FA, 0xA9FE},
- {0xAA00, 0xAA28}, {0xAA40, 0xAA42}, {0xAA44, 0xAA4B},
- {0xAA60, 0xAA76}, {0xAA7A, 0xAA7A}, {0xAA7E, 0xAAAF},
- {0xAAB1, 0xAAB1}, {0xAAB5, 0xAAB6}, {0xAAB9, 0xAABD},
- {0xAAC0, 0xAAC0}, {0xAAC2, 0xAAC2}, {0xAADB, 0xAADD},
- {0xAAE0, 0xAAEA}, {0xAAF2, 0xAAF4}, {0xAB01, 0xAB06},
- {0xAB09, 0xAB0E}, {0xAB11, 0xAB16}, {0xAB20, 0xAB26},
- {0xAB28, 0xAB2E}, {0xAB30, 0xAB5A}, {0xAB5C, 0xAB69},
- {0xAB6C, 0xAB6D}, {0xAB70, 0xABE2}, {0xAC00, 0xD7A3},
- {0xD7B0, 0xD7C6}, {0xD7CB, 0xD7FB}, {0xF900, 0xFA6D},
- {0xFA70, 0xFAD9}, {0xFB00, 0xFB06}, {0xFB13, 0xFB17},
- {0xFB1D, 0xFB1D}, {0xFB1F, 0xFB28}, {0xFB2A, 0xFB36},
- {0xFB38, 0xFB3C}, {0xFB3E, 0xFB3E}, {0xFB40, 0xFB41},
- {0xFB43, 0xFB44}, {0xFB46, 0xFBB1}, {0xFBD3, 0xFC5D},
- {0xFC64, 0xFD3D}, {0xFD50, 0xFD8F}, {0xFD92, 0xFDC7},
- {0xFDF0, 0xFDF9}, {0xFE71, 0xFE71}, {0xFE73, 0xFE73},
- {0xFE77, 0xFE77}, {0xFE79, 0xFE79}, {0xFE7B, 0xFE7B},
- {0xFE7D, 0xFE7D}, {0xFE7F, 0xFEFC}, {0xFF21, 0xFF3A},
- {0xFF41, 0xFF5A}, {0xFF66, 0xFF9D}, {0xFFA0, 0xFFBE},
- {0xFFC2, 0xFFC7}, {0xFFCA, 0xFFCF}, {0xFFD2, 0xFFD7},
- {0xFFDA, 0xFFDC}, {0x10000, 0x1000B}, {0x1000D, 0x10026},
- {0x10028, 0x1003A}, {0x1003C, 0x1003D}, {0x1003F, 0x1004D},
- {0x10050, 0x1005D}, {0x10080, 0x100FA}, {0x10140, 0x10174},
- {0x10280, 0x1029C}, {0x102A0, 0x102D0}, {0x10300, 0x1031F},
- {0x1032D, 0x1034A}, {0x10350, 0x10375}, {0x10380, 0x1039D},
- {0x103A0, 0x103C3}, {0x103C8, 0x103CF}, {0x103D1, 0x103D5},
- {0x10400, 0x1049D}, {0x104B0, 0x104D3}, {0x104D8, 0x104FB},
- {0x10500, 0x10527}, {0x10530, 0x10563}, {0x10570, 0x1057A},
- {0x1057C, 0x1058A}, {0x1058C, 0x10592}, {0x10594, 0x10595},
- {0x10597, 0x105A1}, {0x105A3, 0x105B1}, {0x105B3, 0x105B9},
- {0x105BB, 0x105BC}, {0x105C0, 0x105F3}, {0x10600, 0x10736},
- {0x10740, 0x10755}, {0x10760, 0x10767}, {0x10780, 0x10785},
- {0x10787, 0x107B0}, {0x107B2, 0x107BF}, {0x10800, 0x10805},
- {0x10808, 0x10808}, {0x1080A, 0x10835}, {0x10837, 0x10838},
- {0x1083C, 0x1083C}, {0x1083F, 0x10855}, {0x10860, 0x10876},
- {0x10880, 0x1089E}, {0x108E0, 0x108F2}, {0x108F4, 0x108F5},
- {0x10900, 0x10915}, {0x10920, 0x10939}, {0x10940, 0x10959},
- {0x10980, 0x109B7}, {0x109BE, 0x109BF}, {0x10A00, 0x10A00},
- {0x10A10, 0x10A13}, {0x10A15, 0x10A17}, {0x10A19, 0x10A35},
- {0x10A60, 0x10A7C}, {0x10A80, 0x10A9C}, {0x10AC0, 0x10AC7},
- {0x10AC9, 0x10AE4}, {0x10B00, 0x10B35}, {0x10B40, 0x10B55},
- {0x10B60, 0x10B72}, {0x10B80, 0x10B91}, {0x10C00, 0x10C48},
- {0x10C80, 0x10CB2}, {0x10CC0, 0x10CF2}, {0x10D00, 0x10D23},
- {0x10D4A, 0x10D65}, {0x10D6F, 0x10D85}, {0x10E80, 0x10EA9},
- {0x10EB0, 0x10EB1}, {0x10EC2, 0x10EC7}, {0x10ED9, 0x10EEE},
- {0x10F00, 0x10F1C}, {0x10F27, 0x10F27}, {0x10F30, 0x10F45},
- {0x10F70, 0x10F81}, {0x10FB0, 0x10FC4}, {0x10FE0, 0x10FF6},
- {0x11003, 0x11037}, {0x11071, 0x11072}, {0x11075, 0x11075},
- {0x11083, 0x110AF}, {0x110D0, 0x110E8}, {0x11103, 0x11126},
- {0x11144, 0x11144}, {0x11147, 0x11147}, {0x11150, 0x11172},
- {0x11176, 0x11176}, {0x11183, 0x111B2}, {0x111C1, 0x111C4},
- {0x111DA, 0x111DA}, {0x111DC, 0x111DC}, {0x11200, 0x11211},
- {0x11213, 0x1122B}, {0x1123F, 0x11240}, {0x11280, 0x11286},
- {0x11288, 0x11288}, {0x1128A, 0x1128D}, {0x1128F, 0x1129D},
- {0x1129F, 0x112A8}, {0x112B0, 0x112DE}, {0x11305, 0x1130C},
- {0x1130F, 0x11310}, {0x11313, 0x11328}, {0x1132A, 0x11330},
- {0x11332, 0x11333}, {0x11335, 0x11339}, {0x1133D, 0x1133D},
- {0x11350, 0x11350}, {0x1135D, 0x11361}, {0x11380, 0x11389},
- {0x1138B, 0x1138B}, {0x1138E, 0x1138E}, {0x11390, 0x113B5},
- {0x113B7, 0x113B7}, {0x113D1, 0x113D1}, {0x113D3, 0x113D3},
- {0x11400, 0x11434}, {0x11447, 0x1144A}, {0x1145F, 0x11461},
- {0x11480, 0x114AF}, {0x114C4, 0x114C5}, {0x114C7, 0x114C7},
- {0x11580, 0x115AE}, {0x115D8, 0x115DB}, {0x11600, 0x1162F},
- {0x11644, 0x11644}, {0x11680, 0x116AA}, {0x116B8, 0x116B8},
- {0x11700, 0x1171A}, {0x11740, 0x11746}, {0x11800, 0x1182B},
- {0x118A0, 0x118DF}, {0x118FF, 0x11906}, {0x11909, 0x11909},
- {0x1190C, 0x11913}, {0x11915, 0x11916}, {0x11918, 0x1192F},
- {0x1193F, 0x1193F}, {0x11941, 0x11941}, {0x119A0, 0x119A7},
- {0x119AA, 0x119D0}, {0x119E1, 0x119E1}, {0x119E3, 0x119E3},
- {0x11A00, 0x11A00}, {0x11A0B, 0x11A32}, {0x11A3A, 0x11A3A},
- {0x11A50, 0x11A50}, {0x11A5C, 0x11A89}, {0x11A9D, 0x11A9D},
- {0x11AB0, 0x11AF8}, {0x11B0A, 0x11B0A}, {0x11BC0, 0x11BE0},
- {0x11C00, 0x11C08}, {0x11C0A, 0x11C2E}, {0x11C40, 0x11C40},
- {0x11C72, 0x11C8F}, {0x11D00, 0x11D06}, {0x11D08, 0x11D09},
- {0x11D0B, 0x11D30}, {0x11D46, 0x11D46}, {0x11D60, 0x11D65},
- {0x11D67, 0x11D68}, {0x11D6A, 0x11D89}, {0x11D98, 0x11D98},
- {0x11DB0, 0x11DDB}, {0x11DF1, 0x11DF1}, {0x11EE0, 0x11EF2},
- {0x11F02, 0x11F02}, {0x11F04, 0x11F10}, {0x11F12, 0x11F33},
- {0x11FB0, 0x11FB0}, {0x12000, 0x12399}, {0x12400, 0x1246F},
- {0x12475, 0x12543}, {0x12550, 0x12686}, {0x12F90, 0x12FF0},
- {0x13000, 0x1342F}, {0x13441, 0x13446}, {0x13460, 0x143FA},
- {0x14400, 0x14646}, {0x16100, 0x1611D}, {0x16800, 0x16A38},
- {0x16A40, 0x16A5E}, {0x16A70, 0x16ABE}, {0x16AD0, 0x16AED},
- {0x16B00, 0x16B2F}, {0x16B40, 0x16B43}, {0x16B63, 0x16B77},
- {0x16B7D, 0x16B8F}, {0x16D40, 0x16D6C}, {0x16D80, 0x16D97},
- {0x16D99, 0x16D9C}, {0x16E40, 0x16E7F}, {0x16EA0, 0x16EB8},
- {0x16EBB, 0x16ED3}, {0x16F00, 0x16F4A}, {0x16F50, 0x16F50},
- {0x16F93, 0x16F9F}, {0x16FE0, 0x16FE1}, {0x16FE3, 0x16FE3},
- {0x16FF2, 0x16FF6}, {0x17000, 0x18CDA}, {0x18CFF, 0x18D20},
- {0x18D80, 0x18DF2}, {0x18E00, 0x19191}, {0x191A0, 0x191D2},
- {0x1AFF0, 0x1AFF3}, {0x1AFF5, 0x1AFFB}, {0x1AFFD, 0x1AFFE},
- {0x1B000, 0x1B128}, {0x1B132, 0x1B132}, {0x1B150, 0x1B152},
- {0x1B155, 0x1B155}, {0x1B164, 0x1B168}, {0x1B170, 0x1B2FB},
- {0x1BC00, 0x1BC6A}, {0x1BC70, 0x1BC7C}, {0x1BC80, 0x1BC88},
- {0x1BC90, 0x1BC99}, {0x1D400, 0x1D454}, {0x1D456, 0x1D49C},
- {0x1D49E, 0x1D49F}, {0x1D4A2, 0x1D4A2}, {0x1D4A5, 0x1D4A6},
- {0x1D4A9, 0x1D4AC}, {0x1D4AE, 0x1D4B9}, {0x1D4BB, 0x1D4BB},
- {0x1D4BD, 0x1D4C3}, {0x1D4C5, 0x1D505}, {0x1D507, 0x1D50A},
- {0x1D50D, 0x1D514}, {0x1D516, 0x1D51C}, {0x1D51E, 0x1D539},
- {0x1D53B, 0x1D53E}, {0x1D540, 0x1D544}, {0x1D546, 0x1D546},
- {0x1D54A, 0x1D550}, {0x1D552, 0x1D6A6}, {0x1D6A8, 0x1D6C0},
- {0x1D6C2, 0x1D6DA}, {0x1D6DC, 0x1D6FA}, {0x1D6FC, 0x1D714},
- {0x1D716, 0x1D734}, {0x1D736, 0x1D74E}, {0x1D750, 0x1D76E},
- {0x1D770, 0x1D788}, {0x1D78A, 0x1D7A8}, {0x1D7AA, 0x1D7C2},
- {0x1D7C4, 0x1D7CB}, {0x1DF00, 0x1DF81}, {0x1DF90, 0x1DF96},
- {0x1DFCD, 0x1DFFF}, {0x1E030, 0x1E06D}, {0x1E100, 0x1E12C},
- {0x1E137, 0x1E13D}, {0x1E14E, 0x1E14E}, {0x1E290, 0x1E2AD},
- {0x1E2C0, 0x1E2EB}, {0x1E4D0, 0x1E4EB}, {0x1E5D0, 0x1E5ED},
- {0x1E5F0, 0x1E5F0}, {0x1E6C0, 0x1E6DE}, {0x1E6E0, 0x1E6E2},
- {0x1E6E4, 0x1E6E5}, {0x1E6E7, 0x1E6ED}, {0x1E6F0, 0x1E6F4},
- {0x1E6FE, 0x1E6FF}, {0x1E7E0, 0x1E7E6}, {0x1E7E8, 0x1E7EB},
- {0x1E7ED, 0x1E7EE}, {0x1E7F0, 0x1E7FE}, {0x1E800, 0x1E8C4},
- {0x1E900, 0x1E943}, {0x1E94B, 0x1E94B}, {0x1EE00, 0x1EE03},
- {0x1EE05, 0x1EE1F}, {0x1EE21, 0x1EE22}, {0x1EE24, 0x1EE24},
- {0x1EE27, 0x1EE27}, {0x1EE29, 0x1EE32}, {0x1EE34, 0x1EE37},
- {0x1EE39, 0x1EE39}, {0x1EE3B, 0x1EE3B}, {0x1EE42, 0x1EE42},
- {0x1EE47, 0x1EE47}, {0x1EE49, 0x1EE49}, {0x1EE4B, 0x1EE4B},
- {0x1EE4D, 0x1EE4F}, {0x1EE51, 0x1EE52}, {0x1EE54, 0x1EE54},
- {0x1EE57, 0x1EE57}, {0x1EE59, 0x1EE59}, {0x1EE5B, 0x1EE5B},
- {0x1EE5D, 0x1EE5D}, {0x1EE5F, 0x1EE5F}, {0x1EE61, 0x1EE62},
- {0x1EE64, 0x1EE64}, {0x1EE67, 0x1EE6A}, {0x1EE6C, 0x1EE72},
- {0x1EE74, 0x1EE77}, {0x1EE79, 0x1EE7C}, {0x1EE7E, 0x1EE7E},
- {0x1EE80, 0x1EE89}, {0x1EE8B, 0x1EE9B}, {0x1EEA1, 0x1EEA3},
- {0x1EEA5, 0x1EEA9}, {0x1EEAB, 0x1EEBB}, {0x20000, 0x2A6DF},
- {0x2A700, 0x2B81E}, {0x2B820, 0x2CEAD}, {0x2CEB0, 0x2EBE0},
- {0x2EBF0, 0x2EE5D}, {0x2F800, 0x2FA1D}, {0x30000, 0x3134A},
- {0x31350, 0x33479}, {0x3D000, 0x3FC3F},
-};
-
-// Unicode 18.0 XID_Continue, excluding XID_Start
-// The Unicode Property XID_Continue is a super set of XID_Start.
-// To save Space, the table below only contains the codepoints
-// that are not also in XID_Start.
-static const llvm::sys::UnicodeCharRange XIDContinueRanges[] = {
- {0x0030, 0x0039}, {0x005F, 0x005F}, {0x00B7, 0x00B7},
- {0x0300, 0x036F}, {0x0387, 0x0387}, {0x0483, 0x0487},
- {0x0591, 0x05BD}, {0x05BF, 0x05BF}, {0x05C1, 0x05C2},
- {0x05C4, 0x05C5}, {0x05C7, 0x05C9}, {0x0610, 0x061A},
- {0x064B, 0x0669}, {0x0670, 0x0670}, {0x06D6, 0x06DC},
- {0x06DF, 0x06E4}, {0x06E7, 0x06E8}, {0x06EA, 0x06ED},
- {0x06F0, 0x06F9}, {0x0711, 0x0711}, {0x0730, 0x074A},
- {0x07A6, 0x07B0}, {0x07C0, 0x07C9}, {0x07EB, 0x07F3},
- {0x07FD, 0x07FD}, {0x0816, 0x0819}, {0x081B, 0x0823},
- {0x0825, 0x0827}, {0x0829, 0x082D}, {0x0859, 0x085B},
- {0x0897, 0x089F}, {0x08CA, 0x08E1}, {0x08E3, 0x0903},
- {0x093A, 0x093C}, {0x093E, 0x094F}, {0x0951, 0x0957},
- {0x0962, 0x0963}, {0x0966, 0x096F}, {0x0981, 0x0983},
- {0x09BC, 0x09BC}, {0x09BE, 0x09C4}, {0x09C7, 0x09C8},
- {0x09CB, 0x09CD}, {0x09D7, 0x09D7}, {0x09E2, 0x09E3},
- {0x09E6, 0x09EF}, {0x09FE, 0x09FE}, {0x0A01, 0x0A03},
- {0x0A3C, 0x0A3C}, {0x0A3E, 0x0A42}, {0x0A47, 0x0A48},
- {0x0A4B, 0x0A4D}, {0x0A51, 0x0A51}, {0x0A66, 0x0A71},
- {0x0A75, 0x0A75}, {0x0A81, 0x0A83}, {0x0ABC, 0x0ABC},
- {0x0ABE, 0x0AC5}, {0x0AC7, 0x0AC9}, {0x0ACB, 0x0ACD},
- {0x0AE2, 0x0AE3}, {0x0AE6, 0x0AEF}, {0x0AFA, 0x0AFF},
- {0x0B01, 0x0B03}, {0x0B3C, 0x0B3C}, {0x0B3E, 0x0B44},
- {0x0B47, 0x0B48}, {0x0B4B, 0x0B4D}, {0x0B53, 0x0B57},
- {0x0B62, 0x0B63}, {0x0B66, 0x0B6F}, {0x0B82, 0x0B82},
- {0x0BBE, 0x0BC2}, {0x0BC6, 0x0BC8}, {0x0BCA, 0x0BCD},
- {0x0BD7, 0x0BD7}, {0x0BE6, 0x0BEF}, {0x0C00, 0x0C04},
- {0x0C3C, 0x0C3C}, {0x0C3E, 0x0C44}, {0x0C46, 0x0C48},
- {0x0C4A, 0x0C4D}, {0x0C55, 0x0C56}, {0x0C62, 0x0C63},
- {0x0C66, 0x0C6F}, {0x0C81, 0x0C83}, {0x0CBC, 0x0CBC},
- {0x0CBE, 0x0CC4}, {0x0CC6, 0x0CC8}, {0x0CCA, 0x0CCD},
- {0x0CD5, 0x0CD6}, {0x0CE2, 0x0CE3}, {0x0CE6, 0x0CEF},
- {0x0CF3, 0x0CF3}, {0x0D00, 0x0D03}, {0x0D3B, 0x0D3C},
- {0x0D3E, 0x0D44}, {0x0D46, 0x0D48}, {0x0D4A, 0x0D4D},
- {0x0D57, 0x0D57}, {0x0D62, 0x0D63}, {0x0D66, 0x0D6F},
- {0x0D81, 0x0D83}, {0x0DCA, 0x0DCA}, {0x0DCF, 0x0DD4},
- {0x0DD6, 0x0DD6}, {0x0DD8, 0x0DDF}, {0x0DE6, 0x0DEF},
- {0x0DF2, 0x0DF3}, {0x0E31, 0x0E31}, {0x0E33, 0x0E3A},
- {0x0E47, 0x0E4E}, {0x0E50, 0x0E59}, {0x0EB1, 0x0EB1},
- {0x0EB3, 0x0EBC}, {0x0EC8, 0x0ECE}, {0x0ED0, 0x0ED9},
- {0x0F18, 0x0F19}, {0x0F20, 0x0F29}, {0x0F35, 0x0F35},
- {0x0F37, 0x0F37}, {0x0F39, 0x0F39}, {0x0F3E, 0x0F3F},
- {0x0F71, 0x0F84}, {0x0F86, 0x0F87}, {0x0F8D, 0x0F97},
- {0x0F99, 0x0FBC}, {0x0FC6, 0x0FC6}, {0x102B, 0x103E},
- {0x1040, 0x1049}, {0x1056, 0x1059}, {0x105E, 0x1060},
- {0x1062, 0x1064}, {0x1067, 0x106D}, {0x1071, 0x1074},
- {0x1082, 0x108D}, {0x108F, 0x109D}, {0x135D, 0x135F},
- {0x1369, 0x1371}, {0x1712, 0x1715}, {0x1732, 0x1734},
- {0x1752, 0x1753}, {0x1772, 0x1773}, {0x17B4, 0x17D3},
- {0x17DD, 0x17DD}, {0x17E0, 0x17E9}, {0x180B, 0x180D},
- {0x180F, 0x1819}, {0x18A9, 0x18A9}, {0x1920, 0x192B},
- {0x1930, 0x193B}, {0x1946, 0x194F}, {0x19D0, 0x19DA},
- {0x1A17, 0x1A1B}, {0x1A55, 0x1A5E}, {0x1A60, 0x1A7C},
- {0x1A7F, 0x1A89}, {0x1A90, 0x1A99}, {0x1AB0, 0x1ABD},
- {0x1ABF, 0x1AF0}, {0x1B00, 0x1B04}, {0x1B34, 0x1B44},
- {0x1B50, 0x1B59}, {0x1B6B, 0x1B73}, {0x1B80, 0x1B82},
- {0x1BA1, 0x1BAD}, {0x1BB0, 0x1BB9}, {0x1BE6, 0x1BF3},
- {0x1C24, 0x1C37}, {0x1C40, 0x1C49}, {0x1C50, 0x1C59},
- {0x1CD0, 0x1CD2}, {0x1CD4, 0x1CE8}, {0x1CED, 0x1CED},
- {0x1CF4, 0x1CF4}, {0x1CF7, 0x1CF9}, {0x1DC0, 0x1DFF},
- {0x200C, 0x200D}, {0x203F, 0x2040}, {0x2054, 0x2054},
- {0x20D0, 0x20DC}, {0x20E1, 0x20E1}, {0x20E5, 0x20F0},
- {0x2CEF, 0x2CF1}, {0x2D7F, 0x2D7F}, {0x2DE0, 0x2DFF},
- {0x302A, 0x302F}, {0x3099, 0x309A}, {0x30FB, 0x30FB},
- {0xA620, 0xA629}, {0xA66F, 0xA66F}, {0xA674, 0xA67D},
- {0xA69E, 0xA69F}, {0xA6F0, 0xA6F1}, {0xA802, 0xA802},
- {0xA806, 0xA806}, {0xA80B, 0xA80B}, {0xA823, 0xA827},
- {0xA82C, 0xA82C}, {0xA880, 0xA881}, {0xA8B4, 0xA8C5},
- {0xA8D0, 0xA8D9}, {0xA8E0, 0xA8F1}, {0xA8FF, 0xA909},
- {0xA926, 0xA92D}, {0xA947, 0xA953}, {0xA980, 0xA983},
- {0xA9B3, 0xA9C0}, {0xA9D0, 0xA9D9}, {0xA9E5, 0xA9E5},
- {0xA9F0, 0xA9F9}, {0xAA29, 0xAA36}, {0xAA43, 0xAA43},
- {0xAA4C, 0xAA4D}, {0xAA50, 0xAA59}, {0xAA7B, 0xAA7D},
- {0xAAB0, 0xAAB0}, {0xAAB2, 0xAAB4}, {0xAAB7, 0xAAB8},
- {0xAABE, 0xAABF}, {0xAAC1, 0xAAC1}, {0xAAEB, 0xAAEF},
- {0xAAF5, 0xAAF6}, {0xABE3, 0xABEA}, {0xABEC, 0xABED},
- {0xABF0, 0xABF9}, {0xFB1E, 0xFB1E}, {0xFE00, 0xFE0F},
- {0xFE20, 0xFE2F}, {0xFE33, 0xFE34}, {0xFE4D, 0xFE4F},
- {0xFF10, 0xFF19}, {0xFF3F, 0xFF3F}, {0xFF65, 0xFF65},
- {0xFF9E, 0xFF9F}, {0x101FD, 0x101FD}, {0x102E0, 0x102E0},
- {0x10376, 0x1037A}, {0x104A0, 0x104A9}, {0x10A01, 0x10A03},
- {0x10A05, 0x10A06}, {0x10A0C, 0x10A0F}, {0x10A38, 0x10A3A},
- {0x10A3F, 0x10A3F}, {0x10AE5, 0x10AE6}, {0x10D24, 0x10D27},
- {0x10D30, 0x10D39}, {0x10D40, 0x10D49}, {0x10D69, 0x10D6D},
- {0x10EAB, 0x10EAC}, {0x10ECB, 0x10ECF}, {0x10EF0, 0x10EFF},
- {0x10F46, 0x10F50}, {0x10F82, 0x10F85}, {0x11000, 0x11002},
- {0x11038, 0x11046}, {0x11066, 0x11070}, {0x11073, 0x11074},
- {0x1107F, 0x11082}, {0x110B0, 0x110BA}, {0x110C2, 0x110C2},
- {0x110F0, 0x110F9}, {0x11100, 0x11102}, {0x11127, 0x11134},
- {0x11136, 0x1113F}, {0x11145, 0x11146}, {0x11173, 0x11173},
- {0x11180, 0x11182}, {0x111B3, 0x111C0}, {0x111C9, 0x111CC},
- {0x111CE, 0x111D9}, {0x1122C, 0x11237}, {0x1123E, 0x1123E},
- {0x11241, 0x11241}, {0x112DF, 0x112EA}, {0x112F0, 0x112F9},
- {0x11300, 0x11303}, {0x1133B, 0x1133C}, {0x1133E, 0x11344},
- {0x11347, 0x11348}, {0x1134B, 0x1134D}, {0x11357, 0x11357},
- {0x11362, 0x11363}, {0x11366, 0x1136C}, {0x11370, 0x11374},
- {0x113B8, 0x113C0}, {0x113C2, 0x113C2}, {0x113C5, 0x113C5},
- {0x113C7, 0x113CA}, {0x113CC, 0x113D0}, {0x113D2, 0x113D2},
- {0x113E1, 0x113E2}, {0x11435, 0x11446}, {0x11450, 0x11459},
- {0x1145E, 0x1145E}, {0x114B0, 0x114C3}, {0x114D0, 0x114D9},
- {0x115AF, 0x115B5}, {0x115B8, 0x115C0}, {0x115DC, 0x115DD},
- {0x11630, 0x11640}, {0x11650, 0x11659}, {0x116AB, 0x116B7},
- {0x116C0, 0x116C9}, {0x116D0, 0x116E3}, {0x1171D, 0x1172B},
- {0x11730, 0x11739}, {0x1182C, 0x1183A}, {0x118E0, 0x118E9},
- {0x11930, 0x11935}, {0x11937, 0x11938}, {0x1193B, 0x1193E},
- {0x11940, 0x11940}, {0x11942, 0x11943}, {0x11950, 0x11959},
- {0x119D1, 0x119D7}, {0x119DA, 0x119E0}, {0x119E4, 0x119E4},
- {0x11A01, 0x11A0A}, {0x11A33, 0x11A39}, {0x11A3B, 0x11A3E},
- {0x11A47, 0x11A47}, {0x11A51, 0x11A5B}, {0x11A8A, 0x11A99},
- {0x11B60, 0x11B67}, {0x11BF0, 0x11BF9}, {0x11C2F, 0x11C36},
- {0x11C38, 0x11C3F}, {0x11C50, 0x11C59}, {0x11C92, 0x11CA7},
- {0x11CA9, 0x11CB6}, {0x11D31, 0x11D36}, {0x11D3A, 0x11D3A},
- {0x11D3C, 0x11D3D}, {0x11D3F, 0x11D45}, {0x11D47, 0x11D47},
- {0x11D50, 0x11D59}, {0x11D8A, 0x11D8E}, {0x11D90, 0x11D91},
- {0x11D93, 0x11D97}, {0x11DA0, 0x11DA9}, {0x11DE0, 0x11DE9},
- {0x11DF0, 0x11DF0}, {0x11EF3, 0x11EF6}, {0x11F00, 0x11F01},
- {0x11F03, 0x11F03}, {0x11F34, 0x11F3A}, {0x11F3E, 0x11F42},
- {0x11F50, 0x11F5A}, {0x13440, 0x13440}, {0x13447, 0x13455},
- {0x1611E, 0x16139}, {0x16A60, 0x16A69}, {0x16AC0, 0x16AC9},
- {0x16AF0, 0x16AF4}, {0x16B30, 0x16B36}, {0x16B50, 0x16B59},
- {0x16D70, 0x16D79}, {0x16D98, 0x16D98}, {0x16D9D, 0x16D9D},
- {0x16DA0, 0x16DA9}, {0x16F4F, 0x16F4F}, {0x16F51, 0x16F87},
- {0x16F8F, 0x16F92}, {0x16FE4, 0x16FE4}, {0x16FF0, 0x16FF1},
- {0x1BC9D, 0x1BC9E}, {0x1CCF0, 0x1CCF9}, {0x1CF00, 0x1CF2D},
- {0x1CF30, 0x1CF46}, {0x1D127, 0x1D128}, {0x1D165, 0x1D169},
- {0x1D16D, 0x1D172}, {0x1D17B, 0x1D182}, {0x1D185, 0x1D18B},
- {0x1D1AA, 0x1D1AD}, {0x1D242, 0x1D244}, {0x1D250, 0x1D252},
- {0x1D25B, 0x1D25C}, {0x1D25F, 0x1D25F}, {0x1D280, 0x1D281},
- {0x1D7CE, 0x1D7FF}, {0x1DA00, 0x1DA36}, {0x1DA3B, 0x1DA6C},
- {0x1DA75, 0x1DA75}, {0x1DA84, 0x1DA84}, {0x1DA9B, 0x1DA9F},
- {0x1DAA1, 0x1DAAF}, {0x1E000, 0x1E006}, {0x1E008, 0x1E018},
- {0x1E01B, 0x1E021}, {0x1E023, 0x1E024}, {0x1E026, 0x1E02A},
- {0x1E08F, 0x1E08F}, {0x1E130, 0x1E136}, {0x1E140, 0x1E149},
- {0x1E2AE, 0x1E2AE}, {0x1E2EC, 0x1E2F9}, {0x1E4EC, 0x1E4F9},
- {0x1E5EE, 0x1E5EF}, {0x1E5F1, 0x1E5FA}, {0x1E6E3, 0x1E6E3},
- {0x1E6E6, 0x1E6E6}, {0x1E6EE, 0x1E6EF}, {0x1E6F5, 0x1E6F5},
- {0x1E8D0, 0x1E8D6}, {0x1E944, 0x1E94A}, {0x1E950, 0x1E959},
- {0x1FBF0, 0x1FBF9}, {0xE0100, 0xE01EF},
-};
-
-// Unicode 18.0 ID_Compat_Math_Start
-static const llvm::sys::UnicodeCharRange
- MathematicalNotationProfileIDStartRanges[] = {
- {0x02202, 0x02202}, // โ
- {0x02207, 0x02207}, // โ
- {0x0221E, 0x0221E}, // โ
- {0x1D6C1, 0x1D6C1}, // ๐
- {0x1D6DB, 0x1D6DB}, // ๐
- {0x1D6FB, 0x1D6FB}, // ๐ป
- {0x1D715, 0x1D715}, // ๐
- {0x1D735, 0x1D735}, // ๐ต
- {0x1D74F, 0x1D74F}, // ๐
- {0x1D76F, 0x1D76F}, // ๐ฏ
- {0x1D789, 0x1D789}, // ๐
- {0x1D7A9, 0x1D7A9}, // ๐ฉ
- {0x1D7C3, 0x1D7C3}, // ๐
-};
-
-// Unicode 18.0 ID_Compat_Math_Continue, excluding ID_Compat_Math_Start
-static const llvm::sys::UnicodeCharRange
- MathematicalNotationProfileIDContinueRanges[] = {
- {0x000B2, 0x000B3}, // ยฒ-ยณ
- {0x000B9, 0x000B9}, // ยน
- {0x02070, 0x02070}, // โฐ
- {0x02074, 0x0207E}, // โด-โพ
- {0x02080, 0x0208E}, // โ-โ
-};
+namespace clang {
+extern const llvm::sys::UnicodeCharSet::CharRanges GeneratedXIDStartRanges;
+extern const llvm::sys::UnicodeCharSet::CharRanges GeneratedXIDContinueRanges;
+extern const llvm::sys::UnicodeCharSet::CharRanges
+ GeneratedMathematicalNotationProfileIDStartRanges;
+extern const llvm::sys::UnicodeCharSet::CharRanges
+ GeneratedMathematicalNotationProfileIDContinueRanges;
+} // namespace clang
// C11 D.1, C++11 [charname.allowed]
static const llvm::sys::UnicodeCharRange C11AllowedIDCharRanges[] = {
diff --git a/clang/lib/Lex/UnicodeCharSetsGenerated.cpp b/clang/lib/Lex/UnicodeCharSetsGenerated.cpp
new file mode 100644
index 0000000000000..f1967d496d9da
--- /dev/null
+++ b/clang/lib/Lex/UnicodeCharSetsGenerated.cpp
@@ -0,0 +1,423 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file was generated by llvm/utils/UnicodeData/UnicodeCharSetsGenerator
+// from the Unicode 18.0 Character Database. Do not edit manually.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Support/UnicodeCharRanges.h"
+
+using llvm::sys::UnicodeCharRange;
+using CharRanges = llvm::sys::UnicodeCharSet::CharRanges;
+
+namespace clang {
+
+// XID_Start
+static constexpr UnicodeCharRange GeneratedXIDStartRangesData[] = {
+ {0x0041, 0x005A}, {0x0061, 0x007A}, {0x00AA, 0x00AA},
+ {0x00B5, 0x00B5}, {0x00BA, 0x00BA}, {0x00C0, 0x00D6},
+ {0x00D8, 0x00F6}, {0x00F8, 0x02C1}, {0x02C6, 0x02D1},
+ {0x02E0, 0x02E4}, {0x02EC, 0x02EC}, {0x02EE, 0x02EE},
+ {0x0370, 0x0374}, {0x0376, 0x0377}, {0x037B, 0x037D},
+ {0x037F, 0x037F}, {0x0386, 0x0386}, {0x0388, 0x038A},
+ {0x038C, 0x038C}, {0x038E, 0x03A1}, {0x03A3, 0x03F5},
+ {0x03F7, 0x0481}, {0x048A, 0x052F}, {0x0531, 0x0556},
+ {0x0558, 0x0559}, {0x0560, 0x0588}, {0x058B, 0x058C},
+ {0x05D0, 0x05EA}, {0x05EF, 0x05F2}, {0x0620, 0x064A},
+ {0x066E, 0x066F}, {0x0671, 0x06D3}, {0x06D5, 0x06D5},
+ {0x06E5, 0x06E6}, {0x06EE, 0x06EF}, {0x06FA, 0x06FC},
+ {0x06FF, 0x06FF}, {0x0710, 0x0710}, {0x0712, 0x072F},
+ {0x074D, 0x07A5}, {0x07B1, 0x07B1}, {0x07CA, 0x07EA},
+ {0x07F4, 0x07F5}, {0x07FA, 0x07FA}, {0x0800, 0x0815},
+ {0x081A, 0x081A}, {0x0824, 0x0824}, {0x0828, 0x0828},
+ {0x0840, 0x0858}, {0x0860, 0x086A}, {0x0870, 0x0887},
+ {0x0889, 0x088F}, {0x08A0, 0x08C9}, {0x0904, 0x0939},
+ {0x093D, 0x093D}, {0x0950, 0x0950}, {0x0958, 0x0961},
+ {0x0971, 0x0980}, {0x0985, 0x098C}, {0x098F, 0x0990},
+ {0x0993, 0x09A8}, {0x09AA, 0x09B0}, {0x09B2, 0x09B2},
+ {0x09B6, 0x09B9}, {0x09BD, 0x09BD}, {0x09CE, 0x09CE},
+ {0x09DC, 0x09DD}, {0x09DF, 0x09E1}, {0x09F0, 0x09F1},
+ {0x09FC, 0x09FC}, {0x0A05, 0x0A0A}, {0x0A0F, 0x0A10},
+ {0x0A13, 0x0A28}, {0x0A2A, 0x0A30}, {0x0A32, 0x0A33},
+ {0x0A35, 0x0A36}, {0x0A38, 0x0A39}, {0x0A59, 0x0A5C},
+ {0x0A5E, 0x0A5E}, {0x0A72, 0x0A74}, {0x0A85, 0x0A8D},
+ {0x0A8F, 0x0A91}, {0x0A93, 0x0AA8}, {0x0AAA, 0x0AB0},
+ {0x0AB2, 0x0AB3}, {0x0AB5, 0x0AB9}, {0x0ABD, 0x0ABD},
+ {0x0AD0, 0x0AD0}, {0x0AE0, 0x0AE1}, {0x0AF9, 0x0AF9},
+ {0x0B05, 0x0B0C}, {0x0B0F, 0x0B10}, {0x0B13, 0x0B28},
+ {0x0B2A, 0x0B30}, {0x0B32, 0x0B33}, {0x0B35, 0x0B39},
+ {0x0B3D, 0x0B3D}, {0x0B5C, 0x0B5D}, {0x0B5F, 0x0B61},
+ {0x0B71, 0x0B71}, {0x0B83, 0x0B83}, {0x0B85, 0x0B8A},
+ {0x0B8E, 0x0B90}, {0x0B92, 0x0B95}, {0x0B99, 0x0B9A},
+ {0x0B9C, 0x0B9C}, {0x0B9E, 0x0B9F}, {0x0BA3, 0x0BA4},
+ {0x0BA8, 0x0BAA}, {0x0BAE, 0x0BB9}, {0x0BD0, 0x0BD0},
+ {0x0C05, 0x0C0C}, {0x0C0E, 0x0C10}, {0x0C12, 0x0C28},
+ {0x0C2A, 0x0C39}, {0x0C3D, 0x0C3D}, {0x0C58, 0x0C5A},
+ {0x0C5C, 0x0C5D}, {0x0C60, 0x0C61}, {0x0C80, 0x0C80},
+ {0x0C85, 0x0C8C}, {0x0C8E, 0x0C90}, {0x0C92, 0x0CA8},
+ {0x0CAA, 0x0CB3}, {0x0CB5, 0x0CB9}, {0x0CBD, 0x0CBD},
+ {0x0CDC, 0x0CDE}, {0x0CE0, 0x0CE1}, {0x0CF1, 0x0CF2},
+ {0x0D04, 0x0D0C}, {0x0D0E, 0x0D10}, {0x0D12, 0x0D3A},
+ {0x0D3D, 0x0D3D}, {0x0D4E, 0x0D4E}, {0x0D54, 0x0D56},
+ {0x0D5F, 0x0D61}, {0x0D7A, 0x0D7F}, {0x0D85, 0x0D96},
+ {0x0D9A, 0x0DB1}, {0x0DB3, 0x0DBB}, {0x0DBD, 0x0DBD},
+ {0x0DC0, 0x0DC6}, {0x0E01, 0x0E30}, {0x0E32, 0x0E32},
+ {0x0E40, 0x0E46}, {0x0E81, 0x0E82}, {0x0E84, 0x0E84},
+ {0x0E86, 0x0E8A}, {0x0E8C, 0x0EA3}, {0x0EA5, 0x0EA5},
+ {0x0EA7, 0x0EB0}, {0x0EB2, 0x0EB2}, {0x0EBD, 0x0EBD},
+ {0x0EC0, 0x0EC4}, {0x0EC6, 0x0EC6}, {0x0EDC, 0x0EDF},
+ {0x0F00, 0x0F00}, {0x0F40, 0x0F47}, {0x0F49, 0x0F6C},
+ {0x0F88, 0x0F8C}, {0x1000, 0x102A}, {0x103F, 0x103F},
+ {0x1050, 0x1055}, {0x105A, 0x105D}, {0x1061, 0x1061},
+ {0x1065, 0x1066}, {0x106E, 0x1070}, {0x1075, 0x1081},
+ {0x108E, 0x108E}, {0x10A0, 0x10C5}, {0x10C7, 0x10C7},
+ {0x10CD, 0x10CD}, {0x10D0, 0x10FA}, {0x10FC, 0x1248},
+ {0x124A, 0x124D}, {0x1250, 0x1256}, {0x1258, 0x1258},
+ {0x125A, 0x125D}, {0x1260, 0x1288}, {0x128A, 0x128D},
+ {0x1290, 0x12B0}, {0x12B2, 0x12B5}, {0x12B8, 0x12BE},
+ {0x12C0, 0x12C0}, {0x12C2, 0x12C5}, {0x12C8, 0x12D6},
+ {0x12D8, 0x1310}, {0x1312, 0x1315}, {0x1318, 0x135A},
+ {0x1380, 0x138F}, {0x13A0, 0x13F5}, {0x13F8, 0x13FD},
+ {0x1401, 0x166C}, {0x166F, 0x167F}, {0x1681, 0x169A},
+ {0x16A0, 0x16EA}, {0x16EE, 0x16F8}, {0x1700, 0x1711},
+ {0x171F, 0x1731}, {0x1740, 0x1751}, {0x1760, 0x176C},
+ {0x176E, 0x1770}, {0x1780, 0x17B3}, {0x17D7, 0x17D7},
+ {0x17DC, 0x17DC}, {0x1820, 0x1878}, {0x1880, 0x18A8},
+ {0x18AA, 0x18AA}, {0x18B0, 0x18F5}, {0x1900, 0x191E},
+ {0x1950, 0x196D}, {0x1970, 0x1974}, {0x1980, 0x19AB},
+ {0x19B0, 0x19C9}, {0x1A00, 0x1A16}, {0x1A20, 0x1A54},
+ {0x1AA7, 0x1AA7}, {0x1B05, 0x1B33}, {0x1B45, 0x1B4C},
+ {0x1B83, 0x1BA0}, {0x1BAE, 0x1BAF}, {0x1BBA, 0x1BE5},
+ {0x1C00, 0x1C23}, {0x1C4D, 0x1C4F}, {0x1C5A, 0x1C7D},
+ {0x1C80, 0x1C8A}, {0x1C90, 0x1CBA}, {0x1CBD, 0x1CBF},
+ {0x1CE9, 0x1CEC}, {0x1CEE, 0x1CF3}, {0x1CF5, 0x1CF6},
+ {0x1CFA, 0x1CFA}, {0x1D00, 0x1DBF}, {0x1E00, 0x1F15},
+ {0x1F18, 0x1F1D}, {0x1F20, 0x1F45}, {0x1F48, 0x1F4D},
+ {0x1F50, 0x1F57}, {0x1F59, 0x1F59}, {0x1F5B, 0x1F5B},
+ {0x1F5D, 0x1F5D}, {0x1F5F, 0x1F7D}, {0x1F80, 0x1FB4},
+ {0x1FB6, 0x1FBC}, {0x1FBE, 0x1FBE}, {0x1FC2, 0x1FC4},
+ {0x1FC6, 0x1FCC}, {0x1FD0, 0x1FD3}, {0x1FD6, 0x1FDB},
+ {0x1FE0, 0x1FEC}, {0x1FF2, 0x1FF4}, {0x1FF6, 0x1FFC},
+ {0x2071, 0x2071}, {0x207F, 0x207F}, {0x208F, 0x209F},
+ {0x2102, 0x2102}, {0x2107, 0x2107}, {0x210A, 0x2113},
+ {0x2115, 0x2115}, {0x2118, 0x211D}, {0x2124, 0x2124},
+ {0x2126, 0x2126}, {0x2128, 0x2128}, {0x212A, 0x2139},
+ {0x213C, 0x213F}, {0x2145, 0x2149}, {0x214E, 0x214E},
+ {0x2160, 0x2188}, {0x2C00, 0x2CE4}, {0x2CEB, 0x2CEE},
+ {0x2CF2, 0x2CF3}, {0x2D00, 0x2D25}, {0x2D27, 0x2D27},
+ {0x2D2D, 0x2D2D}, {0x2D30, 0x2D67}, {0x2D6F, 0x2D6F},
+ {0x2D80, 0x2D96}, {0x2DA0, 0x2DA6}, {0x2DA8, 0x2DAE},
+ {0x2DB0, 0x2DB6}, {0x2DB8, 0x2DBE}, {0x2DC0, 0x2DC6},
+ {0x2DC8, 0x2DCE}, {0x2DD0, 0x2DD6}, {0x2DD8, 0x2DDE},
+ {0x3005, 0x3007}, {0x3021, 0x3029}, {0x3031, 0x3035},
+ {0x3038, 0x303C}, {0x3041, 0x3096}, {0x309D, 0x309F},
+ {0x30A1, 0x30FA}, {0x30FC, 0x30FF}, {0x3105, 0x312F},
+ {0x3131, 0x318E}, {0x31A0, 0x31BF}, {0x31F0, 0x31FF},
+ {0x3400, 0x4DBF}, {0x4E00, 0xA48C}, {0xA4D0, 0xA4FD},
+ {0xA500, 0xA60C}, {0xA610, 0xA61F}, {0xA62A, 0xA62B},
+ {0xA640, 0xA66E}, {0xA67F, 0xA69D}, {0xA6A0, 0xA6EF},
+ {0xA717, 0xA71F}, {0xA722, 0xA788}, {0xA78B, 0xA7DD},
+ {0xA7E2, 0xA7E2}, {0xA7F1, 0xA801}, {0xA803, 0xA805},
+ {0xA807, 0xA80A}, {0xA80C, 0xA822}, {0xA840, 0xA873},
+ {0xA882, 0xA8B3}, {0xA8F2, 0xA8F7}, {0xA8FB, 0xA8FB},
+ {0xA8FD, 0xA8FE}, {0xA90A, 0xA925}, {0xA930, 0xA946},
+ {0xA960, 0xA97C}, {0xA984, 0xA9B2}, {0xA9CF, 0xA9CF},
+ {0xA9E0, 0xA9E4}, {0xA9E6, 0xA9EF}, {0xA9FA, 0xA9FE},
+ {0xAA00, 0xAA28}, {0xAA40, 0xAA42}, {0xAA44, 0xAA4B},
+ {0xAA60, 0xAA76}, {0xAA7A, 0xAA7A}, {0xAA7E, 0xAAAF},
+ {0xAAB1, 0xAAB1}, {0xAAB5, 0xAAB6}, {0xAAB9, 0xAABD},
+ {0xAAC0, 0xAAC0}, {0xAAC2, 0xAAC2}, {0xAADB, 0xAADD},
+ {0xAAE0, 0xAAEA}, {0xAAF2, 0xAAF4}, {0xAB01, 0xAB06},
+ {0xAB09, 0xAB0E}, {0xAB11, 0xAB16}, {0xAB20, 0xAB26},
+ {0xAB28, 0xAB2E}, {0xAB30, 0xAB5A}, {0xAB5C, 0xAB69},
+ {0xAB6C, 0xAB6D}, {0xAB70, 0xABE2}, {0xAC00, 0xD7A3},
+ {0xD7B0, 0xD7C6}, {0xD7CB, 0xD7FB}, {0xF900, 0xFA6D},
+ {0xFA70, 0xFAD9}, {0xFB00, 0xFB06}, {0xFB13, 0xFB17},
+ {0xFB1D, 0xFB1D}, {0xFB1F, 0xFB28}, {0xFB2A, 0xFB36},
+ {0xFB38, 0xFB3C}, {0xFB3E, 0xFB3E}, {0xFB40, 0xFB41},
+ {0xFB43, 0xFB44}, {0xFB46, 0xFBB1}, {0xFBD3, 0xFC5D},
+ {0xFC64, 0xFD3D}, {0xFD50, 0xFD8F}, {0xFD92, 0xFDC7},
+ {0xFDF0, 0xFDF9}, {0xFE71, 0xFE71}, {0xFE73, 0xFE73},
+ {0xFE77, 0xFE77}, {0xFE79, 0xFE79}, {0xFE7B, 0xFE7B},
+ {0xFE7D, 0xFE7D}, {0xFE7F, 0xFEFC}, {0xFF21, 0xFF3A},
+ {0xFF41, 0xFF5A}, {0xFF66, 0xFF9D}, {0xFFA0, 0xFFBE},
+ {0xFFC2, 0xFFC7}, {0xFFCA, 0xFFCF}, {0xFFD2, 0xFFD7},
+ {0xFFDA, 0xFFDC}, {0x10000, 0x1000B}, {0x1000D, 0x10026},
+ {0x10028, 0x1003A}, {0x1003C, 0x1003D}, {0x1003F, 0x1004D},
+ {0x10050, 0x1005D}, {0x10080, 0x100FA}, {0x10140, 0x10174},
+ {0x10280, 0x1029C}, {0x102A0, 0x102D0}, {0x10300, 0x1031F},
+ {0x1032D, 0x1034A}, {0x10350, 0x10375}, {0x10380, 0x1039D},
+ {0x103A0, 0x103C3}, {0x103C8, 0x103CF}, {0x103D1, 0x103D5},
+ {0x10400, 0x1049D}, {0x104B0, 0x104D3}, {0x104D8, 0x104FB},
+ {0x10500, 0x10527}, {0x10530, 0x10563}, {0x10570, 0x1057A},
+ {0x1057C, 0x1058A}, {0x1058C, 0x10592}, {0x10594, 0x10595},
+ {0x10597, 0x105A1}, {0x105A3, 0x105B1}, {0x105B3, 0x105B9},
+ {0x105BB, 0x105BC}, {0x105C0, 0x105F3}, {0x10600, 0x10736},
+ {0x10740, 0x10755}, {0x10760, 0x10767}, {0x10780, 0x10785},
+ {0x10787, 0x107B0}, {0x107B2, 0x107BF}, {0x10800, 0x10805},
+ {0x10808, 0x10808}, {0x1080A, 0x10835}, {0x10837, 0x10838},
+ {0x1083C, 0x1083C}, {0x1083F, 0x10855}, {0x10860, 0x10876},
+ {0x10880, 0x1089E}, {0x108E0, 0x108F2}, {0x108F4, 0x108F5},
+ {0x10900, 0x10915}, {0x10920, 0x10939}, {0x10940, 0x10959},
+ {0x10980, 0x109B7}, {0x109BE, 0x109BF}, {0x10A00, 0x10A00},
+ {0x10A10, 0x10A13}, {0x10A15, 0x10A17}, {0x10A19, 0x10A35},
+ {0x10A60, 0x10A7C}, {0x10A80, 0x10A9C}, {0x10AC0, 0x10AC7},
+ {0x10AC9, 0x10AE4}, {0x10B00, 0x10B35}, {0x10B40, 0x10B55},
+ {0x10B60, 0x10B72}, {0x10B80, 0x10B91}, {0x10C00, 0x10C48},
+ {0x10C80, 0x10CB2}, {0x10CC0, 0x10CF2}, {0x10D00, 0x10D23},
+ {0x10D4A, 0x10D65}, {0x10D6F, 0x10D85}, {0x10E80, 0x10EA9},
+ {0x10EB0, 0x10EB1}, {0x10EC2, 0x10EC7}, {0x10ED9, 0x10EEE},
+ {0x10F00, 0x10F1C}, {0x10F27, 0x10F27}, {0x10F30, 0x10F45},
+ {0x10F70, 0x10F81}, {0x10FB0, 0x10FC4}, {0x10FE0, 0x10FF6},
+ {0x11003, 0x11037}, {0x11071, 0x11072}, {0x11075, 0x11075},
+ {0x11083, 0x110AF}, {0x110D0, 0x110E8}, {0x11103, 0x11126},
+ {0x11144, 0x11144}, {0x11147, 0x11147}, {0x11150, 0x11172},
+ {0x11176, 0x11176}, {0x11183, 0x111B2}, {0x111C1, 0x111C4},
+ {0x111DA, 0x111DA}, {0x111DC, 0x111DC}, {0x11200, 0x11211},
+ {0x11213, 0x1122B}, {0x1123F, 0x11240}, {0x11280, 0x11286},
+ {0x11288, 0x11288}, {0x1128A, 0x1128D}, {0x1128F, 0x1129D},
+ {0x1129F, 0x112A8}, {0x112B0, 0x112DE}, {0x11305, 0x1130C},
+ {0x1130F, 0x11310}, {0x11313, 0x11328}, {0x1132A, 0x11330},
+ {0x11332, 0x11333}, {0x11335, 0x11339}, {0x1133D, 0x1133D},
+ {0x11350, 0x11350}, {0x1135D, 0x11361}, {0x11380, 0x11389},
+ {0x1138B, 0x1138B}, {0x1138E, 0x1138E}, {0x11390, 0x113B5},
+ {0x113B7, 0x113B7}, {0x113D1, 0x113D1}, {0x113D3, 0x113D3},
+ {0x11400, 0x11434}, {0x11447, 0x1144A}, {0x1145F, 0x11461},
+ {0x11480, 0x114AF}, {0x114C4, 0x114C5}, {0x114C7, 0x114C7},
+ {0x11580, 0x115AE}, {0x115D8, 0x115DB}, {0x11600, 0x1162F},
+ {0x11644, 0x11644}, {0x11680, 0x116AA}, {0x116B8, 0x116B8},
+ {0x11700, 0x1171A}, {0x11740, 0x11746}, {0x11800, 0x1182B},
+ {0x118A0, 0x118DF}, {0x118FF, 0x11906}, {0x11909, 0x11909},
+ {0x1190C, 0x11913}, {0x11915, 0x11916}, {0x11918, 0x1192F},
+ {0x1193F, 0x1193F}, {0x11941, 0x11941}, {0x119A0, 0x119A7},
+ {0x119AA, 0x119D0}, {0x119E1, 0x119E1}, {0x119E3, 0x119E3},
+ {0x11A00, 0x11A00}, {0x11A0B, 0x11A32}, {0x11A3A, 0x11A3A},
+ {0x11A50, 0x11A50}, {0x11A5C, 0x11A89}, {0x11A9D, 0x11A9D},
+ {0x11AB0, 0x11AF8}, {0x11B0A, 0x11B0A}, {0x11BC0, 0x11BE0},
+ {0x11C00, 0x11C08}, {0x11C0A, 0x11C2E}, {0x11C40, 0x11C40},
+ {0x11C72, 0x11C8F}, {0x11D00, 0x11D06}, {0x11D08, 0x11D09},
+ {0x11D0B, 0x11D30}, {0x11D46, 0x11D46}, {0x11D60, 0x11D65},
+ {0x11D67, 0x11D68}, {0x11D6A, 0x11D89}, {0x11D98, 0x11D98},
+ {0x11DB0, 0x11DDB}, {0x11DF1, 0x11DF1}, {0x11EE0, 0x11EF2},
+ {0x11F02, 0x11F02}, {0x11F04, 0x11F10}, {0x11F12, 0x11F33},
+ {0x11FB0, 0x11FB0}, {0x12000, 0x12399}, {0x12400, 0x1246F},
+ {0x12475, 0x12543}, {0x12550, 0x12686}, {0x12F90, 0x12FF0},
+ {0x13000, 0x1342F}, {0x13441, 0x13446}, {0x13460, 0x143FA},
+ {0x14400, 0x14646}, {0x16100, 0x1611D}, {0x16800, 0x16A38},
+ {0x16A40, 0x16A5E}, {0x16A70, 0x16ABE}, {0x16AD0, 0x16AED},
+ {0x16B00, 0x16B2F}, {0x16B40, 0x16B43}, {0x16B63, 0x16B77},
+ {0x16B7D, 0x16B8F}, {0x16D40, 0x16D6C}, {0x16E40, 0x16E7F},
+ {0x16EA0, 0x16EB8}, {0x16EBB, 0x16ED3}, {0x16F00, 0x16F4A},
+ {0x16F50, 0x16F50}, {0x16F93, 0x16F9F}, {0x16FE0, 0x16FE1},
+ {0x16FE3, 0x16FE3}, {0x16FF2, 0x16FF6}, {0x17000, 0x18CDA},
+ {0x18CFF, 0x18D20}, {0x18D80, 0x18DF2}, {0x18E00, 0x19191},
+ {0x191A0, 0x191D2}, {0x1AFF0, 0x1AFF3}, {0x1AFF5, 0x1AFFB},
+ {0x1AFFD, 0x1AFFE}, {0x1B000, 0x1B128}, {0x1B132, 0x1B132},
+ {0x1B150, 0x1B152}, {0x1B155, 0x1B155}, {0x1B164, 0x1B168},
+ {0x1B170, 0x1B2FB}, {0x1BC00, 0x1BC6A}, {0x1BC70, 0x1BC7C},
+ {0x1BC80, 0x1BC88}, {0x1BC90, 0x1BC99}, {0x1D400, 0x1D454},
+ {0x1D456, 0x1D49C}, {0x1D49E, 0x1D49F}, {0x1D4A2, 0x1D4A2},
+ {0x1D4A5, 0x1D4A6}, {0x1D4A9, 0x1D4AC}, {0x1D4AE, 0x1D4B9},
+ {0x1D4BB, 0x1D4BB}, {0x1D4BD, 0x1D4C3}, {0x1D4C5, 0x1D505},
+ {0x1D507, 0x1D50A}, {0x1D50D, 0x1D514}, {0x1D516, 0x1D51C},
+ {0x1D51E, 0x1D539}, {0x1D53B, 0x1D53E}, {0x1D540, 0x1D544},
+ {0x1D546, 0x1D546}, {0x1D54A, 0x1D550}, {0x1D552, 0x1D6A6},
+ {0x1D6A8, 0x1D6C0}, {0x1D6C2, 0x1D6DA}, {0x1D6DC, 0x1D6FA},
+ {0x1D6FC, 0x1D714}, {0x1D716, 0x1D734}, {0x1D736, 0x1D74E},
+ {0x1D750, 0x1D76E}, {0x1D770, 0x1D788}, {0x1D78A, 0x1D7A8},
+ {0x1D7AA, 0x1D7C2}, {0x1D7C4, 0x1D7CB}, {0x1DF00, 0x1DF81},
+ {0x1DF90, 0x1DF96}, {0x1DFCD, 0x1DFFF}, {0x1E030, 0x1E06D},
+ {0x1E100, 0x1E12C}, {0x1E137, 0x1E13D}, {0x1E14E, 0x1E14E},
+ {0x1E290, 0x1E2AD}, {0x1E2C0, 0x1E2EB}, {0x1E4D0, 0x1E4EB},
+ {0x1E5D0, 0x1E5ED}, {0x1E5F0, 0x1E5F0}, {0x1E6C0, 0x1E6DE},
+ {0x1E6E0, 0x1E6E2}, {0x1E6E4, 0x1E6E5}, {0x1E6E7, 0x1E6ED},
+ {0x1E6F0, 0x1E6F4}, {0x1E6FE, 0x1E6FF}, {0x1E7E0, 0x1E7E6},
+ {0x1E7E8, 0x1E7EB}, {0x1E7ED, 0x1E7EE}, {0x1E7F0, 0x1E7FE},
+ {0x1E800, 0x1E8C4}, {0x1E900, 0x1E943}, {0x1E94B, 0x1E94B},
+ {0x1EE00, 0x1EE03}, {0x1EE05, 0x1EE1F}, {0x1EE21, 0x1EE22},
+ {0x1EE24, 0x1EE24}, {0x1EE27, 0x1EE27}, {0x1EE29, 0x1EE32},
+ {0x1EE34, 0x1EE37}, {0x1EE39, 0x1EE39}, {0x1EE3B, 0x1EE3B},
+ {0x1EE42, 0x1EE42}, {0x1EE47, 0x1EE47}, {0x1EE49, 0x1EE49},
+ {0x1EE4B, 0x1EE4B}, {0x1EE4D, 0x1EE4F}, {0x1EE51, 0x1EE52},
+ {0x1EE54, 0x1EE54}, {0x1EE57, 0x1EE57}, {0x1EE59, 0x1EE59},
+ {0x1EE5B, 0x1EE5B}, {0x1EE5D, 0x1EE5D}, {0x1EE5F, 0x1EE5F},
+ {0x1EE61, 0x1EE62}, {0x1EE64, 0x1EE64}, {0x1EE67, 0x1EE6A},
+ {0x1EE6C, 0x1EE72}, {0x1EE74, 0x1EE77}, {0x1EE79, 0x1EE7C},
+ {0x1EE7E, 0x1EE7E}, {0x1EE80, 0x1EE89}, {0x1EE8B, 0x1EE9B},
+ {0x1EEA1, 0x1EEA3}, {0x1EEA5, 0x1EEA9}, {0x1EEAB, 0x1EEBB},
+ {0x20000, 0x2A6DF}, {0x2A700, 0x2B81E}, {0x2B820, 0x2CEAD},
+ {0x2CEB0, 0x2EBE0}, {0x2EBF0, 0x2EE5D}, {0x2F800, 0x2FA1D},
+ {0x30000, 0x3134A}, {0x31350, 0x33479}, {0x3D000, 0x3FC3F},
+};
+extern const CharRanges GeneratedXIDStartRanges = GeneratedXIDStartRangesData;
+
+// XID_Continue, excluding XID_Start
+static constexpr UnicodeCharRange GeneratedXIDContinueRangesData[] = {
+ {0x0030, 0x0039}, {0x005F, 0x005F}, {0x00B7, 0x00B7},
+ {0x0300, 0x036F}, {0x0387, 0x0387}, {0x0483, 0x0487},
+ {0x0591, 0x05BD}, {0x05BF, 0x05BF}, {0x05C1, 0x05C2},
+ {0x05C4, 0x05C5}, {0x05C7, 0x05C9}, {0x0610, 0x061A},
+ {0x064B, 0x0669}, {0x0670, 0x0670}, {0x06D6, 0x06DC},
+ {0x06DF, 0x06E4}, {0x06E7, 0x06E8}, {0x06EA, 0x06ED},
+ {0x06F0, 0x06F9}, {0x0711, 0x0711}, {0x0730, 0x074A},
+ {0x07A6, 0x07B0}, {0x07C0, 0x07C9}, {0x07EB, 0x07F3},
+ {0x07FD, 0x07FD}, {0x0816, 0x0819}, {0x081B, 0x0823},
+ {0x0825, 0x0827}, {0x0829, 0x082D}, {0x0859, 0x085B},
+ {0x0897, 0x089F}, {0x08CA, 0x08E1}, {0x08E3, 0x0903},
+ {0x093A, 0x093C}, {0x093E, 0x094F}, {0x0951, 0x0957},
+ {0x0962, 0x0963}, {0x0966, 0x096F}, {0x0981, 0x0983},
+ {0x09BC, 0x09BC}, {0x09BE, 0x09C4}, {0x09C7, 0x09C8},
+ {0x09CB, 0x09CD}, {0x09D7, 0x09D7}, {0x09E2, 0x09E3},
+ {0x09E6, 0x09EF}, {0x09FE, 0x09FE}, {0x0A01, 0x0A03},
+ {0x0A3C, 0x0A3C}, {0x0A3E, 0x0A42}, {0x0A47, 0x0A48},
+ {0x0A4B, 0x0A4D}, {0x0A51, 0x0A51}, {0x0A66, 0x0A71},
+ {0x0A75, 0x0A75}, {0x0A81, 0x0A83}, {0x0ABC, 0x0ABC},
+ {0x0ABE, 0x0AC5}, {0x0AC7, 0x0AC9}, {0x0ACB, 0x0ACD},
+ {0x0AE2, 0x0AE3}, {0x0AE6, 0x0AEF}, {0x0AFA, 0x0AFF},
+ {0x0B01, 0x0B03}, {0x0B3C, 0x0B3C}, {0x0B3E, 0x0B44},
+ {0x0B47, 0x0B48}, {0x0B4B, 0x0B4D}, {0x0B53, 0x0B57},
+ {0x0B62, 0x0B63}, {0x0B66, 0x0B6F}, {0x0B82, 0x0B82},
+ {0x0BBE, 0x0BC2}, {0x0BC6, 0x0BC8}, {0x0BCA, 0x0BCD},
+ {0x0BD7, 0x0BD7}, {0x0BE6, 0x0BEF}, {0x0C00, 0x0C04},
+ {0x0C3C, 0x0C3C}, {0x0C3E, 0x0C44}, {0x0C46, 0x0C48},
+ {0x0C4A, 0x0C4D}, {0x0C55, 0x0C56}, {0x0C62, 0x0C63},
+ {0x0C66, 0x0C6F}, {0x0C81, 0x0C83}, {0x0CBC, 0x0CBC},
+ {0x0CBE, 0x0CC4}, {0x0CC6, 0x0CC8}, {0x0CCA, 0x0CCD},
+ {0x0CD5, 0x0CD6}, {0x0CE2, 0x0CE3}, {0x0CE6, 0x0CEF},
+ {0x0CF3, 0x0CF3}, {0x0D00, 0x0D03}, {0x0D3B, 0x0D3C},
+ {0x0D3E, 0x0D44}, {0x0D46, 0x0D48}, {0x0D4A, 0x0D4D},
+ {0x0D57, 0x0D57}, {0x0D62, 0x0D63}, {0x0D66, 0x0D6F},
+ {0x0D81, 0x0D83}, {0x0DCA, 0x0DCA}, {0x0DCF, 0x0DD4},
+ {0x0DD6, 0x0DD6}, {0x0DD8, 0x0DDF}, {0x0DE6, 0x0DEF},
+ {0x0DF2, 0x0DF3}, {0x0E31, 0x0E31}, {0x0E33, 0x0E3A},
+ {0x0E47, 0x0E4E}, {0x0E50, 0x0E59}, {0x0EB1, 0x0EB1},
+ {0x0EB3, 0x0EBC}, {0x0EC8, 0x0ECE}, {0x0ED0, 0x0ED9},
+ {0x0F18, 0x0F19}, {0x0F20, 0x0F29}, {0x0F35, 0x0F35},
+ {0x0F37, 0x0F37}, {0x0F39, 0x0F39}, {0x0F3E, 0x0F3F},
+ {0x0F71, 0x0F84}, {0x0F86, 0x0F87}, {0x0F8D, 0x0F97},
+ {0x0F99, 0x0FBC}, {0x0FC6, 0x0FC6}, {0x102B, 0x103E},
+ {0x1040, 0x1049}, {0x1056, 0x1059}, {0x105E, 0x1060},
+ {0x1062, 0x1064}, {0x1067, 0x106D}, {0x1071, 0x1074},
+ {0x1082, 0x108D}, {0x108F, 0x109D}, {0x135D, 0x135F},
+ {0x1369, 0x1371}, {0x1712, 0x1715}, {0x1732, 0x1734},
+ {0x1752, 0x1753}, {0x1772, 0x1773}, {0x17B4, 0x17D3},
+ {0x17DD, 0x17DD}, {0x17E0, 0x17E9}, {0x180B, 0x180D},
+ {0x180F, 0x1819}, {0x18A9, 0x18A9}, {0x1920, 0x192B},
+ {0x1930, 0x193B}, {0x1946, 0x194F}, {0x19D0, 0x19DA},
+ {0x1A17, 0x1A1B}, {0x1A55, 0x1A5E}, {0x1A60, 0x1A7C},
+ {0x1A7F, 0x1A89}, {0x1A90, 0x1A99}, {0x1AB0, 0x1ABD},
+ {0x1ABF, 0x1AF0}, {0x1B00, 0x1B04}, {0x1B34, 0x1B44},
+ {0x1B50, 0x1B59}, {0x1B6B, 0x1B73}, {0x1B80, 0x1B82},
+ {0x1BA1, 0x1BAD}, {0x1BB0, 0x1BB9}, {0x1BE6, 0x1BF3},
+ {0x1C24, 0x1C37}, {0x1C40, 0x1C49}, {0x1C50, 0x1C59},
+ {0x1CD0, 0x1CD2}, {0x1CD4, 0x1CE8}, {0x1CED, 0x1CED},
+ {0x1CF4, 0x1CF4}, {0x1CF7, 0x1CF9}, {0x1DC0, 0x1DFF},
+ {0x200C, 0x200D}, {0x203F, 0x2040}, {0x2054, 0x2054},
+ {0x20D0, 0x20DC}, {0x20E1, 0x20E1}, {0x20E5, 0x20F0},
+ {0x2CEF, 0x2CF1}, {0x2D7F, 0x2D7F}, {0x2DE0, 0x2DFF},
+ {0x302A, 0x302F}, {0x3099, 0x309A}, {0x30FB, 0x30FB},
+ {0xA620, 0xA629}, {0xA66F, 0xA66F}, {0xA674, 0xA67D},
+ {0xA69E, 0xA69F}, {0xA6F0, 0xA6F1}, {0xA802, 0xA802},
+ {0xA806, 0xA806}, {0xA80B, 0xA80B}, {0xA823, 0xA827},
+ {0xA82C, 0xA82C}, {0xA880, 0xA881}, {0xA8B4, 0xA8C5},
+ {0xA8D0, 0xA8D9}, {0xA8E0, 0xA8F1}, {0xA8FF, 0xA909},
+ {0xA926, 0xA92D}, {0xA947, 0xA953}, {0xA980, 0xA983},
+ {0xA9B3, 0xA9C0}, {0xA9D0, 0xA9D9}, {0xA9E5, 0xA9E5},
+ {0xA9F0, 0xA9F9}, {0xAA29, 0xAA36}, {0xAA43, 0xAA43},
+ {0xAA4C, 0xAA4D}, {0xAA50, 0xAA59}, {0xAA7B, 0xAA7D},
+ {0xAAB0, 0xAAB0}, {0xAAB2, 0xAAB4}, {0xAAB7, 0xAAB8},
+ {0xAABE, 0xAABF}, {0xAAC1, 0xAAC1}, {0xAAEB, 0xAAEF},
+ {0xAAF5, 0xAAF6}, {0xABE3, 0xABEA}, {0xABEC, 0xABED},
+ {0xABF0, 0xABF9}, {0xFB1E, 0xFB1E}, {0xFE00, 0xFE0F},
+ {0xFE20, 0xFE2F}, {0xFE33, 0xFE34}, {0xFE4D, 0xFE4F},
+ {0xFF10, 0xFF19}, {0xFF3F, 0xFF3F}, {0xFF65, 0xFF65},
+ {0xFF9E, 0xFF9F}, {0x101FD, 0x101FD}, {0x102E0, 0x102E0},
+ {0x10376, 0x1037A}, {0x104A0, 0x104A9}, {0x10A01, 0x10A03},
+ {0x10A05, 0x10A06}, {0x10A0C, 0x10A0F}, {0x10A38, 0x10A3A},
+ {0x10A3F, 0x10A3F}, {0x10AE5, 0x10AE6}, {0x10D24, 0x10D27},
+ {0x10D30, 0x10D39}, {0x10D40, 0x10D49}, {0x10D69, 0x10D6D},
+ {0x10EAB, 0x10EAC}, {0x10ECB, 0x10ECF}, {0x10EF0, 0x10EFF},
+ {0x10F46, 0x10F50}, {0x10F82, 0x10F85}, {0x11000, 0x11002},
+ {0x11038, 0x11046}, {0x11066, 0x11070}, {0x11073, 0x11074},
+ {0x1107F, 0x11082}, {0x110B0, 0x110BA}, {0x110C2, 0x110C2},
+ {0x110F0, 0x110F9}, {0x11100, 0x11102}, {0x11127, 0x11134},
+ {0x11136, 0x1113F}, {0x11145, 0x11146}, {0x11173, 0x11173},
+ {0x11180, 0x11182}, {0x111B3, 0x111C0}, {0x111C9, 0x111CC},
+ {0x111CE, 0x111D9}, {0x1122C, 0x11237}, {0x1123E, 0x1123E},
+ {0x11241, 0x11241}, {0x112DF, 0x112EA}, {0x112F0, 0x112F9},
+ {0x11300, 0x11303}, {0x1133B, 0x1133C}, {0x1133E, 0x11344},
+ {0x11347, 0x11348}, {0x1134B, 0x1134D}, {0x11357, 0x11357},
+ {0x11362, 0x11363}, {0x11366, 0x1136C}, {0x11370, 0x11374},
+ {0x113B8, 0x113C0}, {0x113C2, 0x113C2}, {0x113C5, 0x113C5},
+ {0x113C7, 0x113CA}, {0x113CC, 0x113D0}, {0x113D2, 0x113D2},
+ {0x113E1, 0x113E2}, {0x11435, 0x11446}, {0x11450, 0x11459},
+ {0x1145E, 0x1145E}, {0x114B0, 0x114C3}, {0x114D0, 0x114D9},
+ {0x115AF, 0x115B5}, {0x115B8, 0x115C0}, {0x115DC, 0x115DD},
+ {0x11630, 0x11640}, {0x11650, 0x11659}, {0x116AB, 0x116B7},
+ {0x116C0, 0x116C9}, {0x116D0, 0x116E3}, {0x1171D, 0x1172B},
+ {0x11730, 0x11739}, {0x1182C, 0x1183A}, {0x118E0, 0x118E9},
+ {0x11930, 0x11935}, {0x11937, 0x11938}, {0x1193B, 0x1193E},
+ {0x11940, 0x11940}, {0x11942, 0x11943}, {0x11950, 0x11959},
+ {0x119D1, 0x119D7}, {0x119DA, 0x119E0}, {0x119E4, 0x119E4},
+ {0x11A01, 0x11A0A}, {0x11A33, 0x11A39}, {0x11A3B, 0x11A3E},
+ {0x11A47, 0x11A47}, {0x11A51, 0x11A5B}, {0x11A8A, 0x11A99},
+ {0x11B60, 0x11B67}, {0x11BF0, 0x11BF9}, {0x11C2F, 0x11C36},
+ {0x11C38, 0x11C3F}, {0x11C50, 0x11C59}, {0x11C92, 0x11CA7},
+ {0x11CA9, 0x11CB6}, {0x11D31, 0x11D36}, {0x11D3A, 0x11D3A},
+ {0x11D3C, 0x11D3D}, {0x11D3F, 0x11D45}, {0x11D47, 0x11D47},
+ {0x11D50, 0x11D59}, {0x11D8A, 0x11D8E}, {0x11D90, 0x11D91},
+ {0x11D93, 0x11D97}, {0x11DA0, 0x11DA9}, {0x11DE0, 0x11DE9},
+ {0x11DF0, 0x11DF0}, {0x11EF3, 0x11EF6}, {0x11F00, 0x11F01},
+ {0x11F03, 0x11F03}, {0x11F34, 0x11F3A}, {0x11F3E, 0x11F42},
+ {0x11F50, 0x11F5A}, {0x13440, 0x13440}, {0x13447, 0x13455},
+ {0x1611E, 0x16139}, {0x16A60, 0x16A69}, {0x16AC0, 0x16AC9},
+ {0x16AF0, 0x16AF4}, {0x16B30, 0x16B36}, {0x16B50, 0x16B59},
+ {0x16D70, 0x16D79}, {0x16F4F, 0x16F4F}, {0x16F51, 0x16F87},
+ {0x16F8F, 0x16F92}, {0x16FE4, 0x16FE4}, {0x16FF0, 0x16FF1},
+ {0x1BC9D, 0x1BC9E}, {0x1CCF0, 0x1CCF9}, {0x1CF00, 0x1CF2D},
+ {0x1CF30, 0x1CF46}, {0x1D127, 0x1D128}, {0x1D165, 0x1D169},
+ {0x1D16D, 0x1D172}, {0x1D17B, 0x1D182}, {0x1D185, 0x1D18B},
+ {0x1D1AA, 0x1D1AD}, {0x1D242, 0x1D244}, {0x1D250, 0x1D252},
+ {0x1D25B, 0x1D25C}, {0x1D25F, 0x1D25F}, {0x1D280, 0x1D281},
+ {0x1D7CE, 0x1D7FF}, {0x1DA00, 0x1DA36}, {0x1DA3B, 0x1DA6C},
+ {0x1DA75, 0x1DA75}, {0x1DA84, 0x1DA84}, {0x1DA9B, 0x1DA9F},
+ {0x1DAA1, 0x1DAAF}, {0x1E000, 0x1E006}, {0x1E008, 0x1E018},
+ {0x1E01B, 0x1E021}, {0x1E023, 0x1E024}, {0x1E026, 0x1E02A},
+ {0x1E08F, 0x1E08F}, {0x1E130, 0x1E136}, {0x1E140, 0x1E149},
+ {0x1E2AE, 0x1E2AE}, {0x1E2EC, 0x1E2F9}, {0x1E4EC, 0x1E4F9},
+ {0x1E5EE, 0x1E5EF}, {0x1E5F1, 0x1E5FA}, {0x1E6E3, 0x1E6E3},
+ {0x1E6E6, 0x1E6E6}, {0x1E6EE, 0x1E6EF}, {0x1E6F5, 0x1E6F5},
+ {0x1E8D0, 0x1E8D6}, {0x1E944, 0x1E94A}, {0x1E950, 0x1E959},
+ {0x1FBF0, 0x1FBF9}, {0xE0100, 0xE01EF},
+};
+extern const CharRanges GeneratedXIDContinueRanges =
+ GeneratedXIDContinueRangesData;
+
+// ID_Compat_Math_Start
+static constexpr UnicodeCharRange
+ GeneratedMathematicalNotationProfileIDStartRangesData[] = {
+ {0x2202, 0x2202}, {0x2207, 0x2207}, {0x221E, 0x221E},
+ {0x1D6C1, 0x1D6C1}, {0x1D6DB, 0x1D6DB}, {0x1D6FB, 0x1D6FB},
+ {0x1D715, 0x1D715}, {0x1D735, 0x1D735}, {0x1D74F, 0x1D74F},
+ {0x1D76F, 0x1D76F}, {0x1D789, 0x1D789}, {0x1D7A9, 0x1D7A9},
+ {0x1D7C3, 0x1D7C3},
+};
+extern const CharRanges GeneratedMathematicalNotationProfileIDStartRanges =
+ GeneratedMathematicalNotationProfileIDStartRangesData;
+
+// ID_Compat_Math_Continue, excluding ID_Compat_Math_Start
+static constexpr UnicodeCharRange
+ GeneratedMathematicalNotationProfileIDContinueRangesData[] = {
+ {0x00B2, 0x00B3}, {0x00B9, 0x00B9}, {0x2070, 0x2070},
+ {0x2074, 0x207E}, {0x2080, 0x208E},
+};
+extern const CharRanges GeneratedMathematicalNotationProfileIDContinueRanges =
+ GeneratedMathematicalNotationProfileIDContinueRangesData;
+
+} // namespace clang
diff --git a/llvm/lib/Support/CMakeLists.txt b/llvm/lib/Support/CMakeLists.txt
index 65ec233299495..40c5c392addd5 100644
--- a/llvm/lib/Support/CMakeLists.txt
+++ b/llvm/lib/Support/CMakeLists.txt
@@ -290,6 +290,7 @@ add_llvm_component_library(LLVMSupport
Twine.cpp
Unicode.cpp
UnicodeCaseFold.cpp
+ UnicodeCharSetsGenerated.cpp
UnicodeNameToCodepoint.cpp
UnicodeNameToCodepointGenerated.cpp
VersionTuple.cpp
diff --git a/llvm/lib/Support/Unicode.cpp b/llvm/lib/Support/Unicode.cpp
index cbf772b41f972..6c89270d29f76 100644
--- a/llvm/lib/Support/Unicode.cpp
+++ b/llvm/lib/Support/Unicode.cpp
@@ -19,266 +19,18 @@ namespace llvm {
namespace sys {
namespace unicode {
+extern const UnicodeCharSet::CharRanges GeneratedPrintableRanges;
+extern const UnicodeCharSet::CharRanges GeneratedFormatCharacterRanges;
+extern const UnicodeCharSet::CharRanges GeneratedCombiningCharacterRanges;
+extern const UnicodeCharSet::CharRanges GeneratedDoubleWidthCharacterRanges;
+
/// Unicode code points of the categories L, M, N, P, S and Zs are considered
/// printable.
/// In addition, U+00AD SOFT HYPHEN is also considered printable, as
/// it's actually displayed on most terminals. \return true if the character is
/// considered printable.
bool isPrintable(int UCS) {
- // https://unicode.org/Public/18.0.0/ucdxml/
- static const UnicodeCharRange PrintableRanges[] = {
- {0x0020, 0x007E}, {0x00A0, 0x00AC}, {0x00AE, 0x0377},
- {0x037A, 0x037F}, {0x0384, 0x038A}, {0x038C, 0x038C},
- {0x038E, 0x03A1}, {0x03A3, 0x052F}, {0x0531, 0x0556},
- {0x0558, 0x058F}, {0x0591, 0x05C9}, {0x05D0, 0x05EA},
- {0x05EF, 0x05F4}, {0x0606, 0x061B}, {0x061D, 0x06DC},
- {0x06DE, 0x070D}, {0x0710, 0x074A}, {0x074D, 0x07B1},
- {0x07C0, 0x07FA}, {0x07FD, 0x082D}, {0x0830, 0x083E},
- {0x0840, 0x085B}, {0x085E, 0x085E}, {0x0860, 0x086A},
- {0x0870, 0x088F}, {0x0897, 0x08E1}, {0x08E3, 0x0983},
- {0x0985, 0x098C}, {0x098F, 0x0990}, {0x0993, 0x09A8},
- {0x09AA, 0x09B0}, {0x09B2, 0x09B2}, {0x09B6, 0x09B9},
- {0x09BC, 0x09C4}, {0x09C7, 0x09C8}, {0x09CB, 0x09CE},
- {0x09D7, 0x09D7}, {0x09DC, 0x09DD}, {0x09DF, 0x09E3},
- {0x09E6, 0x09FE}, {0x0A01, 0x0A03}, {0x0A05, 0x0A0A},
- {0x0A0F, 0x0A10}, {0x0A13, 0x0A28}, {0x0A2A, 0x0A30},
- {0x0A32, 0x0A33}, {0x0A35, 0x0A36}, {0x0A38, 0x0A39},
- {0x0A3C, 0x0A3C}, {0x0A3E, 0x0A42}, {0x0A47, 0x0A48},
- {0x0A4B, 0x0A4D}, {0x0A51, 0x0A51}, {0x0A59, 0x0A5C},
- {0x0A5E, 0x0A5E}, {0x0A66, 0x0A76}, {0x0A81, 0x0A83},
- {0x0A85, 0x0A8D}, {0x0A8F, 0x0A91}, {0x0A93, 0x0AA8},
- {0x0AAA, 0x0AB0}, {0x0AB2, 0x0AB3}, {0x0AB5, 0x0AB9},
- {0x0ABC, 0x0AC5}, {0x0AC7, 0x0AC9}, {0x0ACB, 0x0ACD},
- {0x0AD0, 0x0AD0}, {0x0AE0, 0x0AE3}, {0x0AE6, 0x0AF1},
- {0x0AF9, 0x0AFF}, {0x0B01, 0x0B03}, {0x0B05, 0x0B0C},
- {0x0B0F, 0x0B10}, {0x0B13, 0x0B28}, {0x0B2A, 0x0B30},
- {0x0B32, 0x0B33}, {0x0B35, 0x0B39}, {0x0B3C, 0x0B44},
- {0x0B47, 0x0B48}, {0x0B4B, 0x0B4D}, {0x0B53, 0x0B57},
- {0x0B5C, 0x0B5D}, {0x0B5F, 0x0B63}, {0x0B66, 0x0B77},
- {0x0B82, 0x0B83}, {0x0B85, 0x0B8A}, {0x0B8E, 0x0B90},
- {0x0B92, 0x0B95}, {0x0B99, 0x0B9A}, {0x0B9C, 0x0B9C},
- {0x0B9E, 0x0B9F}, {0x0BA3, 0x0BA4}, {0x0BA8, 0x0BAA},
- {0x0BAE, 0x0BB9}, {0x0BBE, 0x0BC2}, {0x0BC6, 0x0BC8},
- {0x0BCA, 0x0BCD}, {0x0BD0, 0x0BD0}, {0x0BD7, 0x0BD7},
- {0x0BE6, 0x0BFA}, {0x0C00, 0x0C0C}, {0x0C0E, 0x0C10},
- {0x0C12, 0x0C28}, {0x0C2A, 0x0C39}, {0x0C3C, 0x0C44},
- {0x0C46, 0x0C48}, {0x0C4A, 0x0C4D}, {0x0C55, 0x0C56},
- {0x0C58, 0x0C5A}, {0x0C5C, 0x0C5D}, {0x0C60, 0x0C63},
- {0x0C66, 0x0C6F}, {0x0C77, 0x0C8C}, {0x0C8E, 0x0C90},
- {0x0C92, 0x0CA8}, {0x0CAA, 0x0CB3}, {0x0CB5, 0x0CB9},
- {0x0CBC, 0x0CC4}, {0x0CC6, 0x0CC8}, {0x0CCA, 0x0CCD},
- {0x0CD5, 0x0CD6}, {0x0CDC, 0x0CDE}, {0x0CE0, 0x0CE3},
- {0x0CE6, 0x0CEF}, {0x0CF1, 0x0CF3}, {0x0D00, 0x0D0C},
- {0x0D0E, 0x0D10}, {0x0D12, 0x0D44}, {0x0D46, 0x0D48},
- {0x0D4A, 0x0D4F}, {0x0D54, 0x0D63}, {0x0D66, 0x0D7F},
- {0x0D81, 0x0D83}, {0x0D85, 0x0D96}, {0x0D9A, 0x0DB1},
- {0x0DB3, 0x0DBB}, {0x0DBD, 0x0DBD}, {0x0DC0, 0x0DC6},
- {0x0DCA, 0x0DCA}, {0x0DCF, 0x0DD4}, {0x0DD6, 0x0DD6},
- {0x0DD8, 0x0DDF}, {0x0DE6, 0x0DEF}, {0x0DF2, 0x0DF4},
- {0x0E01, 0x0E3A}, {0x0E3F, 0x0E5B}, {0x0E81, 0x0E82},
- {0x0E84, 0x0E84}, {0x0E86, 0x0E8A}, {0x0E8C, 0x0EA3},
- {0x0EA5, 0x0EA5}, {0x0EA7, 0x0EBD}, {0x0EC0, 0x0EC4},
- {0x0EC6, 0x0EC6}, {0x0EC8, 0x0ECE}, {0x0ED0, 0x0ED9},
- {0x0EDC, 0x0EDF}, {0x0F00, 0x0F47}, {0x0F49, 0x0F6C},
- {0x0F71, 0x0F97}, {0x0F99, 0x0FBC}, {0x0FBE, 0x0FCC},
- {0x0FCE, 0x0FDA}, {0x1000, 0x10C5}, {0x10C7, 0x10C7},
- {0x10CD, 0x10CD}, {0x10D0, 0x1248}, {0x124A, 0x124D},
- {0x1250, 0x1256}, {0x1258, 0x1258}, {0x125A, 0x125D},
- {0x1260, 0x1288}, {0x128A, 0x128D}, {0x1290, 0x12B0},
- {0x12B2, 0x12B5}, {0x12B8, 0x12BE}, {0x12C0, 0x12C0},
- {0x12C2, 0x12C5}, {0x12C8, 0x12D6}, {0x12D8, 0x1310},
- {0x1312, 0x1315}, {0x1318, 0x135A}, {0x135D, 0x137C},
- {0x1380, 0x1399}, {0x13A0, 0x13F5}, {0x13F8, 0x13FD},
- {0x1400, 0x169C}, {0x16A0, 0x16F8}, {0x1700, 0x1715},
- {0x171F, 0x1736}, {0x1740, 0x1753}, {0x1760, 0x176C},
- {0x176E, 0x1770}, {0x1772, 0x1773}, {0x1780, 0x17DD},
- {0x17E0, 0x17E9}, {0x17F0, 0x17F9}, {0x1800, 0x180D},
- {0x180F, 0x1819}, {0x1820, 0x1878}, {0x1880, 0x18AA},
- {0x18B0, 0x18F5}, {0x1900, 0x191E}, {0x1920, 0x192B},
- {0x1930, 0x193B}, {0x1940, 0x1940}, {0x1944, 0x196D},
- {0x1970, 0x1974}, {0x1980, 0x19AB}, {0x19B0, 0x19C9},
- {0x19D0, 0x19DA}, {0x19DE, 0x1A1B}, {0x1A1E, 0x1A5E},
- {0x1A60, 0x1A7C}, {0x1A7F, 0x1A89}, {0x1A90, 0x1A99},
- {0x1AA0, 0x1AAD}, {0x1AB0, 0x1AF0}, {0x1B00, 0x1B4C},
- {0x1B4E, 0x1BF3}, {0x1BFC, 0x1C37}, {0x1C3B, 0x1C49},
- {0x1C4D, 0x1C8A}, {0x1C90, 0x1CBA}, {0x1CBD, 0x1CC7},
- {0x1CD0, 0x1CFA}, {0x1D00, 0x1F15}, {0x1F18, 0x1F1D},
- {0x1F20, 0x1F45}, {0x1F48, 0x1F4D}, {0x1F50, 0x1F57},
- {0x1F59, 0x1F59}, {0x1F5B, 0x1F5B}, {0x1F5D, 0x1F5D},
- {0x1F5F, 0x1F7D}, {0x1F80, 0x1FB4}, {0x1FB6, 0x1FC4},
- {0x1FC6, 0x1FD3}, {0x1FD6, 0x1FDB}, {0x1FDD, 0x1FEF},
- {0x1FF2, 0x1FF4}, {0x1FF6, 0x1FFE}, {0x2000, 0x200A},
- {0x2010, 0x2027}, {0x202F, 0x205F}, {0x2070, 0x2071},
- {0x2074, 0x20C4}, {0x20D0, 0x20F0}, {0x2100, 0x218B},
- {0x2190, 0x2429}, {0x2440, 0x244A}, {0x2460, 0x2B73},
- {0x2B76, 0x2CF3}, {0x2CF9, 0x2D25}, {0x2D27, 0x2D27},
- {0x2D2D, 0x2D2D}, {0x2D30, 0x2D67}, {0x2D6F, 0x2D70},
- {0x2D7F, 0x2D96}, {0x2DA0, 0x2DA6}, {0x2DA8, 0x2DAE},
- {0x2DB0, 0x2DB6}, {0x2DB8, 0x2DBE}, {0x2DC0, 0x2DC6},
- {0x2DC8, 0x2DCE}, {0x2DD0, 0x2DD6}, {0x2DD8, 0x2DDE},
- {0x2DE0, 0x2E5D}, {0x2E60, 0x2E63}, {0x2E80, 0x2E99},
- {0x2E9B, 0x2EF3}, {0x2F00, 0x2FD5}, {0x2FF0, 0x303F},
- {0x3041, 0x3096}, {0x3099, 0x30FF}, {0x3105, 0x312F},
- {0x3131, 0x318E}, {0x3190, 0x31E5}, {0x31EF, 0x321E},
- {0x3220, 0xA48C}, {0xA490, 0xA4C6}, {0xA4D0, 0xA62B},
- {0xA640, 0xA6F7}, {0xA700, 0xA7DD}, {0xA7E2, 0xA7E2},
- {0xA7F1, 0xA82C}, {0xA830, 0xA839}, {0xA840, 0xA877},
- {0xA880, 0xA8C5}, {0xA8CE, 0xA8D9}, {0xA8E0, 0xA953},
- {0xA95F, 0xA97C}, {0xA980, 0xA9CD}, {0xA9CF, 0xA9D9},
- {0xA9DE, 0xA9FE}, {0xAA00, 0xAA36}, {0xAA40, 0xAA4D},
- {0xAA50, 0xAA59}, {0xAA5C, 0xAAC2}, {0xAADB, 0xAAF6},
- {0xAB01, 0xAB06}, {0xAB09, 0xAB0E}, {0xAB11, 0xAB16},
- {0xAB20, 0xAB26}, {0xAB28, 0xAB2E}, {0xAB30, 0xAB6D},
- {0xAB70, 0xABED}, {0xABF0, 0xABF9}, {0xAC00, 0xD7A3},
- {0xD7B0, 0xD7C6}, {0xD7CB, 0xD7FB}, {0xF900, 0xFA6D},
- {0xFA70, 0xFAD9}, {0xFB00, 0xFB06}, {0xFB13, 0xFB17},
- {0xFB1D, 0xFB36}, {0xFB38, 0xFB3C}, {0xFB3E, 0xFB3E},
- {0xFB40, 0xFB41}, {0xFB43, 0xFB44}, {0xFB46, 0xFDCF},
- {0xFDF0, 0xFE19}, {0xFE20, 0xFE52}, {0xFE54, 0xFE66},
- {0xFE68, 0xFE6B}, {0xFE70, 0xFE74}, {0xFE76, 0xFEFC},
- {0xFF01, 0xFFBE}, {0xFFC2, 0xFFC7}, {0xFFCA, 0xFFCF},
- {0xFFD2, 0xFFD7}, {0xFFDA, 0xFFDC}, {0xFFE0, 0xFFE6},
- {0xFFE8, 0xFFEE}, {0xFFFC, 0xFFFD}, {0x10000, 0x1000B},
- {0x1000D, 0x10026}, {0x10028, 0x1003A}, {0x1003C, 0x1003D},
- {0x1003F, 0x1004D}, {0x10050, 0x1005D}, {0x10080, 0x100FA},
- {0x10100, 0x10102}, {0x10107, 0x10133}, {0x10137, 0x1018E},
- {0x10190, 0x1019C}, {0x101A0, 0x101A0}, {0x101D0, 0x101FD},
- {0x10280, 0x1029C}, {0x102A0, 0x102D0}, {0x102E0, 0x102FB},
- {0x10300, 0x10323}, {0x1032D, 0x1034A}, {0x10350, 0x1037A},
- {0x10380, 0x1039D}, {0x1039F, 0x103C3}, {0x103C8, 0x103D5},
- {0x10400, 0x1049D}, {0x104A0, 0x104A9}, {0x104B0, 0x104D3},
- {0x104D8, 0x104FB}, {0x10500, 0x10527}, {0x10530, 0x10563},
- {0x1056F, 0x1057A}, {0x1057C, 0x1058A}, {0x1058C, 0x10592},
- {0x10594, 0x10595}, {0x10597, 0x105A1}, {0x105A3, 0x105B1},
- {0x105B3, 0x105B9}, {0x105BB, 0x105BC}, {0x105C0, 0x105F3},
- {0x10600, 0x10736}, {0x10740, 0x10755}, {0x10760, 0x10767},
- {0x10780, 0x10785}, {0x10787, 0x107B0}, {0x107B2, 0x107BF},
- {0x10800, 0x10805}, {0x10808, 0x10808}, {0x1080A, 0x10835},
- {0x10837, 0x10838}, {0x1083C, 0x1083C}, {0x1083F, 0x10855},
- {0x10857, 0x1089E}, {0x108A7, 0x108AF}, {0x108E0, 0x108F2},
- {0x108F4, 0x108F5}, {0x108FB, 0x1091B}, {0x1091F, 0x10939},
- {0x1093F, 0x10959}, {0x10980, 0x109B7}, {0x109BC, 0x109CF},
- {0x109D2, 0x10A03}, {0x10A05, 0x10A06}, {0x10A0C, 0x10A13},
- {0x10A15, 0x10A17}, {0x10A19, 0x10A35}, {0x10A38, 0x10A3A},
- {0x10A3F, 0x10A48}, {0x10A50, 0x10A58}, {0x10A60, 0x10A9F},
- {0x10AC0, 0x10AE6}, {0x10AEB, 0x10AF6}, {0x10B00, 0x10B35},
- {0x10B39, 0x10B55}, {0x10B58, 0x10B72}, {0x10B78, 0x10B91},
- {0x10B99, 0x10B9C}, {0x10BA9, 0x10BAF}, {0x10C00, 0x10C48},
- {0x10C80, 0x10CB2}, {0x10CC0, 0x10CF2}, {0x10CFA, 0x10D27},
- {0x10D30, 0x10D39}, {0x10D40, 0x10D65}, {0x10D69, 0x10D85},
- {0x10D8E, 0x10D8F}, {0x10E60, 0x10E7E}, {0x10E80, 0x10EA9},
- {0x10EAB, 0x10EAD}, {0x10EB0, 0x10EB1}, {0x10EC2, 0x10EC7},
- {0x10EC9, 0x10EEE}, {0x10EF0, 0x10F27}, {0x10F30, 0x10F59},
- {0x10F70, 0x10F89}, {0x10FB0, 0x10FCB}, {0x10FE0, 0x10FF6},
- {0x11000, 0x1104D}, {0x11052, 0x11075}, {0x1107F, 0x110BC},
- {0x110BE, 0x110C2}, {0x110D0, 0x110E8}, {0x110F0, 0x110F9},
- {0x11100, 0x11134}, {0x11136, 0x11147}, {0x11150, 0x11176},
- {0x11180, 0x111DF}, {0x111E1, 0x111F4}, {0x11200, 0x11211},
- {0x11213, 0x11241}, {0x11280, 0x11286}, {0x11288, 0x11288},
- {0x1128A, 0x1128D}, {0x1128F, 0x1129D}, {0x1129F, 0x112A9},
- {0x112B0, 0x112EA}, {0x112F0, 0x112F9}, {0x11300, 0x11303},
- {0x11305, 0x1130C}, {0x1130F, 0x11310}, {0x11313, 0x11328},
- {0x1132A, 0x11330}, {0x11332, 0x11333}, {0x11335, 0x11339},
- {0x1133B, 0x11344}, {0x11347, 0x11348}, {0x1134B, 0x1134D},
- {0x11350, 0x11350}, {0x11357, 0x11357}, {0x1135D, 0x11363},
- {0x11366, 0x1136C}, {0x11370, 0x11374}, {0x11380, 0x11389},
- {0x1138B, 0x1138B}, {0x1138E, 0x1138E}, {0x11390, 0x113B5},
- {0x113B7, 0x113C0}, {0x113C2, 0x113C2}, {0x113C5, 0x113C5},
- {0x113C7, 0x113CA}, {0x113CC, 0x113D5}, {0x113D7, 0x113D8},
- {0x113E1, 0x113E2}, {0x11400, 0x1145B}, {0x1145D, 0x11461},
- {0x11480, 0x114C7}, {0x114D0, 0x114D9}, {0x11580, 0x115B5},
- {0x115B8, 0x115DD}, {0x11600, 0x11644}, {0x11650, 0x11659},
- {0x11660, 0x1166C}, {0x11680, 0x116B9}, {0x116C0, 0x116C9},
- {0x116D0, 0x116E3}, {0x11700, 0x1171A}, {0x1171D, 0x1172B},
- {0x11730, 0x11746}, {0x11800, 0x1183B}, {0x118A0, 0x118F2},
- {0x118FF, 0x11906}, {0x11909, 0x11909}, {0x1190C, 0x11913},
- {0x11915, 0x11916}, {0x11918, 0x11935}, {0x11937, 0x11938},
- {0x1193B, 0x11946}, {0x11950, 0x11959}, {0x119A0, 0x119A7},
- {0x119AA, 0x119D7}, {0x119DA, 0x119E4}, {0x11A00, 0x11A47},
- {0x11A50, 0x11AA2}, {0x11AB0, 0x11AF8}, {0x11B00, 0x11B0A},
- {0x11B60, 0x11B67}, {0x11BC0, 0x11BE1}, {0x11BF0, 0x11BF9},
- {0x11C00, 0x11C08}, {0x11C0A, 0x11C36}, {0x11C38, 0x11C45},
- {0x11C50, 0x11C6C}, {0x11C70, 0x11C8F}, {0x11C92, 0x11CA7},
- {0x11CA9, 0x11CB6}, {0x11D00, 0x11D06}, {0x11D08, 0x11D09},
- {0x11D0B, 0x11D36}, {0x11D3A, 0x11D3A}, {0x11D3C, 0x11D3D},
- {0x11D3F, 0x11D47}, {0x11D50, 0x11D59}, {0x11D60, 0x11D65},
- {0x11D67, 0x11D68}, {0x11D6A, 0x11D8E}, {0x11D90, 0x11D91},
- {0x11D93, 0x11D98}, {0x11DA0, 0x11DA9}, {0x11DB0, 0x11DDB},
- {0x11DE0, 0x11DE9}, {0x11DF0, 0x11DF1}, {0x11EE0, 0x11EF8},
- {0x11F00, 0x11F10}, {0x11F12, 0x11F3A}, {0x11F3E, 0x11F5A},
- {0x11FB0, 0x11FB0}, {0x11FC0, 0x11FF1}, {0x11FFF, 0x12399},
- {0x12400, 0x12543}, {0x12550, 0x12686}, {0x12F90, 0x12FF2},
- {0x13000, 0x1342F}, {0x13440, 0x13455}, {0x13460, 0x143FA},
- {0x14400, 0x14646}, {0x16100, 0x16139}, {0x16800, 0x16A38},
- {0x16A40, 0x16A5E}, {0x16A60, 0x16A69}, {0x16A6E, 0x16ABE},
- {0x16AC0, 0x16AC9}, {0x16AD0, 0x16AED}, {0x16AF0, 0x16AF5},
- {0x16B00, 0x16B45}, {0x16B50, 0x16B59}, {0x16B5B, 0x16B61},
- {0x16B63, 0x16B77}, {0x16B7D, 0x16B8F}, {0x16D40, 0x16D79},
- {0x16D80, 0x16D9D}, {0x16DA0, 0x16DA9}, {0x16E40, 0x16E9A},
- {0x16EA0, 0x16EB8}, {0x16EBB, 0x16ED3}, {0x16F00, 0x16F4A},
- {0x16F4F, 0x16F87}, {0x16F8F, 0x16F9F}, {0x16FE0, 0x16FE4},
- {0x16FF0, 0x16FF6}, {0x17000, 0x18CDA}, {0x18CFF, 0x18D20},
- {0x18D80, 0x18DF2}, {0x18E00, 0x19191}, {0x191A0, 0x191D2},
- {0x1AFF0, 0x1AFF3}, {0x1AFF5, 0x1AFFB}, {0x1AFFD, 0x1AFFE},
- {0x1B000, 0x1B128}, {0x1B132, 0x1B132}, {0x1B150, 0x1B152},
- {0x1B155, 0x1B155}, {0x1B164, 0x1B168}, {0x1B170, 0x1B2FB},
- {0x1BC00, 0x1BC6A}, {0x1BC70, 0x1BC7C}, {0x1BC80, 0x1BC88},
- {0x1BC90, 0x1BC99}, {0x1BC9C, 0x1BC9F}, {0x1CC00, 0x1CCFC},
- {0x1CD00, 0x1CEB3}, {0x1CEBA, 0x1CED0}, {0x1CED2, 0x1CED4},
- {0x1CEDD, 0x1CEFD}, {0x1CF00, 0x1CF2D}, {0x1CF30, 0x1CF46},
- {0x1CF50, 0x1CFC3}, {0x1D000, 0x1D0F5}, {0x1D100, 0x1D172},
- {0x1D17B, 0x1D245}, {0x1D250, 0x1D281}, {0x1D2C0, 0x1D2D3},
- {0x1D2E0, 0x1D2F3}, {0x1D300, 0x1D356}, {0x1D360, 0x1D378},
- {0x1D400, 0x1D454}, {0x1D456, 0x1D49C}, {0x1D49E, 0x1D49F},
- {0x1D4A2, 0x1D4A2}, {0x1D4A5, 0x1D4A6}, {0x1D4A9, 0x1D4AC},
- {0x1D4AE, 0x1D4B9}, {0x1D4BB, 0x1D4BB}, {0x1D4BD, 0x1D4C3},
- {0x1D4C5, 0x1D505}, {0x1D507, 0x1D50A}, {0x1D50D, 0x1D514},
- {0x1D516, 0x1D51C}, {0x1D51E, 0x1D539}, {0x1D53B, 0x1D53E},
- {0x1D540, 0x1D544}, {0x1D546, 0x1D546}, {0x1D54A, 0x1D550},
- {0x1D552, 0x1D6A6}, {0x1D6A8, 0x1D7CB}, {0x1D7CE, 0x1DA8B},
- {0x1DA9B, 0x1DA9F}, {0x1DAA1, 0x1DAAF}, {0x1DB00, 0x1DB1C},
- {0x1DF00, 0x1DF81}, {0x1DF90, 0x1DF96}, {0x1DFCD, 0x1E006},
- {0x1E008, 0x1E018}, {0x1E01B, 0x1E021}, {0x1E023, 0x1E024},
- {0x1E026, 0x1E02A}, {0x1E030, 0x1E06D}, {0x1E08F, 0x1E08F},
- {0x1E100, 0x1E12C}, {0x1E130, 0x1E13D}, {0x1E140, 0x1E149},
- {0x1E14E, 0x1E14F}, {0x1E290, 0x1E2AE}, {0x1E2C0, 0x1E2F9},
- {0x1E2FF, 0x1E2FF}, {0x1E4D0, 0x1E4F9}, {0x1E5D0, 0x1E5FA},
- {0x1E5FF, 0x1E5FF}, {0x1E6C0, 0x1E6DE}, {0x1E6E0, 0x1E6F5},
- {0x1E6FE, 0x1E6FF}, {0x1E7E0, 0x1E7E6}, {0x1E7E8, 0x1E7EB},
- {0x1E7ED, 0x1E7EE}, {0x1E7F0, 0x1E7FE}, {0x1E800, 0x1E8C4},
- {0x1E8C7, 0x1E8D6}, {0x1E900, 0x1E94B}, {0x1E950, 0x1E959},
- {0x1E95E, 0x1E95F}, {0x1EC71, 0x1ECB4}, {0x1ED01, 0x1ED3D},
- {0x1EE00, 0x1EE03}, {0x1EE05, 0x1EE1F}, {0x1EE21, 0x1EE22},
- {0x1EE24, 0x1EE24}, {0x1EE27, 0x1EE27}, {0x1EE29, 0x1EE32},
- {0x1EE34, 0x1EE37}, {0x1EE39, 0x1EE39}, {0x1EE3B, 0x1EE3B},
- {0x1EE42, 0x1EE42}, {0x1EE47, 0x1EE47}, {0x1EE49, 0x1EE49},
- {0x1EE4B, 0x1EE4B}, {0x1EE4D, 0x1EE4F}, {0x1EE51, 0x1EE52},
- {0x1EE54, 0x1EE54}, {0x1EE57, 0x1EE57}, {0x1EE59, 0x1EE59},
- {0x1EE5B, 0x1EE5B}, {0x1EE5D, 0x1EE5D}, {0x1EE5F, 0x1EE5F},
- {0x1EE61, 0x1EE62}, {0x1EE64, 0x1EE64}, {0x1EE67, 0x1EE6A},
- {0x1EE6C, 0x1EE72}, {0x1EE74, 0x1EE77}, {0x1EE79, 0x1EE7C},
- {0x1EE7E, 0x1EE7E}, {0x1EE80, 0x1EE89}, {0x1EE8B, 0x1EE9B},
- {0x1EEA1, 0x1EEA3}, {0x1EEA5, 0x1EEA9}, {0x1EEAB, 0x1EEBB},
- {0x1EEF0, 0x1EEF1}, {0x1F000, 0x1F02B}, {0x1F030, 0x1F093},
- {0x1F0A0, 0x1F0AE}, {0x1F0B1, 0x1F0BF}, {0x1F0C1, 0x1F0CF},
- {0x1F0D1, 0x1F0F5}, {0x1F100, 0x1F1AE}, {0x1F1E6, 0x1F202},
- {0x1F210, 0x1F23B}, {0x1F240, 0x1F248}, {0x1F250, 0x1F251},
- {0x1F260, 0x1F265}, {0x1F300, 0x1F6D9}, {0x1F6DC, 0x1F6EC},
- {0x1F6F0, 0x1F6FC}, {0x1F700, 0x1F7DB}, {0x1F7E0, 0x1F7EB},
- {0x1F7F0, 0x1F80B}, {0x1F810, 0x1F847}, {0x1F850, 0x1F859},
- {0x1F860, 0x1F887}, {0x1F890, 0x1F8AD}, {0x1F8B0, 0x1F8BB},
- {0x1F8C0, 0x1F8C1}, {0x1F8D0, 0x1F8D8}, {0x1F900, 0x1FA57},
- {0x1FA60, 0x1FA6D}, {0x1FA70, 0x1FA7C}, {0x1FA80, 0x1FAC6},
- {0x1FAC8, 0x1FAC8}, {0x1FACC, 0x1FADD}, {0x1FADF, 0x1FAEB},
- {0x1FAEF, 0x1FAFA}, {0x1FB00, 0x1FB92}, {0x1FB94, 0x1FBFA},
- {0x20000, 0x2A6DF}, {0x2A700, 0x2B81E}, {0x2B820, 0x2CEAD},
- {0x2CEB0, 0x2EBE0}, {0x2EBF0, 0x2EE5D}, {0x2F800, 0x2FA1D},
- {0x30000, 0x3134A}, {0x31350, 0x33479}, {0x3D000, 0x3FC3F},
- {0xE0100, 0xE01EF},
-
- };
-
- static const UnicodeCharSet Printables(PrintableRanges);
+ static const UnicodeCharSet Printables(GeneratedPrintableRanges);
// Clang special cases 0x00AD (SOFT HYPHEN) which is rendered as an actual
// hyphen in most terminals.
return UCS == 0x00AD || Printables.contains(UCS);
@@ -287,19 +39,7 @@ bool isPrintable(int UCS) {
/// Unicode code points of the Cf category are considered
/// formatting characters.
bool isFormatting(int UCS) {
-
- // https://unicode.org/Public/18.0.0/ucdxml/
- static const UnicodeCharRange Cf[] = {
- {0x00AD, 0x00AD}, {0x0600, 0x0605}, {0x061C, 0x061C},
- {0x06DD, 0x06DD}, {0x070F, 0x070F}, {0x0890, 0x0891},
- {0x08E2, 0x08E2}, {0x180E, 0x180E}, {0x200B, 0x200F},
- {0x202A, 0x202E}, {0x2060, 0x2064}, {0x2066, 0x206F},
- {0xFEFF, 0xFEFF}, {0xFFF9, 0xFFFB}, {0x110BD, 0x110BD},
- {0x110CD, 0x110CD}, {0x13430, 0x1343F}, {0x1BCA0, 0x1BCA3},
- {0x1D173, 0x1D17A}, {0xE0001, 0xE0001}, {0xE0020, 0xE007F},
- };
-
- static const UnicodeCharSet Format(Cf);
+ static const UnicodeCharSet Format(GeneratedFormatCharacterRanges);
return Format.contains(UCS);
}
@@ -318,188 +58,14 @@ static inline int charWidth(int UCS) {
if (!isPrintable(UCS))
return ErrorNonPrintableCharacter;
- // Sorted list of non-spacing and enclosing combining mark intervals as
- // defined in "3.6 Combination" of
- // https://www.unicode.org/versions/Unicode18.0.0/UnicodeStandard-18.0.pdf
- static const UnicodeCharRange CombiningCharacterRanges[] = {
- {0x0300, 0x036F}, {0x0483, 0x0489}, {0x0591, 0x05BD},
- {0x05BF, 0x05BF}, {0x05C1, 0x05C2}, {0x05C4, 0x05C5},
- {0x05C7, 0x05C9}, {0x0610, 0x061A}, {0x064B, 0x065F},
- {0x0670, 0x0670}, {0x06D6, 0x06DC}, {0x06DF, 0x06E4},
- {0x06E7, 0x06E8}, {0x06EA, 0x06ED}, {0x0711, 0x0711},
- {0x0730, 0x074A}, {0x07A6, 0x07B0}, {0x07EB, 0x07F3},
- {0x07FD, 0x07FD}, {0x0816, 0x0819}, {0x081B, 0x0823},
- {0x0825, 0x0827}, {0x0829, 0x082D}, {0x0859, 0x085B},
- {0x0897, 0x089F}, {0x08CA, 0x08E1}, {0x08E3, 0x0902},
- {0x093A, 0x093A}, {0x093C, 0x093C}, {0x0941, 0x0948},
- {0x094D, 0x094D}, {0x0951, 0x0957}, {0x0962, 0x0963},
- {0x0981, 0x0981}, {0x09BC, 0x09BC}, {0x09C1, 0x09C4},
- {0x09CD, 0x09CD}, {0x09E2, 0x09E3}, {0x09FE, 0x09FE},
- {0x0A01, 0x0A02}, {0x0A3C, 0x0A3C}, {0x0A41, 0x0A42},
- {0x0A47, 0x0A48}, {0x0A4B, 0x0A4D}, {0x0A51, 0x0A51},
- {0x0A70, 0x0A71}, {0x0A75, 0x0A75}, {0x0A81, 0x0A82},
- {0x0ABC, 0x0ABC}, {0x0AC1, 0x0AC5}, {0x0AC7, 0x0AC8},
- {0x0ACD, 0x0ACD}, {0x0AE2, 0x0AE3}, {0x0AFA, 0x0AFF},
- {0x0B01, 0x0B01}, {0x0B3C, 0x0B3C}, {0x0B3F, 0x0B3F},
- {0x0B41, 0x0B44}, {0x0B4D, 0x0B4D}, {0x0B53, 0x0B56},
- {0x0B62, 0x0B63}, {0x0B82, 0x0B82}, {0x0BC0, 0x0BC0},
- {0x0BCD, 0x0BCD}, {0x0C00, 0x0C00}, {0x0C04, 0x0C04},
- {0x0C3C, 0x0C3C}, {0x0C3E, 0x0C40}, {0x0C46, 0x0C48},
- {0x0C4A, 0x0C4D}, {0x0C55, 0x0C56}, {0x0C62, 0x0C63},
- {0x0C81, 0x0C81}, {0x0CBC, 0x0CBC}, {0x0CBF, 0x0CBF},
- {0x0CC6, 0x0CC6}, {0x0CCC, 0x0CCD}, {0x0CE2, 0x0CE3},
- {0x0D00, 0x0D01}, {0x0D3B, 0x0D3C}, {0x0D41, 0x0D44},
- {0x0D4D, 0x0D4D}, {0x0D62, 0x0D63}, {0x0D81, 0x0D81},
- {0x0DCA, 0x0DCA}, {0x0DD2, 0x0DD4}, {0x0DD6, 0x0DD6},
- {0x0E31, 0x0E31}, {0x0E34, 0x0E3A}, {0x0E47, 0x0E4E},
- {0x0EB1, 0x0EB1}, {0x0EB4, 0x0EBC}, {0x0EC8, 0x0ECE},
- {0x0F18, 0x0F19}, {0x0F35, 0x0F35}, {0x0F37, 0x0F37},
- {0x0F39, 0x0F39}, {0x0F71, 0x0F7E}, {0x0F80, 0x0F84},
- {0x0F86, 0x0F87}, {0x0F8D, 0x0F97}, {0x0F99, 0x0FBC},
- {0x0FC6, 0x0FC6}, {0x102D, 0x1030}, {0x1032, 0x1037},
- {0x1039, 0x103A}, {0x103D, 0x103E}, {0x1058, 0x1059},
- {0x105E, 0x1060}, {0x1071, 0x1074}, {0x1082, 0x1082},
- {0x1085, 0x1086}, {0x108D, 0x108D}, {0x109D, 0x109D},
- {0x135D, 0x135F}, {0x1712, 0x1714}, {0x1732, 0x1733},
- {0x1752, 0x1753}, {0x1772, 0x1773}, {0x17B4, 0x17B5},
- {0x17B7, 0x17BD}, {0x17C6, 0x17C6}, {0x17C9, 0x17D3},
- {0x17DD, 0x17DD}, {0x180B, 0x180D}, {0x180F, 0x180F},
- {0x1885, 0x1886}, {0x18A9, 0x18A9}, {0x1920, 0x1922},
- {0x1927, 0x1928}, {0x1932, 0x1932}, {0x1939, 0x193B},
- {0x1A17, 0x1A18}, {0x1A1B, 0x1A1B}, {0x1A56, 0x1A56},
- {0x1A58, 0x1A5E}, {0x1A60, 0x1A60}, {0x1A62, 0x1A62},
- {0x1A65, 0x1A6C}, {0x1A73, 0x1A7C}, {0x1A7F, 0x1A7F},
- {0x1AB0, 0x1AF0}, {0x1B00, 0x1B03}, {0x1B34, 0x1B34},
- {0x1B36, 0x1B3A}, {0x1B3C, 0x1B3C}, {0x1B42, 0x1B42},
- {0x1B6B, 0x1B73}, {0x1B80, 0x1B81}, {0x1BA2, 0x1BA5},
- {0x1BA8, 0x1BA9}, {0x1BAB, 0x1BAD}, {0x1BE6, 0x1BE6},
- {0x1BE8, 0x1BE9}, {0x1BED, 0x1BED}, {0x1BEF, 0x1BF1},
- {0x1C2C, 0x1C33}, {0x1C36, 0x1C37}, {0x1CD0, 0x1CD2},
- {0x1CD4, 0x1CE0}, {0x1CE2, 0x1CE8}, {0x1CED, 0x1CED},
- {0x1CF4, 0x1CF4}, {0x1CF8, 0x1CF9}, {0x1DC0, 0x1DFF},
- {0x20D0, 0x20F0}, {0x2CEF, 0x2CF1}, {0x2D7F, 0x2D7F},
- {0x2DE0, 0x2DFF}, {0x302A, 0x302D}, {0x3099, 0x309A},
- {0xA66F, 0xA672}, {0xA674, 0xA67D}, {0xA69E, 0xA69F},
- {0xA6F0, 0xA6F1}, {0xA802, 0xA802}, {0xA806, 0xA806},
- {0xA80B, 0xA80B}, {0xA825, 0xA826}, {0xA82C, 0xA82C},
- {0xA8C4, 0xA8C5}, {0xA8E0, 0xA8F1}, {0xA8FF, 0xA8FF},
- {0xA926, 0xA92D}, {0xA947, 0xA951}, {0xA980, 0xA982},
- {0xA9B3, 0xA9B3}, {0xA9B6, 0xA9B9}, {0xA9BC, 0xA9BD},
- {0xA9E5, 0xA9E5}, {0xAA29, 0xAA2E}, {0xAA31, 0xAA32},
- {0xAA35, 0xAA36}, {0xAA43, 0xAA43}, {0xAA4C, 0xAA4C},
- {0xAA7C, 0xAA7C}, {0xAAB0, 0xAAB0}, {0xAAB2, 0xAAB4},
- {0xAAB7, 0xAAB8}, {0xAABE, 0xAABF}, {0xAAC1, 0xAAC1},
- {0xAAEC, 0xAAED}, {0xAAF6, 0xAAF6}, {0xABE5, 0xABE5},
- {0xABE8, 0xABE8}, {0xABED, 0xABED}, {0xFB1E, 0xFB1E},
- {0xFE00, 0xFE0F}, {0xFE20, 0xFE2F}, {0x101FD, 0x101FD},
- {0x102E0, 0x102E0}, {0x10376, 0x1037A}, {0x10A01, 0x10A03},
- {0x10A05, 0x10A06}, {0x10A0C, 0x10A0F}, {0x10A38, 0x10A3A},
- {0x10A3F, 0x10A3F}, {0x10AE5, 0x10AE6}, {0x10D24, 0x10D27},
- {0x10D69, 0x10D6D}, {0x10EAB, 0x10EAC}, {0x10ECB, 0x10ECF},
- {0x10EF0, 0x10EFF}, {0x10F46, 0x10F50}, {0x10F82, 0x10F85},
- {0x11001, 0x11001}, {0x11038, 0x11046}, {0x11070, 0x11070},
- {0x11073, 0x11074}, {0x1107F, 0x11081}, {0x110B3, 0x110B6},
- {0x110B9, 0x110BA}, {0x110C2, 0x110C2}, {0x11100, 0x11102},
- {0x11127, 0x1112B}, {0x1112D, 0x11134}, {0x11173, 0x11173},
- {0x11180, 0x11181}, {0x111B6, 0x111BE}, {0x111C9, 0x111CC},
- {0x111CF, 0x111CF}, {0x1122F, 0x11231}, {0x11234, 0x11234},
- {0x11236, 0x11237}, {0x1123E, 0x1123E}, {0x11241, 0x11241},
- {0x112DF, 0x112DF}, {0x112E3, 0x112EA}, {0x11300, 0x11301},
- {0x1133B, 0x1133C}, {0x11340, 0x11340}, {0x11366, 0x1136C},
- {0x11370, 0x11374}, {0x113BB, 0x113C0}, {0x113CE, 0x113CE},
- {0x113D0, 0x113D0}, {0x113D2, 0x113D2}, {0x113E1, 0x113E2},
- {0x11438, 0x1143F}, {0x11442, 0x11444}, {0x11446, 0x11446},
- {0x1145E, 0x1145E}, {0x114B3, 0x114B8}, {0x114BA, 0x114BA},
- {0x114BF, 0x114C0}, {0x114C2, 0x114C3}, {0x115B2, 0x115B5},
- {0x115BC, 0x115BD}, {0x115BF, 0x115C0}, {0x115DC, 0x115DD},
- {0x11633, 0x1163A}, {0x1163D, 0x1163D}, {0x1163F, 0x11640},
- {0x116AB, 0x116AB}, {0x116AD, 0x116AD}, {0x116B0, 0x116B5},
- {0x116B7, 0x116B7}, {0x1171D, 0x1171D}, {0x1171F, 0x1171F},
- {0x11722, 0x11725}, {0x11727, 0x1172B}, {0x1182F, 0x11837},
- {0x11839, 0x1183A}, {0x1193B, 0x1193C}, {0x1193E, 0x1193E},
- {0x11943, 0x11943}, {0x119D4, 0x119D7}, {0x119DA, 0x119DB},
- {0x119E0, 0x119E0}, {0x11A01, 0x11A0A}, {0x11A33, 0x11A38},
- {0x11A3B, 0x11A3E}, {0x11A47, 0x11A47}, {0x11A51, 0x11A56},
- {0x11A59, 0x11A5B}, {0x11A8A, 0x11A96}, {0x11A98, 0x11A99},
- {0x11B60, 0x11B60}, {0x11B62, 0x11B64}, {0x11B66, 0x11B66},
- {0x11C30, 0x11C36}, {0x11C38, 0x11C3D}, {0x11C3F, 0x11C3F},
- {0x11C92, 0x11CA7}, {0x11CAA, 0x11CB0}, {0x11CB2, 0x11CB3},
- {0x11CB5, 0x11CB6}, {0x11D31, 0x11D36}, {0x11D3A, 0x11D3A},
- {0x11D3C, 0x11D3D}, {0x11D3F, 0x11D45}, {0x11D47, 0x11D47},
- {0x11D90, 0x11D91}, {0x11D95, 0x11D95}, {0x11D97, 0x11D97},
- {0x11DF0, 0x11DF0}, {0x11EF3, 0x11EF4}, {0x11F00, 0x11F01},
- {0x11F36, 0x11F3A}, {0x11F40, 0x11F40}, {0x11F42, 0x11F42},
- {0x11F5A, 0x11F5A}, {0x13440, 0x13440}, {0x13447, 0x13455},
- {0x1611E, 0x16129}, {0x1612D, 0x1612F}, {0x16AF0, 0x16AF4},
- {0x16B30, 0x16B36}, {0x16D98, 0x16D98}, {0x16D9D, 0x16D9D},
- {0x16F4F, 0x16F4F}, {0x16F8F, 0x16F92}, {0x16FE4, 0x16FE4},
- {0x1BC9D, 0x1BC9E}, {0x1CF00, 0x1CF2D}, {0x1CF30, 0x1CF46},
- {0x1D127, 0x1D128}, {0x1D167, 0x1D169}, {0x1D17B, 0x1D182},
- {0x1D185, 0x1D18B}, {0x1D1AA, 0x1D1AD}, {0x1D242, 0x1D244},
- {0x1D25B, 0x1D25C}, {0x1DA00, 0x1DA36}, {0x1DA3B, 0x1DA6C},
- {0x1DA75, 0x1DA75}, {0x1DA84, 0x1DA84}, {0x1DA9B, 0x1DA9F},
- {0x1DAA1, 0x1DAAF}, {0x1E000, 0x1E006}, {0x1E008, 0x1E018},
- {0x1E01B, 0x1E021}, {0x1E023, 0x1E024}, {0x1E026, 0x1E02A},
- {0x1E08F, 0x1E08F}, {0x1E130, 0x1E136}, {0x1E2AE, 0x1E2AE},
- {0x1E2EC, 0x1E2EF}, {0x1E4EC, 0x1E4EF}, {0x1E5EE, 0x1E5EF},
- {0x1E6E3, 0x1E6E3}, {0x1E6E6, 0x1E6E6}, {0x1E6EE, 0x1E6EF},
- {0x1E6F5, 0x1E6F5}, {0x1E8D0, 0x1E8D6}, {0x1E944, 0x1E94A},
- {0xE0100, 0xE01EF},
- };
- static const UnicodeCharSet CombiningCharacters(CombiningCharacterRanges);
+ static const UnicodeCharSet CombiningCharacters(
+ GeneratedCombiningCharacterRanges);
if (CombiningCharacters.contains(UCS))
return 0;
- // We consider double width codepoints any codepoint with
- // the property East_Asian_Width=F|W
- // + Misc Symbols and Pictographs (U+1F300...U+1F5FF)
- // + Supplemental Symbols and Pictographs (U+1F900...U+1F9FF)
- static const UnicodeCharRange DoubleWidthCharacterRanges[] = {
- {0x1100, 0x115F}, {0x231A, 0x231B}, {0x2329, 0x232A},
- {0x23E9, 0x23EC}, {0x23F0, 0x23F0}, {0x23F3, 0x23F3},
- {0x25FD, 0x25FE}, {0x2614, 0x2615}, {0x2630, 0x2637},
- {0x2648, 0x2653}, {0x267F, 0x267F}, {0x268A, 0x268F},
- {0x2693, 0x2693}, {0x26A1, 0x26A1}, {0x26AA, 0x26AB},
- {0x26BD, 0x26BE}, {0x26C4, 0x26C5}, {0x26CE, 0x26CE},
- {0x26D4, 0x26D4}, {0x26EA, 0x26EA}, {0x26F2, 0x26F3},
- {0x26F5, 0x26F5}, {0x26FA, 0x26FA}, {0x26FD, 0x26FD},
- {0x2705, 0x2705}, {0x270A, 0x270B}, {0x2728, 0x2728},
- {0x274C, 0x274C}, {0x274E, 0x274E}, {0x2753, 0x2755},
- {0x2757, 0x2757}, {0x2795, 0x2797}, {0x27B0, 0x27B0},
- {0x27BF, 0x27BF}, {0x2B1B, 0x2B1C}, {0x2B50, 0x2B50},
- {0x2B55, 0x2B55}, {0x2E80, 0x2E99}, {0x2E9B, 0x2EF3},
- {0x2F00, 0x2FD5}, {0x2FF0, 0x303E}, {0x3041, 0x3096},
- {0x3099, 0x30FF}, {0x3105, 0x312F}, {0x3131, 0x318E},
- {0x3190, 0x31E5}, {0x31EF, 0x321E}, {0x3220, 0x3247},
- {0x3250, 0xA48C}, {0xA490, 0xA4C6}, {0xA960, 0xA97C},
- {0xAC00, 0xD7A3}, {0xF900, 0xFAFF}, {0xFE10, 0xFE19},
- {0xFE30, 0xFE52}, {0xFE54, 0xFE66}, {0xFE68, 0xFE6B},
- {0xFF01, 0xFF60}, {0xFFE0, 0xFFE6}, {0x16FE0, 0x16FE4},
- {0x16FF0, 0x16FF6}, {0x17000, 0x18CDA}, {0x18CFF, 0x18D20},
- {0x18D80, 0x18DF2}, {0x18E00, 0x19191}, {0x191A0, 0x191D2},
- {0x1AFF0, 0x1AFF3}, {0x1AFF5, 0x1AFFB}, {0x1AFFD, 0x1AFFE},
- {0x1B000, 0x1B128}, {0x1B132, 0x1B132}, {0x1B150, 0x1B152},
- {0x1B155, 0x1B155}, {0x1B164, 0x1B168}, {0x1B170, 0x1B2FB},
- {0x1D300, 0x1D356}, {0x1D360, 0x1D376}, {0x1F004, 0x1F004},
- {0x1F0CF, 0x1F0CF}, {0x1F18E, 0x1F18E}, {0x1F191, 0x1F19A},
- {0x1F1AE, 0x1F1AE}, {0x1F200, 0x1F202}, {0x1F210, 0x1F23B},
- {0x1F240, 0x1F248}, {0x1F250, 0x1F251}, {0x1F260, 0x1F265},
- {0x1F300, 0x1F320}, {0x1F32D, 0x1F335}, {0x1F337, 0x1F37C},
- {0x1F37E, 0x1F393}, {0x1F3A0, 0x1F3CA}, {0x1F3CF, 0x1F3D3},
- {0x1F3E0, 0x1F3F0}, {0x1F3F4, 0x1F3F4}, {0x1F3F8, 0x1F43E},
- {0x1F440, 0x1F440}, {0x1F442, 0x1F4FC}, {0x1F4FF, 0x1F53D},
- {0x1F54B, 0x1F54E}, {0x1F550, 0x1F567}, {0x1F57A, 0x1F57A},
- {0x1F595, 0x1F596}, {0x1F5A4, 0x1F5A4}, {0x1F5FB, 0x1F64F},
- {0x1F680, 0x1F6C5}, {0x1F6CC, 0x1F6CC}, {0x1F6D0, 0x1F6D2},
- {0x1F6D5, 0x1F6D9}, {0x1F6DC, 0x1F6DF}, {0x1F6EB, 0x1F6EC},
- {0x1F6F4, 0x1F6FC}, {0x1F7DA, 0x1F7DA}, {0x1F7E0, 0x1F7EB},
- {0x1F7F0, 0x1F7F0}, {0x1F90C, 0x1F93A}, {0x1F93C, 0x1F945},
- {0x1F947, 0x1F9FF}, {0x1FA70, 0x1FA7C}, {0x1FA80, 0x1FAC6},
- {0x1FAC8, 0x1FAC8}, {0x1FACC, 0x1FADD}, {0x1FADF, 0x1FAEB},
- {0x1FAEF, 0x1FAFA}, {0x20000, 0x2FFFD}, {0x30000, 0x3FFFD},
- };
- static const UnicodeCharSet DoubleWidthCharacters(DoubleWidthCharacterRanges);
+ static const UnicodeCharSet DoubleWidthCharacters(
+ GeneratedDoubleWidthCharacterRanges);
if (DoubleWidthCharacters.contains(UCS))
return 2;
diff --git a/llvm/lib/Support/UnicodeCharSetsGenerated.cpp b/llvm/lib/Support/UnicodeCharSetsGenerated.cpp
new file mode 100644
index 0000000000000..0a30fe11c2ca9
--- /dev/null
+++ b/llvm/lib/Support/UnicodeCharSetsGenerated.cpp
@@ -0,0 +1,462 @@
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file was generated by llvm/utils/UnicodeData/UnicodeCharSetsGenerator
+// from the Unicode 18.0 Character Database. Do not edit manually.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Support/UnicodeCharRanges.h"
+
+using llvm::sys::UnicodeCharRange;
+using CharRanges = llvm::sys::UnicodeCharSet::CharRanges;
+
+namespace llvm::sys::unicode {
+
+// General_Category=L|M|N|P|S|Zs
+static constexpr UnicodeCharRange GeneratedPrintableRangesData[] = {
+ {0x0020, 0x007E}, {0x00A0, 0x00AC}, {0x00AE, 0x0377},
+ {0x037A, 0x037F}, {0x0384, 0x038A}, {0x038C, 0x038C},
+ {0x038E, 0x03A1}, {0x03A3, 0x052F}, {0x0531, 0x0556},
+ {0x0558, 0x058F}, {0x0591, 0x05C9}, {0x05D0, 0x05EA},
+ {0x05EF, 0x05F4}, {0x0606, 0x061B}, {0x061D, 0x06DC},
+ {0x06DE, 0x070D}, {0x0710, 0x074A}, {0x074D, 0x07B1},
+ {0x07C0, 0x07FA}, {0x07FD, 0x082D}, {0x0830, 0x083E},
+ {0x0840, 0x085B}, {0x085E, 0x085E}, {0x0860, 0x086A},
+ {0x0870, 0x088F}, {0x0897, 0x08E1}, {0x08E3, 0x0983},
+ {0x0985, 0x098C}, {0x098F, 0x0990}, {0x0993, 0x09A8},
+ {0x09AA, 0x09B0}, {0x09B2, 0x09B2}, {0x09B6, 0x09B9},
+ {0x09BC, 0x09C4}, {0x09C7, 0x09C8}, {0x09CB, 0x09CE},
+ {0x09D7, 0x09D7}, {0x09DC, 0x09DD}, {0x09DF, 0x09E3},
+ {0x09E6, 0x09FE}, {0x0A01, 0x0A03}, {0x0A05, 0x0A0A},
+ {0x0A0F, 0x0A10}, {0x0A13, 0x0A28}, {0x0A2A, 0x0A30},
+ {0x0A32, 0x0A33}, {0x0A35, 0x0A36}, {0x0A38, 0x0A39},
+ {0x0A3C, 0x0A3C}, {0x0A3E, 0x0A42}, {0x0A47, 0x0A48},
+ {0x0A4B, 0x0A4D}, {0x0A51, 0x0A51}, {0x0A59, 0x0A5C},
+ {0x0A5E, 0x0A5E}, {0x0A66, 0x0A76}, {0x0A81, 0x0A83},
+ {0x0A85, 0x0A8D}, {0x0A8F, 0x0A91}, {0x0A93, 0x0AA8},
+ {0x0AAA, 0x0AB0}, {0x0AB2, 0x0AB3}, {0x0AB5, 0x0AB9},
+ {0x0ABC, 0x0AC5}, {0x0AC7, 0x0AC9}, {0x0ACB, 0x0ACD},
+ {0x0AD0, 0x0AD0}, {0x0AE0, 0x0AE3}, {0x0AE6, 0x0AF1},
+ {0x0AF9, 0x0AFF}, {0x0B01, 0x0B03}, {0x0B05, 0x0B0C},
+ {0x0B0F, 0x0B10}, {0x0B13, 0x0B28}, {0x0B2A, 0x0B30},
+ {0x0B32, 0x0B33}, {0x0B35, 0x0B39}, {0x0B3C, 0x0B44},
+ {0x0B47, 0x0B48}, {0x0B4B, 0x0B4D}, {0x0B53, 0x0B57},
+ {0x0B5C, 0x0B5D}, {0x0B5F, 0x0B63}, {0x0B66, 0x0B77},
+ {0x0B82, 0x0B83}, {0x0B85, 0x0B8A}, {0x0B8E, 0x0B90},
+ {0x0B92, 0x0B95}, {0x0B99, 0x0B9A}, {0x0B9C, 0x0B9C},
+ {0x0B9E, 0x0B9F}, {0x0BA3, 0x0BA4}, {0x0BA8, 0x0BAA},
+ {0x0BAE, 0x0BB9}, {0x0BBE, 0x0BC2}, {0x0BC6, 0x0BC8},
+ {0x0BCA, 0x0BCD}, {0x0BD0, 0x0BD0}, {0x0BD7, 0x0BD7},
+ {0x0BE6, 0x0BFA}, {0x0C00, 0x0C0C}, {0x0C0E, 0x0C10},
+ {0x0C12, 0x0C28}, {0x0C2A, 0x0C39}, {0x0C3C, 0x0C44},
+ {0x0C46, 0x0C48}, {0x0C4A, 0x0C4D}, {0x0C55, 0x0C56},
+ {0x0C58, 0x0C5A}, {0x0C5C, 0x0C5D}, {0x0C60, 0x0C63},
+ {0x0C66, 0x0C6F}, {0x0C77, 0x0C8C}, {0x0C8E, 0x0C90},
+ {0x0C92, 0x0CA8}, {0x0CAA, 0x0CB3}, {0x0CB5, 0x0CB9},
+ {0x0CBC, 0x0CC4}, {0x0CC6, 0x0CC8}, {0x0CCA, 0x0CCD},
+ {0x0CD5, 0x0CD6}, {0x0CDC, 0x0CDE}, {0x0CE0, 0x0CE3},
+ {0x0CE6, 0x0CEF}, {0x0CF1, 0x0CF3}, {0x0D00, 0x0D0C},
+ {0x0D0E, 0x0D10}, {0x0D12, 0x0D44}, {0x0D46, 0x0D48},
+ {0x0D4A, 0x0D4F}, {0x0D54, 0x0D63}, {0x0D66, 0x0D7F},
+ {0x0D81, 0x0D83}, {0x0D85, 0x0D96}, {0x0D9A, 0x0DB1},
+ {0x0DB3, 0x0DBB}, {0x0DBD, 0x0DBD}, {0x0DC0, 0x0DC6},
+ {0x0DCA, 0x0DCA}, {0x0DCF, 0x0DD4}, {0x0DD6, 0x0DD6},
+ {0x0DD8, 0x0DDF}, {0x0DE6, 0x0DEF}, {0x0DF2, 0x0DF4},
+ {0x0E01, 0x0E3A}, {0x0E3F, 0x0E5B}, {0x0E81, 0x0E82},
+ {0x0E84, 0x0E84}, {0x0E86, 0x0E8A}, {0x0E8C, 0x0EA3},
+ {0x0EA5, 0x0EA5}, {0x0EA7, 0x0EBD}, {0x0EC0, 0x0EC4},
+ {0x0EC6, 0x0EC6}, {0x0EC8, 0x0ECE}, {0x0ED0, 0x0ED9},
+ {0x0EDC, 0x0EDF}, {0x0F00, 0x0F47}, {0x0F49, 0x0F6C},
+ {0x0F71, 0x0F97}, {0x0F99, 0x0FBC}, {0x0FBE, 0x0FCC},
+ {0x0FCE, 0x0FDA}, {0x1000, 0x10C5}, {0x10C7, 0x10C7},
+ {0x10CD, 0x10CD}, {0x10D0, 0x1248}, {0x124A, 0x124D},
+ {0x1250, 0x1256}, {0x1258, 0x1258}, {0x125A, 0x125D},
+ {0x1260, 0x1288}, {0x128A, 0x128D}, {0x1290, 0x12B0},
+ {0x12B2, 0x12B5}, {0x12B8, 0x12BE}, {0x12C0, 0x12C0},
+ {0x12C2, 0x12C5}, {0x12C8, 0x12D6}, {0x12D8, 0x1310},
+ {0x1312, 0x1315}, {0x1318, 0x135A}, {0x135D, 0x137C},
+ {0x1380, 0x1399}, {0x13A0, 0x13F5}, {0x13F8, 0x13FD},
+ {0x1400, 0x169C}, {0x16A0, 0x16F8}, {0x1700, 0x1715},
+ {0x171F, 0x1736}, {0x1740, 0x1753}, {0x1760, 0x176C},
+ {0x176E, 0x1770}, {0x1772, 0x1773}, {0x1780, 0x17DD},
+ {0x17E0, 0x17E9}, {0x17F0, 0x17F9}, {0x1800, 0x180D},
+ {0x180F, 0x1819}, {0x1820, 0x1878}, {0x1880, 0x18AA},
+ {0x18B0, 0x18F5}, {0x1900, 0x191E}, {0x1920, 0x192B},
+ {0x1930, 0x193B}, {0x1940, 0x1940}, {0x1944, 0x196D},
+ {0x1970, 0x1974}, {0x1980, 0x19AB}, {0x19B0, 0x19C9},
+ {0x19D0, 0x19DA}, {0x19DE, 0x1A1B}, {0x1A1E, 0x1A5E},
+ {0x1A60, 0x1A7C}, {0x1A7F, 0x1A89}, {0x1A90, 0x1A99},
+ {0x1AA0, 0x1AAD}, {0x1AB0, 0x1AF0}, {0x1B00, 0x1B4C},
+ {0x1B4E, 0x1BF3}, {0x1BFC, 0x1C37}, {0x1C3B, 0x1C49},
+ {0x1C4D, 0x1C8A}, {0x1C90, 0x1CBA}, {0x1CBD, 0x1CC7},
+ {0x1CD0, 0x1CFA}, {0x1D00, 0x1F15}, {0x1F18, 0x1F1D},
+ {0x1F20, 0x1F45}, {0x1F48, 0x1F4D}, {0x1F50, 0x1F57},
+ {0x1F59, 0x1F59}, {0x1F5B, 0x1F5B}, {0x1F5D, 0x1F5D},
+ {0x1F5F, 0x1F7D}, {0x1F80, 0x1FB4}, {0x1FB6, 0x1FC4},
+ {0x1FC6, 0x1FD3}, {0x1FD6, 0x1FDB}, {0x1FDD, 0x1FEF},
+ {0x1FF2, 0x1FF4}, {0x1FF6, 0x1FFE}, {0x2000, 0x200A},
+ {0x2010, 0x2027}, {0x202F, 0x205F}, {0x2070, 0x2071},
+ {0x2074, 0x20C4}, {0x20D0, 0x20F0}, {0x2100, 0x218B},
+ {0x2190, 0x2429}, {0x2440, 0x244A}, {0x2460, 0x2B73},
+ {0x2B76, 0x2CF3}, {0x2CF9, 0x2D25}, {0x2D27, 0x2D27},
+ {0x2D2D, 0x2D2D}, {0x2D30, 0x2D67}, {0x2D6F, 0x2D70},
+ {0x2D7F, 0x2D96}, {0x2DA0, 0x2DA6}, {0x2DA8, 0x2DAE},
+ {0x2DB0, 0x2DB6}, {0x2DB8, 0x2DBE}, {0x2DC0, 0x2DC6},
+ {0x2DC8, 0x2DCE}, {0x2DD0, 0x2DD6}, {0x2DD8, 0x2DDE},
+ {0x2DE0, 0x2E5D}, {0x2E60, 0x2E63}, {0x2E80, 0x2E99},
+ {0x2E9B, 0x2EF3}, {0x2F00, 0x2FD5}, {0x2FF0, 0x303F},
+ {0x3041, 0x3096}, {0x3099, 0x30FF}, {0x3105, 0x312F},
+ {0x3131, 0x318E}, {0x3190, 0x31E5}, {0x31EF, 0x321E},
+ {0x3220, 0xA48C}, {0xA490, 0xA4C6}, {0xA4D0, 0xA62B},
+ {0xA640, 0xA6F7}, {0xA700, 0xA7DD}, {0xA7E2, 0xA7E2},
+ {0xA7F1, 0xA82C}, {0xA830, 0xA839}, {0xA840, 0xA877},
+ {0xA880, 0xA8C5}, {0xA8CE, 0xA8D9}, {0xA8E0, 0xA953},
+ {0xA95F, 0xA97C}, {0xA980, 0xA9CD}, {0xA9CF, 0xA9D9},
+ {0xA9DE, 0xA9FE}, {0xAA00, 0xAA36}, {0xAA40, 0xAA4D},
+ {0xAA50, 0xAA59}, {0xAA5C, 0xAAC2}, {0xAADB, 0xAAF6},
+ {0xAB01, 0xAB06}, {0xAB09, 0xAB0E}, {0xAB11, 0xAB16},
+ {0xAB20, 0xAB26}, {0xAB28, 0xAB2E}, {0xAB30, 0xAB6D},
+ {0xAB70, 0xABED}, {0xABF0, 0xABF9}, {0xAC00, 0xD7A3},
+ {0xD7B0, 0xD7C6}, {0xD7CB, 0xD7FB}, {0xF900, 0xFA6D},
+ {0xFA70, 0xFAD9}, {0xFB00, 0xFB06}, {0xFB13, 0xFB17},
+ {0xFB1D, 0xFB36}, {0xFB38, 0xFB3C}, {0xFB3E, 0xFB3E},
+ {0xFB40, 0xFB41}, {0xFB43, 0xFB44}, {0xFB46, 0xFDCF},
+ {0xFDF0, 0xFE19}, {0xFE20, 0xFE52}, {0xFE54, 0xFE66},
+ {0xFE68, 0xFE6B}, {0xFE70, 0xFE74}, {0xFE76, 0xFEFC},
+ {0xFF01, 0xFFBE}, {0xFFC2, 0xFFC7}, {0xFFCA, 0xFFCF},
+ {0xFFD2, 0xFFD7}, {0xFFDA, 0xFFDC}, {0xFFE0, 0xFFE6},
+ {0xFFE8, 0xFFEE}, {0xFFFC, 0xFFFD}, {0x10000, 0x1000B},
+ {0x1000D, 0x10026}, {0x10028, 0x1003A}, {0x1003C, 0x1003D},
+ {0x1003F, 0x1004D}, {0x10050, 0x1005D}, {0x10080, 0x100FA},
+ {0x10100, 0x10102}, {0x10107, 0x10133}, {0x10137, 0x1018E},
+ {0x10190, 0x1019C}, {0x101A0, 0x101A0}, {0x101D0, 0x101FD},
+ {0x10280, 0x1029C}, {0x102A0, 0x102D0}, {0x102E0, 0x102FB},
+ {0x10300, 0x10323}, {0x1032D, 0x1034A}, {0x10350, 0x1037A},
+ {0x10380, 0x1039D}, {0x1039F, 0x103C3}, {0x103C8, 0x103D5},
+ {0x10400, 0x1049D}, {0x104A0, 0x104A9}, {0x104B0, 0x104D3},
+ {0x104D8, 0x104FB}, {0x10500, 0x10527}, {0x10530, 0x10563},
+ {0x1056F, 0x1057A}, {0x1057C, 0x1058A}, {0x1058C, 0x10592},
+ {0x10594, 0x10595}, {0x10597, 0x105A1}, {0x105A3, 0x105B1},
+ {0x105B3, 0x105B9}, {0x105BB, 0x105BC}, {0x105C0, 0x105F3},
+ {0x10600, 0x10736}, {0x10740, 0x10755}, {0x10760, 0x10767},
+ {0x10780, 0x10785}, {0x10787, 0x107B0}, {0x107B2, 0x107BF},
+ {0x10800, 0x10805}, {0x10808, 0x10808}, {0x1080A, 0x10835},
+ {0x10837, 0x10838}, {0x1083C, 0x1083C}, {0x1083F, 0x10855},
+ {0x10857, 0x1089E}, {0x108A7, 0x108AF}, {0x108E0, 0x108F2},
+ {0x108F4, 0x108F5}, {0x108FB, 0x1091B}, {0x1091F, 0x10939},
+ {0x1093F, 0x10959}, {0x10980, 0x109B7}, {0x109BC, 0x109CF},
+ {0x109D2, 0x10A03}, {0x10A05, 0x10A06}, {0x10A0C, 0x10A13},
+ {0x10A15, 0x10A17}, {0x10A19, 0x10A35}, {0x10A38, 0x10A3A},
+ {0x10A3F, 0x10A48}, {0x10A50, 0x10A58}, {0x10A60, 0x10A9F},
+ {0x10AC0, 0x10AE6}, {0x10AEB, 0x10AF6}, {0x10B00, 0x10B35},
+ {0x10B39, 0x10B55}, {0x10B58, 0x10B72}, {0x10B78, 0x10B91},
+ {0x10B99, 0x10B9C}, {0x10BA9, 0x10BAF}, {0x10C00, 0x10C48},
+ {0x10C80, 0x10CB2}, {0x10CC0, 0x10CF2}, {0x10CFA, 0x10D27},
+ {0x10D30, 0x10D39}, {0x10D40, 0x10D65}, {0x10D69, 0x10D85},
+ {0x10D8E, 0x10D8F}, {0x10E60, 0x10E7E}, {0x10E80, 0x10EA9},
+ {0x10EAB, 0x10EAD}, {0x10EB0, 0x10EB1}, {0x10EC2, 0x10EC7},
+ {0x10EC9, 0x10EEE}, {0x10EF0, 0x10F27}, {0x10F30, 0x10F59},
+ {0x10F70, 0x10F89}, {0x10FB0, 0x10FCB}, {0x10FE0, 0x10FF6},
+ {0x11000, 0x1104D}, {0x11052, 0x11075}, {0x1107F, 0x110BC},
+ {0x110BE, 0x110C2}, {0x110D0, 0x110E8}, {0x110F0, 0x110F9},
+ {0x11100, 0x11134}, {0x11136, 0x11147}, {0x11150, 0x11176},
+ {0x11180, 0x111DF}, {0x111E1, 0x111F4}, {0x11200, 0x11211},
+ {0x11213, 0x11241}, {0x11280, 0x11286}, {0x11288, 0x11288},
+ {0x1128A, 0x1128D}, {0x1128F, 0x1129D}, {0x1129F, 0x112A9},
+ {0x112B0, 0x112EA}, {0x112F0, 0x112F9}, {0x11300, 0x11303},
+ {0x11305, 0x1130C}, {0x1130F, 0x11310}, {0x11313, 0x11328},
+ {0x1132A, 0x11330}, {0x11332, 0x11333}, {0x11335, 0x11339},
+ {0x1133B, 0x11344}, {0x11347, 0x11348}, {0x1134B, 0x1134D},
+ {0x11350, 0x11350}, {0x11357, 0x11357}, {0x1135D, 0x11363},
+ {0x11366, 0x1136C}, {0x11370, 0x11374}, {0x11380, 0x11389},
+ {0x1138B, 0x1138B}, {0x1138E, 0x1138E}, {0x11390, 0x113B5},
+ {0x113B7, 0x113C0}, {0x113C2, 0x113C2}, {0x113C5, 0x113C5},
+ {0x113C7, 0x113CA}, {0x113CC, 0x113D5}, {0x113D7, 0x113D8},
+ {0x113E1, 0x113E2}, {0x11400, 0x1145B}, {0x1145D, 0x11461},
+ {0x11480, 0x114C7}, {0x114D0, 0x114D9}, {0x11580, 0x115B5},
+ {0x115B8, 0x115DD}, {0x11600, 0x11644}, {0x11650, 0x11659},
+ {0x11660, 0x1166C}, {0x11680, 0x116B9}, {0x116C0, 0x116C9},
+ {0x116D0, 0x116E3}, {0x11700, 0x1171A}, {0x1171D, 0x1172B},
+ {0x11730, 0x11746}, {0x11800, 0x1183B}, {0x118A0, 0x118F2},
+ {0x118FF, 0x11906}, {0x11909, 0x11909}, {0x1190C, 0x11913},
+ {0x11915, 0x11916}, {0x11918, 0x11935}, {0x11937, 0x11938},
+ {0x1193B, 0x11946}, {0x11950, 0x11959}, {0x119A0, 0x119A7},
+ {0x119AA, 0x119D7}, {0x119DA, 0x119E4}, {0x11A00, 0x11A47},
+ {0x11A50, 0x11AA2}, {0x11AB0, 0x11AF8}, {0x11B00, 0x11B0A},
+ {0x11B60, 0x11B67}, {0x11BC0, 0x11BE1}, {0x11BF0, 0x11BF9},
+ {0x11C00, 0x11C08}, {0x11C0A, 0x11C36}, {0x11C38, 0x11C45},
+ {0x11C50, 0x11C6C}, {0x11C70, 0x11C8F}, {0x11C92, 0x11CA7},
+ {0x11CA9, 0x11CB6}, {0x11D00, 0x11D06}, {0x11D08, 0x11D09},
+ {0x11D0B, 0x11D36}, {0x11D3A, 0x11D3A}, {0x11D3C, 0x11D3D},
+ {0x11D3F, 0x11D47}, {0x11D50, 0x11D59}, {0x11D60, 0x11D65},
+ {0x11D67, 0x11D68}, {0x11D6A, 0x11D8E}, {0x11D90, 0x11D91},
+ {0x11D93, 0x11D98}, {0x11DA0, 0x11DA9}, {0x11DB0, 0x11DDB},
+ {0x11DE0, 0x11DE9}, {0x11DF0, 0x11DF1}, {0x11EE0, 0x11EF8},
+ {0x11F00, 0x11F10}, {0x11F12, 0x11F3A}, {0x11F3E, 0x11F5A},
+ {0x11FB0, 0x11FB0}, {0x11FC0, 0x11FF1}, {0x11FFF, 0x12399},
+ {0x12400, 0x12543}, {0x12550, 0x12686}, {0x12F90, 0x12FF2},
+ {0x13000, 0x1342F}, {0x13440, 0x13455}, {0x13460, 0x143FA},
+ {0x14400, 0x14646}, {0x16100, 0x16139}, {0x16800, 0x16A38},
+ {0x16A40, 0x16A5E}, {0x16A60, 0x16A69}, {0x16A6E, 0x16ABE},
+ {0x16AC0, 0x16AC9}, {0x16AD0, 0x16AED}, {0x16AF0, 0x16AF5},
+ {0x16B00, 0x16B45}, {0x16B50, 0x16B59}, {0x16B5B, 0x16B61},
+ {0x16B63, 0x16B77}, {0x16B7D, 0x16B8F}, {0x16D40, 0x16D79},
+ {0x16E40, 0x16E9A}, {0x16EA0, 0x16EB8}, {0x16EBB, 0x16ED3},
+ {0x16F00, 0x16F4A}, {0x16F4F, 0x16F87}, {0x16F8F, 0x16F9F},
+ {0x16FE0, 0x16FE4}, {0x16FF0, 0x16FF6}, {0x17000, 0x18CDA},
+ {0x18CFF, 0x18D20}, {0x18D80, 0x18DF2}, {0x18E00, 0x19191},
+ {0x191A0, 0x191D2}, {0x1AFF0, 0x1AFF3}, {0x1AFF5, 0x1AFFB},
+ {0x1AFFD, 0x1AFFE}, {0x1B000, 0x1B128}, {0x1B132, 0x1B132},
+ {0x1B150, 0x1B152}, {0x1B155, 0x1B155}, {0x1B164, 0x1B168},
+ {0x1B170, 0x1B2FB}, {0x1BC00, 0x1BC6A}, {0x1BC70, 0x1BC7C},
+ {0x1BC80, 0x1BC88}, {0x1BC90, 0x1BC99}, {0x1BC9C, 0x1BC9F},
+ {0x1CC00, 0x1CCFC}, {0x1CD00, 0x1CEB3}, {0x1CEBA, 0x1CED0},
+ {0x1CED2, 0x1CED4}, {0x1CEDD, 0x1CEFD}, {0x1CF00, 0x1CF2D},
+ {0x1CF30, 0x1CF46}, {0x1CF50, 0x1CFC3}, {0x1D000, 0x1D0F5},
+ {0x1D100, 0x1D172}, {0x1D17B, 0x1D245}, {0x1D250, 0x1D281},
+ {0x1D2C0, 0x1D2D3}, {0x1D2E0, 0x1D2F3}, {0x1D300, 0x1D356},
+ {0x1D360, 0x1D378}, {0x1D400, 0x1D454}, {0x1D456, 0x1D49C},
+ {0x1D49E, 0x1D49F}, {0x1D4A2, 0x1D4A2}, {0x1D4A5, 0x1D4A6},
+ {0x1D4A9, 0x1D4AC}, {0x1D4AE, 0x1D4B9}, {0x1D4BB, 0x1D4BB},
+ {0x1D4BD, 0x1D4C3}, {0x1D4C5, 0x1D505}, {0x1D507, 0x1D50A},
+ {0x1D50D, 0x1D514}, {0x1D516, 0x1D51C}, {0x1D51E, 0x1D539},
+ {0x1D53B, 0x1D53E}, {0x1D540, 0x1D544}, {0x1D546, 0x1D546},
+ {0x1D54A, 0x1D550}, {0x1D552, 0x1D6A6}, {0x1D6A8, 0x1D7CB},
+ {0x1D7CE, 0x1DA8B}, {0x1DA9B, 0x1DA9F}, {0x1DAA1, 0x1DAAF},
+ {0x1DB00, 0x1DB1C}, {0x1DF00, 0x1DF81}, {0x1DF90, 0x1DF96},
+ {0x1DFCD, 0x1E006}, {0x1E008, 0x1E018}, {0x1E01B, 0x1E021},
+ {0x1E023, 0x1E024}, {0x1E026, 0x1E02A}, {0x1E030, 0x1E06D},
+ {0x1E08F, 0x1E08F}, {0x1E100, 0x1E12C}, {0x1E130, 0x1E13D},
+ {0x1E140, 0x1E149}, {0x1E14E, 0x1E14F}, {0x1E290, 0x1E2AE},
+ {0x1E2C0, 0x1E2F9}, {0x1E2FF, 0x1E2FF}, {0x1E4D0, 0x1E4F9},
+ {0x1E5D0, 0x1E5FA}, {0x1E5FF, 0x1E5FF}, {0x1E6C0, 0x1E6DE},
+ {0x1E6E0, 0x1E6F5}, {0x1E6FE, 0x1E6FF}, {0x1E7E0, 0x1E7E6},
+ {0x1E7E8, 0x1E7EB}, {0x1E7ED, 0x1E7EE}, {0x1E7F0, 0x1E7FE},
+ {0x1E800, 0x1E8C4}, {0x1E8C7, 0x1E8D6}, {0x1E900, 0x1E94B},
+ {0x1E950, 0x1E959}, {0x1E95E, 0x1E95F}, {0x1EC71, 0x1ECB4},
+ {0x1ED01, 0x1ED3D}, {0x1EE00, 0x1EE03}, {0x1EE05, 0x1EE1F},
+ {0x1EE21, 0x1EE22}, {0x1EE24, 0x1EE24}, {0x1EE27, 0x1EE27},
+ {0x1EE29, 0x1EE32}, {0x1EE34, 0x1EE37}, {0x1EE39, 0x1EE39},
+ {0x1EE3B, 0x1EE3B}, {0x1EE42, 0x1EE42}, {0x1EE47, 0x1EE47},
+ {0x1EE49, 0x1EE49}, {0x1EE4B, 0x1EE4B}, {0x1EE4D, 0x1EE4F},
+ {0x1EE51, 0x1EE52}, {0x1EE54, 0x1EE54}, {0x1EE57, 0x1EE57},
+ {0x1EE59, 0x1EE59}, {0x1EE5B, 0x1EE5B}, {0x1EE5D, 0x1EE5D},
+ {0x1EE5F, 0x1EE5F}, {0x1EE61, 0x1EE62}, {0x1EE64, 0x1EE64},
+ {0x1EE67, 0x1EE6A}, {0x1EE6C, 0x1EE72}, {0x1EE74, 0x1EE77},
+ {0x1EE79, 0x1EE7C}, {0x1EE7E, 0x1EE7E}, {0x1EE80, 0x1EE89},
+ {0x1EE8B, 0x1EE9B}, {0x1EEA1, 0x1EEA3}, {0x1EEA5, 0x1EEA9},
+ {0x1EEAB, 0x1EEBB}, {0x1EEF0, 0x1EEF1}, {0x1F000, 0x1F02B},
+ {0x1F030, 0x1F093}, {0x1F0A0, 0x1F0AE}, {0x1F0B1, 0x1F0BF},
+ {0x1F0C1, 0x1F0CF}, {0x1F0D1, 0x1F0F5}, {0x1F100, 0x1F1AE},
+ {0x1F1E6, 0x1F202}, {0x1F210, 0x1F23B}, {0x1F240, 0x1F248},
+ {0x1F250, 0x1F251}, {0x1F260, 0x1F265}, {0x1F300, 0x1F6D9},
+ {0x1F6DC, 0x1F6EC}, {0x1F6F0, 0x1F6FC}, {0x1F700, 0x1F7DB},
+ {0x1F7E0, 0x1F7EB}, {0x1F7F0, 0x1F80B}, {0x1F810, 0x1F847},
+ {0x1F850, 0x1F859}, {0x1F860, 0x1F887}, {0x1F890, 0x1F8AD},
+ {0x1F8B0, 0x1F8BB}, {0x1F8C0, 0x1F8C1}, {0x1F8D0, 0x1F8D8},
+ {0x1F900, 0x1FA57}, {0x1FA60, 0x1FA6D}, {0x1FA70, 0x1FA7C},
+ {0x1FA80, 0x1FAC6}, {0x1FAC8, 0x1FAC8}, {0x1FACC, 0x1FADD},
+ {0x1FADF, 0x1FAEB}, {0x1FAEF, 0x1FAFA}, {0x1FB00, 0x1FB92},
+ {0x1FB94, 0x1FBFA}, {0x20000, 0x2A6DF}, {0x2A700, 0x2B81E},
+ {0x2B820, 0x2CEAD}, {0x2CEB0, 0x2EBE0}, {0x2EBF0, 0x2EE5D},
+ {0x2F800, 0x2FA1D}, {0x30000, 0x3134A}, {0x31350, 0x33479},
+ {0x3D000, 0x3FC3F}, {0xE0100, 0xE01EF},
+};
+extern const CharRanges GeneratedPrintableRanges = GeneratedPrintableRangesData;
+
+// General_Category=Cf
+static constexpr UnicodeCharRange GeneratedFormatCharacterRangesData[] = {
+ {0x00AD, 0x00AD}, {0x0600, 0x0605}, {0x061C, 0x061C},
+ {0x06DD, 0x06DD}, {0x070F, 0x070F}, {0x0890, 0x0891},
+ {0x08E2, 0x08E2}, {0x180E, 0x180E}, {0x200B, 0x200F},
+ {0x202A, 0x202E}, {0x2060, 0x2064}, {0x2066, 0x206F},
+ {0xFEFF, 0xFEFF}, {0xFFF9, 0xFFFB}, {0x110BD, 0x110BD},
+ {0x110CD, 0x110CD}, {0x13430, 0x1343F}, {0x1BCA0, 0x1BCA3},
+ {0x1D173, 0x1D17A}, {0xE0001, 0xE0001}, {0xE0020, 0xE007F},
+};
+extern const CharRanges GeneratedFormatCharacterRanges =
+ GeneratedFormatCharacterRangesData;
+
+// General_Category=Mn|Me
+static constexpr UnicodeCharRange GeneratedCombiningCharacterRangesData[] = {
+ {0x0300, 0x036F}, {0x0483, 0x0489}, {0x0591, 0x05BD},
+ {0x05BF, 0x05BF}, {0x05C1, 0x05C2}, {0x05C4, 0x05C5},
+ {0x05C7, 0x05C9}, {0x0610, 0x061A}, {0x064B, 0x065F},
+ {0x0670, 0x0670}, {0x06D6, 0x06DC}, {0x06DF, 0x06E4},
+ {0x06E7, 0x06E8}, {0x06EA, 0x06ED}, {0x0711, 0x0711},
+ {0x0730, 0x074A}, {0x07A6, 0x07B0}, {0x07EB, 0x07F3},
+ {0x07FD, 0x07FD}, {0x0816, 0x0819}, {0x081B, 0x0823},
+ {0x0825, 0x0827}, {0x0829, 0x082D}, {0x0859, 0x085B},
+ {0x0897, 0x089F}, {0x08CA, 0x08E1}, {0x08E3, 0x0902},
+ {0x093A, 0x093A}, {0x093C, 0x093C}, {0x0941, 0x0948},
+ {0x094D, 0x094D}, {0x0951, 0x0957}, {0x0962, 0x0963},
+ {0x0981, 0x0981}, {0x09BC, 0x09BC}, {0x09C1, 0x09C4},
+ {0x09CD, 0x09CD}, {0x09E2, 0x09E3}, {0x09FE, 0x09FE},
+ {0x0A01, 0x0A02}, {0x0A3C, 0x0A3C}, {0x0A41, 0x0A42},
+ {0x0A47, 0x0A48}, {0x0A4B, 0x0A4D}, {0x0A51, 0x0A51},
+ {0x0A70, 0x0A71}, {0x0A75, 0x0A75}, {0x0A81, 0x0A82},
+ {0x0ABC, 0x0ABC}, {0x0AC1, 0x0AC5}, {0x0AC7, 0x0AC8},
+ {0x0ACD, 0x0ACD}, {0x0AE2, 0x0AE3}, {0x0AFA, 0x0AFF},
+ {0x0B01, 0x0B01}, {0x0B3C, 0x0B3C}, {0x0B3F, 0x0B3F},
+ {0x0B41, 0x0B44}, {0x0B4D, 0x0B4D}, {0x0B53, 0x0B56},
+ {0x0B62, 0x0B63}, {0x0B82, 0x0B82}, {0x0BC0, 0x0BC0},
+ {0x0BCD, 0x0BCD}, {0x0C00, 0x0C00}, {0x0C04, 0x0C04},
+ {0x0C3C, 0x0C3C}, {0x0C3E, 0x0C40}, {0x0C46, 0x0C48},
+ {0x0C4A, 0x0C4D}, {0x0C55, 0x0C56}, {0x0C62, 0x0C63},
+ {0x0C81, 0x0C81}, {0x0CBC, 0x0CBC}, {0x0CBF, 0x0CBF},
+ {0x0CC6, 0x0CC6}, {0x0CCC, 0x0CCD}, {0x0CE2, 0x0CE3},
+ {0x0D00, 0x0D01}, {0x0D3B, 0x0D3C}, {0x0D41, 0x0D44},
+ {0x0D4D, 0x0D4D}, {0x0D62, 0x0D63}, {0x0D81, 0x0D81},
+ {0x0DCA, 0x0DCA}, {0x0DD2, 0x0DD4}, {0x0DD6, 0x0DD6},
+ {0x0E31, 0x0E31}, {0x0E34, 0x0E3A}, {0x0E47, 0x0E4E},
+ {0x0EB1, 0x0EB1}, {0x0EB4, 0x0EBC}, {0x0EC8, 0x0ECE},
+ {0x0F18, 0x0F19}, {0x0F35, 0x0F35}, {0x0F37, 0x0F37},
+ {0x0F39, 0x0F39}, {0x0F71, 0x0F7E}, {0x0F80, 0x0F84},
+ {0x0F86, 0x0F87}, {0x0F8D, 0x0F97}, {0x0F99, 0x0FBC},
+ {0x0FC6, 0x0FC6}, {0x102D, 0x1030}, {0x1032, 0x1037},
+ {0x1039, 0x103A}, {0x103D, 0x103E}, {0x1058, 0x1059},
+ {0x105E, 0x1060}, {0x1071, 0x1074}, {0x1082, 0x1082},
+ {0x1085, 0x1086}, {0x108D, 0x108D}, {0x109D, 0x109D},
+ {0x135D, 0x135F}, {0x1712, 0x1714}, {0x1732, 0x1733},
+ {0x1752, 0x1753}, {0x1772, 0x1773}, {0x17B4, 0x17B5},
+ {0x17B7, 0x17BD}, {0x17C6, 0x17C6}, {0x17C9, 0x17D3},
+ {0x17DD, 0x17DD}, {0x180B, 0x180D}, {0x180F, 0x180F},
+ {0x1885, 0x1886}, {0x18A9, 0x18A9}, {0x1920, 0x1922},
+ {0x1927, 0x1928}, {0x1932, 0x1932}, {0x1939, 0x193B},
+ {0x1A17, 0x1A18}, {0x1A1B, 0x1A1B}, {0x1A56, 0x1A56},
+ {0x1A58, 0x1A5E}, {0x1A60, 0x1A60}, {0x1A62, 0x1A62},
+ {0x1A65, 0x1A6C}, {0x1A73, 0x1A7C}, {0x1A7F, 0x1A7F},
+ {0x1AB0, 0x1AF0}, {0x1B00, 0x1B03}, {0x1B34, 0x1B34},
+ {0x1B36, 0x1B3A}, {0x1B3C, 0x1B3C}, {0x1B42, 0x1B42},
+ {0x1B6B, 0x1B73}, {0x1B80, 0x1B81}, {0x1BA2, 0x1BA5},
+ {0x1BA8, 0x1BA9}, {0x1BAB, 0x1BAD}, {0x1BE6, 0x1BE6},
+ {0x1BE8, 0x1BE9}, {0x1BED, 0x1BED}, {0x1BEF, 0x1BF1},
+ {0x1C2C, 0x1C33}, {0x1C36, 0x1C37}, {0x1CD0, 0x1CD2},
+ {0x1CD4, 0x1CE0}, {0x1CE2, 0x1CE8}, {0x1CED, 0x1CED},
+ {0x1CF4, 0x1CF4}, {0x1CF8, 0x1CF9}, {0x1DC0, 0x1DFF},
+ {0x20D0, 0x20F0}, {0x2CEF, 0x2CF1}, {0x2D7F, 0x2D7F},
+ {0x2DE0, 0x2DFF}, {0x302A, 0x302D}, {0x3099, 0x309A},
+ {0xA66F, 0xA672}, {0xA674, 0xA67D}, {0xA69E, 0xA69F},
+ {0xA6F0, 0xA6F1}, {0xA802, 0xA802}, {0xA806, 0xA806},
+ {0xA80B, 0xA80B}, {0xA825, 0xA826}, {0xA82C, 0xA82C},
+ {0xA8C4, 0xA8C5}, {0xA8E0, 0xA8F1}, {0xA8FF, 0xA8FF},
+ {0xA926, 0xA92D}, {0xA947, 0xA951}, {0xA980, 0xA982},
+ {0xA9B3, 0xA9B3}, {0xA9B6, 0xA9B9}, {0xA9BC, 0xA9BD},
+ {0xA9E5, 0xA9E5}, {0xAA29, 0xAA2E}, {0xAA31, 0xAA32},
+ {0xAA35, 0xAA36}, {0xAA43, 0xAA43}, {0xAA4C, 0xAA4C},
+ {0xAA7C, 0xAA7C}, {0xAAB0, 0xAAB0}, {0xAAB2, 0xAAB4},
+ {0xAAB7, 0xAAB8}, {0xAABE, 0xAABF}, {0xAAC1, 0xAAC1},
+ {0xAAEC, 0xAAED}, {0xAAF6, 0xAAF6}, {0xABE5, 0xABE5},
+ {0xABE8, 0xABE8}, {0xABED, 0xABED}, {0xFB1E, 0xFB1E},
+ {0xFE00, 0xFE0F}, {0xFE20, 0xFE2F}, {0x101FD, 0x101FD},
+ {0x102E0, 0x102E0}, {0x10376, 0x1037A}, {0x10A01, 0x10A03},
+ {0x10A05, 0x10A06}, {0x10A0C, 0x10A0F}, {0x10A38, 0x10A3A},
+ {0x10A3F, 0x10A3F}, {0x10AE5, 0x10AE6}, {0x10D24, 0x10D27},
+ {0x10D69, 0x10D6D}, {0x10EAB, 0x10EAC}, {0x10ECB, 0x10ECF},
+ {0x10EF0, 0x10EFF}, {0x10F46, 0x10F50}, {0x10F82, 0x10F85},
+ {0x11001, 0x11001}, {0x11038, 0x11046}, {0x11070, 0x11070},
+ {0x11073, 0x11074}, {0x1107F, 0x11081}, {0x110B3, 0x110B6},
+ {0x110B9, 0x110BA}, {0x110C2, 0x110C2}, {0x11100, 0x11102},
+ {0x11127, 0x1112B}, {0x1112D, 0x11134}, {0x11173, 0x11173},
+ {0x11180, 0x11181}, {0x111B6, 0x111BE}, {0x111C9, 0x111CC},
+ {0x111CF, 0x111CF}, {0x1122F, 0x11231}, {0x11234, 0x11234},
+ {0x11236, 0x11237}, {0x1123E, 0x1123E}, {0x11241, 0x11241},
+ {0x112DF, 0x112DF}, {0x112E3, 0x112EA}, {0x11300, 0x11301},
+ {0x1133B, 0x1133C}, {0x11340, 0x11340}, {0x11366, 0x1136C},
+ {0x11370, 0x11374}, {0x113BB, 0x113C0}, {0x113CE, 0x113CE},
+ {0x113D0, 0x113D0}, {0x113D2, 0x113D2}, {0x113E1, 0x113E2},
+ {0x11438, 0x1143F}, {0x11442, 0x11444}, {0x11446, 0x11446},
+ {0x1145E, 0x1145E}, {0x114B3, 0x114B8}, {0x114BA, 0x114BA},
+ {0x114BF, 0x114C0}, {0x114C2, 0x114C3}, {0x115B2, 0x115B5},
+ {0x115BC, 0x115BD}, {0x115BF, 0x115C0}, {0x115DC, 0x115DD},
+ {0x11633, 0x1163A}, {0x1163D, 0x1163D}, {0x1163F, 0x11640},
+ {0x116AB, 0x116AB}, {0x116AD, 0x116AD}, {0x116B0, 0x116B5},
+ {0x116B7, 0x116B7}, {0x1171D, 0x1171D}, {0x1171F, 0x1171F},
+ {0x11722, 0x11725}, {0x11727, 0x1172B}, {0x1182F, 0x11837},
+ {0x11839, 0x1183A}, {0x1193B, 0x1193C}, {0x1193E, 0x1193E},
+ {0x11943, 0x11943}, {0x119D4, 0x119D7}, {0x119DA, 0x119DB},
+ {0x119E0, 0x119E0}, {0x11A01, 0x11A0A}, {0x11A33, 0x11A38},
+ {0x11A3B, 0x11A3E}, {0x11A47, 0x11A47}, {0x11A51, 0x11A56},
+ {0x11A59, 0x11A5B}, {0x11A8A, 0x11A96}, {0x11A98, 0x11A99},
+ {0x11B60, 0x11B60}, {0x11B62, 0x11B64}, {0x11B66, 0x11B66},
+ {0x11C30, 0x11C36}, {0x11C38, 0x11C3D}, {0x11C3F, 0x11C3F},
+ {0x11C92, 0x11CA7}, {0x11CAA, 0x11CB0}, {0x11CB2, 0x11CB3},
+ {0x11CB5, 0x11CB6}, {0x11D31, 0x11D36}, {0x11D3A, 0x11D3A},
+ {0x11D3C, 0x11D3D}, {0x11D3F, 0x11D45}, {0x11D47, 0x11D47},
+ {0x11D90, 0x11D91}, {0x11D95, 0x11D95}, {0x11D97, 0x11D97},
+ {0x11DF0, 0x11DF0}, {0x11EF3, 0x11EF4}, {0x11F00, 0x11F01},
+ {0x11F36, 0x11F3A}, {0x11F40, 0x11F40}, {0x11F42, 0x11F42},
+ {0x11F5A, 0x11F5A}, {0x13440, 0x13440}, {0x13447, 0x13455},
+ {0x1611E, 0x16129}, {0x1612D, 0x1612F}, {0x16AF0, 0x16AF4},
+ {0x16B30, 0x16B36}, {0x16F4F, 0x16F4F}, {0x16F8F, 0x16F92},
+ {0x16FE4, 0x16FE4}, {0x1BC9D, 0x1BC9E}, {0x1CF00, 0x1CF2D},
+ {0x1CF30, 0x1CF46}, {0x1D127, 0x1D128}, {0x1D167, 0x1D169},
+ {0x1D17B, 0x1D182}, {0x1D185, 0x1D18B}, {0x1D1AA, 0x1D1AD},
+ {0x1D242, 0x1D244}, {0x1D25B, 0x1D25C}, {0x1DA00, 0x1DA36},
+ {0x1DA3B, 0x1DA6C}, {0x1DA75, 0x1DA75}, {0x1DA84, 0x1DA84},
+ {0x1DA9B, 0x1DA9F}, {0x1DAA1, 0x1DAAF}, {0x1E000, 0x1E006},
+ {0x1E008, 0x1E018}, {0x1E01B, 0x1E021}, {0x1E023, 0x1E024},
+ {0x1E026, 0x1E02A}, {0x1E08F, 0x1E08F}, {0x1E130, 0x1E136},
+ {0x1E2AE, 0x1E2AE}, {0x1E2EC, 0x1E2EF}, {0x1E4EC, 0x1E4EF},
+ {0x1E5EE, 0x1E5EF}, {0x1E6E3, 0x1E6E3}, {0x1E6E6, 0x1E6E6},
+ {0x1E6EE, 0x1E6EF}, {0x1E6F5, 0x1E6F5}, {0x1E8D0, 0x1E8D6},
+ {0x1E944, 0x1E94A}, {0xE0100, 0xE01EF},
+};
+extern const CharRanges GeneratedCombiningCharacterRanges =
+ GeneratedCombiningCharacterRangesData;
+
+// East_Asian_Width=F|W
+static constexpr UnicodeCharRange GeneratedDoubleWidthCharacterRangesData[] = {
+ {0x1100, 0x115F}, {0x231A, 0x231B}, {0x2329, 0x232A},
+ {0x23E9, 0x23EC}, {0x23F0, 0x23F0}, {0x23F3, 0x23F3},
+ {0x25FD, 0x25FE}, {0x2614, 0x2615}, {0x2630, 0x2637},
+ {0x2648, 0x2653}, {0x267F, 0x267F}, {0x268A, 0x268F},
+ {0x2693, 0x2693}, {0x26A1, 0x26A1}, {0x26AA, 0x26AB},
+ {0x26BD, 0x26BE}, {0x26C4, 0x26C5}, {0x26CE, 0x26CE},
+ {0x26D4, 0x26D4}, {0x26EA, 0x26EA}, {0x26F2, 0x26F3},
+ {0x26F5, 0x26F5}, {0x26FA, 0x26FA}, {0x26FD, 0x26FD},
+ {0x2705, 0x2705}, {0x270A, 0x270B}, {0x2728, 0x2728},
+ {0x274C, 0x274C}, {0x274E, 0x274E}, {0x2753, 0x2755},
+ {0x2757, 0x2757}, {0x2795, 0x2797}, {0x27B0, 0x27B0},
+ {0x27BF, 0x27BF}, {0x2B1B, 0x2B1C}, {0x2B50, 0x2B50},
+ {0x2B55, 0x2B55}, {0x2E80, 0x2E99}, {0x2E9B, 0x2EF3},
+ {0x2F00, 0x2FD5}, {0x2FF0, 0x303E}, {0x3041, 0x3096},
+ {0x3099, 0x30FF}, {0x3105, 0x312F}, {0x3131, 0x318E},
+ {0x3190, 0x31E5}, {0x31EF, 0x321E}, {0x3220, 0x3247},
+ {0x3250, 0xA48C}, {0xA490, 0xA4C6}, {0xA960, 0xA97C},
+ {0xAC00, 0xD7A3}, {0xF900, 0xFAFF}, {0xFE10, 0xFE19},
+ {0xFE30, 0xFE52}, {0xFE54, 0xFE66}, {0xFE68, 0xFE6B},
+ {0xFF01, 0xFF60}, {0xFFE0, 0xFFE6}, {0x16FE0, 0x16FE4},
+ {0x16FF0, 0x16FF6}, {0x17000, 0x18CDA}, {0x18CFF, 0x18D20},
+ {0x18D80, 0x18DF2}, {0x18E00, 0x19191}, {0x191A0, 0x191D2},
+ {0x1AFF0, 0x1AFF3}, {0x1AFF5, 0x1AFFB}, {0x1AFFD, 0x1AFFE},
+ {0x1B000, 0x1B128}, {0x1B132, 0x1B132}, {0x1B150, 0x1B152},
+ {0x1B155, 0x1B155}, {0x1B164, 0x1B168}, {0x1B170, 0x1B2FB},
+ {0x1D300, 0x1D356}, {0x1D360, 0x1D376}, {0x1F004, 0x1F004},
+ {0x1F0CF, 0x1F0CF}, {0x1F18E, 0x1F18E}, {0x1F191, 0x1F19A},
+ {0x1F1AE, 0x1F1AE}, {0x1F200, 0x1F202}, {0x1F210, 0x1F23B},
+ {0x1F240, 0x1F248}, {0x1F250, 0x1F251}, {0x1F260, 0x1F265},
+ {0x1F300, 0x1F320}, {0x1F32D, 0x1F335}, {0x1F337, 0x1F37C},
+ {0x1F37E, 0x1F393}, {0x1F3A0, 0x1F3CA}, {0x1F3CF, 0x1F3D3},
+ {0x1F3E0, 0x1F3F0}, {0x1F3F4, 0x1F3F4}, {0x1F3F8, 0x1F43E},
+ {0x1F440, 0x1F440}, {0x1F442, 0x1F4FC}, {0x1F4FF, 0x1F53D},
+ {0x1F54B, 0x1F54E}, {0x1F550, 0x1F567}, {0x1F57A, 0x1F57A},
+ {0x1F595, 0x1F596}, {0x1F5A4, 0x1F5A4}, {0x1F5FB, 0x1F64F},
+ {0x1F680, 0x1F6C5}, {0x1F6CC, 0x1F6CC}, {0x1F6D0, 0x1F6D2},
+ {0x1F6D5, 0x1F6D9}, {0x1F6DC, 0x1F6DF}, {0x1F6EB, 0x1F6EC},
+ {0x1F6F4, 0x1F6FC}, {0x1F7DA, 0x1F7DA}, {0x1F7E0, 0x1F7EB},
+ {0x1F7F0, 0x1F7F0}, {0x1F90C, 0x1F93A}, {0x1F93C, 0x1F945},
+ {0x1F947, 0x1F9FF}, {0x1FA70, 0x1FA7C}, {0x1FA80, 0x1FAC6},
+ {0x1FAC8, 0x1FAC8}, {0x1FACC, 0x1FADD}, {0x1FADF, 0x1FAEB},
+ {0x1FAEF, 0x1FAFA}, {0x20000, 0x2FFFD}, {0x30000, 0x3FFFD},
+};
+extern const CharRanges GeneratedDoubleWidthCharacterRanges =
+ GeneratedDoubleWidthCharacterRangesData;
+
+} // namespace llvm::sys::unicode
diff --git a/llvm/utils/UnicodeData/CMakeLists.txt b/llvm/utils/UnicodeData/CMakeLists.txt
index 75edef55e4059..a43c7ed158caa 100644
--- a/llvm/utils/UnicodeData/CMakeLists.txt
+++ b/llvm/utils/UnicodeData/CMakeLists.txt
@@ -2,4 +2,20 @@ set(LLVM_LINK_COMPONENTS Support)
add_llvm_utility(UnicodeNameMappingGenerator
UnicodeNameMappingGenerator.cpp
+ PARTIAL_SOURCES_INTENDED
)
+
+if (NOT LLVM_ENABLE_LIBXML2)
+ message(STATUS "Not building UnicodeCharSetsGenerator because libxml2 is not available")
+ return()
+endif ()
+
+add_llvm_utility(UnicodeCharSetsGenerator
+ UnicodeCharSetsGenerator.cpp
+ PARTIAL_SOURCES_INTENDED
+)
+if (LLVM_USE_STATIC_LIBXML2)
+ target_link_libraries(UnicodeCharSetsGenerator PRIVATE LibXml2::LibXml2Static)
+else()
+ target_link_libraries(UnicodeCharSetsGenerator PRIVATE LibXml2::LibXml2)
+endif()
diff --git a/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp b/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp
new file mode 100644
index 0000000000000..fb1918bcddd70
--- /dev/null
+++ b/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp
@@ -0,0 +1,257 @@
+//===--- UnicodeCharSetsGenerator.cpp - Unicode character sets generator --===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file is used to generate clang/lib/Lex/UnicodeCharSetsGenerated.cpp and
+// llvm/lib/Support/UnicodeCharSetsGenerated.cpp from the XML representation of
+// the Unicode Character Database available at
+// https://www.unicode.org/Public/UCD/latest/ucdxml/
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/Twine.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/FormatVariadic.h"
+#include "llvm/Support/InitLLVM.h"
+#include "llvm/Support/ToolOutputFile.h"
+#include "llvm/Support/UnicodeCharRanges.h"
+#include "llvm/Support/WithColor.h"
+
+#include <libxml/xmlreader.h>
+#include <vector>
+
+using namespace llvm;
+
+static cl::opt<std::string> UCDFile(cl::Positional, cl::Required,
+ cl::desc("<UCD XML file>"));
+static cl::opt<std::string> ClangOutput(cl::Positional, cl::Required,
+ cl::desc("<clang output file>"));
+static cl::opt<std::string> LLVMOutput(cl::Positional, cl::Required,
+ cl::desc("<LLVM output file>"));
+
+static constexpr unsigned CodePointCount = 0x110000;
+
+static constexpr StringLiteral FileHeader = R"(
+//===----------------------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file was generated by llvm/utils/UnicodeData/UnicodeCharSetsGenerator
+// from the Unicode {0} Character Database. Do not edit manually.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Support/UnicodeCharRanges.h"
+
+using llvm::sys::UnicodeCharRange;
+using CharRanges = llvm::sys::UnicodeCharSet::CharRanges;
+
+namespace {1} {{
+)";
+
+[[noreturn]] static void error(const Twine &Message) {
+ WithColor::error() << Message << '\n';
+ exit(EXIT_FAILURE);
+}
+
+namespace {
+struct Properties {
+ unsigned XIDStart : 1;
+ unsigned XIDContinue : 1;
+ unsigned MathStart : 1;
+ unsigned MathContinue : 1;
+ unsigned Printable : 1;
+ unsigned Format : 1;
+ unsigned Combining : 1;
+ unsigned DoubleWidth : 1;
+};
+
+struct CharacterDatabase {
+ std::string Version;
+ std::vector<Properties> Chars;
+};
+
+struct XmlDeleter {
+ void operator()(xmlTextReader *Reader) { xmlFreeTextReader(Reader); }
+ void operator()(xmlChar *Ptr) { xmlFree(Ptr); }
+};
+} // namespace
+
+static StringRef fromXmlChar(const xmlChar *S) {
+ return S ? StringRef(reinterpret_cast<const char *>(S)) : StringRef();
+}
+
+static bool isPrintableGC(StringRef GC) {
+ return GC == "Zs" || (!GC.empty() && StringRef("LMNPS").contains(GC.front()));
+}
+
+// Loads the properties we care about from the flat XML representation of the
+// UCD, which describes each code point, or range of code points, with all of
+// its properties, e.g.
+// <char cp="0041" age="1.1" na="LATIN CAPITAL LETTER A" gc="Lu" ... />
+static CharacterDatabase loadDatabase(StringRef Path) {
+ std::unique_ptr<xmlTextReader, XmlDeleter> Reader(xmlReaderForFile(
+ Path.str().c_str(), /*encoding=*/nullptr, XML_PARSE_NONET));
+ if (!Reader)
+ error(Path + ": cannot open file");
+
+ CharacterDatabase Database;
+ Database.Chars.resize(CodePointCount);
+ int Status;
+ while ((Status = xmlTextReaderRead(Reader.get())) == 1) {
+ if (xmlTextReaderNodeType(Reader.get()) != XML_READER_TYPE_ELEMENT)
+ continue;
+ StringRef Element = fromXmlChar(xmlTextReaderConstLocalName(Reader.get()));
+ if (Element == "description") {
+ // <description>Unicode 18.0.0</description>
+ std::unique_ptr<xmlChar, XmlDeleter> Description(
+ xmlTextReaderReadString(Reader.get()));
+ if (Description)
+ Database.Version = fromXmlChar(Description.get())
+ .rsplit(' ')
+ .second.rsplit('.')
+ .first.str();
+ continue;
+ }
+ if (!is_contained({"char", "reserved", "noncharacter", "surrogate"},
+ Element))
+ continue;
+
+ unsigned First = 0, Last = 0;
+ bool HasRange = false;
+ unsigned Found = 0;
+ Properties P = {};
+ while (xmlTextReaderMoveToNextAttribute(Reader.get()) == 1) {
+ StringRef Name = fromXmlChar(xmlTextReaderConstLocalName(Reader.get()));
+ StringRef Value = fromXmlChar(xmlTextReaderConstValue(Reader.get()));
+ if (Name == "cp" || Name == "first-cp") {
+ if (Value.getAsInteger(16, First))
+ break;
+ if (Name == "cp")
+ Last = First;
+ HasRange = true;
+ } else if (Name == "last-cp") {
+ if (Value.getAsInteger(16, Last))
+ HasRange = false;
+ } else if (Name == "XIDS") {
+ P.XIDStart = Value == "Y";
+ ++Found;
+ } else if (Name == "XIDC") {
+ P.XIDContinue = Value == "Y";
+ ++Found;
+ } else if (Name == "ID_Compat_Math_Start") {
+ P.MathStart = Value == "Y";
+ ++Found;
+ } else if (Name == "ID_Compat_Math_Continue") {
+ P.MathContinue = Value == "Y";
+ ++Found;
+ } else if (Name == "gc") {
+ P.Printable = isPrintableGC(Value);
+ P.Format = Value == "Cf";
+ P.Combining = Value == "Mn" || Value == "Me";
+ ++Found;
+ } else if (Name == "ea") {
+ P.DoubleWidth = Value == "F" || Value == "W";
+ ++Found;
+ }
+ }
+
+ if (!HasRange || First > Last || Last >= CodePointCount || Found != 6)
+ error(Path + ":" + Twine(xmlTextReaderGetParserLineNumber(Reader.get())) +
+ ": invalid <" + Element + "> element");
+ llvm::fill(MutableArrayRef(Database.Chars).slice(First, Last - First + 1),
+ P);
+ }
+ if (Status != 0)
+ error(Path + ": failed to parse XML");
+ if (Database.Version.empty())
+ error(Path + ": missing Unicode version");
+ return Database;
+}
+
+template <typename Predicate>
+static void emitTable(raw_ostream &OS, StringRef Name, StringRef Description,
+ ArrayRef<Properties> Chars, Predicate Pred) {
+ SmallVector<sys::UnicodeCharRange> Ranges;
+ for (unsigned CodePoint = 0, E = Chars.size(); CodePoint != E; ++CodePoint) {
+ if (!Pred(Chars[CodePoint]))
+ continue;
+ if (!Ranges.empty() && Ranges.back().Upper + 1 == CodePoint)
+ Ranges.back().Upper = CodePoint;
+ else
+ Ranges.push_back({CodePoint, CodePoint});
+ }
+
+ OS << "\n// " << Description << "\nstatic constexpr UnicodeCharRange " << Name
+ << "Data[] = {\n";
+ for (sys::UnicodeCharRange Range : Ranges)
+ OS << formatv(" {{{0:X4}, {1:X4}},\n", Range.Lower, Range.Upper);
+ OS << "};\nextern const CharRanges " << Name << " = " << Name << "Data;\n";
+}
+
+static void writeFile(StringRef Path, StringRef Namespace, StringRef Version,
+ function_ref<void(raw_ostream &)> EmitTables) {
+ std::error_code EC;
+ ToolOutputFile Out(Path, EC, sys::fs::OF_Text);
+ if (EC)
+ error(Path + ": " + EC.message());
+
+ Out.os() << formatv(FileHeader.drop_front().data(), Version, Namespace);
+ EmitTables(Out.os());
+ Out.os() << "\n} // namespace " << Namespace << '\n';
+ Out.keep();
+}
+
+int main(int argc, char **argv) {
+ InitLLVM X(argc, argv);
+ cl::ParseCommandLineOptions(
+ argc, argv,
+ "Unicode character sets generator\n\n"
+ " Generates clang/lib/Lex/UnicodeCharSetsGenerated.cpp and\n"
+ " llvm/lib/Support/UnicodeCharSetsGenerated.cpp from the flat XML\n"
+ " representation of the Unicode Character Database, which can be\n"
+ " downloaded from https://www.unicode.org/Public/UCD/latest/ucdxml/\n"
+ " The generated files must then be formatted with clang-format.\n");
+
+ CharacterDatabase UCD = loadDatabase(UCDFile);
+ ArrayRef<Properties> Chars = UCD.Chars;
+
+ writeFile(ClangOutput, "clang", UCD.Version, [&](raw_ostream &OS) {
+ emitTable(OS, "GeneratedXIDStartRanges", "XID_Start", Chars,
+ [](Properties P) { return P.XIDStart; });
+ emitTable(OS, "GeneratedXIDContinueRanges",
+ "XID_Continue, excluding XID_Start", Chars,
+ [](Properties P) { return P.XIDContinue && !P.XIDStart; });
+ emitTable(OS, "GeneratedMathematicalNotationProfileIDStartRanges",
+ "ID_Compat_Math_Start", Chars,
+ [](Properties P) { return P.MathStart; });
+ emitTable(OS, "GeneratedMathematicalNotationProfileIDContinueRanges",
+ "ID_Compat_Math_Continue, excluding ID_Compat_Math_Start", Chars,
+ [](Properties P) { return P.MathContinue && !P.MathStart; });
+ });
+ writeFile(
+ LLVMOutput, "llvm::sys::unicode", UCD.Version, [&](raw_ostream &OS) {
+ emitTable(OS, "GeneratedPrintableRanges",
+ "General_Category=L|M|N|P|S|Zs", Chars,
+ [](Properties P) { return P.Printable; });
+ emitTable(OS, "GeneratedFormatCharacterRanges", "General_Category=Cf",
+ Chars, [](Properties P) { return P.Format; });
+ emitTable(OS, "GeneratedCombiningCharacterRanges",
+ "General_Category=Mn|Me", Chars,
+ [](Properties P) { return P.Combining; });
+ emitTable(OS, "GeneratedDoubleWidthCharacterRanges",
+ "East_Asian_Width=F|W", Chars,
+ [](Properties P) { return P.DoubleWidth; });
+ });
+}
>From 991f94010b7d8ae90c95a3a60e891638c8f174e9 Mon Sep 17 00:00:00 2001
From: Corentin Jabot <corentinjabot at gmail.com>
Date: Mon, 5 Oct 2026 16:07:46 +0200
Subject: [PATCH 2/5] address feedback, fix build
---
clang/lib/Lex/UnicodeCharSetsGenerated.cpp | 19 ++++---
clang/test/Lexer/unicode.c | 2 +-
llvm/lib/Support/UnicodeCharSetsGenerated.cpp | 16 +++---
.../UnicodeData/UnicodeCharSetsGenerator.cpp | 54 +++++++++----------
4 files changed, 44 insertions(+), 47 deletions(-)
diff --git a/clang/lib/Lex/UnicodeCharSetsGenerated.cpp b/clang/lib/Lex/UnicodeCharSetsGenerated.cpp
index f1967d496d9da..86b0536c19053 100644
--- a/clang/lib/Lex/UnicodeCharSetsGenerated.cpp
+++ b/clang/lib/Lex/UnicodeCharSetsGenerated.cpp
@@ -16,8 +16,6 @@
using llvm::sys::UnicodeCharRange;
using CharRanges = llvm::sys::UnicodeCharSet::CharRanges;
-namespace clang {
-
// XID_Start
static constexpr UnicodeCharRange GeneratedXIDStartRangesData[] = {
{0x0041, 0x005A}, {0x0061, 0x007A}, {0x00AA, 0x00AA},
@@ -255,7 +253,8 @@ static constexpr UnicodeCharRange GeneratedXIDStartRangesData[] = {
{0x2CEB0, 0x2EBE0}, {0x2EBF0, 0x2EE5D}, {0x2F800, 0x2FA1D},
{0x30000, 0x3134A}, {0x31350, 0x33479}, {0x3D000, 0x3FC3F},
};
-extern const CharRanges GeneratedXIDStartRanges = GeneratedXIDStartRangesData;
+extern const CharRanges clang::GeneratedXIDStartRanges =
+ GeneratedXIDStartRangesData;
// XID_Continue, excluding XID_Start
static constexpr UnicodeCharRange GeneratedXIDContinueRangesData[] = {
@@ -396,7 +395,7 @@ static constexpr UnicodeCharRange GeneratedXIDContinueRangesData[] = {
{0x1E8D0, 0x1E8D6}, {0x1E944, 0x1E94A}, {0x1E950, 0x1E959},
{0x1FBF0, 0x1FBF9}, {0xE0100, 0xE01EF},
};
-extern const CharRanges GeneratedXIDContinueRanges =
+extern const CharRanges clang::GeneratedXIDContinueRanges =
GeneratedXIDContinueRangesData;
// ID_Compat_Math_Start
@@ -408,8 +407,9 @@ static constexpr UnicodeCharRange
{0x1D76F, 0x1D76F}, {0x1D789, 0x1D789}, {0x1D7A9, 0x1D7A9},
{0x1D7C3, 0x1D7C3},
};
-extern const CharRanges GeneratedMathematicalNotationProfileIDStartRanges =
- GeneratedMathematicalNotationProfileIDStartRangesData;
+extern const CharRanges
+ clang::GeneratedMathematicalNotationProfileIDStartRanges =
+ GeneratedMathematicalNotationProfileIDStartRangesData;
// ID_Compat_Math_Continue, excluding ID_Compat_Math_Start
static constexpr UnicodeCharRange
@@ -417,7 +417,6 @@ static constexpr UnicodeCharRange
{0x00B2, 0x00B3}, {0x00B9, 0x00B9}, {0x2070, 0x2070},
{0x2074, 0x207E}, {0x2080, 0x208E},
};
-extern const CharRanges GeneratedMathematicalNotationProfileIDContinueRanges =
- GeneratedMathematicalNotationProfileIDContinueRangesData;
-
-} // namespace clang
+extern const CharRanges
+ clang::GeneratedMathematicalNotationProfileIDContinueRanges =
+ GeneratedMathematicalNotationProfileIDContinueRangesData;
diff --git a/clang/test/Lexer/unicode.c b/clang/test/Lexer/unicode.c
index 53acca6ba04b7..963eca04e0908 100644
--- a/clang/test/Lexer/unicode.c
+++ b/clang/test/Lexer/unicode.c
@@ -47,7 +47,7 @@ extern int \U0001E4D0; // ๐ NAG MUNDARI LETTER O - Added in Unicode 15
extern int \u{2EBF0}; // CJK UNIFIED IDEOGRAPH-2EBF0 - Added in Unicode 15.1
extern int \u{10D5A}; // GARAY CAPITAL LETTER DA - Added in Unicode 16.0
extern int \u{16EBE}; // BERIA ERFE SMALL LETTER EH - Added in Unicode 17.0
-extern int \u{16D80}; // CHISOI LETTER A - Added in Unicode 18.0
+extern int \u{18E00}; // JURCHEN CHARACTER-18E00 - Added in Unicode 18.0
extern int a\N{TANGSA LETTER GA};
extern int a\N{ZWJ};
extern int a\N{TANGSALETTERGA}; // expected-error {{'TANGSALETTERGA' is not a valid Unicode character name}} \
diff --git a/llvm/lib/Support/UnicodeCharSetsGenerated.cpp b/llvm/lib/Support/UnicodeCharSetsGenerated.cpp
index 0a30fe11c2ca9..f50dfd9744bf2 100644
--- a/llvm/lib/Support/UnicodeCharSetsGenerated.cpp
+++ b/llvm/lib/Support/UnicodeCharSetsGenerated.cpp
@@ -16,8 +16,6 @@
using llvm::sys::UnicodeCharRange;
using CharRanges = llvm::sys::UnicodeCharSet::CharRanges;
-namespace llvm::sys::unicode {
-
// General_Category=L|M|N|P|S|Zs
static constexpr UnicodeCharRange GeneratedPrintableRangesData[] = {
{0x0020, 0x007E}, {0x00A0, 0x00AC}, {0x00AE, 0x0377},
@@ -268,7 +266,8 @@ static constexpr UnicodeCharRange GeneratedPrintableRangesData[] = {
{0x2F800, 0x2FA1D}, {0x30000, 0x3134A}, {0x31350, 0x33479},
{0x3D000, 0x3FC3F}, {0xE0100, 0xE01EF},
};
-extern const CharRanges GeneratedPrintableRanges = GeneratedPrintableRangesData;
+extern const CharRanges llvm::sys::unicode::GeneratedPrintableRanges =
+ GeneratedPrintableRangesData;
// General_Category=Cf
static constexpr UnicodeCharRange GeneratedFormatCharacterRangesData[] = {
@@ -280,7 +279,7 @@ static constexpr UnicodeCharRange GeneratedFormatCharacterRangesData[] = {
{0x110CD, 0x110CD}, {0x13430, 0x1343F}, {0x1BCA0, 0x1BCA3},
{0x1D173, 0x1D17A}, {0xE0001, 0xE0001}, {0xE0020, 0xE007F},
};
-extern const CharRanges GeneratedFormatCharacterRanges =
+extern const CharRanges llvm::sys::unicode::GeneratedFormatCharacterRanges =
GeneratedFormatCharacterRangesData;
// General_Category=Mn|Me
@@ -408,7 +407,7 @@ static constexpr UnicodeCharRange GeneratedCombiningCharacterRangesData[] = {
{0x1E6EE, 0x1E6EF}, {0x1E6F5, 0x1E6F5}, {0x1E8D0, 0x1E8D6},
{0x1E944, 0x1E94A}, {0xE0100, 0xE01EF},
};
-extern const CharRanges GeneratedCombiningCharacterRanges =
+extern const CharRanges llvm::sys::unicode::GeneratedCombiningCharacterRanges =
GeneratedCombiningCharacterRangesData;
// East_Asian_Width=F|W
@@ -456,7 +455,6 @@ static constexpr UnicodeCharRange GeneratedDoubleWidthCharacterRangesData[] = {
{0x1FAC8, 0x1FAC8}, {0x1FACC, 0x1FADD}, {0x1FADF, 0x1FAEB},
{0x1FAEF, 0x1FAFA}, {0x20000, 0x2FFFD}, {0x30000, 0x3FFFD},
};
-extern const CharRanges GeneratedDoubleWidthCharacterRanges =
- GeneratedDoubleWidthCharacterRangesData;
-
-} // namespace llvm::sys::unicode
+extern const CharRanges
+ llvm::sys::unicode::GeneratedDoubleWidthCharacterRanges =
+ GeneratedDoubleWidthCharacterRangesData;
diff --git a/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp b/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp
index fb1918bcddd70..d549af84858e9 100644
--- a/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp
+++ b/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp
@@ -56,8 +56,6 @@ static constexpr StringLiteral FileHeader = R"(
using llvm::sys::UnicodeCharRange;
using CharRanges = llvm::sys::UnicodeCharSet::CharRanges;
-
-namespace {1} {{
)";
[[noreturn]] static void error(const Twine &Message) {
@@ -181,8 +179,9 @@ static CharacterDatabase loadDatabase(StringRef Path) {
}
template <typename Predicate>
-static void emitTable(raw_ostream &OS, StringRef Name, StringRef Description,
- ArrayRef<Properties> Chars, Predicate Pred) {
+static void emitTable(raw_ostream &OS, StringRef Namespace, StringRef Name,
+ StringRef Description, ArrayRef<Properties> Chars,
+ Predicate Pred) {
SmallVector<sys::UnicodeCharRange> Ranges;
for (unsigned CodePoint = 0, E = Chars.size(); CodePoint != E; ++CodePoint) {
if (!Pred(Chars[CodePoint]))
@@ -197,19 +196,19 @@ static void emitTable(raw_ostream &OS, StringRef Name, StringRef Description,
<< "Data[] = {\n";
for (sys::UnicodeCharRange Range : Ranges)
OS << formatv(" {{{0:X4}, {1:X4}},\n", Range.Lower, Range.Upper);
- OS << "};\nextern const CharRanges " << Name << " = " << Name << "Data;\n";
+ OS << "};\nextern const CharRanges " << Namespace << "::" << Name << " = "
+ << Name << "Data;\n";
}
-static void writeFile(StringRef Path, StringRef Namespace, StringRef Version,
+static void writeFile(StringRef Path, StringRef Version,
function_ref<void(raw_ostream &)> EmitTables) {
std::error_code EC;
ToolOutputFile Out(Path, EC, sys::fs::OF_Text);
if (EC)
error(Path + ": " + EC.message());
- Out.os() << formatv(FileHeader.drop_front().data(), Version, Namespace);
+ Out.os() << formatv(FileHeader.drop_front().data(), Version);
EmitTables(Out.os());
- Out.os() << "\n} // namespace " << Namespace << '\n';
Out.keep();
}
@@ -227,31 +226,32 @@ int main(int argc, char **argv) {
CharacterDatabase UCD = loadDatabase(UCDFile);
ArrayRef<Properties> Chars = UCD.Chars;
- writeFile(ClangOutput, "clang", UCD.Version, [&](raw_ostream &OS) {
- emitTable(OS, "GeneratedXIDStartRanges", "XID_Start", Chars,
+ writeFile(ClangOutput, UCD.Version, [&](raw_ostream &OS) {
+ emitTable(OS, "clang", "GeneratedXIDStartRanges", "XID_Start", Chars,
[](Properties P) { return P.XIDStart; });
- emitTable(OS, "GeneratedXIDContinueRanges",
+ emitTable(OS, "clang", "GeneratedXIDContinueRanges",
"XID_Continue, excluding XID_Start", Chars,
[](Properties P) { return P.XIDContinue && !P.XIDStart; });
- emitTable(OS, "GeneratedMathematicalNotationProfileIDStartRanges",
+ emitTable(OS, "clang", "GeneratedMathematicalNotationProfileIDStartRanges",
"ID_Compat_Math_Start", Chars,
[](Properties P) { return P.MathStart; });
- emitTable(OS, "GeneratedMathematicalNotationProfileIDContinueRanges",
+ emitTable(OS, "clang",
+ "GeneratedMathematicalNotationProfileIDContinueRanges",
"ID_Compat_Math_Continue, excluding ID_Compat_Math_Start", Chars,
[](Properties P) { return P.MathContinue && !P.MathStart; });
});
- writeFile(
- LLVMOutput, "llvm::sys::unicode", UCD.Version, [&](raw_ostream &OS) {
- emitTable(OS, "GeneratedPrintableRanges",
- "General_Category=L|M|N|P|S|Zs", Chars,
- [](Properties P) { return P.Printable; });
- emitTable(OS, "GeneratedFormatCharacterRanges", "General_Category=Cf",
- Chars, [](Properties P) { return P.Format; });
- emitTable(OS, "GeneratedCombiningCharacterRanges",
- "General_Category=Mn|Me", Chars,
- [](Properties P) { return P.Combining; });
- emitTable(OS, "GeneratedDoubleWidthCharacterRanges",
- "East_Asian_Width=F|W", Chars,
- [](Properties P) { return P.DoubleWidth; });
- });
+ writeFile(LLVMOutput, UCD.Version, [&](raw_ostream &OS) {
+ emitTable(OS, "llvm::sys::unicode", "GeneratedPrintableRanges",
+ "General_Category=L|M|N|P|S|Zs", Chars,
+ [](Properties P) { return P.Printable; });
+ emitTable(OS, "llvm::sys::unicode", "GeneratedFormatCharacterRanges",
+ "General_Category=Cf", Chars,
+ [](Properties P) { return P.Format; });
+ emitTable(OS, "llvm::sys::unicode", "GeneratedCombiningCharacterRanges",
+ "General_Category=Mn|Me", Chars,
+ [](Properties P) { return P.Combining; });
+ emitTable(OS, "llvm::sys::unicode", "GeneratedDoubleWidthCharacterRanges",
+ "East_Asian_Width=F|W", Chars,
+ [](Properties P) { return P.DoubleWidth; });
+ });
}
>From 5dee5bb73324b7a3458068968cbfcf69d3b246fa Mon Sep 17 00:00:00 2001
From: Corentin Jabot <corentinjabot at gmail.com>
Date: Mon, 5 Oct 2026 17:28:41 +0200
Subject: [PATCH 3/5] Restore namespaces in generated files.
The namespace does not exist in this file, and
artificially opening it doesn;t lead to better code,
so this revert the previous attempt at qualifying namespaces
---
clang/lib/Lex/UnicodeCharSetsGenerated.cpp | 19 +++----
llvm/lib/Support/UnicodeCharSetsGenerated.cpp | 16 +++---
.../UnicodeData/UnicodeCharSetsGenerator.cpp | 54 +++++++++----------
3 files changed, 46 insertions(+), 43 deletions(-)
diff --git a/clang/lib/Lex/UnicodeCharSetsGenerated.cpp b/clang/lib/Lex/UnicodeCharSetsGenerated.cpp
index 86b0536c19053..f1967d496d9da 100644
--- a/clang/lib/Lex/UnicodeCharSetsGenerated.cpp
+++ b/clang/lib/Lex/UnicodeCharSetsGenerated.cpp
@@ -16,6 +16,8 @@
using llvm::sys::UnicodeCharRange;
using CharRanges = llvm::sys::UnicodeCharSet::CharRanges;
+namespace clang {
+
// XID_Start
static constexpr UnicodeCharRange GeneratedXIDStartRangesData[] = {
{0x0041, 0x005A}, {0x0061, 0x007A}, {0x00AA, 0x00AA},
@@ -253,8 +255,7 @@ static constexpr UnicodeCharRange GeneratedXIDStartRangesData[] = {
{0x2CEB0, 0x2EBE0}, {0x2EBF0, 0x2EE5D}, {0x2F800, 0x2FA1D},
{0x30000, 0x3134A}, {0x31350, 0x33479}, {0x3D000, 0x3FC3F},
};
-extern const CharRanges clang::GeneratedXIDStartRanges =
- GeneratedXIDStartRangesData;
+extern const CharRanges GeneratedXIDStartRanges = GeneratedXIDStartRangesData;
// XID_Continue, excluding XID_Start
static constexpr UnicodeCharRange GeneratedXIDContinueRangesData[] = {
@@ -395,7 +396,7 @@ static constexpr UnicodeCharRange GeneratedXIDContinueRangesData[] = {
{0x1E8D0, 0x1E8D6}, {0x1E944, 0x1E94A}, {0x1E950, 0x1E959},
{0x1FBF0, 0x1FBF9}, {0xE0100, 0xE01EF},
};
-extern const CharRanges clang::GeneratedXIDContinueRanges =
+extern const CharRanges GeneratedXIDContinueRanges =
GeneratedXIDContinueRangesData;
// ID_Compat_Math_Start
@@ -407,9 +408,8 @@ static constexpr UnicodeCharRange
{0x1D76F, 0x1D76F}, {0x1D789, 0x1D789}, {0x1D7A9, 0x1D7A9},
{0x1D7C3, 0x1D7C3},
};
-extern const CharRanges
- clang::GeneratedMathematicalNotationProfileIDStartRanges =
- GeneratedMathematicalNotationProfileIDStartRangesData;
+extern const CharRanges GeneratedMathematicalNotationProfileIDStartRanges =
+ GeneratedMathematicalNotationProfileIDStartRangesData;
// ID_Compat_Math_Continue, excluding ID_Compat_Math_Start
static constexpr UnicodeCharRange
@@ -417,6 +417,7 @@ static constexpr UnicodeCharRange
{0x00B2, 0x00B3}, {0x00B9, 0x00B9}, {0x2070, 0x2070},
{0x2074, 0x207E}, {0x2080, 0x208E},
};
-extern const CharRanges
- clang::GeneratedMathematicalNotationProfileIDContinueRanges =
- GeneratedMathematicalNotationProfileIDContinueRangesData;
+extern const CharRanges GeneratedMathematicalNotationProfileIDContinueRanges =
+ GeneratedMathematicalNotationProfileIDContinueRangesData;
+
+} // namespace clang
diff --git a/llvm/lib/Support/UnicodeCharSetsGenerated.cpp b/llvm/lib/Support/UnicodeCharSetsGenerated.cpp
index f50dfd9744bf2..0a30fe11c2ca9 100644
--- a/llvm/lib/Support/UnicodeCharSetsGenerated.cpp
+++ b/llvm/lib/Support/UnicodeCharSetsGenerated.cpp
@@ -16,6 +16,8 @@
using llvm::sys::UnicodeCharRange;
using CharRanges = llvm::sys::UnicodeCharSet::CharRanges;
+namespace llvm::sys::unicode {
+
// General_Category=L|M|N|P|S|Zs
static constexpr UnicodeCharRange GeneratedPrintableRangesData[] = {
{0x0020, 0x007E}, {0x00A0, 0x00AC}, {0x00AE, 0x0377},
@@ -266,8 +268,7 @@ static constexpr UnicodeCharRange GeneratedPrintableRangesData[] = {
{0x2F800, 0x2FA1D}, {0x30000, 0x3134A}, {0x31350, 0x33479},
{0x3D000, 0x3FC3F}, {0xE0100, 0xE01EF},
};
-extern const CharRanges llvm::sys::unicode::GeneratedPrintableRanges =
- GeneratedPrintableRangesData;
+extern const CharRanges GeneratedPrintableRanges = GeneratedPrintableRangesData;
// General_Category=Cf
static constexpr UnicodeCharRange GeneratedFormatCharacterRangesData[] = {
@@ -279,7 +280,7 @@ static constexpr UnicodeCharRange GeneratedFormatCharacterRangesData[] = {
{0x110CD, 0x110CD}, {0x13430, 0x1343F}, {0x1BCA0, 0x1BCA3},
{0x1D173, 0x1D17A}, {0xE0001, 0xE0001}, {0xE0020, 0xE007F},
};
-extern const CharRanges llvm::sys::unicode::GeneratedFormatCharacterRanges =
+extern const CharRanges GeneratedFormatCharacterRanges =
GeneratedFormatCharacterRangesData;
// General_Category=Mn|Me
@@ -407,7 +408,7 @@ static constexpr UnicodeCharRange GeneratedCombiningCharacterRangesData[] = {
{0x1E6EE, 0x1E6EF}, {0x1E6F5, 0x1E6F5}, {0x1E8D0, 0x1E8D6},
{0x1E944, 0x1E94A}, {0xE0100, 0xE01EF},
};
-extern const CharRanges llvm::sys::unicode::GeneratedCombiningCharacterRanges =
+extern const CharRanges GeneratedCombiningCharacterRanges =
GeneratedCombiningCharacterRangesData;
// East_Asian_Width=F|W
@@ -455,6 +456,7 @@ static constexpr UnicodeCharRange GeneratedDoubleWidthCharacterRangesData[] = {
{0x1FAC8, 0x1FAC8}, {0x1FACC, 0x1FADD}, {0x1FADF, 0x1FAEB},
{0x1FAEF, 0x1FAFA}, {0x20000, 0x2FFFD}, {0x30000, 0x3FFFD},
};
-extern const CharRanges
- llvm::sys::unicode::GeneratedDoubleWidthCharacterRanges =
- GeneratedDoubleWidthCharacterRangesData;
+extern const CharRanges GeneratedDoubleWidthCharacterRanges =
+ GeneratedDoubleWidthCharacterRangesData;
+
+} // namespace llvm::sys::unicode
diff --git a/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp b/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp
index d549af84858e9..fb1918bcddd70 100644
--- a/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp
+++ b/llvm/utils/UnicodeData/UnicodeCharSetsGenerator.cpp
@@ -56,6 +56,8 @@ static constexpr StringLiteral FileHeader = R"(
using llvm::sys::UnicodeCharRange;
using CharRanges = llvm::sys::UnicodeCharSet::CharRanges;
+
+namespace {1} {{
)";
[[noreturn]] static void error(const Twine &Message) {
@@ -179,9 +181,8 @@ static CharacterDatabase loadDatabase(StringRef Path) {
}
template <typename Predicate>
-static void emitTable(raw_ostream &OS, StringRef Namespace, StringRef Name,
- StringRef Description, ArrayRef<Properties> Chars,
- Predicate Pred) {
+static void emitTable(raw_ostream &OS, StringRef Name, StringRef Description,
+ ArrayRef<Properties> Chars, Predicate Pred) {
SmallVector<sys::UnicodeCharRange> Ranges;
for (unsigned CodePoint = 0, E = Chars.size(); CodePoint != E; ++CodePoint) {
if (!Pred(Chars[CodePoint]))
@@ -196,19 +197,19 @@ static void emitTable(raw_ostream &OS, StringRef Namespace, StringRef Name,
<< "Data[] = {\n";
for (sys::UnicodeCharRange Range : Ranges)
OS << formatv(" {{{0:X4}, {1:X4}},\n", Range.Lower, Range.Upper);
- OS << "};\nextern const CharRanges " << Namespace << "::" << Name << " = "
- << Name << "Data;\n";
+ OS << "};\nextern const CharRanges " << Name << " = " << Name << "Data;\n";
}
-static void writeFile(StringRef Path, StringRef Version,
+static void writeFile(StringRef Path, StringRef Namespace, StringRef Version,
function_ref<void(raw_ostream &)> EmitTables) {
std::error_code EC;
ToolOutputFile Out(Path, EC, sys::fs::OF_Text);
if (EC)
error(Path + ": " + EC.message());
- Out.os() << formatv(FileHeader.drop_front().data(), Version);
+ Out.os() << formatv(FileHeader.drop_front().data(), Version, Namespace);
EmitTables(Out.os());
+ Out.os() << "\n} // namespace " << Namespace << '\n';
Out.keep();
}
@@ -226,32 +227,31 @@ int main(int argc, char **argv) {
CharacterDatabase UCD = loadDatabase(UCDFile);
ArrayRef<Properties> Chars = UCD.Chars;
- writeFile(ClangOutput, UCD.Version, [&](raw_ostream &OS) {
- emitTable(OS, "clang", "GeneratedXIDStartRanges", "XID_Start", Chars,
+ writeFile(ClangOutput, "clang", UCD.Version, [&](raw_ostream &OS) {
+ emitTable(OS, "GeneratedXIDStartRanges", "XID_Start", Chars,
[](Properties P) { return P.XIDStart; });
- emitTable(OS, "clang", "GeneratedXIDContinueRanges",
+ emitTable(OS, "GeneratedXIDContinueRanges",
"XID_Continue, excluding XID_Start", Chars,
[](Properties P) { return P.XIDContinue && !P.XIDStart; });
- emitTable(OS, "clang", "GeneratedMathematicalNotationProfileIDStartRanges",
+ emitTable(OS, "GeneratedMathematicalNotationProfileIDStartRanges",
"ID_Compat_Math_Start", Chars,
[](Properties P) { return P.MathStart; });
- emitTable(OS, "clang",
- "GeneratedMathematicalNotationProfileIDContinueRanges",
+ emitTable(OS, "GeneratedMathematicalNotationProfileIDContinueRanges",
"ID_Compat_Math_Continue, excluding ID_Compat_Math_Start", Chars,
[](Properties P) { return P.MathContinue && !P.MathStart; });
});
- writeFile(LLVMOutput, UCD.Version, [&](raw_ostream &OS) {
- emitTable(OS, "llvm::sys::unicode", "GeneratedPrintableRanges",
- "General_Category=L|M|N|P|S|Zs", Chars,
- [](Properties P) { return P.Printable; });
- emitTable(OS, "llvm::sys::unicode", "GeneratedFormatCharacterRanges",
- "General_Category=Cf", Chars,
- [](Properties P) { return P.Format; });
- emitTable(OS, "llvm::sys::unicode", "GeneratedCombiningCharacterRanges",
- "General_Category=Mn|Me", Chars,
- [](Properties P) { return P.Combining; });
- emitTable(OS, "llvm::sys::unicode", "GeneratedDoubleWidthCharacterRanges",
- "East_Asian_Width=F|W", Chars,
- [](Properties P) { return P.DoubleWidth; });
- });
+ writeFile(
+ LLVMOutput, "llvm::sys::unicode", UCD.Version, [&](raw_ostream &OS) {
+ emitTable(OS, "GeneratedPrintableRanges",
+ "General_Category=L|M|N|P|S|Zs", Chars,
+ [](Properties P) { return P.Printable; });
+ emitTable(OS, "GeneratedFormatCharacterRanges", "General_Category=Cf",
+ Chars, [](Properties P) { return P.Format; });
+ emitTable(OS, "GeneratedCombiningCharacterRanges",
+ "General_Category=Mn|Me", Chars,
+ [](Properties P) { return P.Combining; });
+ emitTable(OS, "GeneratedDoubleWidthCharacterRanges",
+ "East_Asian_Width=F|W", Chars,
+ [](Properties P) { return P.DoubleWidth; });
+ });
}
>From fa9355de315e07e667a563fa67d2f49e086d520a Mon Sep 17 00:00:00 2001
From: Corentin Jabot <corentinjabot at gmail.com>
Date: Wed, 7 Oct 2026 12:53:23 +0200
Subject: [PATCH 4/5] Move the doc to LLVM and improve it
---
llvm/docs/HowToUpdateUnicodeTables.md | 81 +++++++++++++++++++++++++++
1 file changed, 81 insertions(+)
create mode 100644 llvm/docs/HowToUpdateUnicodeTables.md
diff --git a/llvm/docs/HowToUpdateUnicodeTables.md b/llvm/docs/HowToUpdateUnicodeTables.md
new file mode 100644
index 0000000000000..06feab39501e3
--- /dev/null
+++ b/llvm/docs/HowToUpdateUnicodeTables.md
@@ -0,0 +1,81 @@
+# How to Update Unicode Tables
+
+LLVM and Clang use a subset of the [Unicode Character Database](https://www.unicode.org/ucd/)
+(UCD) for identifiers, `\N{...}` named characters, diagnostic column width,
+and simple case folding. The corresponding tables are generated from the UCD
+into Clang and LLVM Support.
+
+## Downloading the Unicode data
+
+Download the following files, replacing `<VERSION>` with a Unicode version such
+as `18.0.0`:
+
+```text
+https://www.unicode.org/Public/<VERSION>/ucdxml/ucd.nounihan.flat.zip
+https://www.unicode.org/Public/<VERSION>/ucd/UnicodeData.txt
+https://www.unicode.org/Public/<VERSION>/ucd/NameAliases.txt
+https://www.unicode.org/Public/<VERSION>/ucd/extracted/DerivedName.txt
+```
+
+Unzip `ucd.nounihan.flat.zip` to get `ucd.nounihan.flat.xml`.
+
+## Building the generators
+
+Build the generators from an LLVM build directory that has utilities enabled
+(`LLVM_BUILD_UTILS`, the default) and libxml2 (`LLVM_ENABLE_LIBXML2`, also the
+default):
+
+```bash
+ninja -C <build> UnicodeCharSetsGenerator UnicodeNameMappingGenerator
+```
+
+Then run them from the `llvm-project` root as shown below.
+
+## Character properties
+
+`UnicodeCharSetsGenerator` writes the identifier character sets used by the
+lexer (`XID_Start`, `XID_Continue`, and the mathematical compatibility notation
+profile) and the printable / formatting / combining / East-Asian-width sets
+used by LLVM Support.
+
+```bash
+<build>/bin/UnicodeCharSetsGenerator ucd.nounihan.flat.xml \
+ clang/lib/Lex/UnicodeCharSetsGenerated.cpp \
+ llvm/lib/Support/UnicodeCharSetsGenerated.cpp
+clang-format -i clang/lib/Lex/UnicodeCharSetsGenerated.cpp \
+ llvm/lib/Support/UnicodeCharSetsGenerated.cpp
+```
+
+The C99 and C11 identifier tables and the whitespace table in
+`clang/lib/Lex/UnicodeCharSets.h` are maintained by hand.
+
+## Character names
+
+`UnicodeNameMappingGenerator` writes the name-to-codepoint trie used by
+`\N{...}` and `llvm::sys::unicode::nameToCodepointStrict` /
+`nameToCodepointLooseMatching`.
+
+```bash
+<build>/bin/UnicodeNameMappingGenerator UnicodeData.txt NameAliases.txt \
+ llvm/lib/Support/UnicodeNameToCodepointGenerated.cpp
+clang-format -i llvm/lib/Support/UnicodeNameToCodepointGenerated.cpp
+```
+
+Algorithmically derived names (`CJK UNIFIED IDEOGRAPH-*`, and so on) are not
+in those files. Update `GeneratedNamesDataTable` in
+`llvm/lib/Support/UnicodeNameToCodepoint.cpp` from `DerivedName.txt`.
+
+## Case folding
+
+`llvm/utils/unicode-case-fold.py` fetches `CaseFolding.txt` and writes
+`llvm/lib/Support/UnicodeCaseFold.cpp`:
+
+```bash
+llvm/utils/unicode-case-fold.py \
+ https://www.unicode.org/Public/<VERSION>/ucd/CaseFolding.txt \
+ > llvm/lib/Support/UnicodeCaseFold.cpp
+```
+
+After regenerating the tables, update
+`llvm/unittests/Support/UnicodeTest.cpp` and `clang/test/Lexer/unicode.c` for
+new characters and changed name ranges.
>From bde21dae16dc292558112ad0155f48291881da26 Mon Sep 17 00:00:00 2001
From: Corentin Jabot <corentinjabot at gmail.com>
Date: Wed, 7 Oct 2026 12:53:32 +0200
Subject: [PATCH 5/5] Move the doc to LLVM and improve it
---
clang/docs/UnicodeSupport.md | 83 ------------------------------------
clang/docs/index.md | 2 -
llvm/docs/UserGuides.md | 6 +++
3 files changed, 6 insertions(+), 85 deletions(-)
delete mode 100644 clang/docs/UnicodeSupport.md
diff --git a/clang/docs/UnicodeSupport.md b/clang/docs/UnicodeSupport.md
deleted file mode 100644
index 7aae8ce73cb6b..0000000000000
--- a/clang/docs/UnicodeSupport.md
+++ /dev/null
@@ -1,83 +0,0 @@
-# Unicode Support
-
-Clang uses a subset of the [Unicode Character Database](https://www.unicode.org/ucd/)
-(UCD) for identifiers, `\N{...}` named characters, diagnostic column width,
-and simple case folding. The corresponding tables are generated from the UCD
-into Clang and LLVM Support.
-
-## Updating the Unicode Data
-
-Replace `N.M.0` with the Unicode version being imported. Build the generators
-from an LLVM build directory that has utilities enabled (`LLVM_BUILD_UTILS`,
-the default) and libxml2 (`LLVM_ENABLE_LIBXML2`, also the default):
-
-```bash
-ninja -C <build> UnicodeCharSetsGenerator UnicodeNameMappingGenerator
-```
-
-Then run them from the `llvm-project` root as shown below.
-
-### Character properties
-
-`UnicodeCharSetsGenerator` writes the identifier character sets used by the
-lexer (`XID_Start`, `XID_Continue`, and the mathematical compatibility notation
-profile) and the printable / formatting / combining / East-Asian-width sets
-used by LLVM Support.
-
-Download the [flat XML UCD](https://www.unicode.org/Public/UCD/latest/ucdxml/)
-(`ucd.nounihan.flat.zip` is enough):
-
-```text
-https://www.unicode.org/Public/N.M.0/ucdxml/ucd.nounihan.flat.zip
-```
-
-```bash
-<build>/bin/UnicodeCharSetsGenerator ucd.nounihan.flat.xml \
- clang/lib/Lex/UnicodeCharSetsGenerated.cpp \
- llvm/lib/Support/UnicodeCharSetsGenerated.cpp
-clang-format -i clang/lib/Lex/UnicodeCharSetsGenerated.cpp \
- llvm/lib/Support/UnicodeCharSetsGenerated.cpp
-```
-
-The C99 and C11 identifier tables and the whitespace table in
-`clang/lib/Lex/UnicodeCharSets.h` are maintained by hand.
-
-### Character names
-
-`UnicodeNameMappingGenerator` writes the name-to-codepoint trie used by
-`\N{...}` and `llvm::sys::unicode::nameToCodepointStrict` /
-`nameToCodepointLooseMatching`.
-
-From the [UCD zip](https://www.unicode.org/Public/UCD/latest/ucd/UCD.zip), take
-`UnicodeData.txt` and `NameAliases.txt`:
-
-```text
-https://www.unicode.org/Public/N.M.0/ucd/UnicodeData.txt
-https://www.unicode.org/Public/N.M.0/ucd/NameAliases.txt
-```
-
-```bash
-<build>/bin/UnicodeNameMappingGenerator UnicodeData.txt NameAliases.txt \
- llvm/lib/Support/UnicodeNameToCodepointGenerated.cpp
-clang-format -i llvm/lib/Support/UnicodeNameToCodepointGenerated.cpp
-```
-
-Algorithmically derived names (`CJK UNIFIED IDEOGRAPH-*`, and so on) are not
-in those files. Update `GeneratedNamesDataTable` in
-`llvm/lib/Support/UnicodeNameToCodepoint.cpp` from
-`extracted/DerivedName.txt` in the same UCD zip.
-
-### Case folding
-
-`llvm/utils/unicode-case-fold.py` fetches `CaseFolding.txt` and writes
-`llvm/lib/Support/UnicodeCaseFold.cpp`:
-
-```bash
-llvm/utils/unicode-case-fold.py \
- https://www.unicode.org/Public/N.M.0/ucd/CaseFolding.txt \
- > llvm/lib/Support/UnicodeCaseFold.cpp
-```
-
-After regenerating the tables, update
-`llvm/unittests/Support/UnicodeTest.cpp` and `clang/test/Lexer/unicode.c` for
-new characters and changed name ranges.
diff --git a/clang/docs/index.md b/clang/docs/index.md
index 9c353317fd260..a05210b8e73a0 100644
--- a/clang/docs/index.md
+++ b/clang/docs/index.md
@@ -93,7 +93,6 @@ ClangSYCLLinker
Maintainers
InternalsManual
DriverInternals
-UnicodeSupport
Multilib
OffloadingDesign
PCHInternals
@@ -201,7 +200,6 @@ LLVMExceptionHandlingCodeGen
* {doc}`Maintainers`
* {doc}`InternalsManual`
* {doc}`DriverInternals`
-* {doc}`UnicodeSupport`
* {doc}`Multilib`
* {doc}`OffloadingDesign`
* {doc}`PCHInternals`
diff --git a/llvm/docs/UserGuides.md b/llvm/docs/UserGuides.md
index 108601427697c..f7e971915b669 100644
--- a/llvm/docs/UserGuides.md
+++ b/llvm/docs/UserGuides.md
@@ -36,6 +36,7 @@ GoldPlugin
Remarks
SourceLevelDebugging
HowToUpdateDebugInfo
+HowToUpdateUnicodeTables
Instrumentor
InstrRefDebugInfo
RemoveDIsDebugInfo
@@ -218,6 +219,11 @@ yaml2obj
This document specifies how to correctly update debug info in various kinds
of code transformations.
+- {doc}`How to Update Unicode Tables <HowToUpdateUnicodeTables>`
+
+ How to regenerate LLVM and Clang Unicode character tables from the Unicode
+ Character Database.
+
- {doc}`InstrRefDebugInfo`
This document explains how LLVM uses value tracking, or instruction
More information about the cfe-commits
mailing list