<div dir="ltr">Hi Douglas,<div><br></div><div>Sorry about that! It's an encoding problem, I forgot there were encodings other than UTF-8 :-)</div><div>I'll fix it.</div><div><br></div><div>Thanks for the heads up.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 21, 2017 at 7:56 PM, Yung, Douglas <span dir="ltr"><<a href="mailto:douglas.yung@sony.com" target="_blank">douglas.yung@sony.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Sam,<br>
<br>
Your change is causing the PS4 Windows bot to fail because your test includes a divide by zero exception that the compiler is issuing an error for:<br>
<br>
<a href="http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/builds/13639/" rel="noreferrer" target="_blank">http://lab.llvm.org:8011/<wbr>builders/llvm-clang-lld-x86_<wbr>64-scei-ps4-windows10pro-fast/<wbr>builds/13639/</a><br>
<br>
FAILED: tools/clang/tools/extra/<wbr>unittests/clangd/CMakeFiles/<wbr>ClangdTests.dir/JSONExprTests.<wbr>cpp.obj<br>
C:\PROGRA~2\MICROS~1.0\VC\bin\<wbr>cl.exe /nologo /TP -DGTEST_HAS_RTTI=0 -DGTEST_HAS_TR1_TUPLE=0 -DGTEST_LANG_CXX11=1 -DUNICODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_HAS_EXCEPTIONS=0 -D_LARGEFILE_SOURCE -D_SCL_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_WARNINGS -D_UNICODE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools\clang\tools\extra\<wbr>unittests\clangd -IC:\ps4-buildslave2\llvm-<wbr>clang-lld-x86_64-scei-ps4-<wbr>windows10pro-fast\llvm.src\<wbr>tools\clang\tools\extra\<wbr>unittests\clangd -IC:\ps4-buildslave2\llvm-<wbr>clang-lld-x86_64-scei-ps4-<wbr>windows10pro-fast\llvm.src\<wbr>tools\clang\include -Itools\clang\include -Iinclude -IC:\ps4-buildslave2\llvm-<wbr>clang-lld-x86_64-scei-ps4-<wbr>windows10pro-fast\llvm.src\<wbr>include -IC:\ps4-buildslave2\llvm-<wbr>clang-lld-x86_64-scei-ps4-<wbr>windows10pro-fast\llvm.src\<wbr>tools\clang\tools\extra\clangd -IC:\ps4-buildslave2\llvm-<wbr>clang-lld-x86_64-scei-ps4-<wbr>windows10pro-fast\llvm.src\<wbr>utils\unittest\googletest\<wbr>include -IC:\ps4-buildslave2\llvm-<wbr>clang-lld-x86_64-scei-ps4-<wbr>windows10pro-fast\llvm.src\<wbr>utils\unittest\googlemock\<wbr>include /DWIN32 /D_WINDOWS /Zc:inline /Zc:strictStrings /Oi /Zc:rvalueCast /W4 -wd4141 -wd4146 -wd4180 -wd4244 -wd4258 -wd4267 -wd4291 -wd4345 -wd4351 -wd4355 -wd4456 -wd4457 -wd4458 -wd4459 -wd4503 -wd4624 -wd4722 -wd4800 -wd4100 -wd4127 -wd4512 -wd4505 -wd4610 -wd4510 -wd4702 -wd4245 -wd4706 -wd4310 -wd4701 -wd4703 -wd4389 -wd4611 -wd4805 -wd4204 -wd4577 -wd4091 -wd4592 -wd4319 -wd4324 -w14062 -we4238 /MD /O2 /Ob2 -UNDEBUG /EHs-c- /GR- /showIncludes /Fotools\clang\tools\extra\<wbr>unittests\clangd\CMakeFiles\<wbr>ClangdTests.dir\JSONExprTests.<wbr>cpp.obj /Fdtools\clang\tools\extra\<wbr>unittests\clangd\CMakeFiles\<wbr>ClangdTests.dir\ /FS -c C:\ps4-buildslave2\llvm-clang-<wbr>lld-x86_64-scei-ps4-<wbr>windows10pro-fast\llvm.src\<wbr>tools\clang\tools\extra\<wbr>unittests\clangd\<wbr>JSONExprTests.cpp<br>
C:\ps4-buildslave2\llvm-clang-<wbr>lld-x86_64-scei-ps4-<wbr>windows10pro-fast\llvm.src\<wbr>tools\clang\tools\extra\<wbr>unittests\clangd\<wbr>JSONExprTests.cpp(140): warning C4566: character represented by universal-character-name '\U00010437' cannot be represented in the current code page (1252)<br>
C:\ps4-buildslave2\llvm-clang-<wbr>lld-x86_64-scei-ps4-<wbr>windows10pro-fast\llvm.src\<wbr>tools\clang\tools\extra\<wbr>unittests\clangd\<wbr>JSONExprTests.cpp(141): warning C4566: character represented by universal-character-name '\U0001D11E' cannot be represented in the current code page (1252)<br>
C:\ps4-buildslave2\llvm-clang-<wbr>lld-x86_64-scei-ps4-<wbr>windows10pro-fast\llvm.src\<wbr>tools\clang\tools\extra\<wbr>unittests\clangd\<wbr>JSONExprTests.cpp(142): warning C4566: character represented by universal-character-name '\uFFFD' cannot be represented in the current code page (1252)<br>
C:\ps4-buildslave2\llvm-clang-<wbr>lld-x86_64-scei-ps4-<wbr>windows10pro-fast\llvm.src\<wbr>tools\clang\tools\extra\<wbr>unittests\clangd\<wbr>JSONExprTests.cpp(134): error C2124: divide or mod by zero<br>
<br>
Can you fix this so that we can get the bot green again? Thanks!<br>
<br>
Douglas Yung<br>
<br>
> -----Original Message-----<br>
> From: cfe-commits [mailto:<a href="mailto:cfe-commits-bounces@lists.llvm.org">cfe-commits-bounces@<wbr>lists.llvm.org</a>] On Behalf Of Sam<br>
> McCall via cfe-commits<br>
> Sent: Tuesday, November 21, 2017 8:01<br>
> To: <a href="mailto:cfe-commits@lists.llvm.org">cfe-commits@lists.llvm.org</a><br>
> Subject: [clang-tools-extra] r318774 - [clangd] Add parsing and value<br>
> inspection to JSONExpr.<br>
><br>
> Author: sammccall<br>
> Date: Tue Nov 21 08:00:53 2017<br>
> New Revision: 318774<br>
><br>
> URL: <a href="http://llvm.org/viewvc/llvm-project?rev=318774&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project?rev=318774&view=rev</a><br>
> Log:<br>
> [clangd] Add parsing and value inspection to JSONExpr.<br>
><br>
> Summary:<br>
> This will replace the places where we're using YAMLParser to parse JSON now:<br>
> - the new marshalling code (T::parse()) should handle fewer cases and<br>
> require<br>
> fewer explicit casts<br>
> - we'll early-reject invalid JSON that YAMLParser accepts<br>
> - we'll be able to fix protocol-parsing bugs caused by the fact that YAML<br>
> can<br>
> only parse forward<br>
><br>
> I plan to do the conversion as soon as this lands, but I don't want it in one<br>
> patch as the protocol.cpp changes are conflict-prone.<br>
><br>
> Reviewers: ioeric<br>
><br>
> Subscribers: ilya-biryukov, cfe-commits<br>
><br>
> Differential Revision: <a href="https://reviews.llvm.org/D40182" rel="noreferrer" target="_blank">https://reviews.llvm.org/<wbr>D40182</a><br>
><br>
> Modified:<br>
> clang-tools-extra/trunk/<wbr>clangd/JSONExpr.cpp<br>
> clang-tools-extra/trunk/<wbr>clangd/JSONExpr.h<br>
> clang-tools-extra/trunk/<wbr>unittests/clangd/<wbr>JSONExprTests.cpp<br>
><br>
> Modified: clang-tools-extra/trunk/<wbr>clangd/JSONExpr.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/clang-tools-</a><br>
> extra/trunk/clangd/JSONExpr.<wbr>cpp?rev=318774&r1=318773&r2=<wbr>318774&view=diff<br>
> ==============================<wbr>==============================<wbr>==================<br>
> --- clang-tools-extra/trunk/<wbr>clangd/JSONExpr.cpp (original)<br>
> +++ clang-tools-extra/trunk/<wbr>clangd/JSONExpr.cpp Tue Nov 21 08:00:53 2017<br>
> @@ -22,10 +22,10 @@ void Expr::copyFrom(const Expr &M) {<br>
> create<std::string>(M.as<std::<wbr>string>());<br>
> break;<br>
> case T_Object:<br>
> - create<Object>(M.as<Object>())<wbr>;<br>
> + create<ObjectExpr>(M.as<<wbr>ObjectExpr>());<br>
> break;<br>
> case T_Array:<br>
> - create<Array>(M.as<Array>());<br>
> + create<ArrayExpr>(M.as<<wbr>ArrayExpr>());<br>
> break;<br>
> }<br>
> }<br>
> @@ -46,11 +46,11 @@ void Expr::moveFrom(const Expr &&M) {<br>
> M.Type = T_Null;<br>
> break;<br>
> case T_Object:<br>
> - create<Object>(std::move(M.as<<wbr>Object>()));<br>
> + create<ObjectExpr>(std::move(<wbr>M.as<ObjectExpr>()));<br>
> M.Type = T_Null;<br>
> break;<br>
> case T_Array:<br>
> - create<Array>(std::move(M.as<<wbr>Array>()));<br>
> + create<ArrayExpr>(std::move(M.<wbr>as<ArrayExpr>()));<br>
> M.Type = T_Null;<br>
> break;<br>
> }<br>
> @@ -69,14 +69,318 @@ void Expr::destroy() {<br>
> as<std::string>().~basic_<wbr>string();<br>
> break;<br>
> case T_Object:<br>
> - as<Object>().~Object();<br>
> + as<ObjectExpr>().~ObjectExpr()<wbr>;<br>
> break;<br>
> case T_Array:<br>
> - as<Array>().~Array();<br>
> + as<ArrayExpr>().~ArrayExpr();<br>
> break;<br>
> }<br>
> }<br>
><br>
> +namespace {<br>
> +// Simple recursive-descent JSON parser.<br>
> +class Parser {<br>
> +public:<br>
> + Parser(StringRef JSON)<br>
> + : Start(JSON.begin()), P(JSON.begin()), End(JSON.end()) {}<br>
> +<br>
> + bool parseExpr(Expr &Out);<br>
> +<br>
> + bool assertEnd() {<br>
> + eatWhitespace();<br>
> + if (P == End)<br>
> + return true;<br>
> + return parseError("Text after end of document"); }<br>
> +<br>
> + Error takeError() {<br>
> + assert(Error);<br>
> + return std::move(*Error);<br>
> + }<br>
> +<br>
> +private:<br>
> + void eatWhitespace() {<br>
> + while (P != End && (*P == ' ' || *P == '\r' || *P == '\n' || *P == '\t'))<br>
> + ++P;<br>
> + }<br>
> +<br>
> + // On invalid syntax, parseX() functions return false and and set Error.<br>
> + bool parseNumber(char First, double &Out); bool<br>
> + parseString(std::string &Out); bool parseUnicode(std::string &Out);<br>
> + bool parseError(const char *Msg); // always returns false<br>
> +<br>
> + char next() { return P == End ? 0 : *P++; } char peek() { return P<br>
> + == End ? 0 : *P; } static bool isNumber(char C) {<br>
> + return C == '0' || C == '1' || C == '2' || C == '3' || C == '4' ||<br>
> + C == '5' || C == '6' || C == '7' || C == '8' || C == '9' ||<br>
> + C == 'e' || C == 'E' || C == '+' || C == '-' || C == '.'; }<br>
> + static void encodeUtf8(uint32_t Rune, std::string &Out);<br>
> +<br>
> + Optional<Error> Error;<br>
> + const char *Start, *P, *End;<br>
> +};<br>
> +<br>
> +bool Parser::parseExpr(Expr &Out) {<br>
> + eatWhitespace();<br>
> + if (P == End)<br>
> + return parseError("Unexpected EOF");<br>
> + switch (char C = next()) {<br>
> + // Bare null/true/false are easy - first char identifies them.<br>
> + case 'n':<br>
> + Out = nullptr;<br>
> + return (next() == 'u' && next() == 'l' && next() == 'l') ||<br>
> + parseError("Invalid bareword");<br>
> + case 't':<br>
> + Out = true;<br>
> + return (next() == 'r' && next() == 'u' && next() == 'e') ||<br>
> + parseError("Invalid bareword");<br>
> + case 'f':<br>
> + Out = false;<br>
> + return (next() == 'a' && next() == 'l' && next() == 's' && next() == 'e')<br>
> ||<br>
> + parseError("Invalid bareword");<br>
> + case '"': {<br>
> + std::string S;<br>
> + if (parseString(S)) {<br>
> + Out = std::move(S);<br>
> + return true;<br>
> + }<br>
> + return false;<br>
> + }<br>
> + case '[': {<br>
> + Out = json::ary{};<br>
> + json::ary &A = *Out.array();<br>
> + eatWhitespace();<br>
> + if (peek() == ']') {<br>
> + ++P;<br>
> + return true;<br>
> + }<br>
> + for (;;) {<br>
> + A.emplace_back(nullptr);<br>
> + if (!parseExpr(A.back()))<br>
> + return false;<br>
> + eatWhitespace();<br>
> + switch (next()) {<br>
> + case ',':<br>
> + eatWhitespace();<br>
> + continue;<br>
> + case ']':<br>
> + return true;<br>
> + default:<br>
> + return parseError("Expected , or ] after array element");<br>
> + }<br>
> + }<br>
> + }<br>
> + case '{': {<br>
> + Out = json::obj{};<br>
> + json::obj &O = *Out.object();<br>
> + eatWhitespace();<br>
> + if (peek() == '}') {<br>
> + ++P;<br>
> + return true;<br>
> + }<br>
> + for (;;) {<br>
> + if (next() != '"')<br>
> + return parseError("Expected object key");<br>
> + std::string K;<br>
> + if (!parseString(K))<br>
> + return false;<br>
> + eatWhitespace();<br>
> + if (next() != ':')<br>
> + return parseError("Expected : after object key");<br>
> + eatWhitespace();<br>
> + if (!parseExpr(O[std::move(K)]))<br>
> + return false;<br>
> + eatWhitespace();<br>
> + switch (next()) {<br>
> + case ',':<br>
> + eatWhitespace();<br>
> + continue;<br>
> + case '}':<br>
> + return true;<br>
> + default:<br>
> + return parseError("Expected , or } after object property");<br>
> + }<br>
> + }<br>
> + }<br>
> + default:<br>
> + if (isNumber(C)) {<br>
> + double Num;<br>
> + if (parseNumber(C, Num)) {<br>
> + Out = Num;<br>
> + return true;<br>
> + } else {<br>
> + return false;<br>
> + }<br>
> + }<br>
> + return parseError("Expected JSON value");<br>
> + }<br>
> +}<br>
> +<br>
> +bool Parser::parseNumber(char First, double &Out) {<br>
> + SmallString<24> S;<br>
> + S.push_back(First);<br>
> + while (isNumber(peek()))<br>
> + S.push_back(next());<br>
> + char *End;<br>
> + Out = std::strtod(S.c_str(), &End);<br>
> + return End == S.end() || parseError("Invalid number"); }<br>
> +<br>
> +bool Parser::parseString(std::<wbr>string &Out) {<br>
> + // leading quote was already consumed.<br>
> + for (char C = next(); C != '"'; C = next()) {<br>
> + if (LLVM_UNLIKELY(P == End))<br>
> + return parseError("Unterminated string");<br>
> + if (LLVM_UNLIKELY((C & 0x1f) == C))<br>
> + return parseError("Control character in string");<br>
> + if (LLVM_LIKELY(C != '\\')) {<br>
> + Out.push_back(C);<br>
> + continue;<br>
> + }<br>
> + // Handle escape sequence.<br>
> + switch (C = next()) {<br>
> + case '"':<br>
> + case '\\':<br>
> + case '/':<br>
> + Out.push_back(C);<br>
> + break;<br>
> + case 'b':<br>
> + Out.push_back('\b');<br>
> + break;<br>
> + case 'f':<br>
> + Out.push_back('\f');<br>
> + break;<br>
> + case 'n':<br>
> + Out.push_back('\n');<br>
> + break;<br>
> + case 'r':<br>
> + Out.push_back('\r');<br>
> + break;<br>
> + case 't':<br>
> + Out.push_back('\t');<br>
> + break;<br>
> + case 'u':<br>
> + if (!parseUnicode(Out))<br>
> + return false;<br>
> + break;<br>
> + default:<br>
> + return parseError("Invalid escape sequence");<br>
> + }<br>
> + }<br>
> + return true;<br>
> +}<br>
> +<br>
> +void Parser::encodeUtf8(uint32_t Rune, std::string &Out) {<br>
> + if (Rune <= 0x7F) {<br>
> + Out.push_back(Rune & 0x7F);<br>
> + } else if (Rune <= 0x7FF) {<br>
> + uint8_t FirstByte = 0xC0 | ((Rune & 0x7C0) >> 6);<br>
> + uint8_t SecondByte = 0x80 | (Rune & 0x3F);<br>
> + Out.push_back(FirstByte);<br>
> + Out.push_back(SecondByte);<br>
> + } else if (Rune <= 0xFFFF) {<br>
> + uint8_t FirstByte = 0xE0 | ((Rune & 0xF000) >> 12);<br>
> + uint8_t SecondByte = 0x80 | ((Rune & 0xFC0) >> 6);<br>
> + uint8_t ThirdByte = 0x80 | (Rune & 0x3F);<br>
> + Out.push_back(FirstByte);<br>
> + Out.push_back(SecondByte);<br>
> + Out.push_back(ThirdByte);<br>
> + } else if (Rune <= 0x10FFFF) {<br>
> + uint8_t FirstByte = 0xF0 | ((Rune & 0x1F0000) >> 18);<br>
> + uint8_t SecondByte = 0x80 | ((Rune & 0x3F000) >> 12);<br>
> + uint8_t ThirdByte = 0x80 | ((Rune & 0xFC0) >> 6);<br>
> + uint8_t FourthByte = 0x80 | (Rune & 0x3F);<br>
> + Out.push_back(FirstByte);<br>
> + Out.push_back(SecondByte);<br>
> + Out.push_back(ThirdByte);<br>
> + Out.push_back(FourthByte);<br>
> + } else {<br>
> + llvm_unreachable("Invalid codepoint");<br>
> + }<br>
> +}<br>
> +<br>
> +// Parse a \uNNNN escape sequence, the \u have already been consumed.<br>
> +// May parse multiple escapes in the presence of surrogate pairs.<br>
> +bool Parser::parseUnicode(std::<wbr>string &Out) {<br>
> + // Note that invalid unicode is not a JSON error. It gets replaced by<br>
> U+FFFD.<br>
> + auto Invalid = [&] { Out.append(/* UTF-8 */ {'\xef', '\xbf',<br>
> +'\xbd'}); };<br>
> + auto Parse4Hex = [this](uint16_t &Out) {<br>
> + Out = 0;<br>
> + char Bytes[] = {next(), next(), next(), next()};<br>
> + for (unsigned char C : Bytes) {<br>
> + if (!std::isxdigit(C))<br>
> + return parseError("Invalid \\u escape sequence");<br>
> + Out <<= 4;<br>
> + Out |= (C > '9') ? (C & ~0x20) - 'A' + 10 : (C - '0');<br>
> + }<br>
> + return true;<br>
> + };<br>
> + uint16_t First;<br>
> + if (!Parse4Hex(First))<br>
> + return false;<br>
> +<br>
> + // We loop to allow proper surrogate-pair error handling.<br>
> + while (true) {<br>
> + if (LLVM_LIKELY(First < 0xD800 || First >= 0xE000)) { // BMP.<br>
> + encodeUtf8(First, Out);<br>
> + return true;<br>
> + }<br>
> +<br>
> + if (First >= 0xDC00) {<br>
> + Invalid(); // Lone trailing surrogate.<br>
> + return true;<br>
> + }<br>
> +<br>
> + // We have a leading surrogate, and need a trailing one.<br>
> + // Don't advance P: a lone surrogate is valid JSON (but invalid unicode)<br>
> + if (P + 2 > End || *P != '\\' || *(P + 1) != 'u') {<br>
> + Invalid(); // Lone leading not followed by \u...<br>
> + return true;<br>
> + }<br>
> + P += 2;<br>
> + uint16_t Second;<br>
> + if (!Parse4Hex(Second))<br>
> + return false;<br>
> + if (Second < 0xDC00 && Second >= 0xE000) {<br>
> + Invalid(); // Leading surrogate not followed by trailing.<br>
> + First = Second; // Second escape still needs to be processed.<br>
> + continue;<br>
> + }<br>
> +<br>
> + // Valid surrogate pair.<br>
> + encodeUtf8(0x10000 | ((First - 0xD800) << 10) | (Second - 0xDC00), Out);<br>
> + return true;<br>
> + }<br>
> +}<br>
> +<br>
> +bool Parser::parseError(const char *Msg) {<br>
> + int Line = 1;<br>
> + const char *StartOfLine = Start;<br>
> + for (const char *X = Start; X < P; ++X) {<br>
> + if (*X == 0x0A) {<br>
> + ++Line;<br>
> + StartOfLine = X + 1;<br>
> + }<br>
> + }<br>
> + Error.emplace(<br>
> + llvm::make_unique<ParseError>(<wbr>Msg, Line, P - StartOfLine, P -<br>
> +Start));<br>
> + return false;<br>
> +}<br>
> +} // namespace<br>
> +<br>
> +Expected<Expr> parse(StringRef JSON) {<br>
> + Parser P(JSON);<br>
> + json::Expr E = nullptr;<br>
> + if (P.parseExpr(E))<br>
> + if (P.assertEnd())<br>
> + return std::move(E);<br>
> + return P.takeError();<br>
> +}<br>
> +char ParseError::ID = 0;<br>
> +<br>
> } // namespace json<br>
> } // namespace clangd<br>
> } // namespace clang<br>
> @@ -144,7 +448,7 @@ void clang::clangd::json::Expr::<wbr>print(ra<br>
> bool Comma = false;<br>
> OS << '{';<br>
> I(Indent);<br>
> - for (const auto &P : as<Expr::Object>()) {<br>
> + for (const auto &P : as<Expr::ObjectExpr>()) {<br>
> if (Comma)<br>
> OS << ',';<br>
> Comma = true;<br>
> @@ -164,7 +468,7 @@ void clang::clangd::json::Expr::<wbr>print(ra<br>
> bool Comma = false;<br>
> OS << '[';<br>
> I(Indent);<br>
> - for (const auto &E : as<Expr::Array>()) {<br>
> + for (const auto &E : as<Expr::ArrayExpr>()) {<br>
> if (Comma)<br>
> OS << ',';<br>
> Comma = true;<br>
> @@ -187,6 +491,25 @@ llvm::raw_ostream &operator<<(raw_ostrea<br>
> E.print(OS, [](IndenterAction A) { /*ignore*/ });<br>
> return OS;<br>
> }<br>
> +<br>
> +bool operator==(const Expr &L, const Expr &R) {<br>
> + if (L.kind() != R.kind())<br>
> + return false;<br>
> + switch (L.kind()) {<br>
> + case Expr::Null:<br>
> + return L.null() == R.null();<br>
> + case Expr::Boolean:<br>
> + return L.boolean() == R.boolean();<br>
> + case Expr::Number:<br>
> + return L.boolean() == R.boolean();<br>
> + case Expr::String:<br>
> + return L.string() == R.string();<br>
> + case Expr::Array:<br>
> + return *L.array() == *R.array();<br>
> + case Expr::Object:<br>
> + return *L.object() == *R.object();<br>
> + }<br>
> +}<br>
> } // namespace json<br>
> } // namespace clangd<br>
> } // namespace clang<br>
><br>
> Modified: clang-tools-extra/trunk/<wbr>clangd/JSONExpr.h<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/clang-tools-</a><br>
> extra/trunk/clangd/JSONExpr.h?<wbr>rev=318774&r1=318773&r2=<wbr>318774&view=diff<br>
> ==============================<wbr>==============================<wbr>==================<br>
> --- clang-tools-extra/trunk/<wbr>clangd/JSONExpr.h (original)<br>
> +++ clang-tools-extra/trunk/<wbr>clangd/JSONExpr.h Tue Nov 21 08:00:53 2017<br>
> @@ -1,4 +1,4 @@<br>
> -//===--- JSONExpr.h - composable JSON expressions ---------------*- C++ -*-<br>
> ===//<br>
> +//===--- JSONExpr.h - JSON expressions, parsing and serialization - C++<br>
> +-*-===//<br>
> //<br>
> // The LLVM Compiler Infrastructure<br>
> //<br>
> @@ -7,6 +7,8 @@<br>
> //<br>
> //===-------------------------<wbr>------------------------------<wbr>--------------<br>
> ===//<br>
><br>
> +// FIXME: rename to JSON.h now that the scope is wider?<br>
> +<br>
> #ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_<wbr>JSON_H<br>
> #define LLVM_CLANG_TOOLS_EXTRA_CLANGD_<wbr>JSON_H<br>
><br>
> @@ -14,6 +16,7 @@<br>
><br>
> #include "llvm/ADT/SmallVector.h"<br>
> #include "llvm/ADT/StringRef.h"<br>
> +#include "llvm/Support/Error.h"<br>
> #include "llvm/Support/FormatVariadic.<wbr>h"<br>
> #include "llvm/Support/raw_ostream.h"<br>
><br>
> @@ -21,10 +24,12 @@ namespace clang {<br>
> namespace clangd {<br>
> namespace json {<br>
><br>
> -// An Expr is an opaque temporary JSON structure used to compose documents.<br>
> +// An Expr is an JSON value of unknown type.<br>
> // They can be copied, but should generally be moved.<br>
> //<br>
> -// You can implicitly construct literals from:<br>
> +// === Composing expressions ===<br>
> +//<br>
> +// You can implicitly construct Exprs from:<br>
> // - strings: std::string, SmallString, formatv, StringRef, char*<br>
> // (char*, and StringRef are references, not copies!)<br>
> // - numbers<br>
> @@ -39,25 +44,62 @@ namespace json {<br>
> // These can be list-initialized, or used to build up collections in a loop.<br>
> // json::ary(Collection) converts all items in a collection to Exprs.<br>
> //<br>
> +// === Inspecting expressions ===<br>
> +//<br>
> +// Each Expr is one of the JSON kinds:<br>
> +// null (nullptr_t)<br>
> +// boolean (bool)<br>
> +// number (double)<br>
> +// string (StringRef)<br>
> +// array (json::ary)<br>
> +// object (json::obj)<br>
> +//<br>
> +// The kind can be queried directly, or implicitly via the typed accessors:<br>
> +// if (Optional<StringRef> S = E.string())<br>
> +// assert(E.kind() == Expr::String);<br>
> +//<br>
> +// Array and Object also have typed indexing accessors for easy traversal:<br>
> +// Expected<Expr> E = parse(R"( {"options": {"font": "sans-serif"}} )");<br>
> +// if (json::obj* O = E->object())<br>
> +// if (json::obj* Opts = O->object("options"))<br>
> +// if (Optional<StringRef> Font = Opts->string("font"))<br>
> +// assert(Opts->at("font").kind() == Expr::String);<br>
> +//<br>
> +// === Serialization ===<br>
> +//<br>
> // Exprs can be serialized to JSON:<br>
> // 1) raw_ostream << Expr // Basic formatting.<br>
> // 2) raw_ostream << formatv("{0}", Expr) // Basic formatting.<br>
> // 3) raw_ostream << formatv("{0:2}", Expr) // Pretty-print with indent 2.<br>
> +//<br>
> +// And parsed:<br>
> +// Expected<Expr> E = json::parse("[1, 2, null]");<br>
> +// assert(E && E->kind() == Expr::Array);<br>
> class Expr {<br>
> public:<br>
> - class Object;<br>
> + enum Kind {<br>
> + Null,<br>
> + Boolean,<br>
> + Number,<br>
> + String,<br>
> + Array,<br>
> + Object,<br>
> + };<br>
> + class ObjectExpr;<br>
> class ObjectKey;<br>
> - class Array;<br>
> + class ArrayExpr;<br>
><br>
> // It would be nice to have Expr() be null. But that would make {} null<br>
> too...<br>
> Expr(const Expr &M) { copyFrom(M); }<br>
> Expr(Expr &&M) { moveFrom(std::move(M)); }<br>
> // "cheating" move-constructor for moving from initializer_list.<br>
> Expr(const Expr &&M) { moveFrom(std::move(M)); }<br>
> - Expr(std::initializer_list<<wbr>Expr> Elements) : Expr(Array(Elements)) {}<br>
> - Expr(Array &&Elements) : Type(T_Array) {<br>
> create<Array>(std::move(<wbr>Elements)); }<br>
> - Expr(Object &&Properties) : Type(T_Object) {<br>
> - create<Object>(std::move(<wbr>Properties));<br>
> + Expr(std::initializer_list<<wbr>Expr> Elements) :<br>
> + Expr(ArrayExpr(Elements)) {} Expr(ArrayExpr &&Elements) : Type(T_Array) {<br>
> + create<ArrayExpr>(std::move(<wbr>Elements));<br>
> + }<br>
> + Expr(ObjectExpr &&Properties) : Type(T_Object) {<br>
> + create<ObjectExpr>(std::move(<wbr>Properties));<br>
> }<br>
> // Strings: types with value semantics.<br>
> Expr(std::string &&V) : Type(T_String) { create<std::string>(std::move(<wbr>V));<br>
> } @@ -104,6 +146,60 @@ public:<br>
> }<br>
> ~Expr() { destroy(); }<br>
><br>
> + Kind kind() const {<br>
> + switch (Type) {<br>
> + case T_Null:<br>
> + return Null;<br>
> + case T_Boolean:<br>
> + return Boolean;<br>
> + case T_Number:<br>
> + return Number;<br>
> + case T_String:<br>
> + case T_StringRef:<br>
> + return String;<br>
> + case T_Object:<br>
> + return Object;<br>
> + case T_Array:<br>
> + return Array;<br>
> + }<br>
> + }<br>
> +<br>
> + // Typed accessors return None/nullptr if the Expr is not of this type.<br>
> + llvm::Optional<std::nullptr_t> null() const {<br>
> + if (LLVM_LIKELY(Type == T_Null))<br>
> + return nullptr;<br>
> + return llvm::None;<br>
> + }<br>
> + llvm::Optional<bool> boolean() const {<br>
> + if (LLVM_LIKELY(Type == T_Null))<br>
> + return as<bool>();<br>
> + return llvm::None;<br>
> + }<br>
> + llvm::Optional<double> number() const {<br>
> + if (LLVM_LIKELY(Type == T_Number))<br>
> + return as<double>();<br>
> + return llvm::None;<br>
> + }<br>
> + llvm::Optional<llvm::<wbr>StringRef> string() const {<br>
> + if (Type == T_String)<br>
> + return llvm::StringRef(as<std::<wbr>string>());<br>
> + if (LLVM_LIKELY(Type == T_StringRef))<br>
> + return as<llvm::StringRef>();<br>
> + return llvm::None;<br>
> + }<br>
> + const ObjectExpr *object() const {<br>
> + return LLVM_LIKELY(Type == T_Object) ? &as<ObjectExpr>() : nullptr;<br>
> + } ObjectExpr *object() {<br>
> + return LLVM_LIKELY(Type == T_Object) ? &as<ObjectExpr>() : nullptr;<br>
> + } const ArrayExpr *array() const {<br>
> + return LLVM_LIKELY(Type == T_Array) ? &as<ArrayExpr>() : nullptr;<br>
> + } ArrayExpr *array() {<br>
> + return LLVM_LIKELY(Type == T_Array) ? &as<ArrayExpr>() : nullptr;<br>
> + }<br>
> +<br>
> friend llvm::raw_ostream &operator<<(llvm::raw_ostream &, const Expr &);<br>
><br>
> private:<br>
> @@ -137,10 +233,8 @@ private:<br>
> mutable ExprType Type;<br>
><br>
> public:<br>
> - // ObjectKey is a used to capture keys in Expr::Objects. It's like Expr<br>
> but:<br>
> + // ObjectKey is a used to capture keys in Expr::ObjectExpr. Like Expr but:<br>
> // - only strings are allowed<br>
> - // - it's copyable (for std::map)<br>
> - // - we're slightly more eager to copy, to allow efficient key compares<br>
> // - it's optimized for the string literal case (Owned == nullptr)<br>
> class ObjectKey {<br>
> public:<br>
> @@ -183,12 +277,12 @@ public:<br>
> llvm::StringRef Data;<br>
> };<br>
><br>
> - class Object : public std::map<ObjectKey, Expr> {<br>
> + class ObjectExpr : public std::map<ObjectKey, Expr> {<br>
> public:<br>
> - explicit Object() {}<br>
> + explicit ObjectExpr() {}<br>
> // Use a custom struct for list-init, because pair forces extra copies.<br>
> struct KV;<br>
> - explicit Object(std::initializer_list<<wbr>KV> Properties);<br>
> + explicit ObjectExpr(std::initializer_<wbr>list<KV> Properties);<br>
><br>
> // Allow [] as if Expr was default-constructible as null.<br>
> Expr &operator[](const ObjectKey &K) { @@ -199,15 +293,15 @@ public:<br>
> }<br>
> };<br>
><br>
> - class Array : public std::vector<Expr> {<br>
> + class ArrayExpr : public std::vector<Expr> {<br>
> public:<br>
> - explicit Array() {}<br>
> - explicit Array(std::initializer_list<<wbr>Expr> Elements) {<br>
> + explicit ArrayExpr() {}<br>
> + explicit ArrayExpr(std::initializer_<wbr>list<Expr> Elements) {<br>
> reserve(Elements.size());<br>
> for (const Expr &V : Elements)<br>
> emplace_back(std::move(V));<br>
> };<br>
> - template <typename Collection> explicit Array(const Collection &C) {<br>
> + template <typename Collection> explicit ArrayExpr(const Collection<br>
> + &C) {<br>
> for (const auto &V : C)<br>
> emplace_back(V);<br>
> }<br>
> @@ -215,23 +309,50 @@ public:<br>
><br>
> private:<br>
> mutable llvm::AlignedCharArrayUnion<<wbr>bool, double, llvm::StringRef,<br>
> - std::string, Array, Object><br>
> + std::string, ArrayExpr,<br>
> + ObjectExpr><br>
> Union;<br>
> };<br>
><br>
> -struct Expr::Object::KV {<br>
> +bool operator==(const Expr &, const Expr &); inline bool<br>
> +operator!=(const Expr &L, const Expr &R) { return !(L == R); } inline<br>
> +bool operator==(const Expr::ObjectKey &L, const Expr::ObjectKey &R) {<br>
> + return llvm::StringRef(L) == llvm::StringRef(R); } inline bool<br>
> +operator!=(const Expr::ObjectKey &L, const Expr::ObjectKey &R) {<br>
> + return !(L == R);<br>
> +}<br>
> +<br>
> +struct Expr::ObjectExpr::KV {<br>
> ObjectKey K;<br>
> Expr V;<br>
> };<br>
><br>
> -inline Expr::Object::Object(std::<wbr>initializer_list<KV> Properties) {<br>
> +inline Expr::ObjectExpr::ObjectExpr(<wbr>std::initializer_list<KV><br>
> +Properties) {<br>
> for (const auto &P : Properties)<br>
> emplace(std::move(P.K), std::move(P.V)); }<br>
><br>
> // Give Expr::{Object,Array} more convenient names for literal use.<br>
> -using obj = Expr::Object;<br>
> -using ary = Expr::Array;<br>
> +using obj = Expr::ObjectExpr;<br>
> +using ary = Expr::ArrayExpr;<br>
> +<br>
> +llvm::Expected<Expr> parse(llvm::StringRef JSON);<br>
> +<br>
> +class ParseError : public llvm::ErrorInfo<ParseError> {<br>
> + const char *Msg;<br>
> + unsigned Line, Column, Offset;<br>
> +<br>
> +public:<br>
> + static char ID;<br>
> + ParseError(const char *Msg, unsigned Line, unsigned Column, unsigned<br>
> Offset)<br>
> + : Msg(Msg), Line(Line), Column(Column), Offset(Offset) {}<br>
> + void log(llvm::raw_ostream &OS) const override {<br>
> + OS << llvm::formatv("[{0}:{1}, byte={2}]: {3}", Line, Column,<br>
> +Offset, Msg);<br>
> + }<br>
> + std::error_code convertToErrorCode() const override {<br>
> + return llvm::inconvertibleErrorCode()<wbr>;<br>
> + }<br>
> +};<br>
><br>
> } // namespace json<br>
> } // namespace clangd<br>
><br>
> Modified: clang-tools-extra/trunk/<wbr>unittests/clangd/<wbr>JSONExprTests.cpp<br>
> URL: <a href="http://llvm.org/viewvc/llvm-project/clang-tools-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/clang-tools-</a><br>
> extra/trunk/unittests/clangd/<wbr>JSONExprTests.cpp?rev=318774&<wbr>r1=318773&r2=318774&<br>
> view=diff<br>
> ==============================<wbr>==============================<wbr>==================<br>
> --- clang-tools-extra/trunk/<wbr>unittests/clangd/<wbr>JSONExprTests.cpp (original)<br>
> +++ clang-tools-extra/trunk/<wbr>unittests/clangd/<wbr>JSONExprTests.cpp Tue Nov<br>
> +++ 21 08:00:53 2017<br>
> @@ -15,6 +15,9 @@<br>
> namespace clang {<br>
> namespace clangd {<br>
> namespace json {<br>
> +void PrintTo(const Expr &E, std::ostream *OS) {<br>
> + llvm::raw_os_ostream(*OS) << llvm::formatv("{0:2}", E); }<br>
> namespace {<br>
><br>
> std::string s(const Expr &E) { return llvm::formatv("{0}", E).str(); } @@ -<br>
> 108,6 +111,77 @@ TEST(JSONExprTests, PrettyPrinting) {<br>
> }));<br>
> }<br>
><br>
> +TEST(JSONTest, Parse) {<br>
> + auto Compare = [](llvm::StringRef S, Expr Expected) {<br>
> + if (auto E = parse(S)) {<br>
> + // Compare both string forms and with operator==, in case we have bugs.<br>
> + EXPECT_EQ(*E, Expected);<br>
> + EXPECT_EQ(sp(*E), sp(Expected));<br>
> + } else {<br>
> + handleAllErrors(E.takeError(), [S](const llvm::ErrorInfoBase &E) {<br>
> + FAIL() << "Failed to parse JSON >>> " << S << " <<<: " <<<br>
> E.message();<br>
> + });<br>
> + }<br>
> + };<br>
> +<br>
> + Compare(R"(true)", true);<br>
> + Compare(R"(false)", false);<br>
> + Compare(R"(null)", nullptr);<br>
> +<br>
> + Compare(R"(42)", 42);<br>
> + Compare(R"(2.5)", 2.5);<br>
> + Compare(R"(2e50)", 2e50);<br>
> + Compare(R"(1.2e3456789)", 1.0 / 0.0);<br>
> +<br>
> + Compare(R"("foo")", "foo");<br>
> + Compare(R"("\"\\\b\f\n\r\t")", "\"\\\b\f\n\r\t");<br>
> + Compare(R"("\u0000")", llvm::StringRef("\0", 1)); Compare("\"\x7f\"",<br>
> + "\x7f"); Compare(R"("\ud801\udc37")", "\U00010437"); // UTF16<br>
> + surrogate pair escape.<br>
> + Compare("\"\xE2\x82\xAC\xF0\<wbr>x9D\x84\x9E\"", "\u20ac\U0001d11e"); //<br>
> + UTF8 Compare(R"("\ud801")", "\ufffd"); // Invalid codepoint.<br>
> +<br>
> + Compare(R"({"":0,"":0})", obj{{"", 0}});<br>
> + Compare(R"({"obj":{},"arr":[]}<wbr>)", obj{{"obj", obj{}}, {"arr", {}}});<br>
> + Compare(R"({"\n":{"\u0000":[[[<wbr>[]]]]}})",<br>
> + obj{{"\n", obj{<br>
> + {llvm::StringRef("\0", 1), {{{{}}}}},<br>
> + }}});<br>
> + Compare("\r[\n\t] ", {});<br>
> +}<br>
> +<br>
> +TEST(JSONTest, ParseErrors) {<br>
> + auto ExpectErr = [](llvm::StringRef Msg, llvm::StringRef S) {<br>
> + if (auto E = parse(S)) {<br>
> + // Compare both string forms and with operator==, in case we have bugs.<br>
> + FAIL() << "Parsed JSON >>> " << S << " <<< but wanted error: " << Msg;<br>
> + } else {<br>
> + handleAllErrors(E.takeError(), [S, Msg](const llvm::ErrorInfoBase &E) {<br>
> + EXPECT_THAT(E.message(), testing::HasSubstr(Msg)) << S;<br>
> + });<br>
> + }<br>
> + };<br>
> + ExpectErr("Unexpected EOF", "");<br>
> + ExpectErr("Unexpected EOF", "[");<br>
> + ExpectErr("Text after end of document", "[][]");<br>
> + ExpectErr("Text after end of document", "[][]");<br>
> + ExpectErr("Invalid bareword", "fuzzy");<br>
> + ExpectErr("Expected , or ]", "[2?]");<br>
> + ExpectErr("Expected object key", "{a:2}");<br>
> + ExpectErr("Expected : after object key", R"({"a",2})");<br>
> + ExpectErr("Expected , or } after object property", R"({"a":2<br>
> +"b":3})");<br>
> + ExpectErr("Expected JSON value", R"([&%!])");<br>
> + ExpectErr("Invalid number", "1e1.0");<br>
> + ExpectErr("Unterminated string", R"("abc\"def)");<br>
> + ExpectErr("Control character in string", "\"abc\ndef\"");<br>
> + ExpectErr("Invalid escape sequence", R"("\030")");<br>
> + ExpectErr("Invalid \\u escape sequence", R"("\usuck")");<br>
> + ExpectErr("[3:3, byte=19]", R"({<br>
> + "valid": 1,<br>
> + invalid: 2<br>
> +})");<br>
> +}<br>
> +<br>
> } // namespace<br>
> } // namespace json<br>
> } // namespace clangd<br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> cfe-commits mailing list<br>
> <a href="mailto:cfe-commits@lists.llvm.org">cfe-commits@lists.llvm.org</a><br>
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/cfe-commits</a><br>
</blockquote></div><br></div>