r332458 - [AST] Added a helper to extract a user-friendly text of a comment.
Clement Courbet via cfe-commits
cfe-commits at lists.llvm.org
Thu May 17 00:04:10 PDT 2018
I should have fixed it in r332576.
On Wed, May 16, 2018 at 11:49 PM, Galina Kistanova via cfe-commits <
cfe-commits at lists.llvm.org> wrote:
> Also few other builders are affected:
>
> http://lab.llvm.org:8011/builders/clang-x86_64-linux-abi-test
> http://lab.llvm.org:8011/builders/clang-lld-x86_64-2stage
> http://lab.llvm.org:8011/builders/clang-with-lto-ubuntu
>
>
> Thanks
>
> Galina
>
> On Wed, May 16, 2018 at 12:58 PM, Galina Kistanova <gkistanova at gmail.com>
> wrote:
>
>> Hello Ilya,
>>
>> This commit broke build step for couple of our builders:
>>
>> http://lab.llvm.org:8011/builders/clang-with-lto-ubuntu/builds/8541
>> http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu
>>
>> . . .
>> FAILED: tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o
>>
>> /usr/bin/c++ -DGTEST_HAS_RTTI=0 -DGTEST_HAS_TR1_TUPLE=0
>> -DGTEST_LANG_CXX11=1 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS
>> -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/clang/unittests/AST
>> -I/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/tools/clang/unittests/AST
>> -I/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src/tools/clang/include
>> -Itools/clang/include -Iinclude -I/home/buildslave/buildslave1
>> a/clang-with-lto-ubuntu/llvm.src/include -I/home/buildslave/buildslave1
>> a/clang-with-lto-ubuntu/llvm.src/utils/unittest/googletest/include
>> -I/home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.
>> src/utils/unittest/googlemock/include -fPIC -fvisibility-inlines-hidden
>> -std=c++11 -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual
>> -Wno-missing-field-initializers -pedantic -Wno-long-long
>> -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment
>> -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual
>> -fno-strict-aliasing -O3 -DNDEBUG -Wno-variadic-macros -fno-exceptions
>> -fno-rtti -MD -MT tools/clang/unittests/AST/CMak
>> eFiles/ASTTests.dir/CommentTextTest.cpp.o -MF
>> tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o.d
>> -o tools/clang/unittests/AST/CMakeFiles/ASTTests.dir/CommentTextTest.cpp.o
>> -c /home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src
>> /tools/clang/unittests/AST/CommentTextTest.cpp
>> /home/buildslave/buildslave1a/clang-with-lto-ubuntu/llvm.src
>> /tools/clang/unittests/AST/CommentTextTest.cpp:62:1: error: unterminated
>> raw string
>> R"cpp(
>> ^
>> . . .
>>
>> Please have a look?
>>
>> The builder was already red and did not send notifications.
>>
>> Thanks
>>
>> Galina
>>
>>
>>
>> On Wed, May 16, 2018 at 5:30 AM, Ilya Biryukov via cfe-commits <
>> cfe-commits at lists.llvm.org> wrote:
>>
>>> Author: ibiryukov
>>> Date: Wed May 16 05:30:09 2018
>>> New Revision: 332458
>>>
>>> URL: http://llvm.org/viewvc/llvm-project?rev=332458&view=rev
>>> Log:
>>> [AST] Added a helper to extract a user-friendly text of a comment.
>>>
>>> Summary:
>>> The helper is used in clangd for documentation shown in code completion
>>> and storing the docs in the symbols. See D45999.
>>>
>>> This patch reuses the code of the Doxygen comment lexer, disabling the
>>> bits that do command and html tag parsing.
>>> The new helper works on all comments, including non-doxygen comments.
>>> However, it does not understand or transform any doxygen directives,
>>> i.e. cannot extract brief text, etc.
>>>
>>> Reviewers: sammccall, hokein, ioeric
>>>
>>> Reviewed By: ioeric
>>>
>>> Subscribers: mgorny, cfe-commits
>>>
>>> Differential Revision: https://reviews.llvm.org/D46000
>>>
>>> Added:
>>> cfe/trunk/unittests/AST/CommentTextTest.cpp
>>> Modified:
>>> cfe/trunk/include/clang/AST/CommentLexer.h
>>> cfe/trunk/include/clang/AST/RawCommentList.h
>>> cfe/trunk/lib/AST/CommentLexer.cpp
>>> cfe/trunk/lib/AST/RawCommentList.cpp
>>> cfe/trunk/unittests/AST/CMakeLists.txt
>>>
>>> Modified: cfe/trunk/include/clang/AST/CommentLexer.h
>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/
>>> AST/CommentLexer.h?rev=332458&r1=332457&r2=332458&view=diff
>>> ============================================================
>>> ==================
>>> --- cfe/trunk/include/clang/AST/CommentLexer.h (original)
>>> +++ cfe/trunk/include/clang/AST/CommentLexer.h Wed May 16 05:30:09 2018
>>> @@ -281,6 +281,11 @@ private:
>>> /// command, including command marker.
>>> SmallString<16> VerbatimBlockEndCommandName;
>>>
>>> + /// If true, the commands, html tags, etc will be parsed and reported
>>> as
>>> + /// separate tokens inside the comment body. If false, the comment
>>> text will
>>> + /// be parsed into text and newline tokens.
>>> + bool ParseCommands;
>>> +
>>> /// Given a character reference name (e.g., "lt"), return the
>>> character that
>>> /// it stands for (e.g., "<").
>>> StringRef resolveHTMLNamedCharacterReference(StringRef Name) const;
>>> @@ -315,12 +320,11 @@ private:
>>> /// Eat string matching regexp \code \s*\* \endcode.
>>> void skipLineStartingDecorations();
>>>
>>> - /// Lex stuff inside comments. CommentEnd should be set correctly.
>>> + /// Lex comment text, including commands if ParseCommands is set to
>>> true.
>>> void lexCommentText(Token &T);
>>>
>>> - void setupAndLexVerbatimBlock(Token &T,
>>> - const char *TextBegin,
>>> - char Marker, const CommandInfo *Info);
>>> + void setupAndLexVerbatimBlock(Token &T, const char *TextBegin, char
>>> Marker,
>>> + const CommandInfo *Info);
>>>
>>> void lexVerbatimBlockFirstLine(Token &T);
>>>
>>> @@ -343,14 +347,13 @@ private:
>>>
>>> public:
>>> Lexer(llvm::BumpPtrAllocator &Allocator, DiagnosticsEngine &Diags,
>>> - const CommandTraits &Traits,
>>> - SourceLocation FileLoc,
>>> - const char *BufferStart, const char *BufferEnd);
>>> + const CommandTraits &Traits, SourceLocation FileLoc,
>>> + const char *BufferStart, const char *BufferEnd,
>>> + bool ParseCommands = true);
>>>
>>> void lex(Token &T);
>>>
>>> - StringRef getSpelling(const Token &Tok,
>>> - const SourceManager &SourceMgr,
>>> + StringRef getSpelling(const Token &Tok, const SourceManager
>>> &SourceMgr,
>>> bool *Invalid = nullptr) const;
>>> };
>>>
>>>
>>> Modified: cfe/trunk/include/clang/AST/RawCommentList.h
>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/include/clang/
>>> AST/RawCommentList.h?rev=332458&r1=332457&r2=332458&view=diff
>>> ============================================================
>>> ==================
>>> --- cfe/trunk/include/clang/AST/RawCommentList.h (original)
>>> +++ cfe/trunk/include/clang/AST/RawCommentList.h Wed May 16 05:30:09
>>> 2018
>>> @@ -111,6 +111,30 @@ public:
>>> return extractBriefText(Context);
>>> }
>>>
>>> + /// Returns sanitized comment text, suitable for presentation in
>>> editor UIs.
>>> + /// E.g. will transform:
>>> + /// // This is a long multiline comment.
>>> + /// // Parts of it might be indented.
>>> + /// /* The comments styles might be mixed. */
>>> + /// into
>>> + /// "This is a long multiline comment.\n"
>>> + /// " Parts of it might be indented.\n"
>>> + /// "The comments styles might be mixed."
>>> + /// Also removes leading indentation and sanitizes some common cases:
>>> + /// /* This is a first line.
>>> + /// * This is a second line. It is indented.
>>> + /// * This is a third line. */
>>> + /// and
>>> + /// /* This is a first line.
>>> + /// This is a second line. It is indented.
>>> + /// This is a third line. */
>>> + /// will both turn into:
>>> + /// "This is a first line.\n"
>>> + /// " This is a second line. It is indented.\n"
>>> + /// "This is a third line."
>>> + std::string getFormattedText(const SourceManager &SourceMgr,
>>> + DiagnosticsEngine &Diags) const;
>>> +
>>> /// Parse the comment, assuming it is attached to decl \c D.
>>> comments::FullComment *parse(const ASTContext &Context,
>>> const Preprocessor *PP, const Decl *D)
>>> const;
>>>
>>> Modified: cfe/trunk/lib/AST/CommentLexer.cpp
>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/AST/Commen
>>> tLexer.cpp?rev=332458&r1=332457&r2=332458&view=diff
>>> ============================================================
>>> ==================
>>> --- cfe/trunk/lib/AST/CommentLexer.cpp (original)
>>> +++ cfe/trunk/lib/AST/CommentLexer.cpp Wed May 16 05:30:09 2018
>>> @@ -294,6 +294,39 @@ void Lexer::lexCommentText(Token &T) {
>>> assert(CommentState == LCS_InsideBCPLComment ||
>>> CommentState == LCS_InsideCComment);
>>>
>>> + // Handles lexing non-command text, i.e. text and newline.
>>> + auto HandleNonCommandToken = [&]() -> void {
>>> + assert(State == LS_Normal);
>>> +
>>> + const char *TokenPtr = BufferPtr;
>>> + assert(TokenPtr < CommentEnd);
>>> + switch (*TokenPtr) {
>>> + case '\n':
>>> + case '\r':
>>> + TokenPtr = skipNewline(TokenPtr, CommentEnd);
>>> + formTokenWithChars(T, TokenPtr, tok::newline);
>>> +
>>> + if (CommentState == LCS_InsideCComment)
>>> + skipLineStartingDecorations();
>>> + return;
>>> +
>>> + default: {
>>> + StringRef TokStartSymbols = ParseCommands ? "\n\r\\@&<" :
>>> "\n\r";
>>> + size_t End = StringRef(TokenPtr, CommentEnd - TokenPtr)
>>> + .find_first_of(TokStartSymbols);
>>> + if (End != StringRef::npos)
>>> + TokenPtr += End;
>>> + else
>>> + TokenPtr = CommentEnd;
>>> + formTextToken(T, TokenPtr);
>>> + return;
>>> + }
>>> + }
>>> + };
>>> +
>>> + if (!ParseCommands)
>>> + return HandleNonCommandToken();
>>> +
>>> switch (State) {
>>> case LS_Normal:
>>> break;
>>> @@ -315,136 +348,116 @@ void Lexer::lexCommentText(Token &T) {
>>> }
>>>
>>> assert(State == LS_Normal);
>>> -
>>> const char *TokenPtr = BufferPtr;
>>> assert(TokenPtr < CommentEnd);
>>> - while (TokenPtr != CommentEnd) {
>>> - switch(*TokenPtr) {
>>> - case '\\':
>>> - case '@': {
>>> - // Commands that start with a backslash and commands that start
>>> with
>>> - // 'at' have equivalent semantics. But we keep information
>>> about the
>>> - // exact syntax in AST for comments.
>>> - tok::TokenKind CommandKind =
>>> - (*TokenPtr == '@') ? tok::at_command :
>>> tok::backslash_command;
>>> + switch(*TokenPtr) {
>>> + case '\\':
>>> + case '@': {
>>> + // Commands that start with a backslash and commands that start
>>> with
>>> + // 'at' have equivalent semantics. But we keep information about
>>> the
>>> + // exact syntax in AST for comments.
>>> + tok::TokenKind CommandKind =
>>> + (*TokenPtr == '@') ? tok::at_command : tok::backslash_command;
>>> + TokenPtr++;
>>> + if (TokenPtr == CommentEnd) {
>>> + formTextToken(T, TokenPtr);
>>> + return;
>>> + }
>>> + char C = *TokenPtr;
>>> + switch (C) {
>>> + default:
>>> + break;
>>> +
>>> + case '\\': case '@': case '&': case '$':
>>> + case '#': case '<': case '>': case '%':
>>> + case '\"': case '.': case ':':
>>> + // This is one of \\ \@ \& \$ etc escape sequences.
>>> TokenPtr++;
>>> - if (TokenPtr == CommentEnd) {
>>> - formTextToken(T, TokenPtr);
>>> - return;
>>> - }
>>> - char C = *TokenPtr;
>>> - switch (C) {
>>> - default:
>>> - break;
>>> -
>>> - case '\\': case '@': case '&': case '$':
>>> - case '#': case '<': case '>': case '%':
>>> - case '\"': case '.': case ':':
>>> - // This is one of \\ \@ \& \$ etc escape sequences.
>>> + if (C == ':' && TokenPtr != CommentEnd && *TokenPtr == ':') {
>>> + // This is the \:: escape sequence.
>>> TokenPtr++;
>>> - if (C == ':' && TokenPtr != CommentEnd && *TokenPtr == ':') {
>>> - // This is the \:: escape sequence.
>>> - TokenPtr++;
>>> - }
>>> - StringRef UnescapedText(BufferPtr + 1, TokenPtr - (BufferPtr
>>> + 1));
>>> - formTokenWithChars(T, TokenPtr, tok::text);
>>> - T.setText(UnescapedText);
>>> - return;
>>> }
>>> + StringRef UnescapedText(BufferPtr + 1, TokenPtr - (BufferPtr +
>>> 1));
>>> + formTokenWithChars(T, TokenPtr, tok::text);
>>> + T.setText(UnescapedText);
>>> + return;
>>> + }
>>>
>>> - // Don't make zero-length commands.
>>> - if (!isCommandNameStartCharacter(*TokenPtr)) {
>>> - formTextToken(T, TokenPtr);
>>> - return;
>>> - }
>>> + // Don't make zero-length commands.
>>> + if (!isCommandNameStartCharacter(*TokenPtr)) {
>>> + formTextToken(T, TokenPtr);
>>> + return;
>>> + }
>>>
>>> - TokenPtr = skipCommandName(TokenPtr, CommentEnd);
>>> - unsigned Length = TokenPtr - (BufferPtr + 1);
>>> + TokenPtr = skipCommandName(TokenPtr, CommentEnd);
>>> + unsigned Length = TokenPtr - (BufferPtr + 1);
>>>
>>> - // Hardcoded support for lexing LaTeX formula commands
>>> - // \f$ \f[ \f] \f{ \f} as a single command.
>>> - if (Length == 1 && TokenPtr[-1] == 'f' && TokenPtr !=
>>> CommentEnd) {
>>> - C = *TokenPtr;
>>> - if (C == '$' || C == '[' || C == ']' || C == '{' || C == '}')
>>> {
>>> - TokenPtr++;
>>> - Length++;
>>> - }
>>> + // Hardcoded support for lexing LaTeX formula commands
>>> + // \f$ \f[ \f] \f{ \f} as a single command.
>>> + if (Length == 1 && TokenPtr[-1] == 'f' && TokenPtr != CommentEnd)
>>> {
>>> + C = *TokenPtr;
>>> + if (C == '$' || C == '[' || C == ']' || C == '{' || C == '}') {
>>> + TokenPtr++;
>>> + Length++;
>>> }
>>> + }
>>>
>>> - StringRef CommandName(BufferPtr + 1, Length);
>>> + StringRef CommandName(BufferPtr + 1, Length);
>>>
>>> - const CommandInfo *Info = Traits.getCommandInfoOrNULL(Co
>>> mmandName);
>>> - if (!Info) {
>>> - if ((Info = Traits.getTypoCorrectCommandInfo(CommandName))) {
>>> - StringRef CorrectedName = Info->Name;
>>> - SourceLocation Loc = getSourceLocation(BufferPtr);
>>> - SourceLocation EndLoc = getSourceLocation(TokenPtr);
>>> - SourceRange FullRange = SourceRange(Loc, EndLoc);
>>> - SourceRange CommandRange(Loc.getLocWithOffset(1), EndLoc);
>>> - Diag(Loc, diag::warn_correct_comment_command_name)
>>> - << FullRange << CommandName << CorrectedName
>>> - << FixItHint::CreateReplacement(CommandRange,
>>> CorrectedName);
>>> - } else {
>>> - formTokenWithChars(T, TokenPtr, tok::unknown_command);
>>> - T.setUnknownCommandName(CommandName);
>>> - Diag(T.getLocation(), diag::warn_unknown_comment_com
>>> mand_name)
>>> - << SourceRange(T.getLocation(), T.getEndLocation());
>>> - return;
>>> - }
>>> - }
>>> - if (Info->IsVerbatimBlockCommand) {
>>> - setupAndLexVerbatimBlock(T, TokenPtr, *BufferPtr, Info);
>>> + const CommandInfo *Info = Traits.getCommandInfoOrNULL(Co
>>> mmandName);
>>> + if (!Info) {
>>> + if ((Info = Traits.getTypoCorrectCommandInfo(CommandName))) {
>>> + StringRef CorrectedName = Info->Name;
>>> + SourceLocation Loc = getSourceLocation(BufferPtr);
>>> + SourceLocation EndLoc = getSourceLocation(TokenPtr);
>>> + SourceRange FullRange = SourceRange(Loc, EndLoc);
>>> + SourceRange CommandRange(Loc.getLocWithOffset(1), EndLoc);
>>> + Diag(Loc, diag::warn_correct_comment_command_name)
>>> + << FullRange << CommandName << CorrectedName
>>> + << FixItHint::CreateReplacement(CommandRange,
>>> CorrectedName);
>>> + } else {
>>> + formTokenWithChars(T, TokenPtr, tok::unknown_command);
>>> + T.setUnknownCommandName(CommandName);
>>> + Diag(T.getLocation(), diag::warn_unknown_comment_com
>>> mand_name)
>>> + << SourceRange(T.getLocation(), T.getEndLocation());
>>> return;
>>> }
>>> - if (Info->IsVerbatimLineCommand) {
>>> - setupAndLexVerbatimLine(T, TokenPtr, Info);
>>> - return;
>>> - }
>>> - formTokenWithChars(T, TokenPtr, CommandKind);
>>> - T.setCommandID(Info->getID());
>>> - return;
>>> }
>>> -
>>> - case '&':
>>> - lexHTMLCharacterReference(T);
>>> + if (Info->IsVerbatimBlockCommand) {
>>> + setupAndLexVerbatimBlock(T, TokenPtr, *BufferPtr, Info);
>>> return;
>>> -
>>> - case '<': {
>>> - TokenPtr++;
>>> - if (TokenPtr == CommentEnd) {
>>> - formTextToken(T, TokenPtr);
>>> - return;
>>> - }
>>> - const char C = *TokenPtr;
>>> - if (isHTMLIdentifierStartingCharacter(C))
>>> - setupAndLexHTMLStartTag(T);
>>> - else if (C == '/')
>>> - setupAndLexHTMLEndTag(T);
>>> - else
>>> - formTextToken(T, TokenPtr);
>>> + }
>>> + if (Info->IsVerbatimLineCommand) {
>>> + setupAndLexVerbatimLine(T, TokenPtr, Info);
>>> return;
>>> }
>>> + formTokenWithChars(T, TokenPtr, CommandKind);
>>> + T.setCommandID(Info->getID());
>>> + return;
>>> + }
>>>
>>> - case '\n':
>>> - case '\r':
>>> - TokenPtr = skipNewline(TokenPtr, CommentEnd);
>>> - formTokenWithChars(T, TokenPtr, tok::newline);
>>> -
>>> - if (CommentState == LCS_InsideCComment)
>>> - skipLineStartingDecorations();
>>> - return;
>>> + case '&':
>>> + lexHTMLCharacterReference(T);
>>> + return;
>>>
>>> - default: {
>>> - size_t End = StringRef(TokenPtr, CommentEnd - TokenPtr).
>>> - find_first_of("\n\r\\@&<");
>>> - if (End != StringRef::npos)
>>> - TokenPtr += End;
>>> - else
>>> - TokenPtr = CommentEnd;
>>> + case '<': {
>>> + TokenPtr++;
>>> + if (TokenPtr == CommentEnd) {
>>> formTextToken(T, TokenPtr);
>>> return;
>>> }
>>> + const char C = *TokenPtr;
>>> + if (isHTMLIdentifierStartingCharacter(C))
>>> + setupAndLexHTMLStartTag(T);
>>> + else if (C == '/')
>>> + setupAndLexHTMLEndTag(T);
>>> + else
>>> + formTextToken(T, TokenPtr);
>>> + return;
>>> }
>>> +
>>> + default:
>>> + return HandleNonCommandToken();
>>> }
>>> }
>>>
>>> @@ -727,14 +740,13 @@ void Lexer::lexHTMLEndTag(Token &T) {
>>> }
>>>
>>> Lexer::Lexer(llvm::BumpPtrAllocator &Allocator, DiagnosticsEngine
>>> &Diags,
>>> - const CommandTraits &Traits,
>>> - SourceLocation FileLoc,
>>> - const char *BufferStart, const char *BufferEnd):
>>> - Allocator(Allocator), Diags(Diags), Traits(Traits),
>>> - BufferStart(BufferStart), BufferEnd(BufferEnd),
>>> - FileLoc(FileLoc), BufferPtr(BufferStart),
>>> - CommentState(LCS_BeforeComment), State(LS_Normal) {
>>> -}
>>> + const CommandTraits &Traits, SourceLocation FileLoc,
>>> + const char *BufferStart, const char *BufferEnd,
>>> + bool ParseCommands)
>>> + : Allocator(Allocator), Diags(Diags), Traits(Traits),
>>> + BufferStart(BufferStart), BufferEnd(BufferEnd), FileLoc(FileLoc),
>>> + BufferPtr(BufferStart), CommentState(LCS_BeforeComment),
>>> State(LS_Normal),
>>> + ParseCommands(ParseCommands) {}
>>>
>>> void Lexer::lex(Token &T) {
>>> again:
>>>
>>> Modified: cfe/trunk/lib/AST/RawCommentList.cpp
>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/AST/RawCom
>>> mentList.cpp?rev=332458&r1=332457&r2=332458&view=diff
>>> ============================================================
>>> ==================
>>> --- cfe/trunk/lib/AST/RawCommentList.cpp (original)
>>> +++ cfe/trunk/lib/AST/RawCommentList.cpp Wed May 16 05:30:09 2018
>>> @@ -335,3 +335,94 @@ void RawCommentList::addDeserializedComm
>>> BeforeThanCompare<RawComment>(SourceMgr));
>>> std::swap(Comments, MergedComments);
>>> }
>>> +
>>> +std::string RawComment::getFormattedText(const SourceManager
>>> &SourceMgr,
>>> + DiagnosticsEngine &Diags)
>>> const {
>>> + llvm::StringRef CommentText = getRawText(SourceMgr);
>>> + if (CommentText.empty())
>>> + return "";
>>> +
>>> + llvm::BumpPtrAllocator Allocator;
>>> + // We do not parse any commands, so CommentOptions are ignored by
>>> + // comments::Lexer. Therefore, we just use default-constructed
>>> options.
>>> + CommentOptions DefOpts;
>>> + comments::CommandTraits EmptyTraits(Allocator, DefOpts);
>>> + comments::Lexer L(Allocator, Diags, EmptyTraits,
>>> getSourceRange().getBegin(),
>>> + CommentText.begin(), CommentText.end(),
>>> + /*ParseCommands=*/false);
>>> +
>>> + std::string Result;
>>> + // A column number of the first non-whitespace token in the comment
>>> text.
>>> + // We skip whitespace up to this column, but keep the whitespace
>>> after this
>>> + // column. IndentColumn is calculated when lexing the first line and
>>> reused
>>> + // for the rest of lines.
>>> + unsigned IndentColumn = 0;
>>> +
>>> + // Processes one line of the comment and adds it to the result.
>>> + // Handles skipping the indent at the start of the line.
>>> + // Returns false when eof is reached and true otherwise.
>>> + auto LexLine = [&](bool IsFirstLine) -> bool {
>>> + comments::Token Tok;
>>> + // Lex the first token on the line. We handle it separately,
>>> because we to
>>> + // fix up its indentation.
>>> + L.lex(Tok);
>>> + if (Tok.is(comments::tok::eof))
>>> + return false;
>>> + if (Tok.is(comments::tok::newline)) {
>>> + Result += "\n";
>>> + return true;
>>> + }
>>> + llvm::StringRef TokText = L.getSpelling(Tok, SourceMgr);
>>> + bool LocInvalid = false;
>>> + unsigned TokColumn =
>>> + SourceMgr.getSpellingColumnNumber(Tok.getLocation(),
>>> &LocInvalid);
>>> + assert(!LocInvalid && "getFormattedText for invalid location");
>>> +
>>> + // Amount of leading whitespace in TokText.
>>> + size_t WhitespaceLen = TokText.find_first_not_of(" \t");
>>> + if (WhitespaceLen == StringRef::npos)
>>> + WhitespaceLen = TokText.size();
>>> + // Remember the amount of whitespace we skipped in the first line
>>> to remove
>>> + // indent up to that column in the following lines.
>>> + if (IsFirstLine)
>>> + IndentColumn = TokColumn + WhitespaceLen;
>>> +
>>> + // Amount of leading whitespace we actually want to skip.
>>> + // For the first line we skip all the whitespace.
>>> + // For the rest of the lines, we skip whitespace up to IndentColumn.
>>> + unsigned SkipLen =
>>> + IsFirstLine
>>> + ? WhitespaceLen
>>> + : std::min<size_t>(
>>> + WhitespaceLen,
>>> + std::max<int>(static_cast<int>(IndentColumn) -
>>> TokColumn, 0));
>>> + llvm::StringRef Trimmed = TokText.drop_front(SkipLen);
>>> + Result += Trimmed;
>>> + // Lex all tokens in the rest of the line.
>>> + for (L.lex(Tok); Tok.isNot(comments::tok::eof); L.lex(Tok)) {
>>> + if (Tok.is(comments::tok::newline)) {
>>> + Result += "\n";
>>> + return true;
>>> + }
>>> + Result += L.getSpelling(Tok, SourceMgr);
>>> + }
>>> + // We've reached the end of file token.
>>> + return false;
>>> + };
>>> +
>>> + auto DropTrailingNewLines = [](std::string &Str) {
>>> + while (Str.back() == '\n')
>>> + Str.pop_back();
>>> + };
>>> +
>>> + // Proces first line separately to remember indent for the following
>>> lines.
>>> + if (!LexLine(/*IsFirstLine=*/true)) {
>>> + DropTrailingNewLines(Result);
>>> + return Result;
>>> + }
>>> + // Process the rest of the lines.
>>> + while (LexLine(/*IsFirstLine=*/false))
>>> + ;
>>> + DropTrailingNewLines(Result);
>>> + return Result;
>>> +}
>>>
>>> Modified: cfe/trunk/unittests/AST/CMakeLists.txt
>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/unittests/AST/
>>> CMakeLists.txt?rev=332458&r1=332457&r2=332458&view=diff
>>> ============================================================
>>> ==================
>>> --- cfe/trunk/unittests/AST/CMakeLists.txt (original)
>>> +++ cfe/trunk/unittests/AST/CMakeLists.txt Wed May 16 05:30:09 2018
>>> @@ -9,6 +9,7 @@ add_clang_unittest(ASTTests
>>> ASTVectorTest.cpp
>>> CommentLexer.cpp
>>> CommentParser.cpp
>>> + CommentTextTest.cpp
>>> DataCollectionTest.cpp
>>> DeclPrinterTest.cpp
>>> DeclTest.cpp
>>>
>>> Added: cfe/trunk/unittests/AST/CommentTextTest.cpp
>>> URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/unittests/AST/
>>> CommentTextTest.cpp?rev=332458&view=auto
>>> ============================================================
>>> ==================
>>> --- cfe/trunk/unittests/AST/CommentTextTest.cpp (added)
>>> +++ cfe/trunk/unittests/AST/CommentTextTest.cpp Wed May 16 05:30:09 2018
>>> @@ -0,0 +1,122 @@
>>> +//===- unittest/AST/CommentTextTest.cpp - Comment text extraction test
>>> ----===//
>>> +//
>>> +// The LLVM Compiler Infrastructure
>>> +//
>>> +// This file is distributed under the University of Illinois Open Source
>>> +// License. See LICENSE.TXT for details.
>>> +//
>>> +//===------------------------------------------------------
>>> ----------------===//
>>> +//
>>> +// Tests for user-friendly output formatting of comments, i.e.
>>> +// RawComment::getFormattedText().
>>> +//
>>> +//===------------------------------------------------------
>>> ----------------===//
>>> +
>>> +#include "clang/AST/RawCommentList.h"
>>> +#include "clang/Basic/CommentOptions.h"
>>> +#include "clang/Basic/Diagnostic.h"
>>> +#include "clang/Basic/DiagnosticIDs.h"
>>> +#include "clang/Basic/FileManager.h"
>>> +#include "clang/Basic/FileSystemOptions.h"
>>> +#include "clang/Basic/SourceLocation.h"
>>> +#include "clang/Basic/SourceManager.h"
>>> +#include "clang/Basic/VirtualFileSystem.h"
>>> +#include "llvm/Support/MemoryBuffer.h"
>>> +#include <gtest/gtest.h>
>>> +
>>> +namespace clang {
>>> +
>>> +class CommentTextTest : public ::testing::Test {
>>> +protected:
>>> + std::string formatComment(llvm::StringRef CommentText) {
>>> + SourceManagerForFile FileSourceMgr("comment-test.cpp",
>>> CommentText);
>>> + SourceManager& SourceMgr = FileSourceMgr.get();
>>> +
>>> + auto CommentStartOffset = CommentText.find("/");
>>> + assert(CommentStartOffset != llvm::StringRef::npos);
>>> + FileID File = SourceMgr.getMainFileID();
>>> +
>>> + SourceRange CommentRange(
>>> + SourceMgr.getLocForStartOfFile(File).getLocWithOffset(
>>> + CommentStartOffset),
>>> + SourceMgr.getLocForEndOfFile(File));
>>> + CommentOptions EmptyOpts;
>>> + // FIXME: technically, merged that we set here is incorrect, but
>>> that
>>> + // shouldn't matter.
>>> + RawComment Comment(SourceMgr, CommentRange, EmptyOpts,
>>> /*Merged=*/true);
>>> + DiagnosticsEngine Diags(new DiagnosticIDs, new DiagnosticOptions);
>>> + return Comment.getFormattedText(SourceMgr, Diags);
>>> + }
>>> +};
>>> +
>>> +TEST_F(CommentTextTest, FormattedText) {
>>> + // clang-format off
>>> + auto ExpectedOutput =
>>> +R"(This function does this and that.
>>> +For example,
>>> + Runnning it in that case will give you
>>> + this result.
>>> +That's about it.)";
>>> + // Two-slash comments.
>>> + EXPECT_EQ(ExpectedOutput, formatComment(
>>> +R"cpp(
>>> +// This function does this and that.
>>> +// For example,
>>> +// Runnning it in that case will give you
>>> +// this result.
>>> +// That's about it.)cpp"));
>>> +
>>> + // Three-slash comments.
>>> + EXPECT_EQ(ExpectedOutput, formatComment(
>>> +R"cpp(
>>> +/// This function does this and that.
>>> +/// For example,
>>> +/// Runnning it in that case will give you
>>> +/// this result.
>>> +/// That's about it.)cpp"));
>>> +
>>> + // Block comments.
>>> + EXPECT_EQ(ExpectedOutput, formatComment(
>>> +R"cpp(
>>> +/* This function does this and that.
>>> + * For example,
>>> + * Runnning it in that case will give you
>>> + * this result.
>>> + * That's about it.*/)cpp"));
>>> +
>>> + // Doxygen-style block comments.
>>> + EXPECT_EQ(ExpectedOutput, formatComment(
>>> +R"cpp(
>>> +/** This function does this and that.
>>> + * For example,
>>> + * Runnning it in that case will give you
>>> + * this result.
>>> + * That's about it.*/)cpp"));
>>> +
>>> + // Weird indentation.
>>> + EXPECT_EQ(ExpectedOutput, formatComment(
>>> +R"cpp(
>>> + // This function does this and that.
>>> + // For example,
>>> + // Runnning it in that case will give you
>>> + // this result.
>>> + // That's about it.)cpp"));
>>> + // clang-format on
>>> +}
>>> +
>>> +TEST_F(CommentTextTest, KeepsDoxygenControlSeqs) {
>>> + // clang-format off
>>> + auto ExpectedOutput =
>>> +R"(\brief This is the brief part of the comment.
>>> +\param a something about a.
>>> + at param b something about b.)";
>>> +
>>> + EXPECT_EQ(ExpectedOutput, formatComment(
>>> +R"cpp(
>>> +/// \brief This is the brief part of the comment.
>>> +/// \param a something about a.
>>> +/// @param b something about b.)cpp"));
>>> + // clang-format on
>>> +}
>>> +
>>> +} // namespace clang
>>>
>>>
>>> _______________________________________________
>>> cfe-commits mailing list
>>> cfe-commits at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
>>>
>>
>>
>
> _______________________________________________
> cfe-commits mailing list
> cfe-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20180517/04ee9771/attachment-0001.html>
More information about the cfe-commits
mailing list