[llvm] r315943 - [MC] Lex CRLF as one token

[Reid Kleckner via llvm-commits]

Author: rnk
Date: Mon Oct 16 15:20:03 2017
New Revision: 315943

URL: http://llvm.org/viewvc/llvm-project?rev=315943&view=rev
Log:
[MC] Lex CRLF as one token

This will prevent doubling of line endings when parsing assembly and
emitting assembly.

Otherwise we'd parse the directive, consume the end of statement, hit
the next end of statement, and emit a fresh newline.

Added:
    llvm/trunk/test/MC/X86/crlf.test
Modified:
    llvm/trunk/lib/MC/MCParser/AsmLexer.cpp

Modified: llvm/trunk/lib/MC/MCParser/AsmLexer.cpp
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/MC/MCParser/AsmLexer.cpp?rev=315943&r1=315942&r2=315943&view=diff
==============================================================================
--- llvm/trunk/lib/MC/MCParser/AsmLexer.cpp (original)
+++ llvm/trunk/lib/MC/MCParser/AsmLexer.cpp Mon Oct 16 15:20:03 2017
@@ -606,8 +606,16 @@ AsmToken AsmLexer::LexToken() {
       return LexToken(); // Ignore whitespace.
     else
       return AsmToken(AsmToken::Space, StringRef(TokStart, CurPtr - TokStart));
+  case '\r': {
+    IsAtStartOfLine = true;
+    IsAtStartOfStatement = true;
+    // If this is a CR followed by LF, treat that as one token.
+    if (CurPtr != CurBuf.end() && *CurPtr == '\n')
+      ++CurPtr;
+    return AsmToken(AsmToken::EndOfStatement,
+                    StringRef(TokStart, CurPtr - TokStart));
+  }
   case '\n':
-  case '\r':
     IsAtStartOfLine = true;
     IsAtStartOfStatement = true;
     return AsmToken(AsmToken::EndOfStatement, StringRef(TokStart, 1));

Added: llvm/trunk/test/MC/X86/crlf.test
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/test/MC/X86/crlf.test?rev=315943&view=auto
==============================================================================
--- llvm/trunk/test/MC/X86/crlf.test (added)
+++ llvm/trunk/test/MC/X86/crlf.test Mon Oct 16 15:20:03 2017
@@ -0,0 +1,5 @@
+RUN: printf '\r\n\r\n' | llvm-mc -as-lex | FileCheck %s
+There should only be two end of statements.
+CHECK: EndOfStatement
+CHECK: EndOfStatement
+CHECK-NOT: EndOfStatement