[PATCH] Don't remove unknown tokens that are not whitespace (PR17215).

[Daniel Jasper]

================
Comment at: lib/Format/TokenAnnotator.cpp:575
@@ +574,3 @@
+      Current.Type = TT_ImplicitStringLiteral;
+    if (Current.Type != TT_Unknown)
+      return;
----------------
This change is not a no-op!! Move to line 608 and add:

// FIXME: Add tests that break if this gets moved up.

if no tests are failing now...

================
Comment at: lib/Format/TokenAnnotator.cpp:573
@@ -571,1 +572,3 @@
   void determineTokenType(FormatToken &Current) {
+    if (Current.Previous && Current.Previous->Tok.is(tok::unknown))
+      Current.Type = TT_ImplicitStringLiteral;
----------------
Can this actually make a difference? If yes, how? It seems like we convert tok::unknown to tok::string_literal or set Type = TT_ImplicitStringLiteral earlier or merge this into another token's whitespace..

================
Comment at: lib/Format/Format.cpp:685
@@ -679,3 +684,3 @@
     unsigned WhitespaceLength = TrailingWhitespace;
-    while (FormatTok->Tok.is(tok::unknown)) {
+    while (isWhitespaceToken(*FormatTok)) {
       for (int i = 0, e = FormatTok->TokenText.size(); i != e; ++i) {
----------------
I'd probably inline this function. It is used only once and I find it important at this point to know that it does not look into FormatTok's content. Not quite sure whether it is better, though.


http://llvm-reviews.chandlerc.com/D1858