[cfe-commits] r154204 - /cfe/trunk/lib/Basic/SourceManager.cpp

[Benjamin Kramer]

Author: d0k
Date: Fri Apr  6 15:49:55 2012
New Revision: 154204

URL: http://llvm.org/viewvc/llvm-project?rev=154204&view=rev
Log:
SourceManager: Vectorize ComputeLineNumbers for SSE2.

This method is very hot, it is called when emitting diagnostics, in -E mode
and for many #pragma handlers. It scans through the whole source file to
count newlines, records and caches them in a vector.

The speedup from vectorization isn't very large, as we fall back to bytewise
scanning when we hit a newline. There might be a way to avoid leaving the sse
loop but everything I tried didn't work out because a call to push_back
clobbers xmm registers.

About 2% speedup on average on "clang -E > /dev/null" of all .cpp files in
clang's lib/Sema.

Modified:
    cfe/trunk/lib/Basic/SourceManager.cpp

Modified: cfe/trunk/lib/Basic/SourceManager.cpp
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Basic/SourceManager.cpp?rev=154204&r1=154203&r2=154204&view=diff
==============================================================================
--- cfe/trunk/lib/Basic/SourceManager.cpp (original)
+++ cfe/trunk/lib/Basic/SourceManager.cpp Fri Apr  6 15:49:55 2012
@@ -1037,6 +1037,10 @@
   return getPresumedLoc(Loc).getColumn();
 }
 
+#ifdef __SSE2__
+#include <emmintrin.h>
+#endif
+
 static LLVM_ATTRIBUTE_NOINLINE void
 ComputeLineNumbers(DiagnosticsEngine &Diag, ContentCache *FI,
                    llvm::BumpPtrAllocator &Alloc,
@@ -1062,11 +1066,44 @@
   unsigned Offs = 0;
   while (1) {
     // Skip over the contents of the line.
-    // TODO: Vectorize this?  This is very performance sensitive for programs
-    // with lots of diagnostics and in -E mode.
     const unsigned char *NextBuf = (const unsigned char *)Buf;
+
+#ifdef __SSE2__
+    // Try to skip to the next newline using SSE instructions. This is very
+    // performance sensitive for programs with lots of diagnostics and in -E
+    // mode.
+    __m128i CRs = _mm_set1_epi8('\r');
+    __m128i LFs = _mm_set1_epi8('\n');
+
+    // First fix up the alignment to 16 bytes.
+    while (((uintptr_t)NextBuf & 0xF) != 0) {
+      if (*NextBuf == '\n' || *NextBuf == '\r' || *NextBuf == '\0')
+        goto FoundSpecialChar;
+      ++NextBuf;
+    }
+
+    // Scan 16 byte chunks for '\r' and '\n'. Ignore '\0'.
+    while (NextBuf+16 <= End) {
+      __m128i Chunk = *(__m128i*)NextBuf;
+      __m128i Cmp = _mm_or_si128(_mm_cmpeq_epi8(Chunk, CRs),
+                                 _mm_cmpeq_epi8(Chunk, LFs));
+      unsigned Mask = _mm_movemask_epi8(Cmp);
+
+      // If we found a newline, adjust the pointer and jump to the handling code.
+      if (Mask != 0) {
+        NextBuf += llvm::CountTrailingZeros_32(Mask);
+        goto FoundSpecialChar;
+      }
+      NextBuf += 16;
+    }
+#endif
+
     while (*NextBuf != '\n' && *NextBuf != '\r' && *NextBuf != '\0')
       ++NextBuf;
+
+#ifdef __SSE2__
+FoundSpecialChar:
+#endif
     Offs += NextBuf-Buf;
     Buf = NextBuf;