[PATCH] D15553: PR25717: fatal IO error writing large outputs to console on Windows

[Yunzhong Gao via llvm-commits]

ygao created this revision.
ygao added reviewers: silvas, aaron.ballman, Bigcheese.
ygao added a subscriber: llvm-commits.

The fix to this bug is similar to the Python issue#11395. We need to cap the output size to
32767 on Windows to work around the size limit of WriteConsole().
Reference: https://bugs.python.org/issue11395

At least on Visual Studio 2013, the _isatty() check should be fast because it is implemented
by reading a field from the preloaded ioinfo struct.

http://reviews.llvm.org/D15553

Files:
  lib/Support/raw_ostream.cpp

Index: lib/Support/raw_ostream.cpp
===================================================================
--- lib/Support/raw_ostream.cpp
+++ lib/Support/raw_ostream.cpp
@@ -567,20 +567,34 @@
   assert(FD >= 0 && "File already closed.");
   pos += Size;
 
+#ifndef LLVM_ON_WIN32
+  bool ShouldWriteInChunks = false;
+#else
+  // PR25717: writing a large size of output to Windows console returns ENOMEM.
+  // Seems that WriteFile() is redirecting to WriteConsole() prior to Windows 8,
+  // and WriteConsole() has a size limit (66000 bytes or less, depending on
+  // heap usage).
+  bool ShouldWriteInChunks = !!::_isatty(FD);
+#endif
+
   do {
     ssize_t ret;
 
+    size_t ChunkSize = Size;
+    if (ChunkSize > 32767 && ShouldWriteInChunks)
+        ChunkSize = 32767;
+
     // Check whether we should attempt to use atomic writes.
     if (LLVM_LIKELY(!UseAtomicWrites)) {
-      ret = ::write(FD, Ptr, Size);
+      ret = ::write(FD, Ptr, ChunkSize);
     } else {
       // Use ::writev() where available.
 #if defined(HAVE_WRITEV)
       const void *Addr = static_cast<const void *>(Ptr);
-      struct iovec IOV = {const_cast<void *>(Addr), Size };
+      struct iovec IOV = {const_cast<void *>(Addr), ChunkSize };
       ret = ::writev(FD, &IOV, 1);
 #else
-      ret = ::write(FD, Ptr, Size);
+      ret = ::write(FD, Ptr, ChunkSize);
 #endif
     }
 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D15553.42953.patch
Type: text/x-patch
Size: 1364 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20151216/76e10c76/attachment.bin>