[PATCH] D39444: [Support] Make the default chunk size of raw_fd_ostream to 1 GiB.

[Rafael Avila de Espindola via llvm-commits]

LGTM

Rui Ueyama via Phabricator via llvm-commits
<llvm-commits at lists.llvm.org> writes:

> ruiu created this revision.
> Herald added subscribers: hiraditya, krytarowski.
>
> Previously, we call write(2) for each 32767 byte chunk. That is not
> efficient because Linux can handle much larger write requests.
> This patch changes the chunk size on Linux to 1 GiB.
>
> This patch doesn't change the existing behavior for Windows.
>
>
> https://reviews.llvm.org/D39444
>
> Files:
>   llvm/lib/Support/raw_ostream.cpp
>
>
> Index: llvm/lib/Support/raw_ostream.cpp
> ===================================================================
> --- llvm/lib/Support/raw_ostream.cpp
> +++ llvm/lib/Support/raw_ostream.cpp
> @@ -578,24 +578,20 @@
>    assert(FD >= 0 && "File already closed.");
>    pos += Size;
>  
> -#ifndef LLVM_ON_WIN32
> -#if defined(__linux__)
> -  bool ShouldWriteInChunks = true;
> -#else
> -  bool ShouldWriteInChunks = false;
> -#endif
> -#else
> +  // It is observed that Linux returns EINVAL for a very large write (>2G),
> +  // so the default write size is limited to 1 GiB.
> +  size_t MaxWriteSize = 1024 * 1024 * 1024;
> +
> +#ifdef LLVM_ON_WIN32
>    // Writing a large size of output to Windows console returns ENOMEM. It seems
>    // that, prior to Windows 8, WriteFile() is redirecting to WriteConsole(), and
>    // the latter has a size limit (66000 bytes or less, depending on heap usage).
> -  bool ShouldWriteInChunks = !!::_isatty(FD) && !RunningWindows8OrGreater();
> +  if (!!::_isatty(FD) && !RunningWindows8OrGreater())
> +    MaxWriteSize = 32767;
>  #endif
>  
>    do {
> -    size_t ChunkSize = Size;
> -    if (ChunkSize > 32767 && ShouldWriteInChunks)
> -        ChunkSize = 32767;
> -
> +    size_t ChunkSize = std::min(Size, MaxWriteSize);
>      ssize_t ret = ::write(FD, Ptr, ChunkSize);
>  
>      if (ret < 0) {
>
>
> Index: llvm/lib/Support/raw_ostream.cpp
> ===================================================================
> --- llvm/lib/Support/raw_ostream.cpp
> +++ llvm/lib/Support/raw_ostream.cpp
> @@ -578,24 +578,20 @@
>    assert(FD >= 0 && "File already closed.");
>    pos += Size;
>  
> -#ifndef LLVM_ON_WIN32
> -#if defined(__linux__)
> -  bool ShouldWriteInChunks = true;
> -#else
> -  bool ShouldWriteInChunks = false;
> -#endif
> -#else
> +  // It is observed that Linux returns EINVAL for a very large write (>2G),
> +  // so the default write size is limited to 1 GiB.
> +  size_t MaxWriteSize = 1024 * 1024 * 1024;
> +
> +#ifdef LLVM_ON_WIN32
>    // Writing a large size of output to Windows console returns ENOMEM. It seems
>    // that, prior to Windows 8, WriteFile() is redirecting to WriteConsole(), and
>    // the latter has a size limit (66000 bytes or less, depending on heap usage).
> -  bool ShouldWriteInChunks = !!::_isatty(FD) && !RunningWindows8OrGreater();
> +  if (!!::_isatty(FD) && !RunningWindows8OrGreater())
> +    MaxWriteSize = 32767;
>  #endif
>  
>    do {
> -    size_t ChunkSize = Size;
> -    if (ChunkSize > 32767 && ShouldWriteInChunks)
> -        ChunkSize = 32767;
> -
> +    size_t ChunkSize = std::min(Size, MaxWriteSize);
>      ssize_t ret = ::write(FD, Ptr, ChunkSize);
>  
>      if (ret < 0) {
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits