r256865 - [Clang/Support/Windows/Unix] Command lines created by clang may exceed the command length limit set by the OS

[Oleg Ranevskyy via cfe-commits]

Author: oleg
Date: Tue Jan  5 13:54:39 2016
New Revision: 256865

URL: http://llvm.org/viewvc/llvm-project?rev=256865&view=rev
Log:
[Clang/Support/Windows/Unix] Command lines created by clang may exceed the command length limit set by the OS

Summary:
LLVM part of the patch is D15831.

When clang runs an external tool such as a linker it may create a command line that exceeds the length limit.

Clang uses the llvm::sys::argumentsFitWithinSystemLimits function to check if command line length fits the OS 

limitation. There are two problems in this function that may cause exceeding of the limit:

1. It ignores the length of the program path in its calculations. On the other hand, clang adds the program 

path to the command line when it runs the program.

2. It assumes no space character is inserted after the last argument, which is not true for Windows. The flattenArgs function adds the trailing space for *each* argument. The result of this is that the terminating NULL character is not counted and may be placed beyond the length limit if the command line is exactly 32768 characters long. The WinAPI's CreateProcess does not find the NULL character and fails.


Reviewers: rafael, asl

Subscribers: asl, llvm-commits

Differential Revision: http://reviews.llvm.org/D15832

Modified:
    cfe/trunk/lib/Driver/Driver.cpp

Modified: cfe/trunk/lib/Driver/Driver.cpp
URL: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Driver/Driver.cpp?rev=256865&r1=256864&r2=256865&view=diff
==============================================================================
--- cfe/trunk/lib/Driver/Driver.cpp (original)
+++ cfe/trunk/lib/Driver/Driver.cpp Tue Jan  5 13:54:39 2016
@@ -699,11 +699,11 @@ void Driver::generateCompilationDiagnost
 }
 
 void Driver::setUpResponseFiles(Compilation &C, Command &Cmd) {
-  // Since argumentsFitWithinSystemLimits() may underestimate system's capacity
+  // Since commandLineFitsWithinSystemLimits() may underestimate system's capacity
   // if the tool does not support response files, there is a chance/ that things
   // will just work without a response file, so we silently just skip it.
   if (Cmd.getCreator().getResponseFilesSupport() == Tool::RF_None ||
-      llvm::sys::argumentsFitWithinSystemLimits(Cmd.getArguments()))
+      llvm::sys::commandLineFitsWithinSystemLimits(Cmd.getExecutable(), Cmd.getArguments()))
     return;
 
   std::string TmpName = GetTemporaryPath("response", "txt");