[llvm] r256866 - [Clang/Support/Windows/Unix] Command lines created by clang may exceed the command length limit set by the OS

[Oleg Ranevskyy via llvm-commits]

Author: oleg
Date: Tue Jan  5 13:56:12 2016
New Revision: 256866

URL: http://llvm.org/viewvc/llvm-project?rev=256866&view=rev
Log:
[Clang/Support/Windows/Unix] Command lines created by clang may exceed the command length limit set by the OS

Summary:
Hi Rafael,

Would you be able to review this patch, please?

(Clang part of the patch is D15832).

When clang runs an external tool, e.g. a linker, it may create a command line that exceeds the length limit.

Clang uses the llvm::sys::argumentsFitWithinSystemLimits function to check if command line length fits the OS 

limitation. There are two problems in this function that may cause exceeding of the limit:

1. It ignores the length of the program path in its calculations. On the other hand, clang adds the program 

path to the command line when it runs the program.

2. It assumes no space character is inserted after the last argument, which is not true for Windows. The flattenArgs function adds the trailing space for *each* argument. The result of this is that the terminating NULL character is not counted and may be placed beyond the length limit if the command line is exactly 32768 characters long. The WinAPI's CreateProcess does not find the NULL character and fails.

Reviewers: rafael, ygao, probinson

Subscribers: asl, llvm-commits

Differential Revision: http://reviews.llvm.org/D15831

Modified:
    llvm/trunk/include/llvm/Support/Program.h
    llvm/trunk/lib/Support/Unix/Program.inc
    llvm/trunk/lib/Support/Windows/Program.inc

Modified: llvm/trunk/include/llvm/Support/Program.h
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/Support/Program.h?rev=256866&r1=256865&r2=256866&view=diff
==============================================================================
--- llvm/trunk/include/llvm/Support/Program.h (original)
+++ llvm/trunk/include/llvm/Support/Program.h Tue Jan  5 13:56:12 2016
@@ -130,7 +130,7 @@ struct ProcessInfo {
 
   /// Return true if the given arguments fit within system-specific
   /// argument length limits.
-  bool argumentsFitWithinSystemLimits(ArrayRef<const char*> Args);
+  bool commandLineFitsWithinSystemLimits(StringRef Program, ArrayRef<const char*> Args);
 
   /// File encoding options when writing contents that a non-UTF8 tool will
   /// read (on Windows systems). For UNIX, we always use UTF-8.

Modified: llvm/trunk/lib/Support/Unix/Program.inc
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Support/Unix/Program.inc?rev=256866&r1=256865&r2=256866&view=diff
==============================================================================
--- llvm/trunk/lib/Support/Unix/Program.inc (original)
+++ llvm/trunk/lib/Support/Unix/Program.inc Tue Jan  5 13:56:12 2016
@@ -446,7 +446,7 @@ llvm::sys::writeFileWithEncoding(StringR
   return EC;
 }
 
-bool llvm::sys::argumentsFitWithinSystemLimits(ArrayRef<const char*> Args) {
+bool llvm::sys::commandLineFitsWithinSystemLimits(StringRef Program, ArrayRef<const char*> Args) {
   static long ArgMax = sysconf(_SC_ARG_MAX);
 
   // System says no practical limit.
@@ -456,7 +456,7 @@ bool llvm::sys::argumentsFitWithinSystem
   // Conservatively account for space required by environment variables.
   long HalfArgMax = ArgMax / 2;
 
-  size_t ArgLength = 0;
+  size_t ArgLength = Program.size() + 1;
   for (ArrayRef<const char*>::iterator I = Args.begin(), E = Args.end();
        I != E; ++I) {
     ArgLength += strlen(*I) + 1;

Modified: llvm/trunk/lib/Support/Windows/Program.inc
URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/Support/Windows/Program.inc?rev=256866&r1=256865&r2=256866&view=diff
==============================================================================
--- llvm/trunk/lib/Support/Windows/Program.inc (original)
+++ llvm/trunk/lib/Support/Windows/Program.inc Tue Jan  5 13:56:12 2016
@@ -535,14 +535,15 @@ llvm::sys::writeFileWithEncoding(StringR
   return EC;
 }
 
-bool llvm::sys::argumentsFitWithinSystemLimits(ArrayRef<const char*> Args) {
+bool llvm::sys::commandLineFitsWithinSystemLimits(StringRef Program, ArrayRef<const char*> Args) {
   // The documented max length of the command line passed to CreateProcess.
   static const size_t MaxCommandStringLength = 32768;
-  size_t ArgLength = 0;
+  // Account for the trailing space for the program path and the
+  // trailing NULL of the last argument.
+  size_t ArgLength = ArgLenWithQuotes(Program.str().c_str()) + 2;
   for (ArrayRef<const char*>::iterator I = Args.begin(), E = Args.end();
        I != E; ++I) {
-    // Account for the trailing space for every arg but the last one and the
-    // trailing NULL of the last argument.
+    // Account for the trailing space for every arg
     ArgLength += ArgLenWithQuotes(*I) + 1;
     if (ArgLength > MaxCommandStringLength) {
       return false;