r278882 - If possible, set the stack rlimit to at least 8MiB on cc1 startup, and work

[Joerg Sonnenberger via cfe-commits]

On Fri, Aug 19, 2016 at 01:16:59PM -0700, Richard Smith wrote:
> On Fri, Aug 19, 2016 at 1:10 PM, Joerg Sonnenberger via cfe-commits <
> cfe-commits at lists.llvm.org> wrote:
> 
> > On Fri, Aug 19, 2016 at 01:03:51PM -0700, Richard Smith wrote:
> > > On Fri, Aug 19, 2016 at 12:58 PM, Joerg Sonnenberger via cfe-commits <
> > > cfe-commits at lists.llvm.org> wrote:
> > >
> > > > On Thu, Aug 18, 2016 at 11:33:49AM -0700, Richard Smith wrote:
> > > > > On Wed, Aug 17, 2016 at 6:35 AM, Joerg Sonnenberger via cfe-commits <
> > > > > cfe-commits at lists.llvm.org> wrote:
> > > > >
> > > > > > On Wed, Aug 17, 2016 at 01:05:08AM -0000, Richard Smith via
> > cfe-commits
> > > > > > wrote:
> > > > > > > Author: rsmith
> > > > > > > Date: Tue Aug 16 20:05:07 2016
> > > > > > > New Revision: 278882
> > > > > > >
> > > > > > > URL: http://llvm.org/viewvc/llvm-project?rev=278882&view=rev
> > > > > > > Log:
> > > > > > > If possible, set the stack rlimit to at least 8MiB on cc1
> > startup,
> > > > and
> > > > > > work
> > > > > > > around a Linux kernel bug where the actual amount of available
> > stack
> > > > may
> > > > > > be a
> > > > > > > *lot* lower than the rlimit.
> > > > > >
> > > > > > Can you please restrict this to Linux? I'm quite opposed to
> > overriding
> > > > > > system default limits, they exist for a reason.
> > > > >
> > > > >
> > > > > No, that wouldn't make any sense. It's not up to the operating
> > system how
> > > > > an application decides to allocate memory (on the heap versus on the
> > > > > stack), and Clang's stack usage isn't going to be significantly
> > lower on
> > > > > other kernels. If some BSD kernel's VM is unable to cope with this,
> > we
> > > > > could spawn a thread with a suitable amount of stack space instead.
> > > >
> > > > This is not about kernel bugs. It is about POLA -- programs are not
> > > > supposed to alter process limits. If GCC does it, it is a GCC bug.
> > > > That's no reason to introduce the same bug here. Using excessive stack
> > > > space due to deep recursion might be a QoI issue, but it is
> > > > fundamentally no different from any other out of memory condition.
> > Those
> > > > kill clang just as easily.
> > >
> > >
> > > Hitting this limit does not imply that memory was exhausted, it instead
> > > means the OS killed a process that was functioning just fine, for no good
> > > reason.
> >
> > You are allocating too much stack space.
> 
> 
> Whether we choose to put data on the heap or stack is not up to you or the
> operating system.

How is that relevant to honoring system limits? At the very least your
commit says "clang knows better than the system administrator".

> > There is no difference to heap
> > allocations which are bound by different flags. It is just a different
> > allocation mechanism. Even hitting a 4MB stack space limit on 64bit
> > platforms takes quite a bit -- not even boost does that by default. As
> > such, I hardly find it normal.
> 
> It's not normal; that's why we're explicitly opting into it here, using the
> mechanism that POSIX provides to do so.

Read again. Input triggering such deep stack case is not normal. Just
because POSIX provides mechanisms for a shell to implement ulimit
doesn't mean that it is considered bad form for *applications* to
override them, especially increasing them. Shall we raise the address
space limits next, because there is input that easily requires more than
16GB of memory to compile? What about bumping the CPU time limit,
because we have input where clang needs hours to compile a single file?

All your commit has done is introduce another funny way to screw users.
There are users of the clang libraries that are not the clang driver.
They don't get this magic stack size boost. Now we have source files
that can be handled by clang but not by library consumers. Lovely edge
case. It all comes back to "raise limits for exceptional programs".
Just like users of compilers for functional languages have learned:
if you write source that triggers such limits, make sure your system is
up for handling them.

Joerg