r278882 - If possible, set the stack rlimit to at least 8MiB on cc1 startup, and work

[Richard Smith via cfe-commits]

On Fri, Aug 19, 2016 at 1:43 PM, Joerg Sonnenberger via cfe-commits <
cfe-commits at lists.llvm.org> wrote:

> On Fri, Aug 19, 2016 at 01:16:59PM -0700, Richard Smith wrote:
> > On Fri, Aug 19, 2016 at 1:10 PM, Joerg Sonnenberger via cfe-commits <
> > cfe-commits at lists.llvm.org> wrote:
> >
> > > On Fri, Aug 19, 2016 at 01:03:51PM -0700, Richard Smith wrote:
> > > > On Fri, Aug 19, 2016 at 12:58 PM, Joerg Sonnenberger via cfe-commits
> <
> > > > cfe-commits at lists.llvm.org> wrote:
> > > >
> > > > > On Thu, Aug 18, 2016 at 11:33:49AM -0700, Richard Smith wrote:
> > > > > > On Wed, Aug 17, 2016 at 6:35 AM, Joerg Sonnenberger via
> cfe-commits <
> > > > > > cfe-commits at lists.llvm.org> wrote:
> > > > > >
> > > > > > > On Wed, Aug 17, 2016 at 01:05:08AM -0000, Richard Smith via
> > > cfe-commits
> > > > > > > wrote:
> > > > > > > > Author: rsmith
> > > > > > > > Date: Tue Aug 16 20:05:07 2016
> > > > > > > > New Revision: 278882
> > > > > > > >
> > > > > > > > URL: http://llvm.org/viewvc/llvm-project?rev=278882&view=rev
> > > > > > > > Log:
> > > > > > > > If possible, set the stack rlimit to at least 8MiB on cc1
> > > startup,
> > > > > and
> > > > > > > work
> > > > > > > > around a Linux kernel bug where the actual amount of
> available
> > > stack
> > > > > may
> > > > > > > be a
> > > > > > > > *lot* lower than the rlimit.
> > > > > > >
> > > > > > > Can you please restrict this to Linux? I'm quite opposed to
> > > overriding
> > > > > > > system default limits, they exist for a reason.
> > > > > >
> > > > > >
> > > > > > No, that wouldn't make any sense. It's not up to the operating
> > > system how
> > > > > > an application decides to allocate memory (on the heap versus on
> the
> > > > > > stack), and Clang's stack usage isn't going to be significantly
> > > lower on
> > > > > > other kernels. If some BSD kernel's VM is unable to cope with
> this,
> > > we
> > > > > > could spawn a thread with a suitable amount of stack space
> instead.
> > > > >
> > > > > This is not about kernel bugs. It is about POLA -- programs are not
> > > > > supposed to alter process limits. If GCC does it, it is a GCC bug.
> > > > > That's no reason to introduce the same bug here. Using excessive
> stack
> > > > > space due to deep recursion might be a QoI issue, but it is
> > > > > fundamentally no different from any other out of memory condition.
> > > Those
> > > > > kill clang just as easily.
> > > >
> > > >
> > > > Hitting this limit does not imply that memory was exhausted, it
> instead
> > > > means the OS killed a process that was functioning just fine, for no
> good
> > > > reason.
> > >
> > > You are allocating too much stack space.
> >
> >
> > Whether we choose to put data on the heap or stack is not up to you or
> the
> > operating system.
>
> How is that relevant to honoring system limits? At the very least your
> commit says "clang knows better than the system administrator".


It typically does; the default stack ulimit is likely tuned for "normal"
applications that are not expected (by an ISO standard) to cope with
recursing a thousand levels deep. If the system administrator really wants
to control the stack space for some stupid reason, they can set a hard
ulimit. If people actually do that, and then file bugs on clang crashing
(which they do today), we may want to resort to spawning a separate thread
with a higher ulimit if we detect the problem.

> > There is no difference to heap
> > > allocations which are bound by different flags. It is just a different
> > > allocation mechanism. Even hitting a 4MB stack space limit on 64bit
> > > platforms takes quite a bit -- not even boost does that by default. As
> > > such, I hardly find it normal.
> >
> > It's not normal; that's why we're explicitly opting into it here, using
> the
> > mechanism that POSIX provides to do so.
>
> Read again. Input triggering such deep stack case is not normal.


Nonetheless, it is within the range that the standard expects us to
process, and it does happen in practice.

Just
> because POSIX provides mechanisms for a shell to implement ulimit
> doesn't mean that it is considered bad form for *applications* to
> override them, especially increasing them. Shall we raise the address
> space limits next, because there is input that easily requires more than
> 16GB of memory to compile? What about bumping the CPU time limit,
> because we have input where clang needs hours to compile a single file?
>

That's a strawman argument. These situations are not comparable.

All your commit has done is introduce another funny way to screw users.
> There are users of the clang libraries that are not the clang driver.
> They don't get this magic stack size boost.


Actually, they do in a lot of cases. libclang parses code in a separate
thread with at least this much stack space by default. Implicit module
compilations run in a separate thread with at least this much stack space.

Now we have source files
> that can be handled by clang but not by library consumers. Lovely edge
> case.


OK, this is a reasonable point, and it seems like the conclusion is that we
should document the amount of stack space that clang expects, so that tools
using clang can provide that much, and perhaps provide some convenience
functions for such tools to use to ensure they get the amount we want. That
seems very reasonable to me. There's even a FIXME in this commit to that
effect. (Clang's 8MiB expectation is now duplicated in a handful of places;
unifying these seems like a very sensible move.)

It all comes back to "raise limits for exceptional programs".
> Just like users of compilers for functional languages have learned:
> if you write source that triggers such limits, make sure your system is
> up for handling them.


If we were talking about programs that exceeded the recommended limits from
the C++ standard, then sure, this seems like a reasonable position to take.
But we're not. It's not unreasonable for C++ programmers to expect clang to
accept programs that fit within the limits recommended by the C++ standard.
It *is* unreasonable to expect them to fiddle with stack ulimits to get the
compiler to accept programs that, say, use certain components of boost (and
fit within the standard's recommended limits).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20160819/71510211/attachment.html>