[llvm-bugs] [Bug 47479] New: inlining of stack-protected functions into non-stack-protected functions dangerous

[via llvm-bugs]

https://bugs.llvm.org/show_bug.cgi?id=47479

            Bug ID: 47479
           Summary: inlining of stack-protected functions into
                    non-stack-protected functions dangerous
           Product: libraries
           Version: trunk
          Hardware: PC
                OS: All
            Status: NEW
          Severity: enhancement
          Priority: P
         Component: Interprocedural Optimizations
          Assignee: unassignedbugs at nondot.org
          Reporter: ndesaulniers at google.com
                CC: llozano at chromium.org, llvm-bugs at lists.llvm.org,
                    manojgupta at google.com, peter at pcc.me.uk,
                    samitolvanen at google.com, srhines at google.com
            Blocks: 4068

Forked from internal google bug b/166163480.

If we have code that is compiled without a stack protector (such as code in the
Linux kernel that's trying to restore registers containing the stack canaries),
then inlining code that was compiled with stack protector into it can break
code.  LTO makes this more likely to occur across TUs which may or may not be
compiled with the same -fstack-protector flag.

The inliner probably should not inline functions if the caller does not use a
stack protector, but the callee does.

Toy example:
// foo.c
#include <alloca.h>
void foo(void *);

void bar(size_t n) {
  foo(alloca(n));
}

__attribute__((no_stack_protector))
void baz(void) {
  bar(1024);
}

$ clang -O2 baz.c -c -fstack-protector-strong
$ llvm-objdump -dr baz.o
0000000000000000 <bar>:
...
      41: c3                            retq
      42: e8 00 00 00 00                callq   0x47 <bar+0x47>
                0000000000000043:  R_X86_64_PLT32       __stack_chk_fail-0x4
...
0000000000000050 <baz>:
...
      8a: c3                            retq
      8b: e8 00 00 00 00                callq   0x90 <baz+0x40>
                000000000000008c:  R_X86_64_PLT32       __stack_chk_fail-0x4


Oh, no! `bar` was compiled with stack protection, was inlined into `baz` (bad),
which now has stack protection, even though we explicitly disabled it for that
function.


Referenced Bugs:

https://bugs.llvm.org/show_bug.cgi?id=4068
[Bug 4068] [Meta] Compiling the Linux kernel with clang
-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20200910/b51b2396/attachment.html>