<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Apr 21, 2020, at 2:20 PM, Kees Cook <<a href="mailto:keescook@chromium.org" class="">keescook@chromium.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Hi,<br class=""><br class="">tl;dr: I'd like to revisit making -ftrivial-auto-var-init=zero an expressly<br class="">supported option. To do this, I think we need to either entirely remove<br class="">"-enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang"<br class="">or rename it to something more directly reflecting the issue, like<br class="">"-enable-trivial-auto-var-init-zero-knowing-it-forks-the-language".<br class=""><br class="">This is currently open as <a href="https://bugs.llvm.org/show_bug.cgi?id=45497" class="">https://bugs.llvm.org/show_bug.cgi?id=45497</a><br class=""><br class="">Here is the situation: -ftrivial-auto-var-init=pattern is great for<br class="">debugging, but -ftrivial-auto-var-init=zero is needed for production<br class="">systems for mainly two reasons, each of which I will try to express context<br class="">for:<br class=""><br class="">1) performance and size<br class=""><br class="">As measured by various Google folks across a few projects and in<br class="">various places, there's a fairly significant performance impact of<br class="">using pattern-init over zero-init. I can let other folks chime in<br class="">with their exact numbers, but I can at least share some measurements<br class="">Alexander Potapenko made with the Linux kernel (see "Performance costs"):<br class=""><a href="https://clangbuiltlinux.github.io/CBL-meetup-2020-slides/glider/Fighting_uninitialized_memory_%40_CBL_Meetup_2020.pdf" class="">https://clangbuiltlinux.github.io/CBL-meetup-2020-slides/glider/Fighting_uninitialized_memory_%40_CBL_Meetup_2020.pdf</a><br class="">tl;dr: zero-init tended to be half the cost of pattern-init, though it<br class="">varied based on workload, and binary size impact fell over 95% going<br class="">from pattern-init to zero-init.<br class=""></div></div></blockquote><div><br class=""></div><div>The main reason folks were OK with me putting in zero init *at all* was to compare performance between 1) no init 2) zero init 3) pattern init, and *then* tune pattern init to the “floor” set by zero init. In other words: try to get to as low a cost as possible. I and other optimizer folks put <i class="">significant</i> efforts towards this, and we still are, but we’ve only done so for our binaries, and only on front-end / middle-end / ARM64. There’s still very low-lying fruit for pattern init on x86, and there’s more to be optimized on ARM64 and general middle-end. I’m therefore not sure we’re honoring the spirit of the agreement without putting in more optimization effort. To clarify: the performance got better, but we haven’t tried enough to declare “pattern init will definitely never get acceptably close to zero init”.</div><div><br class=""></div><div>At the same time, it’s pretty obvious that synthesizing zero will always be cheaper than synthesizing repeated bytes. The question we’ve still to answer is: does this matter?</div><div><br class=""></div><div>I’d like to see the remainder of the optimization work happen. It benefits generic code, not just pattern init. I don’t think it’s a mountain of work.</div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div class="">2) security<br class=""><br class="">Another driving factor (see below from various vendors/projects), is the<br class="">security stance. Putting non-zero values into most variables types ends<br class="">up making them arguably more dangerous than if they were zero-filled.<br class="">Most notably, sizes and indexes and less likely to be used out of bounds<br class="">if they are zero-initialized. The same holds for bool values that tend<br class="">to indicate success instead of failing safe with a false value. While<br class="">pointers in the non-canonical range are nice, zero tends to be just<br class="">as good. There are certainly exceptions here, but the bulk of the<br class="">historical record on how "uninitialized" variables have been used in<br class="">real world exploitation involve their being non-zero, and analysis of<br class="">those bugs support that conclusion.<br class=""><br class=""><br class="">Various positions from vendors and projects:<br class=""><br class="">Google (Android, Chrome OS)<br class=""><br class="">Both Android and Chrome OS initially started using pattern-init, but due<br class="">to each of: the performance characteristics, the binary size changes, and<br class="">the less robust security stance, both projects have recently committed<br class="">to switching to zero-init.<br class=""><br class=""><br class="">Microsoft (Windows)<br class=""><br class="">I'm repeating what Joe Bialek has told me, so he can clarify if I'm not<br class="">representing this correctly... While not using Clang/LLVM, Microsoft is<br class="">part of the larger C/C++ ecosystem and has implemented both zero-init<br class="">(for production builds) and pattern-init (for debug builds) in their<br class="">compiler too. They also chose zero-init for production expressly due<br class="">to the security benefits.<br class=""><br class="">Some details of their work:<br class=""><a href="https://github.com/microsoft/MSRC-Security-Research/blob/master/presentations/2019_09_CppCon/CppCon2019%20-%20Killing%20Uninitialized%20Memory.pdf" class="">https://github.com/microsoft/MSRC-Security-Research/blob/master/presentations/2019_09_CppCon/CppCon2019%20-%20Killing%20Uninitialized%20Memory.pdf</a><br class=""><br class=""><br class="">Upstream Linux kernel<br class=""><br class="">Linus Torvalds has directly stated that he wants zero-init:<br class="">"So I'd like the zeroing of local variables to be a native compiler<br class="">option..."<br class="">"This, btw, is why I also think that the "initialize with poison" is<br class="">pointless and wrong."<br class="">https://lore.kernel.org/lkml/CAHk-=wgTM+cN7zyUZacGQDv3DuuoA4LORNPWgb1Y_Z1p4iedNQ@mail.gmail.com/<br class="">Unsurprisingly, I strongly agree. ;)<br class=""><br class=""><br class="">GrapheneOS is using zero-init (rather than patching Clang as it used to, to get<br class="">the same result):<br class="">https://twitter.com/DanielMicay/status/1248384468181643272<br class=""><br class=""><br class="">GCC<br class="">There's been mostly silence on the entire topic of automatic variable<br class="">initialization, though there have been patches proposed in the past for<br class="">zero-init:<br class="">https://gcc.gnu.org/legacy-ml/gcc-patches/2014-06/msg00615.html<br class=""><br class=""><br class="">Apple<br class=""><br class="">I can't speak meaningfully here, but I've heard rumors that they are<br class="">depending on zero-init as well. Perhaps someone there can clarify how<br class="">they are using these features?<br class=""></div></div></blockquote><div><br class=""></div><div>What I can easily share is in my LLVM dev meeting talk: <a href="https://www.youtube.com/watch?v=I-XUHPimq3o" class="">https://www.youtube.com/watch?v=I-XUHPimq3o</a></div><div>There I say that we’ve at least deployed it in Secure Enclave code, and XNU. I’ve seen people on the internet find other places we seem to have deployed it :-)</div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div class="">So, while I understand the earlier objections to zero-init from a<br class="">"language fork" concern, I think this isn't a position that can really<br class="">stand up to the reality of how many projects are using the feature (even<br class="">via non-Clang compilers). Given that so much code is going to be built<br class="">using zero-init, what's the best way for Clang to adapt here? I would<br class="">prefer to just drop the -enable... option entirely, but I think just<br class="">renaming it would be fine too.<br class=""><br class="">Thoughts/flames? ;)<br class=""><br class="">-- <br class="">Kees Cook<br class=""></div></div></blockquote></div><br class=""></body></html>