<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Apr 21, 2020 at 9:18 PM Joe Bialek via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">To add in, we (Microsoft) currently use zero initialization technology in Visual Studio in a large amount of production code we ship to customers (all kernel components, a number of user-mode components). This code is both C and C++.<br>
<br>
We already have had multiple vulnerabilities killed because we shipped this technology in production. We received bug reports with repros that worked on older versions of Windows without the mitigation and new versions of Windows that do have it. The new versions don't repro, the old ones do.<br>
<br>
Using this sort of technology only in development (and not in production) is not sufficient. Some of these bugs will never produce crashes, the uninitialized data is copied across a trust boundary (i.e. from the kernel in to a untrusted user-mode process). This will never result in a crash but does result in a security issue. This is why shipping in production is a requirement even if you had perfect test coverage that exercises all code paths (which nobody has).<br>
<br>
We do enable pattern initialization for debug builds and internal retail builds (using a developer mode in the build environment). We do this to help prevent "forking of the language" and also to force non-determinism. If your code relies on the zero-init, it won't work when we do pattern init. If your code only works with a non-zero value but doesn't care what that value is (Booleans, certain bit tests, etc.), it won't work with zero-init. Developers cannot depend on the automatic initialization for program correctness.<br></blockquote><div><br></div><div>How is this, as a compiler feature, shipped to users? Users don't have a direct toggle available, but they say "I want memory init" with some compiler flag, then if they're building in debug (any unoptimized build, or only those with debug information emitted?) that translates to pattern, and in a build with any optimizations enabled (even the lowest level) they get zero init?<br><br>While in a constrained environment within a company I can see how you could hold that bar - once you ship that to end users I'd be afraid that some of them would figure out the least-optimizing (most debuggable) way to get zero init and avoid the pattern init behavior & write code that depended on that zero init - thus forking the language.<br><br>- Dave</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Joe<br>
<br>
-----Original Message-----<br>
From: Kees Cook <<a href="mailto:keescook@chromium.org" target="_blank">keescook@chromium.org</a>> <br>
Sent: Tuesday, April 21, 2020 4:20 PM<br>
To: Hubert Tong <<a href="mailto:hubert.reinterpretcast@gmail.com" target="_blank">hubert.reinterpretcast@gmail.com</a>><br>
Cc: Clang Dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>>; Joe Bialek <<a href="mailto:jobialek@microsoft.com" target="_blank">jobialek@microsoft.com</a>><br>
Subject: [EXTERNAL] Re: [cfe-dev] making -ftrivial-auto-var-init=zero a first-class option<br>
<br>
On Tue, Apr 21, 2020 at 06:29:07PM -0400, Hubert Tong wrote:<br>
> On Tue, Apr 21, 2020 at 5:20 PM Kees Cook via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<br>
> > <a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcl" rel="noreferrer" target="_blank">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcl</a><br>
> > <a href="http://angbuiltlinux.github.io" rel="noreferrer" target="_blank">angbuiltlinux.github.io</a>%2FCBL-meetup-2020-slides%2Fglider%2FFighting<br>
> > _uninitialized_memory_%2540_CBL_Meetup_2020.pdf&data=02%7C01%7Cj<br>
> > obialek%<a href="http://40microsoft.com" rel="noreferrer" target="_blank">40microsoft.com</a>%7C4bce6c76554b4dcf4c2b08d7e64a7848%7C72f988b<br>
> > f86f141af91ab2d7cd011db47%7C1%7C0%7C637231080235713420&sdata=uMZ<br>
> > PAiQvnjfLxORQNOAdUHGLY8czk8Mlsxc8dXpLSYg%3D&reserved=0<br>
> > tl;dr: zero-init tended to be half the cost of pattern-init, though <br>
> > it varied based on workload, and binary size impact fell over 95% <br>
> > going from pattern-init to zero-init.<br>
> ><br>
> This does not seem to indicate why zero-init is preferred over a <br>
> default of using no explicit policy in production.<br>
<br>
Do you mean "leaving things uninitialized" when you say "no explicit policy"? Maybe I've misunderstood. Google's goal of using auto-init is to eliminate uninitialized variables in production as a security defense. When examining zero-init vs pattern-init, there is a clear advantage on performance and size for zero-init.<br>
<br>
> > as good. There are certainly exceptions here, but the bulk of the <br>
> > historical record on how "uninitialized" variables have been used in<br>
> ><br>
> Maybe an explanation of the scare quotes around "uninitialized" would <br>
> help clarify your position.<br>
<br>
Ah, sorry, I always use quotes (they are not intended to scare but to<br>
clarify) when discussing uninitialized variables in real-world contexts, because they are, of course, not uninitialized in the sense of them not having a value. The RAM contents have a value. Many people without compiler backgrounds think of such variables as being uncontrollable or meaningless, when in fact they are usually highly controllable by an attacker, etc.<br>
<br>
> > Google (Android, Chrome OS)<br>
> ><br>
> > Both Android and Chrome OS initially started using pattern-init, but <br>
> > due to each of: the performance characteristics, the binary size <br>
> > changes, and the less robust security stance, both projects have <br>
> > recently committed to switching to zero-init.<br>
> ><br>
> I'm not sure that this is clear in terms of whether the statements <br>
> apply to debug/development or production. I don't think pattern-init <br>
> is meant to be a tool for production builds, which leads me to think <br>
> that the above statement is about debug builds, at which point I'm <br>
> thinking that using zero-init only serves to hide problems.<br>
<br>
The context for Google's use of zero-init was meant here to be about production builds.<br>
<br>
> > Upstream Linux kernel<br>
> ><br>
> > Linus Torvalds has directly stated that he wants zero-init:<br>
> > "So I'd like the zeroing of local variables to be a native compiler <br>
> > option..."<br>
> > "This, btw, is why I also think that the "initialize with poison" is <br>
> > pointless and wrong."<br>
> ><br>
> > <a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flo" rel="noreferrer" target="_blank">https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flo</a><br>
> > <a href="http://re.kernel.org" rel="noreferrer" target="_blank">re.kernel.org</a>%2Flkml%2FCAHk-%3DwgTM%2BcN7zyUZacGQDv3DuuoA4LORNPWgb1Y<br>
> > _Z1p4iedNQ%<a href="http://40mail.gmail.com" rel="noreferrer" target="_blank">40mail.gmail.com</a>%2F&data=02%7C01%7Cjobialek%40microso<br>
> > <a href="http://ft.com" rel="noreferrer" target="_blank">ft.com</a>%7C4bce6c76554b4dcf4c2b08d7e64a7848%7C72f988bf86f141af91ab2d7c<br>
> > d011db47%7C1%7C0%7C637231080235723408&sdata=256CyAUusLIf8IetQfyd<br>
> > 3KNAlIqVwV8uvjuPc6daP14%3D&reserved=0<br>
> > Unsurprisingly, I strongly agree. ;)<br>
> ><br>
> I don't see why claiming that pattern-init is bad helps make the case <br>
> for zero-init.<br>
<br>
Perhaps I did not express it well enough, but both have meaningful and important uses. My goal here is to illustrate how zero-init is being used (or preferred) in many real situations, as an argument for why it should not be hidden behind what some might see as a scary enable flag.<br>
<br>
> > Apple<br>
> ><br>
> > I can't speak meaningfully here, but I've heard rumors that they are <br>
> > depending on zero-init as well. Perhaps someone there can clarify <br>
> > how they are using these features?<br>
> ><br>
> There's a difference between "depending on zero-init" (as in, the <br>
> group in question is okay with relying on implicit zeroing on code <br>
> reviews, etc.) and the use of zero-init as some sort of <br>
> defence-in-depth approach. Are these rumours clear as to which?<br>
<br>
My understanding was the latter, but I hope to find out for real via this thread! :) It's not clear to me either.<br>
<br>
> > So, while I understand the earlier objections to zero-init from a <br>
> > "language fork" concern, I think this isn't a position that can <br>
> > really stand up to the reality of how many projects are using the <br>
> > feature (even via non-Clang compilers). Given that so much code is <br>
> > going to be built using zero-init, what's the best way for Clang to adapt here?<br>
> <br>
> It happens that there is zero-init and it's at least close enough to <br>
> what these projects want, but it is actually what they need?<br>
<br>
Yes, it's expressly what is desired from a security perspective. (And quite to the relief of that same community, comes with the least performance impact, which is an unfortunately uncommon scenario in security flaw mitigations.) I tried to detail that earlier in my email where it's directly what is indicated as a meaningful defense against the long history of real-world "uninitialized" variable attacks: setting everything to zero is the best defense for the entire class of flaws.<br>
<br>
--<br>
Kees Cook<br>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
</blockquote></div></div>