<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Mar 4, 2015, at 7:23 PM, Sean Silva <<a href="mailto:chisophugis@gmail.com" class="">chisophugis@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><br class="Apple-interchange-newline"><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_quote" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">On Mon, Mar 2, 2015 at 5:11 PM, Richard Smith<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:richard@metafoo.co.uk" target="_blank" class="">richard@metafoo.co.uk</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><div class="h5">On Mon, Mar 2, 2015 at 4:20 PM, Douglas Gregor<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:dgregor@apple.com" target="_blank" class="">dgregor@apple.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><br class=""><div class=""><span class=""><blockquote type="cite" class=""><div class="">On Mar 2, 2015, at 3:34 PM, Richard Smith <<a href="mailto:richard@metafoo.co.uk" target="_blank" class="">richard@metafoo.co.uk</a>> wrote:</div><br class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote">On Mon, Mar 2, 2015 at 1:22 PM, Douglas Gregor<span class=""> </span><span dir="ltr" class=""><<a href="mailto:dgregor@apple.com" target="_blank" class="">dgregor@apple.com</a>></span><span class=""> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class="">Hello all,<div class=""><br class=""></div><div class="">Null pointers are a significant source of problems in applications. Whether it’s SIGSEGV taking down a process or a foolhardy attempt to recover from NullPointerException breaking invariants everywhere, it’s a problem that’s bad enough for Tony Hoare to call the invention of the null reference his billion dollar mistake [1]. It’s not the ability to create a null pointer that is a problem—having a common sentinel value meaning “no value” is extremely useful—but that it’s very hard to determine whether, for a particular pointer, one is expected to be able to use null. C doesn’t distinguish between “nullable” and “nonnull” pointers, so we turn to documentation and experimentation. Consider strchr from the C standard library:</div><div class=""><br class=""></div><div class=""><div class=""><font face="Menlo" class=""><span style="white-space: pre-wrap;" class=""> </span>char *strchr(const char *s, int c);</font></div></div><div class=""><br class=""></div><div class=""><div class="">It is “obvious” to a programmer who knows the semantics of strchr that it’s important to check for a returned null, because null is used as the sentinel for “not found”. Of course, your tools don’t know that, so they cannot help when you completely forget to check for the null case. Bugs ensue.</div><div class=""><br class=""></div><div class="">Can I pass a null string to strchr? The standard is unclear [2], and my platform’s implementation happily accepts a null parameter and returns null, so obviously I shouldn’t worry about it… until I port my code, or the underlying implementation changes because my expectations and the library implementor’s expectations differ. Given the age of strchr, I suspect that every implementation out there has an explicit, defensive check for a null string, because it’s easier to add yet more defensive (and generally useless) null checks than it is to ask your clients to fix their code. Scale this up, and code bloat ensues, as well as wasted programmer effort that obscures the places where checking for null really does matter.</div><div class=""><br class=""></div><div class="">In a recent version of Xcode, Apple introduced an extension to C/C++/Objective-C that expresses the nullability of pointers in the type system via new nullability qualifiers . Nullability qualifiers express nullability as part of the declaration of <font face="Menlo" class="">strchr</font> <span class=""> </span>[2]:</div></div><div class=""><br class=""></div><div class=""><div class=""><font face="Menlo" class=""><span style="white-space: pre-wrap;" class=""> </span>__nullable char *strchr(__nonnull const char *s, int c);</font></div></div><div class=""><font face="Menlo" class=""><br class=""></font></div><div class="">With this, programmers and tools alike can better reason about the use of strchr with null pointers. </div><div class=""><br class=""></div><div class="">We’d like to contribute the implementation (and there is a patch attached at the end [3]), but since this is a nontrivial extension to all of the C family of languages that Clang supports, we believe that it needs to be discussed here first.</div><div class=""><br class=""></div><div class=""><b class="">Goals</b></div><div class="">We have several specific goals that informed the design of this feature. </div><div class=""><br class=""></div><div class=""><ul class=""><li class=""><b class="">Allow the intended nullability to be expressed on all pointers</b>: Pointers are used throughout library interfaces, and the nullability of those pointers is an important part of the API contract with users. It’s too simplistic to only allow function parameters to have nullability, for example, because it’s also important information for data members, pointers-to-pointers (e.g., "a nonnull pointer to a nullable pointer to an integer”), arrays of pointers, etc.</li><li class=""><b class="">Enable better tools support for detecting nullability problems:</b> The nullability annotations should be useful for tools (especially the static analyzer) that can reason about the use of null, to give warnings about both missed null checks (the result of strchr could be null…) as well as for unnecessarily-defensive code.</li><li class=""><b class="">Support workflows where all interfaces provide nullability annotations:</b> In moving from a world where there are no nullability annotations to one where we hope to see many such annotations, we’ve found it helpful to move header-by-header, auditing a complete header to give it nullability qualifiers. Once one has done that, additions to the header need to be held to the same standard, so we need a design that allows us to warn about pointers that don’t provide nullability annotations for some declarations in a header that already has some nullability annotations.</li></ul></div><div class=""><div class=""><div class=""><ul class=""><li class=""><b class="">Zero effect on ABI or code generation:</b> There are a huge number of interfaces that could benefit from the use of nullability qualifiers, but we won’t get widespread adoption if introducing the nullability qualifiers means breaking existing code, either in the ABI (say, because nullability qualifiers are mangled into the type) or at execution time (e.g., because a non-null pointer ends up being null along some error path and causes undefined behavior).</li></ul></div></div></div></div></blockquote><div class="">A sanitizer for this feature would seem very useful, but this bullet point suggests that such a sanitizer would violate the model. Likewise, I don't see why we should rule out the option of optimizing on the basis of these qualifiers (under a -fstrict-nonnull flag or similar).</div></div></div></div></div></blockquote><div class=""><br class=""></div></span><div class="">I agree that a sanitizer would be useful as a debugging aid. My primary concern here is that optimizing based on this information<span class="Apple-converted-space"> </span><i class="">not</i> be a part of any normal optimization flag (-O2, -Os, whatever), because it will hamper widespread adoption of this feature if adding the annotations to indicate the API contract suddenly starts breaking existing clients by, e.g., optimizing out existing, defensive null checks.</div><span class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><b class="">Why not __attribute__((nonnull))?</b></div><div class="">Clang already has an attribute to express nullability, “nonnull”, which we inherited from GCC [4]. The “nonnull” attribute can be placed on functions to indicate which parameters cannot be null: one either specifies the indices of the arguments that cannot be null, e.g.,</div><div class=""><pre class=""><span style="white-space: pre-wrap;" class=""> </span>extern void *my_memcpy (void *dest, const void *src, size_t len) __attribute__((nonnull (1, 2)));</pre><div class="">or omits the list of indices to state that all pointer arguments cannot be null, e.g.,</div></div><div class=""><pre class=""><span class=""> </span>extern void *my_memcpy (void *dest, const void *src, size_t len) __attribute__((nonnull));</pre></div><div class="">More recently, “nonnull” has grown the ability to be applied to parameters, and one can use the companion attribute returns_nonnull to state that a function returns a non-null pointer:</div><div class=""><pre class=""><span class=""> </span>extern void *my_memcpy (__attribute__((nonnull)) void *dest, __attribute__((nonnull)) const void *src, size_t len) __attribute__((returns_nonnull));</pre></div><div class="">There are a number of problems here. First, there are different attributes to express the same idea at different places in the grammar, and the use of the “nonnull” attribute<span class=""> </span><i class="">on the function</i> actually has an effect<span class=""> </span><i class="">on the function parameters</i> can get very, very confusing. Quick, which pointers are nullable vs. non-null in this example?</div><div class=""><br class=""></div><div class=""><font face="Menlo" class=""><span style="white-space: pre-wrap;" class=""> </span>__attribute__((nonnull)) void *my_realloc (void *ptr, size_t size);</font></div><div class=""><br class=""></div><div class="">According to that declaration, ptr is nonnull and the function returns a nullable pointer… but that’s the opposite of how it reads (and behaves, if this is anything like a realloc that cannot fail). Moreover, because these two attributes are<span class=""> </span><i class="">declaration</i> attributes, not type attributes, you cannot express that nullability of the inner pointer in a multi-level pointer or an array of pointers, which makes these attributes verbose, confusing, and not sufficiently generally. These attributes fail the first of our goals.</div><div class=""><br class=""></div><div class="">These attributes aren’t as useful as they could be for tools support (the second and third goals), because they only express the nonnull case, leaving no way to distinguish between the unannotated case (nobody has documented the nullability of some parameter) and the nullable case (we know the pointer can be null). From a tooling perspective, this is a killer: the static analyzer absolutely cannot warn that one has forgotten to check for null for every unannotated pointer, because the false-positive rate would be astronomical.</div><div class=""><br class=""></div><div class="">Finally, we’ve recently started considering violations of the __attribute__((nonnull)) contract to be undefined behavior, which fails the last of our goals. This is something we could debate further if it were the only problem, but these declaration attributes fall all of our criteria, so it’s not worth discussing.</div><div class=""><br class=""></div><div class=""><b class="">Nullability Qualifiers</b></div><div class="">We propose the addition of a new set of type qualifiers, spelled <font face="Menlo" class="">__nullable</font>, <font face="Menlo" class="">__nonnull</font>, and <font face="Menlo" class="">__null_unspecified</font>, to Clang. These are collectively known as<span class=""> </span><i class="">nullability qualifiers</i> and may be written anywhere any other type qualifier may be written (such as<span class=""> </span><font face="Menlo" class="">const</font>) on any type subject to the following restrictions:</div><div class=""><br class=""></div><div class=""><ul class=""><li class="">Two nullability qualifiers shall not appear in the same set of qualifiers.</li><li class="">A nullability qualifier shall qualify any pointer type, including pointers to objects, pointers to functions, C++ pointers to members, block pointers, and Objective-C object pointers.</li><li class="">A nullability qualifier in the declaration-specifiers applies to the innermost pointer type of each declarator (e.g.,<span class=""> </span><font face="Menlo" class="">__nonnull int *</font><span class=""> </span>is equivalent to<span class=""> </span><font face="Menlo" class="">int * __nonnull</font>).</li></ul></div></div></blockquote><div class="">What happens if there's a mixture of different kinds of declarator? (Can I have '__nonnull int (*p)[3]'? Can I have '__nonnull int *p[3];'?)</div><div class=""><br class=""></div><div class="">I think you're saying that this decision is made based on the syntax of the declarator and not based on the underlying type, right? (So in</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>__nonnull T *</div><div class=""><br class=""></div><div class="">the __nonnull appertains to the *, even if T names a pointer type.)<span class="Apple-converted-space"> </span></div></div></div></div></div></blockquote><div class=""><br class=""></div></span><div class="">I’ve said it poorly. It is based on the type, so</div><div class=""><br class=""></div><div class=""><span style="white-space: pre-wrap;" class=""> </span>__nonnull T *</div><div class=""><br class=""></div><div class="">applies __nonnull to T unless T is a known type that is not a pointer type. In the patch Type::canHaveNullability() computes the operation, and essentially we apply the __nonnull to T when:</div><div class=""><span style="white-space: pre-wrap;" class=""> </span>- T is of pointer, C++ member pointer, block pointer, or Objective-C object pointer</div><div class=""><span style="white-space: pre-wrap;" class=""> </span>- T is a dependent type that could instantiate to some kind of pointer type</div></div></div></blockquote><div class=""><br class=""></div></div></div><div class="">That seems like it could be very confusing:</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>void f(__nonnull int *p);</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>template<typename Integral></div><div class=""> <span class="Apple-converted-space"> </span>void f(__nonnull Integral *p);</div><div class=""><br class=""></div><div class="">... would apply the __nonnull to different components of the type (and I don't even want to think about what happens when T is a member of the current instantiation). The outcome seems to be that people writing templates need to know about both ways of writing this, and they need to know the gotchas and the minutiae of the rules, and they need to be able to reason with precision about which types are dependent.</div><div class=""><br class=""></div><div class="">This is a problem in C too. Consider:</div><div class=""><br class=""></div><div class=""> <span class="Apple-converted-space"> </span>#include <some_library.h> // vends an opaque_t typedef</div><div class=""> <span class="Apple-converted-space"> </span>void use_library(__nonnull opaque_t *handle);</div><div class=""><br class=""></div><div class="">We cannot know what this program means without depending on an implementation detail of some_library.h. And this is not an exotic problem; consider, for instance, some_library == stdio and opaque_t == FILE.</div></div></div></div></blockquote><div class=""><br class=""><br class=""></div><div class="">Random idea: could we bootstrap on programmer's knowledge of of const and volatile and have the rule be "__nonnull <something>" applies to the pointer that points to the const thing in "const <something>"? Since cv are eligible for template specialization and nullability-qualifiers aren't, maybe that just trades-off for another inconsistency?<br class=""></div></div></div></blockquote></div><br class=""><div class="">Personally, I feel like we shouldn’t conflate constness with nullability, because they’re really orthogonal notions. Plus, Richard has convinced me that we shouldn’t move the nullability qualifiers away from the decl-specifiers.</div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>- Doug</div><div class=""><br class=""></div></body></html>