[cfe-dev] RFC: Nullability qualifiers

[Richard Smith]

On Mon, Mar 2, 2015 at 1:22 PM, Douglas Gregor <dgregor at apple.com> wrote:

> Hello all,
>
> Null pointers are a significant source of problems in applications.
> Whether it’s SIGSEGV taking down a process or a foolhardy attempt to
> recover from NullPointerException breaking invariants everywhere, it’s a
> problem that’s bad enough for Tony Hoare to call the invention of the null
> reference his billion dollar mistake [1]. It’s not the ability to create a
> null pointer that is a problem—having a common sentinel value meaning “no
> value” is extremely useful—but that it’s very hard to determine whether,
> for a particular pointer, one is expected to be able to use null. C doesn’t
> distinguish between “nullable” and “nonnull” pointers, so we turn to
> documentation and experimentation. Consider strchr from the C standard
> library:
>
> char *strchr(const char *s, int c);
>
> It is “obvious” to a programmer who knows the semantics of strchr that
> it’s important to check for a returned null, because null is used as the
> sentinel for “not found”. Of course, your tools don’t know that, so they
> cannot help when you completely forget to check for the null case. Bugs
> ensue.
>
> Can I pass a null string to strchr? The standard is unclear [2], and my
> platform’s implementation happily accepts a null parameter and returns
> null, so obviously I shouldn’t worry about it… until I port my code, or the
> underlying implementation changes because my expectations and the library
> implementor’s expectations differ. Given the age of strchr, I suspect that
> every implementation out there has an explicit, defensive check for a null
> string, because it’s easier to add yet more defensive (and generally
> useless) null checks than it is to ask your clients to fix their code.
> Scale this up, and code bloat ensues, as well as wasted programmer effort
> that obscures the places where checking for null really does matter.
>
> In a recent version of Xcode, Apple introduced an extension to
> C/C++/Objective-C that expresses the nullability of pointers in the type
> system via new nullability qualifiers . Nullability qualifiers express
> nullability as part of the declaration of strchr  [2]:
>
> __nullable char *strchr(__nonnull const char *s, int c);
>
> With this, programmers and tools alike can better reason about the use of
> strchr with null pointers.
>
> We’d like to contribute the implementation (and there is a patch attached
> at the end [3]), but since this is a nontrivial extension to all of the C
> family of languages that Clang supports, we believe that it needs to be
> discussed here first.
>
> *Goals*
> We have several specific goals that informed the design of this feature.
>
>
>    - *Allow the intended nullability to be expressed on all pointers*:
>    Pointers are used throughout library interfaces, and the nullability of
>    those pointers is an important part of the API contract with users. It’s
>    too simplistic to only allow function parameters to have nullability, for
>    example, because it’s also important information for data members,
>    pointers-to-pointers (e.g., "a nonnull pointer to a nullable pointer to an
>    integer”), arrays of pointers, etc.
>    - *Enable better tools support for detecting nullability problems:* The
>    nullability annotations should be useful for tools (especially the static
>    analyzer) that can reason about the use of null, to give warnings about
>    both missed null checks (the result of strchr could be null…) as well as
>    for unnecessarily-defensive code.
>    - *Support workflows where all interfaces provide nullability
>    annotations:* In moving from a world where there are no nullability
>    annotations to one where we hope to see many such annotations, we’ve found
>    it helpful to move header-by-header, auditing a complete header to give it
>    nullability qualifiers. Once one has done that, additions to the header
>    need to be held to the same standard, so we need a design that allows us to
>    warn about pointers that don’t provide nullability annotations for some
>    declarations in a header that already has some nullability annotations.
>
>
>    - *Zero effect on ABI or code generation:* There are a huge number of
>    interfaces that could benefit from the use of nullability qualifiers, but
>    we won’t get widespread adoption if introducing the nullability qualifiers
>    means breaking existing code, either in the ABI (say, because nullability
>    qualifiers are mangled into the type) or at execution time (e.g., because a
>    non-null pointer ends up being null along some error path and causes
>    undefined behavior).
>
> A sanitizer for this feature would seem very useful, but this bullet point
suggests that such a sanitizer would violate the model. Likewise, I don't
see why we should rule out the option of optimizing on the basis of these
qualifiers (under a -fstrict-nonnull flag or similar).


> *Why not __attribute__((nonnull))?*
> Clang already has an attribute to express nullability, “nonnull”, which we
> inherited from GCC [4]. The “nonnull” attribute can be placed on functions
> to indicate which parameters cannot be null: one either specifies the
> indices of the arguments that cannot be null, e.g.,
>
> 	extern void *my_memcpy (void *dest, const void *src, size_t len) __attribute__((nonnull (1, 2)));
>
> or omits the list of indices to state that all pointer arguments cannot be
> null, e.g.,
>
> 	extern void *my_memcpy (void *dest, const void *src, size_t len) __attribute__((nonnull));
>
> More recently, “nonnull”  has grown the ability to be applied to
> parameters, and one can use the companion attribute returns_nonnull to
> state that a function returns a non-null pointer:
>
> 	extern void *my_memcpy (__attribute__((nonnull)) void *dest, __attribute__((nonnull)) const void *src, size_t len) __attribute__((returns_nonnull));
>
> There are a number of problems here. First, there are different attributes
> to express the same idea at different places in the grammar, and the use of
> the “nonnull” attribute *on the function* actually has an effect *on the
> function parameters* can get very, very confusing. Quick, which pointers
> are nullable vs. non-null in this example?
>
> __attribute__((nonnull)) void *my_realloc (void *ptr, size_t size);
>
> According to that declaration, ptr is nonnull and the function returns a
> nullable pointer… but that’s the opposite of how it reads (and behaves, if
> this is anything like a realloc that cannot fail). Moreover, because these
> two attributes are *declaration* attributes, not type attributes, you
> cannot express that nullability of the inner pointer in a multi-level
> pointer or an array of pointers, which makes these attributes verbose,
> confusing, and not sufficiently generally. These attributes fail the first
> of our goals.
>
> These attributes aren’t as useful as they could be for tools support (the
> second and third goals), because they only express the nonnull case,
> leaving no way to distinguish between the unannotated case (nobody has
> documented the nullability of some parameter) and the nullable case (we
> know the pointer can be null). From a tooling perspective, this is a
> killer: the static analyzer absolutely cannot warn that one has forgotten
> to check for null for every unannotated pointer, because the false-positive
> rate would be astronomical.
>
> Finally, we’ve recently started considering violations of the
> __attribute__((nonnull)) contract to be undefined behavior, which fails the
> last of our goals. This is something we could debate further if it were the
> only problem, but these declaration attributes fall all of our criteria, so
> it’s not worth discussing.
>
> *Nullability Qualifiers*
> We propose the addition of a new set of type qualifiers,  spelled
> __nullable, __nonnull, and __null_unspecified, to Clang. These are
> collectively known as *nullability qualifiers* and may be written
> anywhere any other type qualifier may be written (such as const) on any
> type subject to the following restrictions:
>
>
>    - Two nullability qualifiers shall not appear in the same set of
>    qualifiers.
>    - A nullability qualifier shall qualify any pointer type, including
>    pointers to objects, pointers to functions, C++ pointers to members, block
>    pointers, and Objective-C object pointers.
>    - A nullability qualifier in the declaration-specifiers applies to the
>    innermost pointer type of each declarator (e.g., __nonnull int * is
>    equivalent to int * __nonnull).
>
> What happens if there's a mixture of different kinds of declarator? (Can I
have '__nonnull int (*p)[3]'? Can I have '__nonnull int *p[3];'?)

I think you're saying that this decision is made based on the syntax of the
declarator and not based on the underlying type, right? (So in

  __nonnull T *

the __nonnull appertains to the *, even if T names a pointer type.) Given
that...

>
>    - A nullability qualifier applied to a typedef of a
>    nullability-qualified pointer type shall specify the same nullability as
>    the underlying type of the typedef.
>
> ... I don't really see what this rule is for. I would expect "__nonnull T"
to be ill-formed because the innermost component of the declarator is not a
pointer, irrespective of whether T is a pointer type and whether it's
nullable. And I'd expect "__nonnull T *" to be valid whether or not T is a
typedef for a __nonnull pointer.

On the whole, I find it a little strange to allow a nullability qualifier
in the decl-specifier-seq / specifiers-and-qualifiers that applies to some
later pointer declarator; I would have expected this to be permitted in the
cv-qualifier-seq / type-qualifier-list after the pointer operator, and
nowhere else (or perhaps permitted in a decl-specifier-seq that also
contains a type-specifier for a pointer type). This kind of flexibility has
proven a disaster for the comprehensibility of GCC's type attributes.

The meanings of the three nullability qualifiers are as follows:
>
> __nullable: the pointer may store a null value at runtime (as part of the
> API contract)
> __nonnull: the pointer should not store a null value at runtime (as part
> of the API contract). it is possible that the value can be null, e.g., in
> erroneous historic uses of an API, and it is up to the library implementor
> to decide to what degree she will accommodate such clients.
> __null_unspecified: it is unclear whether the pointer can be null or not.
> Use of this type qualifier is extremely rare in practice, but it fills a
> small but important niche when auditing a particular header to add
> nullability qualifiers: sometimes the nullability contract for a few APIs
> in the header is unclear *even when looking at the implementation* for
> historical reasons, and establishing the contract requires more extensive
> study. In such cases, it’s often best to mark that pointer as
> __null_unspecified (which will help silence the warning about unannotated
> pointers in a header) and move on, coming back to __null_unspecified
> pointers when the appropriate graybeard has been summoned out of retirement
> [5].
>

Have you considered adding C++11 attributes as synonyms for these?

*Assumes-nonnull Regions*
> We’ve found that it's fairly common for the majority of pointers within a
> particular header to be __nonnull. Therefore, we’ve introduced
> assumes-nonnull regions that assume that certain unannotated pointers
> implicitly get the __nonnull nullability qualifiers. Assumes-nonnull
> regions are marked by pragmas:
>
> #pragma clang assume_nonnull begin
>         __nullable char *strchr(const char *s, int c); // s is inferred to
> be __nonnull
> void *my_realloc (__nullable void *ptr, size_t size); // my_realloc is
> inferred to return __nonnull
> #pragma clang assume_nonnull end
>

These pragmas seem easy to miss when moving declarations around while
refactoring. Do you have enough experience with the feature to know if
that's an issue in practice?

We infer __nonnull within an assumes_nonnull region when:
>
>    - The pointer is a non-typedef declaration, such as a function
>    parameter, variable, or data member, or the result type of a function. It’s
>    very rare for one to warn typedefs to specify nullability information;
>    rather, it’s usually the user of the typedef that needs to specify
>    nullability.
>
> How can they do this, given the earlier rules?

>
>    - The pointer is a single-level pointer, e.g., int* but not int**,
>    because we’ve found that programmers can get confused about the nullability
>    of multi-level pointers (is it a __nullable pointer to __nonnull pointers,
>    or the other way around?) and inferring nullability for any of the pointers
>    in a multi-level pointer compounds the situation.
>
>
> Note that no #include may occur within an assumes_nonnull region, and
> assumes_nonnull regions cannot cross header boundaries.
>

That sounds like it would make the lives of library maintainers using this
feature painful -- they would need to textually duplicate these pragmas and
the surrounding #ifdefs in every system header that needs them, rather than
factoring them out into #includable begin/end files. But I suppose we can
encourage the use of a macro expanding to _Pragma for those cases.


> *Type System Impact*
> Nullability qualifiers are mapped to type attributes within the Clang type
> system, but a nullability-qualified pointer type is not semantically
> distinct from its unqualified pointer type. Therefore, one may freely
> convert between nullability-qualified and non-nullability-qualified
> pointers, or between nullability-qualified pointers with different
> nullability qualifiers. One cannot overload on nullability qualifiers,
> write C++ class template partial specializations that identify nullability
> qualifiers, or inspect nullability via type traits in any way.
>
> Said more strongly, removing nullability qualifiers from a well-formed
> program will not change its behavior in any way, nor will the semantics of
> a program change when any set of (well-formed) nullability qualifiers are
> added to it. Operationally, this means that *nullability qualifiers are
> not part of the canonical type* in Clang’s type system, and that any
> warnings we produce based on nullability information will necessarily be
> dependent on Clang’s ability to retain type sugar during semantic analysis.
>
> While it’s somewhat exceptional for us to introduce new type qualifiers
> that don’t produce semantically distinct types, we feel that this is the
> only plausible design and implementation strategy for this feature: pushing
> nullability qualifiers into the type system semantically would cause
> significant changes to the language (e.g., overloading, partial
> specialization) and break ABI (due to name mangling) that would drastically
> reduce the number of potential users, and we feel that Clang’s support for
> maintaining type sugar throughout semantic analysis is generally good
> enough [6] to get the benefits of nullability annotations in our tools.
>

This seems reasonable to me, given the constraints. (I've had some offline
discussions with various people about template type sugar reconstruction,
which would help to diagnose issues here.)

Looking forward to our discussion.
>
> - Doug (with Jordan Rose and Anna Zaks)
>
> [1] http://en.wikipedia.org/wiki/Tony_Hoare#Apologies_and_retractions
> [2] The standard description of strchr seems to imply that the parameter
> cannot be null
> [3] The patch is complete, but should be reviewed on cfe-commits rather
> than here. There are also several logic parts to this monolithic patch:
> (a) __nonnull/__nullable/__null_unspecified type specifiers
> (b) nonnull/nullable/null_unspecified syntactic sugar for Objective-C
> (c) Warning about inconsistent application of nullability specifiers
> within a given header
> (d) assume_nonnnull begin/end pragmas
> (e) Objective-C null_resettable property attribute
> [4] https://gcc.gnu.org/onlinedocs/gcc/Function-Attributes.html (search
> for “nonnull”)
> [5] No graybeards were harmed in the making of this feature.
> [6] Template instantiation is the notable exception here, because it
> always canonicalizes types.
>
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20150302/a66902f2/attachment.html>