[cfe-dev] RFC: Nullability qualifiers

[b17 c0de]

Apple please implement  __has_feature(nullability) in clang for Xcode 7
release. :-)

On Sat, Jun 27, 2015 at 12:31 AM, Aaron Ballman <aaron at aaronballman.com>
wrote:

> On Fri, Jun 26, 2015 at 6:29 PM, b17 c0de <b17c0de at gmail.com> wrote:
> >  It also appears that the current versions of Apple clang (even the
> newest
> > beta) don't even support __has_feature(nullability). I take it this has
> been
> > fixed in trunk?
>
> Correct, trunk is likely also the only place that has _Nonnull and
> friends, too. If you're developing on OS X and don't need cross
> compiler support for your code base, I would stick with __nonnull
> there and you'll be fine. If you need cross compiler support, you'll
> likely have to piece it together with macros.
>
> ~Aaron
>
> >
> > On Fri, Jun 26, 2015 at 11:53 PM, Aaron Ballman <aaron at aaronballman.com>
> > wrote:
> >>
> >> On Fri, Jun 26, 2015 at 5:44 PM, b17 c0de <b17c0de at gmail.com> wrote:
> >> > OK. What would be the best way to detect if Apple clang supports
> >> > _Nonnull or
> >> > only __nonnull though.
> >>
> >> I cannot speak for how Apple's Clang works in this regard, but perhaps
> >> Doug can.
> >>
> >> ~Aaron
> >>
> >> >
> >> > On Fri, Jun 26, 2015 at 11:40 PM, Aaron Ballman <
> aaron at aaronballman.com>
> >> > wrote:
> >> >>
> >> >> On Fri, Jun 26, 2015 at 5:36 PM, b17 c0de <b17c0de at gmail.com> wrote:
> >> >> > How can one detect if an Apple clang supports the new nullability
> >> >> > attributes. I tried something like:
> >> >> >
> >> >> > #if __has_attribute(_Nonnull)
> >> >> > #elif __has_attribute(__nonnull)
> >> >> > #define _Nonnull __nonnull
> >> >> > #else
> >> >> > #define _Nonnull
> >> >> > #endif
> >> >> >
> >> >> > But this didn't work. Why doesn't _Nonnull/__nonnull work with
> >> >> > __has_attribute?
> >> >>
> >> >> __has_attribute is used to test for GNU-style attribute support only.
> >> >> To test for nullability, you should use: __has_feature(nullability)
> >> >>
> >> >> ~Aaron
> >> >>
> >> >> >
> >> >> > On Wed, Jun 24, 2015 at 10:39 PM, Douglas Gregor <
> dgregor at apple.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> Another addendum: due to the conflict with glibc’s __nonnull,
> we’ll
> >> >> >> be
> >> >> >> renaming the __double_underscored keywords to _Big_underscored
> >> >> >> keywords,
> >> >> >> e.g.,
> >> >> >>
> >> >> >> __nonnull -> _Nonnull
> >> >> >> __nullable -> _Nullable
> >> >> >> __null_unspecified -> _Null_unspecified
> >> >> >>
> >> >> >> On Darwin, we’ll add predefines
> >> >> >>
> >> >> >> #define __nonnull _Nonnull
> >> >> >> #define __nullable _Nullable
> >> >> >> #define __null_unspecified _Null_unspecified
> >> >> >>
> >> >> >> to keep the existing headers working.
> >> >> >>
> >> >> >> - Doug
> >> >> >>
> >> >> >> On Mar 2, 2015, at 1:22 PM, Douglas Gregor <dgregor at apple.com>
> >> >> >> wrote:
> >> >> >>
> >> >> >> Hello all,
> >> >> >>
> >> >> >> Null pointers are a significant source of problems in
> applications.
> >> >> >> Whether it’s SIGSEGV taking down a process or a foolhardy attempt
> to
> >> >> >> recover
> >> >> >> from NullPointerException breaking invariants everywhere, it’s a
> >> >> >> problem
> >> >> >> that’s bad enough for Tony Hoare to call the invention of the null
> >> >> >> reference
> >> >> >> his billion dollar mistake [1]. It’s not the ability to create a
> >> >> >> null
> >> >> >> pointer that is a problem—having a common sentinel value meaning
> “no
> >> >> >> value”
> >> >> >> is extremely useful—but that it’s very hard to determine whether,
> >> >> >> for a
> >> >> >> particular pointer, one is expected to be able to use null. C
> >> >> >> doesn’t
> >> >> >> distinguish between “nullable” and “nonnull” pointers, so we turn
> to
> >> >> >> documentation and experimentation. Consider strchr from the C
> >> >> >> standard
> >> >> >> library:
> >> >> >>
> >> >> >> char *strchr(const char *s, int c);
> >> >> >>
> >> >> >> It is “obvious” to a programmer who knows the semantics of strchr
> >> >> >> that
> >> >> >> it’s important to check for a returned null, because null is used
> as
> >> >> >> the
> >> >> >> sentinel for “not found”. Of course, your tools don’t know that,
> so
> >> >> >> they
> >> >> >> cannot help when you completely forget to check for the null case.
> >> >> >> Bugs
> >> >> >> ensue.
> >> >> >>
> >> >> >> Can I pass a null string to strchr? The standard is unclear [2],
> and
> >> >> >> my
> >> >> >> platform’s implementation happily accepts a null parameter and
> >> >> >> returns
> >> >> >> null,
> >> >> >> so obviously I shouldn’t worry about it… until I port my code, or
> >> >> >> the
> >> >> >> underlying implementation changes because my expectations and the
> >> >> >> library
> >> >> >> implementor’s expectations differ. Given the age of strchr, I
> >> >> >> suspect
> >> >> >> that
> >> >> >> every implementation out there has an explicit, defensive check
> for
> >> >> >> a
> >> >> >> null
> >> >> >> string, because it’s easier to add yet more defensive (and
> generally
> >> >> >> useless) null checks than it is to ask your clients to fix their
> >> >> >> code.
> >> >> >> Scale
> >> >> >> this up, and code bloat ensues, as well as wasted programmer
> effort
> >> >> >> that
> >> >> >> obscures the places where checking for null really does matter.
> >> >> >>
> >> >> >> In a recent version of Xcode, Apple introduced an extension to
> >> >> >> C/C++/Objective-C that expresses the nullability of pointers in
> the
> >> >> >> type
> >> >> >> system via new nullability qualifiers . Nullability qualifiers
> >> >> >> express
> >> >> >> nullability as part of the declaration of strchr  [2]:
> >> >> >>
> >> >> >> __nullable char *strchr(__nonnull const char *s, int c);
> >> >> >>
> >> >> >> With this, programmers and tools alike can better reason about the
> >> >> >> use
> >> >> >> of
> >> >> >> strchr with null pointers.
> >> >> >>
> >> >> >> We’d like to contribute the implementation (and there is a patch
> >> >> >> attached
> >> >> >> at the end [3]), but since this is a nontrivial extension to all
> of
> >> >> >> the
> >> >> >> C
> >> >> >> family of languages that Clang supports, we believe that it needs
> to
> >> >> >> be
> >> >> >> discussed here first.
> >> >> >>
> >> >> >> Goals
> >> >> >> We have several specific goals that informed the design of this
> >> >> >> feature.
> >> >> >>
> >> >> >> Allow the intended nullability to be expressed on all pointers:
> >> >> >> Pointers
> >> >> >> are used throughout library interfaces, and the nullability of
> those
> >> >> >> pointers is an important part of the API contract with users. It’s
> >> >> >> too
> >> >> >> simplistic to only allow function parameters to have nullability,
> >> >> >> for
> >> >> >> example, because it’s also important information for data members,
> >> >> >> pointers-to-pointers (e.g., "a nonnull pointer to a nullable
> pointer
> >> >> >> to
> >> >> >> an
> >> >> >> integer”), arrays of pointers, etc.
> >> >> >> Enable better tools support for detecting nullability problems:
> The
> >> >> >> nullability annotations should be useful for tools (especially the
> >> >> >> static
> >> >> >> analyzer) that can reason about the use of null, to give warnings
> >> >> >> about
> >> >> >> both
> >> >> >> missed null checks (the result of strchr could be null…) as well
> as
> >> >> >> for
> >> >> >> unnecessarily-defensive code.
> >> >> >> Support workflows where all interfaces provide nullability
> >> >> >> annotations:
> >> >> >> In
> >> >> >> moving from a world where there are no nullability annotations to
> >> >> >> one
> >> >> >> where
> >> >> >> we hope to see many such annotations, we’ve found it helpful to
> move
> >> >> >> header-by-header, auditing a complete header to give it
> nullability
> >> >> >> qualifiers. Once one has done that, additions to the header need
> to
> >> >> >> be
> >> >> >> held
> >> >> >> to the same standard, so we need a design that allows us to warn
> >> >> >> about
> >> >> >> pointers that don’t provide nullability annotations for some
> >> >> >> declarations in
> >> >> >> a header that already has some nullability annotations.
> >> >> >>
> >> >> >> Zero effect on ABI or code generation: There are a huge number of
> >> >> >> interfaces that could benefit from the use of nullability
> >> >> >> qualifiers,
> >> >> >> but we
> >> >> >> won’t get widespread adoption if introducing the nullability
> >> >> >> qualifiers
> >> >> >> means breaking existing code, either in the ABI (say, because
> >> >> >> nullability
> >> >> >> qualifiers are mangled into the type) or at execution time (e.g.,
> >> >> >> because a
> >> >> >> non-null pointer ends up being null along some error path and
> causes
> >> >> >> undefined behavior).
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> Why not __attribute__((nonnull))?
> >> >> >> Clang already has an attribute to express nullability, “nonnull”,
> >> >> >> which
> >> >> >> we
> >> >> >> inherited from GCC [4]. The “nonnull” attribute can be placed on
> >> >> >> functions
> >> >> >> to indicate which parameters cannot be null: one either specifies
> >> >> >> the
> >> >> >> indices of the arguments that cannot be null, e.g.,
> >> >> >>
> >> >> >> extern void *my_memcpy (void *dest, const void *src, size_t len)
> >> >> >> __attribute__((nonnull (1, 2)));
> >> >> >>
> >> >> >> or omits the list of indices to state that all pointer arguments
> >> >> >> cannot
> >> >> >> be
> >> >> >> null, e.g.,
> >> >> >>
> >> >> >> extern void *my_memcpy (void *dest, const void *src, size_t len)
> >> >> >> __attribute__((nonnull));
> >> >> >>
> >> >> >> More recently, “nonnull”  has grown the ability to be applied to
> >> >> >> parameters, and one can use the companion attribute
> returns_nonnull
> >> >> >> to
> >> >> >> state
> >> >> >> that a function returns a non-null pointer:
> >> >> >>
> >> >> >> extern void *my_memcpy (__attribute__((nonnull)) void *dest,
> >> >> >> __attribute__((nonnull)) const void *src, size_t len)
> >> >> >> __attribute__((returns_nonnull));
> >> >> >>
> >> >> >> There are a number of problems here. First, there are different
> >> >> >> attributes
> >> >> >> to express the same idea at different places in the grammar, and
> the
> >> >> >> use of
> >> >> >> the “nonnull” attribute on the function actually has an effect on
> >> >> >> the
> >> >> >> function parameters can get very, very confusing. Quick, which
> >> >> >> pointers
> >> >> >> are
> >> >> >> nullable vs. non-null in this example?
> >> >> >>
> >> >> >> __attribute__((nonnull)) void *my_realloc (void *ptr, size_t
> size);
> >> >> >>
> >> >> >> According to that declaration, ptr is nonnull and the function
> >> >> >> returns
> >> >> >> a
> >> >> >> nullable pointer… but that’s the opposite of how it reads (and
> >> >> >> behaves,
> >> >> >> if
> >> >> >> this is anything like a realloc that cannot fail). Moreover,
> because
> >> >> >> these
> >> >> >> two attributes are declaration attributes, not type attributes,
> you
> >> >> >> cannot
> >> >> >> express that nullability of the inner pointer in a multi-level
> >> >> >> pointer
> >> >> >> or an
> >> >> >> array of pointers, which makes these attributes verbose,
> confusing,
> >> >> >> and
> >> >> >> not
> >> >> >> sufficiently generally. These attributes fail the first of our
> >> >> >> goals.
> >> >> >>
> >> >> >> These attributes aren’t as useful as they could be for tools
> support
> >> >> >> (the
> >> >> >> second and third goals), because they only express the nonnull
> case,
> >> >> >> leaving
> >> >> >> no way to distinguish between the unannotated case (nobody has
> >> >> >> documented
> >> >> >> the nullability of some parameter) and the nullable case (we know
> >> >> >> the
> >> >> >> pointer can be null). From a tooling perspective, this is a
> killer:
> >> >> >> the
> >> >> >> static analyzer absolutely cannot warn that one has forgotten to
> >> >> >> check
> >> >> >> for
> >> >> >> null for every unannotated pointer, because the false-positive
> rate
> >> >> >> would be
> >> >> >> astronomical.
> >> >> >>
> >> >> >> Finally, we’ve recently started considering violations of the
> >> >> >> __attribute__((nonnull)) contract to be undefined behavior, which
> >> >> >> fails
> >> >> >> the
> >> >> >> last of our goals. This is something we could debate further if it
> >> >> >> were
> >> >> >> the
> >> >> >> only problem, but these declaration attributes fall all of our
> >> >> >> criteria, so
> >> >> >> it’s not worth discussing.
> >> >> >>
> >> >> >> Nullability Qualifiers
> >> >> >> We propose the addition of a new set of type qualifiers,  spelled
> >> >> >> __nullable, __nonnull, and __null_unspecified, to Clang. These are
> >> >> >> collectively known as nullability qualifiers and may be written
> >> >> >> anywhere any
> >> >> >> other type qualifier may be written (such as const) on any type
> >> >> >> subject
> >> >> >> to
> >> >> >> the following restrictions:
> >> >> >>
> >> >> >> Two nullability qualifiers shall not appear in the same set of
> >> >> >> qualifiers.
> >> >> >> A nullability qualifier shall qualify any pointer type, including
> >> >> >> pointers
> >> >> >> to objects, pointers to functions, C++ pointers to members, block
> >> >> >> pointers,
> >> >> >> and Objective-C object pointers.
> >> >> >> A nullability qualifier in the declaration-specifiers applies to
> the
> >> >> >> innermost pointer type of each declarator (e.g., __nonnull int *
> is
> >> >> >> equivalent to int * __nonnull).
> >> >> >> A nullability qualifier applied to a typedef of a
> >> >> >> nullability-qualified
> >> >> >> pointer type shall specify the same nullability as the underlying
> >> >> >> type
> >> >> >> of
> >> >> >> the typedef.
> >> >> >>
> >> >> >>
> >> >> >> The meanings of the three nullability qualifiers are as follows:
> >> >> >>
> >> >> >> __nullable: the pointer may store a null value at runtime (as part
> >> >> >> of
> >> >> >> the
> >> >> >> API contract)
> >> >> >> __nonnull: the pointer should not store a null value at runtime
> (as
> >> >> >> part
> >> >> >> of the API contract). it is possible that the value can be null,
> >> >> >> e.g.,
> >> >> >> in
> >> >> >> erroneous historic uses of an API, and it is up to the library
> >> >> >> implementor
> >> >> >> to decide to what degree she will accommodate such clients.
> >> >> >> __null_unspecified: it is unclear whether the pointer can be null
> or
> >> >> >> not.
> >> >> >> Use of this type qualifier is extremely rare in practice, but it
> >> >> >> fills
> >> >> >> a
> >> >> >> small but important niche when auditing a particular header to add
> >> >> >> nullability qualifiers: sometimes the nullability contract for a
> few
> >> >> >> APIs in
> >> >> >> the header is unclear even when looking at the implementation for
> >> >> >> historical
> >> >> >> reasons, and establishing the contract requires more extensive
> >> >> >> study.
> >> >> >> In
> >> >> >> such cases, it’s often best to mark that pointer as
> >> >> >> __null_unspecified
> >> >> >> (which will help silence the warning about unannotated pointers
> in a
> >> >> >> header)
> >> >> >> and move on, coming back to __null_unspecified pointers when the
> >> >> >> appropriate
> >> >> >> graybeard has been summoned out of retirement [5].
> >> >> >>
> >> >> >> Assumes-nonnull Regions
> >> >> >> We’ve found that it's fairly common for the majority of pointers
> >> >> >> within
> >> >> >> a
> >> >> >> particular header to be __nonnull. Therefore, we’ve introduced
> >> >> >> assumes-nonnull regions that assume that certain unannotated
> >> >> >> pointers
> >> >> >> implicitly get the __nonnull nullability qualifiers.
> Assumes-nonnull
> >> >> >> regions
> >> >> >> are marked by pragmas:
> >> >> >>
> >> >> >> #pragma clang assume_nonnull begin
> >> >> >>         __nullable char *strchr(const char *s, int c); // s is
> >> >> >> inferred
> >> >> >> to
> >> >> >> be __nonnull
> >> >> >> void *my_realloc (__nullable void *ptr, size_t size); //
> my_realloc
> >> >> >> is
> >> >> >> inferred to return __nonnull
> >> >> >> #pragma clang assume_nonnull end
> >> >> >>
> >> >> >> We infer __nonnull within an assumes_nonnull region when:
> >> >> >>
> >> >> >> The pointer is a non-typedef declaration, such as a function
> >> >> >> parameter,
> >> >> >> variable, or data member, or the result type of a function. It’s
> >> >> >> very
> >> >> >> rare
> >> >> >> for one to warn typedefs to specify nullability information;
> rather,
> >> >> >> it’s
> >> >> >> usually the user of the typedef that needs to specify nullability.
> >> >> >> The pointer is a single-level pointer, e.g., int* but not int**,
> >> >> >> because
> >> >> >> we’ve found that programmers can get confused about the
> nullability
> >> >> >> of
> >> >> >> multi-level pointers (is it a __nullable pointer to __nonnull
> >> >> >> pointers,
> >> >> >> or
> >> >> >> the other way around?) and inferring nullability for any of the
> >> >> >> pointers in
> >> >> >> a multi-level pointer compounds the situation.
> >> >> >>
> >> >> >>
> >> >> >> Note that no #include may occur within an assumes_nonnull region,
> >> >> >> and
> >> >> >> assumes_nonnull regions cannot cross header boundaries.
> >> >> >>
> >> >> >> Type System Impact
> >> >> >> Nullability qualifiers are mapped to type attributes within the
> >> >> >> Clang
> >> >> >> type
> >> >> >> system, but a nullability-qualified pointer type is not
> semantically
> >> >> >> distinct from its unqualified pointer type. Therefore, one may
> >> >> >> freely
> >> >> >> convert between nullability-qualified and
> non-nullability-qualified
> >> >> >> pointers, or between nullability-qualified pointers with different
> >> >> >> nullability qualifiers. One cannot overload on nullability
> >> >> >> qualifiers,
> >> >> >> write
> >> >> >> C++ class template partial specializations that identify
> nullability
> >> >> >> qualifiers, or inspect nullability via type traits in any way.
> >> >> >>
> >> >> >> Said more strongly, removing nullability qualifiers from a
> >> >> >> well-formed
> >> >> >> program will not change its behavior in any way, nor will the
> >> >> >> semantics
> >> >> >> of a
> >> >> >> program change when any set of (well-formed) nullability
> qualifiers
> >> >> >> are
> >> >> >> added to it. Operationally, this means that nullability qualifiers
> >> >> >> are
> >> >> >> not
> >> >> >> part of the canonical type in Clang’s type system, and that any
> >> >> >> warnings we
> >> >> >> produce based on nullability information will necessarily be
> >> >> >> dependent
> >> >> >> on
> >> >> >> Clang’s ability to retain type sugar during semantic analysis.
> >> >> >>
> >> >> >> While it’s somewhat exceptional for us to introduce new type
> >> >> >> qualifiers
> >> >> >> that don’t produce semantically distinct types, we feel that this
> is
> >> >> >> the
> >> >> >> only plausible design and implementation strategy for this
> feature:
> >> >> >> pushing
> >> >> >> nullability qualifiers into the type system semantically would
> cause
> >> >> >> significant changes to the language (e.g., overloading, partial
> >> >> >> specialization) and break ABI (due to name mangling) that would
> >> >> >> drastically
> >> >> >> reduce the number of potential users, and we feel that Clang’s
> >> >> >> support
> >> >> >> for
> >> >> >> maintaining type sugar throughout semantic analysis is generally
> >> >> >> good
> >> >> >> enough
> >> >> >> [6] to get the benefits of nullability annotations in our tools.
> >> >> >>
> >> >> >> Looking forward to our discussion.
> >> >> >>
> >> >> >> - Doug (with Jordan Rose and Anna Zaks)
> >> >> >>
> >> >> >> [1]
> >> >> >> http://en.wikipedia.org/wiki/Tony_Hoare#Apologies_and_retractions
> >> >> >> [2] The standard description of strchr seems to imply that the
> >> >> >> parameter
> >> >> >> cannot be null
> >> >> >> [3] The patch is complete, but should be reviewed on cfe-commits
> >> >> >> rather
> >> >> >> than here. There are also several logic parts to this monolithic
> >> >> >> patch:
> >> >> >> (a) __nonnull/__nullable/__null_unspecified type specifiers
> >> >> >> (b) nonnull/nullable/null_unspecified syntactic sugar for
> >> >> >> Objective-C
> >> >> >> (c) Warning about inconsistent application of nullability
> specifiers
> >> >> >> within a given header
> >> >> >> (d) assume_nonnnull begin/end pragmas
> >> >> >> (e) Objective-C null_resettable property attribute
> >> >> >> [4] https://gcc.gnu.org/onlinedocs/gcc/Function-Attributes.html
> >> >> >> (search
> >> >> >> for “nonnull”)
> >> >> >> [5] No graybeards were harmed in the making of this feature.
> >> >> >> [6] Template instantiation is the notable exception here, because
> it
> >> >> >> always canonicalizes types.
> >> >> >>
> >> >> >> <nullability.patch>
> >> >> >> _______________________________________________
> >> >> >> cfe-dev mailing list
> >> >> >> cfe-dev at cs.uiuc.edu
> >> >> >> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> _______________________________________________
> >> >> >> cfe-dev mailing list
> >> >> >> cfe-dev at cs.uiuc.edu
> >> >> >> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >> >> >>
> >> >> >
> >> >> >
> >> >> > _______________________________________________
> >> >> > cfe-dev mailing list
> >> >> > cfe-dev at cs.uiuc.edu
> >> >> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> >> >> >
> >> >
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20150627/d32ce69f/attachment.html>