[PATCH] D20786: Fix undefined behavior in __tree

[Eric Fiselier via cfe-commits]

EricWF created this revision.
EricWF added a reviewer: mclow.lists.
EricWF added a subscriber: cfe-commits.

This patch attempts to fix the undefined behavior in __tree by changing the node pointer types used throughout. The pointer types are changed for raw pointers in the current ABI and for fancy pointers in ABI V2 (since the fancy pointer types may not be ABI compatible).

The UB in `__tree` arises because tree downcasts the embedded end node and then deferences that pointer. Currently there are 3 node types in __tree.

* `__tree_end_node` which contains the `__left_` pointer. This node is embedded within the container.
* `__tree_node_base` which contains `__right_`, `__parent_` and `__is_black`. This node is used throughout the tree rebalancing algorithms.
* `__tree_node` which contains `__value_`.

Currently both `__tree` stores the start of the tree, `__begin_node_`, as a pointer to a `__tree_node`. Additionally the iterators store their position as a pointer to a `__tree_node`. In both of these cases the pointee can be the end node. This is fixed by changing them to store `__tree_end_node` pointers instead.

To make this change I introduced an `__iter_pointer` typedef which is defined to be a pointer to either `__tree_end_node` in the new ABI or `__tree_node` in the current one.
Both `__tree::__begin_node_` and iterator pointers are now stored as `__iter_pointers`.

The other situation where `__tree_end_node` is stored as a pointer to the wrong type is as the `__parent_` pointer within another node. Currently `__left_`, `__right_`, and `__parent_` are all `__tree_base_node` pointers. Since the end node will only be stored in `__parent_` the fix is to change `__parent_` to be a pointer to `__tree_end_node`.

To make this change I introduced a `__parent_pointer` typedef which is defined to be a pointer to either `__tree_node_base` in the new ABI or `__tree_node` in the current one.

Note that in the new ABI `__iter_pointer` and `__parent_pointer` are the same type (but not the old one). The confusion between these two types is unfortunate but it was the best solution I could come up with that maintains the ABI.

The typedef changes force a ton of explicit type casts to correct pointer types and to make current code compatible with both the old and new pointer typedefs. This is the bulk of the change and it's really messy. Unfortunately I don't know how to avoid it.

Please let me know what you think.





http://reviews.llvm.org/D20786

Files:
  include/__config
  include/__tree

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D20786.58930.patch
Type: text/x-patch
Size: 53748 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20160530/411089e8/attachment-0001.bin>