[libcxx-dev] The per-TU guarantee of libc++

[Louis Dionne via libcxx-dev]

Libc++ currently supports two ABI-level guarantees:

1. All TUs linked together inside the same final linked image need to have use the same libc++ version.
2. TUs can be linked together even if they use different headers of libc++. 

I had a discussion with Eric about ending support for the second guarantee (coined the per-TU guarantee), and I thought it would be good to open up that discussion to a wider audience. Just to clear things up, none of those guarantees require using always_inline anymore, so this discussion is not related to eradicating always_inline.

The rationale for wanting to drop the per-TU guarantee is that this mode is currently not very well tested, and also the cost for supporting it is quite high in terms of source-level annotations (specifically, _LIBCPP_INLINE_VISIBILITY needs to stay). As a result of the poor testing we have for this guarantee, it's unclear whether we're actually providing the guarantee to the extent that we think we are. From a historical perspective, it also appears that the goal of _LIBCPP_INLINE_VISIBILITY was to control which symbols are exported from the dylib, not to allow TUs to be mixed. Eric, please correct me if I misrepresented any of your concerns.

On the other hand, this guarantee is something that we have been supporting for many years now, whether it was initially intended or simply a consequence of Hyrum's law. As a result, it is possible that some users are depending on this use case, and removing the guarantee would prevent them from updating to a newer libc++. Because of this, simply removing the guarantee does not seem like a viable option to me (even though I would directly benefit as a maintainer).

I believe the reasonable options on the table are:
1. Agree on a migration path to remove the per-TU guarantee in a responsible manner, or
2. Improve support for the per-TU guarantee and make it a first-class use case with good support

The current status quo is to do (1), where the migration path is to turn OFF the per-TU guarantee by default in LLVM 8 and see who complains. We could also mention the deprecation and ask people that need the guarantee to reach out to us. If nobody reaches out, we may be safe to assume that nobody is using the guarantee.

The other option is to do (2). If we do this, it would be nice to try and reduce our reliance on manual and error-prone source-code annotations. Eric and I brainstormed a couple ideas:

- We could have a clang-tidy check that ensures that all symbols in libc++ are annotated properly with something that either gives them `internal_linkage` (when the per-TU guarantee is enabled), or that explicitly says we allow this symbol to be ODR-merged across TUs. That would solve the problem of the guarantee not being well-tested, but it would still mean a lot of annotations inside libc++.

- We could alleviate the burden of source annotations by removing all `internal_linkage` annotations and instead making sure that we rename functions (including implementation details) whenever their semantics change. This would ensure that we don't cause non-benign ODR violations when mixing versions of libc++. However, we'd have to maintain a list of all signatures ever used in libc++ to be 100% bullet proof, and the burden to enforce this would be on code review. This is a different way of implementing the per-TU guarantee that has different tradeoffs.

- We could apply the internal_linkage attribute at namespace scope when we want to provide the per-TU guarantee. This would allow dropping all internal_linkage annotations in the code, and it would allow us to keep the per-TU guarantee. Also, testing would be trivial because there would be exactly one place to make a mistake -- where we apply the attribute at namespace scope.

Eric, please add to this if you can remember of other solutions or problems we were trying to address.

I'd like to encourage people to chime in on this discussion. My personal opinion having worked on this issue is that we should try to see whether the per-TU guarantee is actually used by people and simply keep the status quo. I see no urge for a drastic change. If we're lucky, we may just be able to remove the guarantee without doing a lot of work. If we realize we do need it absolutely, we can invest more time into making it a first-class supported use case.

Louis