[clang] [clang] Match -isysroot behaviour with system compiler on Darwin (PR #80524)

[Louis Dionne via cfe-commits]

ldionne wrote:

> @ldionne if downstream is catching upstream, I would love to discuss rationale behind ignoring command line option. I asked this question several times and haven't got answer. Original [diff](https://reviews.llvm.org/D89001) that introduced this behavior also didn't explain why new behavior is "intended" and better than old one. Instead of resolving case of two headers it actually prevents finding header that exists and path is specified in command line.

I think I don't understand your question, but I'm happy to discuss the rationale for the current approach once I do. Let me try to shed some light.

Five-ish years ago, Apple shipped libc++ in the toolchain (roughly in `$(which clang++)/../include/c++/v1`), and did not ship any libc++ headers in the SDK. We wanted to transition over to shipping libc++ only in the SDK, and not in the toolchain anymore. This was motivated by numerous factors, in particular our desire to decouple the shipping of libc++ and Clang to give more flexibility to libc++. To perform that transition, we elaborated the following plan:

1. We started shipping libc++ headers in the SDK (in addition to those in the toolchain)
2. We changed the Clang driver to prefer headers found in the SDK if there were any, and to fallback the toolchain headers otherwise. This behavior only ever made sense for AppleClang during the transition period, as a way for us to support mixing set ups where the headers could come either from the toolchain or from the SDK, while preferring the SDK ones.
3. We then removed the toolchain headers. Now AppleClang would always find only the headers in the SDK.
4. Finally we flipped back the default to what it should always have been, i.e. the behavior where we look in the toolchain and use these headers if they exist, and otherwise we fall back on the SDK headers.

If my memory serves me right, upstream never saw the change (2) since it only made sense for Apple's own toolchain (or at least that was our reasoning back then). The current problem you're seeing is simply that the "switch back" in step (4) is taking a long time to hit a release (due to release-related internal difficulties), and so naturally that makes you think that upstream is broken somehow.

With that being said, can you clarify what you mean by "ignoring the command-line option", and can you expand on why the current state of upstream Clang is broken in your opinion? But for productivity's sake, let's take for granted that the canonical state is upstream Clang and let's pretend that AppleClang also behaves the same (which is not quite true yet).

https://github.com/llvm/llvm-project/pull/80524