[clang] [llvm] Replace autoconversion with TextEncodingConverter converters (PR #206519)

[Corentin Jabot via cfe-commits]

cor3ntin wrote:

> I was promised an RFC for that? @hubert-reinterpretcast

It seems that was a misunderstanding on my part, sorry about that.
That being said, we had a productive discussion about this PR in the clang working group and we concluded we would benefit from an RFC anyway.

In particular, if we have a fallback mechanism, we want to limit the scenarios in which it would get triggered to the scenario where the input charset is ebcdic-based (trying to compile an ebcdic encoded file as utf-8 would fail very quickly but compiling a win1252 as utf-8 might work but have different behavior, which would be very much undesired)

To that extent, one of the solution we think is worth exploring is to determine whether the input encoding is 
in the "ebcdic" family or the "ascii" family. 

It might be difficult to do that based on name reliably, but trying to encode known codepoint that are stable when converting - ie have the same representation in all of the usual encoding of a given family might be an approach worth exploring.

We are also curious to know about the forward-scanning of pragma filetag, and whether we can/should use DependencyScanning for that.

And the questions of running llvm own tests on ebcdic-based platforms also came up.


@AaronBallman @jyknight @hubert-reinterpretcast @Ariel-Burton 

https://github.com/llvm/llvm-project/pull/206519