[llvm] [llvm] Do not use Console API if the output isn't a console device (PR #90230)

[Kacper Michajłow via llvm-commits]

kasper93 wrote:

> What I understand is that if ENABLE_VIRTUAL_TERMINAL_PROCESSING is set when calling SetConsoleMode on a system that does not support VT, it returns a non-zero value to indicate an error, which can be used to determine whether it is supported.

That's correct, but for checking support `GetConsoleMode` can be used. In mpv we first try to set (this is not required for llvm) `ENABLE_VIRTUAL_TERMINAL_PROCESSING` if not already enabled and check it after with `GetConsoleMode`, if `ENABLE_VIRTUAL_TERMINAL_PROCESSING` bit is set, we use ANSI, else fallback to Console API, works well.

I would like to also note that `GetConsoleMode` is relatively slow, when printing lots of data (I've seen that on profile, when outputting images to terminal). Probably not critical for llvm, but we cache the mode, as we expect no runtime changes. LLVM currently already `GetConsoleMode` on each print to test if we are Console, so it is another topic of improvement. In the sense, adding VT check would not regress current status quo.




https://github.com/llvm/llvm-project/pull/90230