[llvm] [llvm] Do not use Console API if the output isn't a console device (PR #90230)

[via llvm-commits]

alvinhochun wrote:

> I'm not entirely following @alvinhochun's comment here - I presume this is about when we're outputting directly to a terminal, whether we should prefer ANSI codes or console APIs for setting color? And that it's possible to query this with `ENABLE_VIRTUAL_TERMINAL_PROCESSING`?

1. Yes
2. What I understand is that if `ENABLE_VIRTUAL_TERMINAL_PROCESSING` is set when calling `SetConsoleMode` on a system that does not support VT, it returns a non-zero value to indicate an error, which can be used to determine whether it is supported.


> > I think this is a good idea. But it would need a release note. Does it need a test of some kind? To make sure the fallback to console api works maybe?
> 
> Some sort of automated test would certainly be good, but I'm not really sure how doable that is in our test infrastructure. (I would expect that all test executions are towards something that isn't directly a console, etc.) So I think it's fine without one... But bonus points for a sequence of easy instructions for verifying that it works, that anyone can repeat both now and when looking into related things in the future.

Just thinking out loud, the conhost properties has an option to "use legacy console" (which should disable VT), and it is backed by the registry key `HKEY_CURRENT_USER\Console\ForceV2`. In theory we can use it to test the fallback, say by screen capturing the conhost window, or by intercepting `SetConsoleTextAttribute` to see whether it is called in the expected way. But these tests will need to be run in a separate conhost window, and should probably not run in parallel with other tests to avoid potentially affecting them. I doubt it is really worth the trouble. but I guess it may be a fun challenge for someone?

https://github.com/llvm/llvm-project/pull/90230