[libcxx-commits] [libcxx] Avoid calling `setlocale` in `do_unshift` when unnecessary (PR #117153)

[Michael Maltsev via libcxx-commits]

https://github.com/m417z created https://github.com/llvm/llvm-project/pull/117153

This is an attempt to mitigate #110954.

As part of the libc++.dll initialization, `static DoIOSInit` is initialized:
https://github.com/llvm/llvm-project/blob/5bdee35544eb21762857390014598748c64ad485/libcxx/src/iostream.cpp#L165-L167

When the dll is unloaded or on process shutdown, `DoIOSInit::~DoIOSInit` is called. It ends up calling `flush`:
https://github.com/llvm/llvm-project/blob/5bdee35544eb21762857390014598748c64ad485/libcxx/src/iostream.cpp#L159

Which calls `pubsync`:
https://github.com/llvm/llvm-project/blob/5bdee35544eb21762857390014598748c64ad485/libcxx/include/__ostream/basic_ostream.h#L659

Which ends up calling `do_unshift`:
https://github.com/llvm/llvm-project/blob/5bdee35544eb21762857390014598748c64ad485/libcxx/src/locale.cpp#L1475-L1479

Which, as can be seen, unconditionally calls `__locale::__wcrtomb`, which ends up calling `setlocale` via `__libcpp_locale_guard`.

All this means that `setlocale` is called on process shutdown even if `wcout` is never used, or even if nothing stream-related is used. Calling `setlocale` on process shutdown causes problems, as described in the mentioned issue.

This PR is an attempt to avoid calling `setlocale` in the vast majority of cases, when there's no output to be flushed. It's not a complete fix to the issue, but it will make it much less common, and it will at least allow to flush output manually to avoid the issue if streams are used.

>From 330b197dafc158e1586890ddf9bd1c64417514bf Mon Sep 17 00:00:00 2001
From: Michael Maltsev <4129781+m417z at users.noreply.github.com>
Date: Thu, 21 Nov 2024 14:29:19 +0200
Subject: [PATCH] Avoid calling setlocale in do_unshift when unnecessary

---
 libcxx/src/locale.cpp | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/libcxx/src/locale.cpp b/libcxx/src/locale.cpp
index a1e10401f0b299..5ecd99c53cd516 100644
--- a/libcxx/src/locale.cpp
+++ b/libcxx/src/locale.cpp
@@ -1475,6 +1475,8 @@ codecvt<wchar_t, char, mbstate_t>::result codecvt<wchar_t, char, mbstate_t>::do_
 codecvt<wchar_t, char, mbstate_t>::result codecvt<wchar_t, char, mbstate_t>::do_unshift(
     state_type& st, extern_type* to, extern_type* to_end, extern_type*& to_nxt) const {
   to_nxt = to;
+  if (std::mbsinit(&st))
+    return ok;
   extern_type tmp[MB_LEN_MAX];
   size_t n = __locale::__wcrtomb(tmp, intern_type(), &st, __l_);
   if (n == size_t(-1) || n == 0) // on error