[llvm-bugs] [Bug 44911] New: codecvt<wchar_t, char, std::mbstate_t>::do_out() does not convert a leading NUL wide character

[via llvm-bugs]

https://bugs.llvm.org/show_bug.cgi?id=44911

            Bug ID: 44911
           Summary: codecvt<wchar_t, char, std::mbstate_t>::do_out() does
                    not convert a leading NUL wide character
           Product: libc++
           Version: 9.0
          Hardware: PC
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P
         Component: All Bugs
          Assignee: unassignedclangbugs at nondot.org
          Reporter: philipp.kutin at dcam.de
                CC: llvm-bugs at lists.llvm.org, mclow.lists at gmail.com

Created attachment 23128
  --> https://bugs.llvm.org/attachment.cgi?id=23128&action=edit
Test case using C++17

The mentioned function (found in src/locale.cpp) converts the input wide
character sequence in chunks of wide-NUL-terminated subsequences (exception:
the last such subsequence may not contain a wide NUL).

If one such subsequence is empty (that is, it consists only of the wide NUL)
before then end of the input sequence has been seen, then do_out() prematurely
returns with codecvt<>::partial when it still could have converted the wide NUL
and proceeded with the next subsequence.

This is easiest demonstrated by attempting to convert a single wide NUL. A test
source file is attached to this bug report. To build:

$ clang++-9 -O0 -g -std=c++17 -Wall -Wextra -Wpedantic
codecvt_wchar_t_char_wNUL.cpp -o test_codecvt_libstdcxx
$ clang++-9 -O0 -g -std=c++17 -stdlib=libc++ -Wall -Wextra -Wpedantic
codecvt_wchar_t_char_wNUL.cpp -o test_codecvt_libcxx

(The environment I am testing this in is an Ubuntu Linux 18.04 with a libc++
whose locale.cpp matches git tag 'llvmorg-9.0.1', but I initially observed the
behavior on a machine running macOS Catalina and having XCode installed there
just recently. I also expect the behavior to be present in master because the
source for the function in question is identical there.)

        static const auto *ws = L"\0";
        const auto res = codeConvert.out(state, ws, ws + 1, fromNext, buf, buf
+ 1, toNext);
        assert(res == CodeConvert::ok);

For comparison, the test program compiled against libstdc++ exhibits the
behavior I expect.

The behavior is easily explained when looking at the code: in case the
subsequence passed to __libcpp_wcsnrtombs_l() is empty (fend - frm == 0), there
is nothing for the C function wrapper to do, and zero as return value seems
like the resonable result. However, further below do_out() returns with with
'partial' in that case, even though the return of zero was just because there
was nothing to convert in the first place!

So, I'm wondering: should the block from

        // save state in case it is needed to recover to_nxt on error

to and excluding

        if (fend != frm_end)  // set up next null terminated sequence

rather not be conditional on "if (fend-frm > 0)"? A quick GDB session using a
'jump' in the (hopefully) right place made my test case pass.

I'm assuming that the observed behavior is unintentional. Looking at the C++
Standard to answer the question whether such behavior is allowed -- I'm reading
n4659 25.4.1.4.2 "codecvt virtual functions" [locale.codecvt.virtuals] -- it
looks to me like it may still be OK (as in no contracts broken -- "not all
source characters converted" is just what happened, and 'from_next' and
'to_next' would be consistent) but it seems odd that the ability to convert a
wide NUL character should be context-dependent / behave like C string functions
-- do_out() does take the length of its input sequence explicitly, after all.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20200214/c303dc86/attachment-0001.html>