[LLVMbugs] [Bug 20481] New: codecvt_utf16 facet with utf-16LE-bom file...

[bugzilla-daemon at llvm.org]

http://llvm.org/bugs/show_bug.cgi?id=20481

            Bug ID: 20481
           Summary: codecvt_utf16 facet with utf-16LE-bom file...
           Product: libraries
           Version: 3.4
          Hardware: PC
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P
         Component: Core LLVM classes
          Assignee: unassignedbugs at nondot.org
          Reporter: frederic.metrich at free.fr
                CC: llvmbugs at cs.uiuc.edu
    Classification: Unclassified

Hi,
It seems, the facet codecvt_utf16 is buggy.

I've a file which is UTF-16LE encoded with BOM.
I define 2 facets like this:
  std::locale utf16_locale( std::locale(), new std::codecvt_utf16< wchar_t,
0x10ffff, std::consume_header > );
  std::locale utf16_localeLEBOM( std::locale(), new std::codecvt_utf16<
wchar_t, 0x10ffff, static_cast< std::codecvt_mode >( 5 ) /*std::little_endian |
std::consume_header*/ > );

I then define 2 streams like this:
std::wifstream xmlfile1;
std::wifstream xmlfile2;

And read them like this:
{
  std::istreambuf_iterator< wchar_t > eos;
  xmlfile1.imbue( utf16_locale );
  xmlfile1.open( szFilename, std::ios::in | std::ios::binary );
  std::wstring strXML( std::istreambuf_iterator< wchar_t >( xmlfile1 ), eos );
}
{
  std::istreambuf_iterator< wchar_t > eos;
  xmlfile2.imbue( utf16_localeLEBOM );
  xmlfile2.open( szFilename, std::ios::in | std::ios::binary );
  std::wstring strXML( std::istreambuf_iterator< wchar_t >( xmlfile2 ), eos );
}

With the first one, the BOM isn't recognized/analyzed, and I end up with a
strXML containing the "raw" content of the file, including the BOM!
With the second one, I end up with the correct string, as I can read it with a
text editor.

If need, I can provide a complete package with test-file.
Thx
Fred

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20140729/d676bf3d/attachment.html>