<div dir="ltr"><div dir="ltr">On Mon, 14 Dec 2020 at 20:59, Tom Honermann <<a href="mailto:Thomas.Honermann@synopsys.com">Thomas.Honermann@synopsys.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>On 12/14/2020 7:40 PM, Richard Smith wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Mon, 14 Dec 2020 at 08:40, Tom Honermann via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>On 12/10/2020 8:59 AM, Abhina Sreeskantharajan via cfe-dev wrote:<br>
</div>
<blockquote type="cite">
<pre>We wish to implement the fexec-charset option to enable support for different execution character sets outside of the default UTF-8. Our proposal is to use UTF-8 as the internal charset. All input source files will be converted to this charset. One side effect is that the file buffer size may change after the translation between single and multi-byte character sets. This Phabricator patch <a href="https://urldefense.com/v3/__https://reviews.llvm.org/D93031__;!!A4F2R9G_pg!OuJ2_1ZvFHtfhljdBP-blYvsVd3FYKi5LL1v0hbh8OYpBmWQrdKjA-_NPoCj4F_l$" target="_blank">https://urldefense.com/v3/__https://reviews.llvm.org/D93031__;!!A4F2R9G_pg!OuJ2_1ZvFHtfhljdBP-blYvsVd3FYKi5LL1v0hbh8OYpBmWQrdKjA-_NPoCj4F_l$</a> shows our initial implementation plan.
First, we create converters using the CharSetConverter class (<a href="https://urldefense.com/v3/__https://reviews.llvm.org/D88741__;!!A4F2R9G_pg!OuJ2_1ZvFHtfhljdBP-blYvsVd3FYKi5LL1v0hbh8OYpBmWQrdKjA-_NPk1e2im-$" target="_blank">https://urldefense.com/v3/__https://reviews.llvm.org/D88741__;!!A4F2R9G_pg!OuJ2_1ZvFHtfhljdBP-blYvsVd3FYKi5LL1v0hbh8OYpBmWQrdKjA-_NPk1e2im-$</a> ) for conversion between the internal charset and the system or execution charset, and vice versa. In the CharSetConverter class, some conversions are provided, but for the unsupported conversions, the class will attempt to create the converter using the iconv library if it exists. Since the iconv library differs between platforms, we cannot guarantee that the behaviour will be the same across all platforms.
Then during the parsing stage we translate the string literals using this converter. Translation cannot be performed after this stage because the context is lost and there is no difference between escaped characters and normal characters. The translated string will be shown in the IR readable format.
In addition, we wish to control translation for different types of strings. In our plan, we introduce an enum, ConversionState, to indicate which output codepage is needed. The table below summarizes what codepage different kinds of string literals should be in:
╔══════════════════════════════════════╤═════════════════════════════════╗
║Context │Output Codepage ║
╠══════════════════════════════════════╪═════════════════════════════════╣
║asm("...") │system charset (IBM-1047 on z/OS)║
╟──────────────────────────────────────┼─────────────────────────────────╢
║typeinfo name │system charset (IBM-1047 on z/OS)║
╟──────────────────────────────────────┼─────────────────────────────────╢
║#include "fn" or #include <fn> │system charset (IBM-1047 on z/OS)║
╟──────────────────────────────────────┼─────────────────────────────────╢
║__FILE__ , __func__ │-fexec-charset ║
╟──────────────────────────────────────┼─────────────────────────────────╢
║literal, string & char │-fexec-charset ║
╟──────────────────────────────────────┼─────────────────────────────────╢
║user-defined literal │-fexec-charset ║
╟──────────────────────────────────────┼─────────────────────────────────╢
║extern "C" ... │n/a (internal charset, UTF-8) ║
╟──────────────────────────────────────┼─────────────────────────────────╢
║_Pragma("message(...)”) │n/a (internal charset, UTF-8) ║
╟──────────────────────────────────────┼─────────────────────────────────╢
║attribute args (GNU, GCC) │n/a (internal charset, UTF-8) ║
╟──────────────────────────────────────┼─────────────────────────────────╢
║module map string literals (-fmodules)│n/a (internal charset, UTF-8) ║
╟──────────────────────────────────────┼─────────────────────────────────╢
║line directive, digit directive │n/a (internal charset, UTF-8) ║
╚══════════════════════════════════════╧═════════════════════════════════╝
</pre>
</blockquote>
<p>The text provided by <font face="monospace">std::type_info::name()</font> is generally displayed in some way, often in conjunction with additional text encoded with the execution encoding. I think this should follow
<font face="monospace">-fexec-charset</font>; that would be consistent with handling of
<font face="monospace">__func__</font>.<br>
</p>
</div>
</blockquote>
<div>The string returned here is assumed to be a mangled type name under both the Itanium ABI and the MS ABI, and is often passed by programs to the demangler / undecorator, so I think it should follow the encoding assumption of the mangling / name decoration
scheme, which is de facto UTF-8 for Itanium, and appears to also be UTF-8 under the MS ABI (even if the execution character set is set to something else:
<a href="https://urldefense.com/v3/__https://godbolt.org/z/3Gx646__;!!A4F2R9G_pg!I1Kr-XQUTghlsUQNrOVHcyHGjGD_Ylfl-pVYhT2VDveDwG2njd4kFem62ShpGc2z4xzHyv8$" target="_blank">
https://godbolt.org/z/3Gx646</a>).</div>
<div><br>
</div>
<div>Similarly for the predefined string variables: __FUNCDNAME__ should be encoded in UTF-8 because it's a mangled name. The rest are human-readable text, so should be in the execution character set (wide execution character set for L__FUNCTION__ and L__FUNCSIG__).</div>
</div>
</div>
</blockquote>
I suspect UTF-8 wouldn't be an appropriate choice (not in general anyway) for the z/OS GOFF format, so perhaps the right answer for these cases is not to attempt to constrain beyond what C and C++ require; a NTBS. The contents therefore have a target/ABI dependent
encoding (if they have one at all).<br></div></blockquote><div><br></div><div>Yes, that's a good point. We can't necessarily assume that there even is a valid encoding for symbols / mangled names on arbitrary (perhaps not in-tree) targets. So we should take whatever sequence of octets comes out of the name mangler, add a trailing 0 byte, and use the result verbatim as the execution-time value. For Itanium and MS ABI that happens to always be a valid UTF-8 encoding, but that's happenstance, and arbitrary binary data (with no embedded 0 bytes, in order to make __FUNCDNAME__ and typeinfo strings usable) should be equally acceptable.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div>The output of the demangler should presumably be in the execution character set, but that's not our concern.</div>
</div>
</div>
</blockquote>
<p>Agreed.</p>
<p>Tom.<br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p></p>
<p>The messages provided for <font face="monospace">#error</font>, <font face="monospace">
static_assert</font>, <font face="monospace">[[deprecated]]</font>, and <font face="monospace">
[[nodiscard]]</font> (following adoption of <a href="https://urldefense.com/v3/__http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2448.pdf__;!!A4F2R9G_pg!I1Kr-XQUTghlsUQNrOVHcyHGjGD_Ylfl-pVYhT2VDveDwG2njd4kFem62ShpGc2znpKAJbU$" target="_blank">
WG14 N2448</a> and <a href="https://urldefense.com/v3/__https://wg21.link/p1301__;!!A4F2R9G_pg!I1Kr-XQUTghlsUQNrOVHcyHGjGD_Ylfl-pVYhT2VDveDwG2njd4kFem62ShpGc2zDRDx15Q$" target="_blank">
WG21 P1301</a>) are another special case. Options are to preserve the provided message in the internal encoding or, as mentioned below, to transcode from the execution encoding to the system encoding for diagnostic display. Per
<a href="https://urldefense.com/v3/__http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2563.pdf__;!!A4F2R9G_pg!I1Kr-XQUTghlsUQNrOVHcyHGjGD_Ylfl-pVYhT2VDveDwG2njd4kFem62ShpGc2zamP3JYE$" target="_blank">
WG14 N2563</a> and <a href="https://urldefense.com/v3/__https://wg21.link/p2246__;!!A4F2R9G_pg!I1Kr-XQUTghlsUQNrOVHcyHGjGD_Ylfl-pVYhT2VDveDwG2njd4kFem62ShpGc2zRLM7Dns$" target="_blank">
WG21 P2246</a>, either approach is acceptable, but the former would improve QoI.<br>
</p>
<blockquote type="cite">
<pre>Several complexities arise when string literals are inspected after translation. In these later stages of compilation, there is an underlying assumption that the literals are encoded in UTF-8 when in fact they can be encoded in one of many charsets. Listed below are three instances where this assumption can be found.
1. During printf/scanf format string validation, format specifiers in UTF-8 are searched for but are not found because the string will already be translated.
2. There are certain optimizations after the parsing stage (e.g. in SimplifyLibCalls.cpp, printf("done\n") gets optimized to puts("done") if the string ends in ‘\n’) which will not work if the string is already translated.
3. When generating messages (e.g. pragma message, warnings, errors) the message (usually in UTF-8) may be merged with string literals which may be translated, resulting in a string with mixed encoding.
4. When using VerifyDiagnosticConsumer to verify expected diagnostics, the expected text is in UTF-8 but the message can refer to string constants and literals which are translated and cannot match the expected text.
Currently, we see no other way than to reverse the translation, disable this optimization or stop certain translations when the string is assumed to be encoded in UTF-8 to resolve these complexities. Although reversing translation may not yield the original string, it can be used to locate format specifiers which are guaranteed to be correctly identified.
Any feedback on this implementation is welcome.
Thanks,
Abhina
_______________________________________________
cfe-dev mailing list
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>
<a href="https://urldefense.com/v3/__https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev__;!!A4F2R9G_pg!OuJ2_1ZvFHtfhljdBP-blYvsVd3FYKi5LL1v0hbh8OYpBmWQrdKjA-_NPrj5FV9-$" target="_blank">https://urldefense.com/v3/__https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev__;!!A4F2R9G_pg!OuJ2_1ZvFHtfhljdBP-blYvsVd3FYKi5LL1v0hbh8OYpBmWQrdKjA-_NPrj5FV9-$</a>
</pre>
</blockquote>
</div>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="https://urldefense.com/v3/__https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev__;!!A4F2R9G_pg!I1Kr-XQUTghlsUQNrOVHcyHGjGD_Ylfl-pVYhT2VDveDwG2njd4kFem62ShpGc2zL34PBLU$" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote></div></div>