[cfe-dev] RFC: Enabling fexec-charset support to LLVM and clang

Tom Honermann via cfe-dev cfe-dev at lists.llvm.org
Mon Dec 14 20:59:40 PST 2020

On 12/14/2020 7:40 PM, Richard Smith wrote:
On Mon, 14 Dec 2020 at 08:40, Tom Honermann via cfe-dev <cfe-dev at lists.llvm.org<mailto:cfe-dev at lists.llvm.org>> wrote:
On 12/10/2020 8:59 AM, Abhina Sreeskantharajan via cfe-dev wrote:

We wish to implement the fexec-charset option to enable support for different execution character sets outside of the default UTF-8. Our proposal is to use UTF-8 as the internal charset. All input source files will be converted to this charset. One side effect is that the file buffer size may change after the translation between single and multi-byte character sets. This Phabricator patch https://urldefense.com/v3/__https://reviews.llvm.org/D93031__;!!A4F2R9G_pg!OuJ2_1ZvFHtfhljdBP-blYvsVd3FYKi5LL1v0hbh8OYpBmWQrdKjA-_NPoCj4F_l$  shows our initial implementation plan.

First, we create converters using the CharSetConverter class (https://urldefense.com/v3/__https://reviews.llvm.org/D88741__;!!A4F2R9G_pg!OuJ2_1ZvFHtfhljdBP-blYvsVd3FYKi5LL1v0hbh8OYpBmWQrdKjA-_NPk1e2im-$ ) for conversion between the internal charset and the system or execution charset, and vice versa. In the CharSetConverter class, some conversions are provided, but for the unsupported conversions, the class will attempt to create the converter using the iconv library if it exists. Since the iconv library differs between platforms, we cannot guarantee that the behaviour will be the same across all platforms.

Then during the parsing stage we translate the string literals using this converter. Translation cannot be performed after this stage because the context is lost and there is no difference between escaped characters and normal characters. The translated string will be shown in the IR readable format.

In addition, we wish to control translation for different types of strings. In our plan, we introduce an enum, ConversionState, to indicate which output codepage is needed. The table below summarizes what codepage different kinds of string literals should be in:
║Context                               │Output Codepage                  ║
║asm("...")                            │system charset (IBM-1047 on z/OS)║
║typeinfo name                         │system charset (IBM-1047 on z/OS)║
║#include "fn" or #include <fn>        │system charset (IBM-1047 on z/OS)║
║__FILE__ , __func__                   │-fexec-charset                   ║
║literal, string & char                │-fexec-charset                   ║
║user-defined literal                  │-fexec-charset                   ║
║extern "C" ...                        │n/a (internal charset, UTF-8)    ║
║_Pragma("message(...)”)               │n/a (internal charset, UTF-8)    ║
║attribute args (GNU, GCC)             │n/a (internal charset, UTF-8)    ║
║module map string literals (-fmodules)│n/a (internal charset, UTF-8)    ║
║line directive, digit directive       │n/a (internal charset, UTF-8)    ║

The text provided by std::type_info::name() is generally displayed in some way, often in conjunction with additional text encoded with the execution encoding.  I think this should follow -fexec-charset; that would be consistent with handling of __func__.

The string returned here is assumed to be a mangled type name under both the Itanium ABI and the MS ABI, and is often passed by programs to the demangler / undecorator, so I think it should follow the encoding assumption of the mangling / name decoration scheme, which is de facto UTF-8 for Itanium, and appears to also be UTF-8 under the MS ABI (even if the execution character set is set to something else: https://godbolt.org/z/3Gx646<https://urldefense.com/v3/__https://godbolt.org/z/3Gx646__;!!A4F2R9G_pg!I1Kr-XQUTghlsUQNrOVHcyHGjGD_Ylfl-pVYhT2VDveDwG2njd4kFem62ShpGc2z4xzHyv8$>).

Similarly for the predefined string variables: __FUNCDNAME__ should be encoded in UTF-8 because it's a mangled name. The rest are human-readable text, so should be in the execution character set (wide execution character set for L__FUNCTION__ and L__FUNCSIG__).
I suspect UTF-8 wouldn't be an appropriate choice (not in general anyway) for the z/OS GOFF format, so perhaps the right answer for these cases is not to attempt to constrain beyond what C and C++ require; a NTBS.  The contents therefore have a target/ABI dependent encoding (if they have one at all).

The output of the demangler should presumably be in the execution character set, but that's not our concern.



The messages provided for #error, static_assert, [[deprecated]], and [[nodiscard]] (following adoption of WG14 N2448<https://urldefense.com/v3/__http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2448.pdf__;!!A4F2R9G_pg!I1Kr-XQUTghlsUQNrOVHcyHGjGD_Ylfl-pVYhT2VDveDwG2njd4kFem62ShpGc2znpKAJbU$> and WG21 P1301<https://urldefense.com/v3/__https://wg21.link/p1301__;!!A4F2R9G_pg!I1Kr-XQUTghlsUQNrOVHcyHGjGD_Ylfl-pVYhT2VDveDwG2njd4kFem62ShpGc2zDRDx15Q$>) are another special case.  Options are to preserve the provided message in the internal encoding or, as mentioned below, to transcode from the execution encoding to the system encoding for diagnostic display.  Per WG14 N2563<https://urldefense.com/v3/__http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2563.pdf__;!!A4F2R9G_pg!I1Kr-XQUTghlsUQNrOVHcyHGjGD_Ylfl-pVYhT2VDveDwG2njd4kFem62ShpGc2zamP3JYE$> and WG21 P2246<https://urldefense.com/v3/__https://wg21.link/p2246__;!!A4F2R9G_pg!I1Kr-XQUTghlsUQNrOVHcyHGjGD_Ylfl-pVYhT2VDveDwG2njd4kFem62ShpGc2zRLM7Dns$>, either approach is acceptable, but the former would improve QoI.

Several complexities arise when string literals are inspected after translation. In these later stages of compilation, there is an underlying assumption that the literals are encoded in UTF-8 when in fact they can be encoded in one of many charsets. Listed below are three instances where this assumption can be found.

1. During printf/scanf format string validation, format specifiers in UTF-8 are searched for but are not found because the string will already be translated.
2. There are certain optimizations after the parsing stage (e.g. in SimplifyLibCalls.cpp, printf("done\n") gets optimized to puts("done") if the string ends in ‘\n’) which will not work if the string is already translated.
3. When generating messages (e.g. pragma message, warnings, errors) the message (usually in UTF-8) may be merged with string literals which may be translated, resulting in a string with mixed encoding.
4. When using VerifyDiagnosticConsumer to verify expected diagnostics, the expected text is in UTF-8 but the message can refer to string constants and literals which are translated and cannot match the expected text.

Currently, we see no other way than to reverse the translation, disable this optimization or stop certain translations when the string is assumed to be encoded in UTF-8 to resolve these complexities. Although reversing translation may not yield the original string, it can be used to locate format specifiers  which are guaranteed to be correctly identified.

Any feedback on this implementation is welcome.


cfe-dev mailing list
cfe-dev at lists.llvm.org<mailto:cfe-dev at lists.llvm.org>

cfe-dev mailing list
cfe-dev at lists.llvm.org<mailto:cfe-dev at lists.llvm.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20201215/1f49de0b/attachment-0001.html>

More information about the cfe-dev mailing list