<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">And I thought Windows was complicated…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Debug info (at least for DWARF) wants strings to be UTF-8; filenames, type/variable names, all that stuff. Just to throw that out there.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">--paulr<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></a></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> cfe-dev [mailto:cfe-dev-bounces@lists.llvm.org]
<b>On Behalf Of </b>Sean Perry via cfe-dev<br>
<b>Sent:</b> Monday, January 29, 2018 5:22 PM<br>
<b>To:</b> cfe-dev@lists.llvm.org<br>
<b>Subject:</b> [cfe-dev] Implementing charsets (-fexec-charset & -finput-charset)<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p style="margin-bottom:12.0pt"><span style="font-size:10.0pt">Hi, I have been investigating how to implementation -finput-charset and -fexec-charset (including -fexec-wide-charset too). I noticed some discussions a couple years ago and was planning on taking
the same approach. In a nutshell, change the source manager to convert the input charset into UTF-8 and do the parsing using UTF-8 (eg. in Lexer::InitLexer()). I would convert strings and character constants into the exec charset when a consumer asks for the
string literal. This seems like a sound concept but there are many details that need to be ironed out. The clang data structure is filled with all kinds of strings (i.e file names, identifiers, literals). What charset should be used when creating the clang
AST's? Should getName() return the name in UTF-8 or an output charset? </span><br>
<br>
<span style="font-size:10.0pt">While looking into this I realized that we need one more charset. We have the input charset for the source code and exec charset for the string literals. And we have the an internal charset (UTF-8) for parsing. But we also need
to have a charset for things like symbol names and file names. These do not use the input or internal charsets. For example, on MVS the user may say the input charset is ASCII or UTF-8 but actual file names and symbol names in the object file need to be EBCDIC.
The same would be true for alternative charsets on Linux. A code point in the a locale other that en_US may map to a difference code point in the file system. The other charset is the output charset. This is the charset that symbol names in the object file
should use as well as the charset for file names.</span><br>
<br>
<span style="font-size:10.0pt">We also need to consider messages. The messages may not be in the same charset as the input charset or internal. We will need to consider translation for messages and the substituted text.
</span><br>
<br>
<span style="font-size:10.0pt">I have thought about some of these issues and would like feedback and/or suggestions.
</span><br>
<br>
<span style="font-size:10.0pt">1) Source file names</span><br>
<span style="font-size:10.0pt">- We'd store these in the SourceManager in the output charset.</span><br>
<span style="font-size:10.0pt">- When preprocessing (#include, etc) we would convert the file names into the output charset and do all file name building and system calls using the output charset</span><br>
<br>
<span style="font-size:10.0pt">2) Identifiers</span><br>
<span style="font-size:10.0pt">- I think the getName() function on IdentifierInfo and similar functions should return the name in the output charset Too many places, even in clang, use the getName() functions and would need to apply a translation if we didn't
do this</span><br>
<span style="font-size:10.0pt">- We need some way to make parsing quick since identifiers will start off in UTF-8 and we won't be able to use getName() to look up identifiers any more. I was thinking about adding a getNameInternal() that would return the UTF-8
spelling and would be used in the hashing.</span><br>
<br>
<span style="font-size:10.0pt">3) String literals & Character constants</span><br>
<span style="font-size:10.0pt">- these are converted to the exec charset and stored in the clang structure in the translated format</span><br>
<br>
<span style="font-size:10.0pt">4) Messages & trace information</span><br>
<span style="font-size:10.0pt">- Going to assume the charset for messages is a variation of the output charset.</span><br>
<span style="font-size:10.0pt">- All substitution text should be in or converted into the output charset before generating the diagnostic message.</span><br>
<span style="font-size:10.0pt">- trace/dump output will be in the output charset too.</span><br>
<br>
<span style="font-size:10.0pt">5) Preprocessed output (including the make depndency rules)<br>
- All preprocessed output will be in the output charset</span><br>
<br>
<span style="font-size:10.0pt">Thanks<br>
--<br>
Sean Perry<br>
Compiler Development<br>
IBM Canada Lab<br>
(905)-413-6031 (tie 313-6031), fax (905)-413-4839</span><o:p></o:p></p>
</div>
</div>
</body>
</html>