<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">On 3/25/20 2:22 PM, Eli Friedman wrote:<br>
</div>
<blockquote type="cite"
cite="mid:BY5PR02MB70926C5CC8E7DA768455558ECACE0@BY5PR02MB7092.namprd02.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:517430285;
mso-list-template-ids:1079805930;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">I agree we shouldn’t try to guess what the
user is trying to do. There shouldn’t be an unbounded set of
heuristic rules; “documented” implies some sort of promise of
stability in addition to the actual text in the manual. And
we shouldn’t try to guess whether the user’s code cares about
the length of a specific instruction.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I think you’re creating a false dichotomy
here, though. There’s some space between “the assembler can
rewrite any instruction” and “the assembler should guess what
the user is doing”. The patches you’re referring to are
dealing with a very specific issue: there are multiple ways
users specify prefixes for certain instructions. Sometimes
the prefixes are integrated into the instruction. Sometimes
they’re written separately, like “rep; stosb”. And sometimes
users end up using raw “.byte” directives. So we add a rule:
if we see something that looks like a prefix, followed by an
instruction, we assume the combination might actually be a
prefixed instruction, and don’t rewrite it. This is a
heuristic in some sense: we can’t prove whether the prefixed
instruction will ever execute at runtime. But adding prefixes
to the instruction is likely problematic.</p>
</div>
</blockquote>
<p>I see this as very much a slipper slope problem. As you mention,
I am trying to draw a hard line, but that's mostly because I can't
figure out what another reasonable line to draw might be. Why is
one idiomatic construct supported, but another not? Why is
"label: nop; mov $(rax), %rax" (with the expectation a fault
occurs at label + 1 byte) any less reasonable? If we say that's
reasonable, and should just work, well, we literally can't
implement prefix padding at all. Unless maybe there's something
I'm missing here?<br>
</p>
<blockquote type="cite"
cite="mid:BY5PR02MB70926C5CC8E7DA768455558ECACE0@BY5PR02MB7092.namprd02.prod.outlook.com">
<div class="WordSection1">
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We could take a different tack on this:
instead of giving up pure “strictness”, we could instead give
the user a diagnostic on patterns we recognize. But forcing
users to rewrite “rep;stosb”->“rep stosb” doesn’t seem
productive.<o:p></o:p></p>
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoNormal">-Eli<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in
0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Philip Reames
<a class="moz-txt-link-rfc2396E" href="mailto:listmail@philipreames.com"><listmail@philipreames.com></a> <br>
<b>Sent:</b> Wednesday, March 25, 2020 1:34 PM<br>
<b>To:</b> Eric Christopher <a class="moz-txt-link-rfc2396E" href="mailto:echristo@gmail.com"><echristo@gmail.com></a>;
Eli Friedman <a class="moz-txt-link-rfc2396E" href="mailto:efriedma@quicinc.com"><efriedma@quicinc.com></a><br>
<b>Cc:</b> Luo, Yuanke <a class="moz-txt-link-rfc2396E" href="mailto:yuanke.luo@intel.com"><yuanke.luo@intel.com></a>;
llvm-dev <a class="moz-txt-link-rfc2396E" href="mailto:llvm-dev@lists.llvm.org"><llvm-dev@lists.llvm.org></a>; Zhang, Annita
<a class="moz-txt-link-rfc2396E" href="mailto:annita.zhang@intel.com"><annita.zhang@intel.com></a>; Craig Topper
<a class="moz-txt-link-rfc2396E" href="mailto:craig.topper@intel.com"><craig.topper@intel.com></a><br>
<b>Subject:</b> [EXT] Re: [llvm-dev] Status of Intel JCC
Mitigations and Next Steps<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>The slightly unexpected bit for me in these responses is
the willingness to accept layout changes if documented. Let
me lay out some options, which of these seem reasonable?<o:p></o:p></p>
<p>1) Default to automatic padding, provide option to disable,
document where padding is inserted.<o:p></o:p></p>
<p>2) Default to not automatically padding, provide option to
enable, document where padding would be inserted if enabled.<o:p></o:p></p>
<p>And orthogonality to the above, two interpretations of
each:<o:p></o:p></p>
<p>a) auto padding is allowed to break common idioms (because
you enabled it explicitly or because you can disable). If
it does, you are expected to either simply not use it, or
possibly use fine grained directives in source.<o:p></o:p></p>
<p>b) auto padding conservative even when explicitly enabled.
One implication of this scheme is that different versions of
the assembler will almost by definition have to tweak
insertion as we find new problematic idioms. This creates
both documentation difficulties, and user confusion. <o:p></o:p></p>
<p>So, four choices total. Which seem reasonable?<o:p></o:p></p>
<p>Personally, I'd have no problem w/ 2a. Any of the other
variants concern me.<o:p></o:p></p>
<p>Philip<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 3/25/20 1:08 PM, Eric Christopher
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">FWIW I'm with Eli here if you need
any more data points. <o:p>
</o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-eric<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Mar 24, 2020 at 8:21 PM Eli
Friedman via llvm-dev <<a
href="mailto:llvm-dev@lists.llvm.org"
moz-do-not-send="true">llvm-dev@lists.llvm.org</a>>
wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC
1.0pt;padding:0in 0in 0in
6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Changing
the length of a sequence of assembly instructions
will break someone’s code at some point. The
length of a sequence of instructions is known, in
general, and people will write code to take
advantage of that. For example, I’ve seen assembly
code using something like Duff’s device, except
that instead of using a jump table, it just
computed the destination as “base+n*caselength”.
Given that, I don’t think it’s reasonable to hide
this mechanism from user control. <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We
definitely should not have any undocumented or
unpredictable behavior in the assembler. The
actual instruction bytes matter. That said, I’m
not sure there’s a strong line between “automagic”
and “explicit”, as long as the rules are
documented.<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">-Eli<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div style="border:none;border-left:solid blue
1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b>
llvm-dev <<a
href="mailto:llvm-dev-bounces@lists.llvm.org"
target="_blank" moz-do-not-send="true">llvm-dev-bounces@lists.llvm.org</a>>
<b>On Behalf Of </b>Philip Reames via
llvm-dev<br>
<b>Sent:</b> Tuesday, March 24, 2020 3:55 PM<br>
<b>To:</b> llvm-dev <<a
href="mailto:llvm-dev@lists.llvm.org"
target="_blank" moz-do-not-send="true">llvm-dev@lists.llvm.org</a>><br>
<b>Cc:</b> Luo, Yuanke <<a
href="mailto:yuanke.luo@intel.com"
target="_blank" moz-do-not-send="true">yuanke.luo@intel.com</a>>;
Zhang, Annita <<a
href="mailto:annita.zhang@intel.com"
target="_blank" moz-do-not-send="true">annita.zhang@intel.com</a>>;
Craig Topper <<a
href="mailto:craig.topper@intel.com"
target="_blank" moz-do-not-send="true">craig.topper@intel.com</a>><br>
<b>Subject:</b> [EXT] [llvm-dev] Status of
Intel JCC Mitigations and Next Steps<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p>TLDR - We have a choice to make about assembler
support, and a disagreement about how to move
forward. Community input needed.<o:p></o:p></p>
<p> <o:p></o:p></p>
<p>Background<o:p></o:p></p>
<p>Intel has a hardware bug in Skylake and later
whose mitigation requires padding of branches to
avoid performance degradation. Background here:
<a
href="https://www.intel.com/content/dam/support/us/en/documents/processors/mitigations-jump-conditional-code-erratum.pdf"
target="_blank" moz-do-not-send="true">
https://www.intel.com/content/dam/support/us/en/documents/processors/mitigations-jump-conditional-code-erratum.pdf</a><o:p></o:p></p>
<p>We now have in tree support for alignment of
such branches via nop padding, and limited
support for padding existing instructions with
either prefixes or larger immediate values.
This has survived several days of dedicated
testing and appears to be reasonably robust.
The padding support applies both to branch
alignment for the mitigation, but also normal
align directives.
<o:p></o:p></p>
<p>The original patches proposed a somewhat
different approach than we've ended up taking -
primarily because of memory overhead concerns.
However, there was also a deeper disagreement on
the original review threads (D70157 and others)
which was never settled, and we seem to be at a
point where this needs attention. In short, the
question is how assembler support should be
handled.<o:p></o:p></p>
<p> <o:p></o:p></p>
<p>The Choice<o:p></o:p></p>
<p>The problematic use case comes when assembling
user provided .s files. (Instead of the more
restricted output of the compiler.) Our basic
choice is do we want to force a new directive
syntax (and thus a source code change to use the
new feature), or attempt to automatically infer
where it's safe to insert padding?<o:p></o:p></p>
<p>The options as I see them:<o:p></o:p></p>
<ul type="disc">
<li class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0
level1 lfo1">
Assembler directives w/explicit opt in - In
this model, assembler input is assumed to only
enable padding in regions where it is safe to
do so.<o:p></o:p></li>
<li class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0
level1 lfo1">
Automagic assembler - In this model, the
assembler is responsible for inferring where
it is legal to pad without breaking user
expectations.
<o:p></o:p></li>
</ul>
<p>(I'll stop and disclaim that I'm strongly in
favor of the former. I've tried to describe the
pros/cons of each, but my perspective is
definitely biased.)<o:p></o:p></p>
<p>The difference between the two is a huge amount
of complexity, and a very fundamental
correctness risk. The basic problem is that
assemblers have to handle unconstrained inputs,
and IMO, the semantics of assembler as used in
practice is so under specified that it's really
hard to infer semantics in any useful way. As a
couple of examples, is the fault behavior of an
instruction well defined? Is the label near an
instruction used by the signal handler? Is that
data byte just before an instruction actually
decoded as part of the instruction?<o:p></o:p></p>
<p>The benefit of the later option is that
existing assembly files can be used without
modification. This is a huge advantage in terms
of ease of mitigation for existing code bases.
It's also the approach the original patch sets
for GCC took.
<o:p></o:p></p>
<p>In the original review thread(s), I had taken
the position that we should reject the automagic
assembler based on the correctness concerns
mentioned. I had thought the consensus in the
review was clearly in that direction as well,
but this has recently come up again. Given
that, I wanted to open it to a wider audience.<o:p></o:p></p>
<p> <o:p></o:p></p>
<p>Why am I pushing for a decision now?<o:p></o:p></p>
<p>There are two major reasons. First, there have
recently been a couple of patches posted and
landed (D76176, and D76052) building towards the
automagic assembler variant. And second, I've
started getting review comments (<a
href="https://reviews.llvm.org/D76398#1930383"
target="_blank" moz-do-not-send="true">https://reviews.llvm.org/D76398#1930383</a>)
which block forward progress on generalized
padding support assuming the automagic
interpretation. Implementing the automatic
assembler variant for prefix and immediate
padding adds substantial complexity and I would
very much like not to bother with if I don't
have to.<o:p></o:p></p>
<p> <o:p></o:p></p>
<p>Current implementation details<o:p></o:p></p>
<p>We have support in the integrated assembler
only for autopadding suppression. This allows a
LLVM based compiler to effectively apply padding
selectively. In particular, we've instrumented
lowering from MI to MC (X86MCInstLowering.cpp)
to selectively disable padding around constructs
which are thought to be problematic. We do not
have an agreed upon syntax for this in
assembler; the code that got checked in is
modeled closely around the last seriously
discussed variant (see below). This support is
able to use all of the padding variants: nop,
prefix, and immediate.<o:p></o:p></p>
<p>We also have limited support in the assembler
for not inserting nops between fragments where
doing so would break known idioms. The list of
such idioms is, IMO, ad hoc. This assembler
support does not include prefix or immediate
padding.
<o:p></o:p></p>
<p> <o:p></o:p></p>
<p>Philip<o:p></o:p></p>
<p>p.s. For those interested, here's roughly what
the last round of assembler syntax I remember
being discussed looked like.<o:p></o:p></p>
<p>.autopadding<br>
.noautopadding<o:p></o:p></p>
<p>These two directives would respectively enable
and disable automatic padding of instructions
within the region defined. It's presumed to be
legal to insert nops between instructions,
modify encodings, or otherwise adjust offsets of
instruction boundaries within the region to
achieve target specific desired alignments.
Similarly, it's presumed not to be legal to
change relative offsets outside an explicitly
enabled region. (Except for existing cases -
e.g. relaxation of branches, etc...)<o:p></o:p></p>
<p>The assembler would provide a command line flag
which conceptually wrapped the whole file in a
pair of enable/disable directives.
<o:p></o:p></p>
<p>We'd previously discussed a variant with
push/pop semantics and more fine grained control
over alignment requests, but I believe we
decided that was overkill in the end. (I walked
away with that impression based on the
integrated assembler work at least.)<o:p></o:p></p>
<p> <o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org"
target="_blank" moz-do-not-send="true">llvm-dev@lists.llvm.org</a><br>
<a
href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev"
target="_blank" moz-do-not-send="true">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><o:p></o:p></p>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</body>
</html>