<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:19554852;
mso-list-type:hybrid;
mso-list-template-ids:-1036486824 -336150466 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:1531147459;
mso-list-type:hybrid;
mso-list-template-ids:-162612770 -265759148 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l1:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi Richard, Paul and other.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Thanks for the input so far. I wanted to point out that it’s not our code-base. Rather, we’re seeing more use of the LLVM technology in the automotive market and as usual we’re faced with existing
code bases that are tried and tested with other toolchains (gcc or others) and when LLVM comes along things don’t always work directly.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">We’ve suggested better ways of structuring their code and your suggestions are certainly good input. However, legacy code is especially sticky in any market that has to handle ‘safety’ concerns,
like automotive, aerospace and medical markets. Code changes are pretty expensive in those fields. So while I hope that over time we see more sensible coding structures, I don’t expect that to happen any time soon. In the mean time, we’re searching for a solution
for this coding pattern that doesn’t play well with clang.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hope that gave some more background of where this question comes from.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal">Do all options that were suggested by Mikhail really require fundamental restructuring of major parts of clang? This surprised me, I had expected that the option 2 to be possible without a complete overhaul. (2 is
<span style="mso-fareast-language:EN-US">“</span>Track until an overflow occurs after that make the lexer output the <invalid location> special value for all subsequent tokens.”) Not nice user experience but maybe doable? I was hoping there was something slightly
better that still works without a major restructuring (maybe something that at least gives a rough location or something that only gives the location of the error and not the include stack under an option or using some kind of heuristic to detect that things
go haywire).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">As an alternative, I was curious if it would be possible and acceptable to make the switch between 32-bit and 64-bit location tracking a build-time/cmake decision? I’ve not done any estimation on the memory size growth, so maybe this is
a dead end.<o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Christof<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> cfe-dev <cfe-dev-bounces@lists.llvm.org>
<b>On Behalf Of </b>Richard Smith via cfe-dev<br>
<b>Sent:</b> 07 October 2019 20:36<br>
<b>To:</b> Mikhail Maltsev <Mikhail.Maltsev@arm.com><br>
<b>Cc:</b> nd <nd@arm.com>; cfe-dev@lists.llvm.org<br>
<b>Subject:</b> Re: [cfe-dev] [RFC] Clang SourceLocation overflow<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, 2 Oct 2019 at 09:26, Mikhail Maltsev via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p>
</div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<p class="MsoNormal">Hi all,<br>
<br>
We are experiencing a problem with Clang SourceLocation overflow.<br>
Currently source locations are 32-bit values, one bit is a flag, which gives<br>
a source location space of 2^31 characters.<br>
<br>
When the Clang lexer processes an #include directive it reserves the total size<br>
of the file being included in the source location space. An overflow can occur<br>
if a large file (which does not have include guards by design) is included many<br>
times into a single TU.<br>
<br>
The pattern of including a file multiple times is for example required by<br>
the AUTOSAR standard [1], which is widely used in the automotive industry.<br>
Specifically the pattern is described in the Specification of Memory Mapping [2]:<br>
<br>
Section 8.2.1, MEMMAP003:<br>
"The start and stop symbols for section control are configured with section<br>
identifiers defined in MemMap.h [...] For instance:<br>
<br>
#define EEP_START_SEC_VAR_16BIT<br>
#include "MemMap.h"<br>
static uint16 EepTimer;<br>
static uint16 EepRemainingBytes;<br>
#define EEP_STOP_SEC_VAR_16BIT<br>
#include "MemMap.h""<br>
<br>
Section 8.2.2, MEMMAP005:<br>
"The file MemMap.h shall provide a mechanism to select different code, variable<br>
or constant sections by checking the definition of the module specific memory<br>
allocation key words for starting a section [...]"<br>
<br>
In practice MemMap.h can reach several MBs and can be included several thousand<br>
times causing an overflow in the source location space.<br>
<br>
The problem does not occur with GCC because it tracks line numbers rather than<br>
file offsets. Column numbers are tracked separately and are optional. I.e., in<br>
GCC a source location can be either a (line+column) tuple packed into 32 bits or<br>
(when the line number exceeds a certain threshold) a 32-bit line number.<br>
<br>
We are looking for an acceptable way of resolving the problem and propose the<br>
following approaches for discussion:<br>
1. Use 64 bits for source location tracking.<br>
2. Track until an overflow occurs after that make the lexer output<br>
the <invalid location> special value for all subsequent tokens.<br>
3. Implement an approach similar to the one used by GCC and start tracking line<br>
numbers instead of file offsets after a certain threshold. Resort to (2)<br>
when even line numbers overflow.<br>
4. (?) Detect the multiple inclusion pattern and track it differently (for now<br>
we don't have specific ideas on how to implement this)<br>
<br>
Is any of these approaches viable? What caveats should we expect? (we already<br>
know about static_asserts guarding the sizes of certain class fields which start<br>
failing in the first approach).<br>
<br>
Other suggestions are welcome.<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I don't think any of the above approaches are reasonable; they would all require fundamental restructuring of major parts of Clang, an efficiency or memory size hit for all other users of Clang, or some combination of those.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Your code pattern seems unreasonable; including a multi-megabyte file thousands of times is not a good idea. Can you split out parts of MemMap.h into a separate header that is only included once, and keep only the parts that actually change
on repeated inclusion in MemMap.h itself?<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>