<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Sounds like we could learn a few lessons from UTF-8 and use the
first several bits to say where to find the rest (ie spread the
rest over multiple 'extra' containers in SourceManager)?</p>
<p>Thanks,</p>
<p>Stephen.<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 02/02/2021 18:40, Keane, Erich via
cfe-dev wrote:<br>
</div>
<blockquote type="cite"
cite="mid:DM6PR11MB44274BD9853A1E709845E8F9F0B59@DM6PR11MB4427.namprd11.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">Presumably, yes, we could hit that limit,
and that limit is closer than we’d think.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Based on my understanding though, we
currently stores the ‘offset’ into the buffer. If we were to
go with your plan, we’d reduce the ‘SourceLocation’ space
significantly, since we’d only have 1 value taken up by
“ALongFunctionName” instead of ~15? Or, at least, only 1
instead of 3 in ‘int’.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I presume that anything short of expanding
the size of SourceLocation is going to be a temporary measure
at best.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">One consideration: What do we think about
making SourceLocation have an underlying type of ‘bitfield’
that is a smaller size than 64 bits? Then at least we could
make SourceRange be a packed value? So make SourceLocation be
40 bits (obviously would still be 64 if stored directly), but
at least we could pack 2 of them into 80 bits for any type
that does some sort of bit-packing?
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Alternatively, what about making
SourceRange (instead of 2 SourceLocation objects) be a
SourceLocation + unsigned? It would make it a little
smaller? Then we could do similar optimizations over time to
the ASTNodes. That is, all of the TypeLoc types could just
store things like open/close parens as
offsets-from-the-original rather than SourceLocations, then
calculate them when needed? I would assume anything that
required re-calculation would be acceptable, since
SourceLocations are seemingly quite rarely used (except for
the few cases that Richard mentioned below).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My preference is obviously to just make
SourceLocation be uint64_t instead, but the impact on AST size
is going to be significant. So I guess I’m hoping that
Richard Smith can comment and help us figure out how much pain
we are willing to go through for this?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> David Rector
<a class="moz-txt-link-rfc2396E" href="mailto:davrecthreads@gmail.com"><davrecthreads@gmail.com></a> <br>
<b>Sent:</b> Tuesday, February 2, 2021 10:28 AM<br>
<b>To:</b> Keane, Erich <a class="moz-txt-link-rfc2396E" href="mailto:erich.keane@intel.com"><erich.keane@intel.com></a><br>
<b>Cc:</b> clang developer list
<a class="moz-txt-link-rfc2396E" href="mailto:cfe-dev@lists.llvm.org"><cfe-dev@lists.llvm.org></a><br>
<b>Subject:</b> Re: [cfe-dev] [RFC] Clang SourceLocation
overflow<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">An additional consideration: if the
files/buffers are so large that each SourceLocation cannot fit
in 30 bits, does that imply that so many SourceLocations will
be generated that the indices to them won’t be able to fit in
30 bits? That may be the case now that I think of it, in
which case the only solution really would be 64 bit
SourceLocations, or something still more creative/costly.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Feb 2, 2021, at 11:53 AM, Keane,
Erich <<a href="mailto:erich.keane@intel.com"
moz-do-not-send="true">erich.keane@intel.com</a>>
wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Huh, that is an interesting idea!
The issue ends up being the callers of getOffset
though, since it is a previate method.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">That said, I wonder if the extra
bit-use is just putting off the inevitable, and we
should just use the vector in SourceManager for
everything. My initial thought is that we could do
away with the ‘IsMacro’ bit as well, and just encode
that in the vector too. It makes isFileID and
isMacroID require the SourceManager as well, but I
wonder if that is OK?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<div>
<p class="MsoNormal"><b>From:</b><span
class="apple-converted-space"> </span>David
Rector <<a
href="mailto:davrecthreads@gmail.com"
moz-do-not-send="true">davrecthreads@gmail.com</a>><span
class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Tuesday,
February 2, 2021 8:46 AM<br>
<b>To:</b><span class="apple-converted-space"> </span>Keane,
Erich <<a href="mailto:erich.keane@intel.com"
moz-do-not-send="true">erich.keane@intel.com</a>><br>
<b>Cc:</b><span class="apple-converted-space"> </span>clang
developer list <<a
href="mailto:cfe-dev@lists.llvm.org"
moz-do-not-send="true">cfe-dev@lists.llvm.org</a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re:
[cfe-dev] [RFC] Clang SourceLocation overflow<o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">A possible solution not
mentioned: reserve a bit in SourceLocation for
"IsBigLoc". When IsBigLoc = true, the remaining 30
bits of the "ID" field is not an offset but an index
into a vector of uintptr_ts in SourceManager, each
holding 64-bit offset data.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Only a few changes to private
methods and isolated details would be needed I
think: e.g. method `unsigned
SourceLocation::getOffset()` would become `uintptr_t
SourceLocation::getOffset(const SourceManager
&SM)`, and would dereference and return the
larger data when IsBigLoc. And more
generally `unsigned` types should be changed to
`uintptr_t` everywhere 32-bit encodings are
currently assumed (mostly in SourceManager).<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">This might be easier and less
intrusive than allowing 64-bit SourceLocations
depending on a build flag, if I understand that
proposal correctly, since that would require
templating virtually everything with the
SourceLocation type.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">On Feb 2, 2021, at 9:21 AM,
Keane, Erich via cfe-dev <<a
href="mailto:cfe-dev@lists.llvm.org"
moz-do-not-send="true">cfe-dev@lists.llvm.org</a>>
wrote:<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">Original thread here:<span
class="apple-converted-space"> </span><a
href="https://lists.llvm.org/pipermail/cfe-dev/2019-October/063459.html"
moz-do-not-send="true"><span
style="color:#0563C1">https://lists.llvm.org/pipermail/cfe-dev/2019-October/063459.html</span></a><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">I’m bringing this back up
since we have a reproduction of this in Boost
now. We haven’t finished analyzing what boost
is doing, but simply doing an include of:<o:p></o:p></p>
</div>
</div>
<pre>#include <libs/vmd/test/test_doc_modifiers_return_type.cxx><o:p></o:p></pre>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Now causes us to run out of
source locations, hitting:<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">/llvm/clang/lib/Basic/SourceManager.cpp:680:
clang::SourceLocation
clang::SourceManager::createExpansionLocImpl(const
clang::SrcMgr::ExpansionInfo &, unsigned
int, int, unsigned int): Assertion
`NextLocalOffset + TokLength + 1 >
NextLocalOffset && NextLocalOffset +
TokLength + 1 <= CurrentLoadedOffset
&& "Ran out of source locations!"'
failed.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">From the last discussion,
it seems that increasing our source-location
size isn’t really acceptable due to how much
it is stored in the AST( Multiple times per
node), and giving up on location isn’t viable
either. Additionally, the source
location/source manager layout is quite
complex and I don’t quite understand it yet,
so I don’t have a good way of suggesting an
alternative.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">SO, I’d like to re-start
the discussion into this, we need to find a
way to make our compiler able to support more
source-locations, as I can’t imagine this is
going to be the only time we run into this.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">-Erihc<span
class="apple-converted-space"> </span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">>>>>>>>>>>>>>>>>>>>> <o:p></o:p></p>
</div>
</div>
<pre>Hi Matt.<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>Thanks for the offer. Whenever you’re back and have a moment is fine by me, I don’t think we’re in a massive hurry.<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>Thanks,<o:p></o:p></pre>
<pre>Christof<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>From: Matt Asplund <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">mwasplund at gmail.com</span></a>><o:p></o:p></pre>
<pre>Sent: 10 October 2019 14:09<o:p></o:p></pre>
<pre>To: Christof Douma <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">Christof.Douma at arm.com</span></a>><o:p></o:p></pre>
<pre>Cc: Richard Smith <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">richard at metafoo.co.uk</span></a>>; Mikhail Maltsev <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">Mikhail.Maltsev at arm.com</span></a>>; Clang Dev <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">cfe-dev at lists.llvm.org</span></a>>; nd <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">nd at arm.com</span></a>><o:p></o:p></pre>
<pre>Subject: Re: [cfe-dev] [RFC] Clang SourceLocation overflow<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>Sure, I can share my changes. In general my changes are very localized to the code paths I was hitting oveflow issues and tried to keep as much as possible using 32bit encodings. Is there an ETA for when you will start investigating, I am out of town for the next week but if needed can get access to my desktop.<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>-Matt<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>On Thu, Oct 10, 2019, 2:13 AM Christof Douma <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">Christof.Douma at arm.com</span></a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">Christof.Douma at arm.com</span></a>>> wrote:<o:p></o:p></pre>
<pre>Hi Richard.<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>Thanks for the clarification, I certainly had not realized that location tracking was needed for correctness of clang. Decoupling this sounds like a painful processes, and the only thing we get is errors without any location information. Sounds like not a great trade-off. We’ll go experiment a bit with the size impact of using 64-bits and will attempt to take the route of a cmake configuration option.<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>If there are people that have already done some experiments on the size impact of 64-bits location pointers, we’re welcome your insights. @Matt Asplund<mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">mwasplund at gmail.com</span></a>> I believe you said that you’ve done some prototyping, is there something you can share?<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>Thanks,<o:p></o:p></pre>
<pre>Christof<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>From: Richard Smith <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">richard at metafoo.co.uk</span></a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">richard at metafoo.co.uk</span></a>>><o:p></o:p></pre>
<pre>Sent: 08 October 2019 19:53<o:p></o:p></pre>
<pre>To: Christof Douma <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">Christof.Douma at arm.com</span></a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">Christof.Douma at arm.com</span></a>>><o:p></o:p></pre>
<pre>Cc: Mikhail Maltsev <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">Mikhail.Maltsev at arm.com</span></a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">Mikhail.Maltsev at arm.com</span></a>>>; Clang Dev <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">cfe-dev at lists.llvm.org</span></a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">cfe-dev at lists.llvm.org</span></a>>>; nd <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">nd at arm.com</span></a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">nd at arm.com</span></a>>><o:p></o:p></pre>
<pre>Subject: Re: [cfe-dev] [RFC] Clang SourceLocation overflow<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>On Tue, 8 Oct 2019, 10:42 Christof Douma via cfe-dev, <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">cfe-dev at lists.llvm.org</span></a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">cfe-dev at lists.llvm.org</span></a>>> wrote:<o:p></o:p></pre>
<pre>Hi Richard, Paul and other.<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>Thanks for the input so far. I wanted to point out that it’s not our code-base. Rather, we’re seeing more use of the LLVM technology in the automotive market and as usual we’re faced with existing code bases that are tried and tested with other toolchains (gcc or others) and when LLVM comes along things don’t always work directly.<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>We’ve suggested better ways of structuring their code and your suggestions are certainly good input. However, legacy code is especially sticky in any market that has to handle ‘safety’ concerns, like automotive, aerospace and medical markets. Code changes are pretty expensive in those fields. So while I hope that over time we see more sensible coding structures, I don’t expect that to happen any time soon. In the mean time, we’re searching for a solution for this coding pattern that doesn’t play well with clang.<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>Hope that gave some more background of where this question comes from.<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>Do all options that were suggested by Mikhail really require fundamental restructuring of major parts of clang? This surprised me, I had expected that the option 2 to be possible without a complete overhaul. (2 is “Track until an overflow occurs after that make the lexer output the <invalid location> special value for all subsequent tokens.”)<o:p></o:p></pre>
<pre>Clang uses source locations as part of the semantic representation of the AST in some cases (simple example: some forms of initialization might use parens or not, with different semantics, and we distinguish between them based on whether we have paren locations; there are also some checks that look at which of two source locations came first when determining which warnings or errors to produce, and so on). Maybe we could go through and fix all of those, but that's still a very large task and it'd be hard to check we got them all.<o:p></o:p></pre>
<pre>Not nice user experience but maybe doable? I was hoping there was something slightly better that still works without a major restructuring (maybe something that at least gives a rough location or something that only gives the location of the error and not the include stack under an option or using some kind of heuristic to detect that things go haywire).<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>As an alternative, I was curious if it would be possible and acceptable to make the switch between 32-bit and 64-bit location tracking a build-time/cmake decision? I’ve not done any estimation on the memory size growth, so maybe this is a dead end.<o:p></o:p></pre>
<pre>If someone is prepared to do the work to add and maintain this build mode, I think this might be a feasible option.<o:p></o:p></pre>
<pre>Thanks,<o:p></o:p></pre>
<pre>Christof<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>From: cfe-dev <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">cfe-dev-bounces at lists.llvm.org</span></a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">cfe-dev-bounces at lists.llvm.org</span></a>>> On Behalf Of Richard Smith via cfe-dev<o:p></o:p></pre>
<pre>Sent: 07 October 2019 20:36<o:p></o:p></pre>
<pre>To: Mikhail Maltsev <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">Mikhail.Maltsev at arm.com</span></a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">Mikhail.Maltsev at arm.com</span></a>>><o:p></o:p></pre>
<pre>Cc: nd <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">nd at arm.com</span></a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">nd at arm.com</span></a>>>; <a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">cfe-dev at lists.llvm.org</span></a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">cfe-dev at lists.llvm.org</span></a>><o:p></o:p></pre>
<pre>Subject: Re: [cfe-dev] [RFC] Clang SourceLocation overflow<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>On Wed, 2 Oct 2019 at 09:26, Mikhail Maltsev via cfe-dev <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">cfe-dev at lists.llvm.org</span></a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">cfe-dev at lists.llvm.org</span></a>>> wrote:<o:p></o:p></pre>
<pre>Hi all,<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>We are experiencing a problem with Clang SourceLocation overflow.<o:p></o:p></pre>
<pre>Currently source locations are 32-bit values, one bit is a flag, which gives<o:p></o:p></pre>
<pre>a source location space of 2^31 characters.<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>When the Clang lexer processes an #include directive it reserves the total size<o:p></o:p></pre>
<pre>of the file being included in the source location space. An overflow can occur<o:p></o:p></pre>
<pre>if a large file (which does not have include guards by design) is included many<o:p></o:p></pre>
<pre>times into a single TU.<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>The pattern of including a file multiple times is for example required by<o:p></o:p></pre>
<pre>the AUTOSAR standard [1], which is widely used in the automotive industry.<o:p></o:p></pre>
<pre>Specifically the pattern is described in the Specification of Memory Mapping [2]:<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>Section 8.2.1, MEMMAP003:<o:p></o:p></pre>
<pre>"The start and stop symbols for section control are configured with section<o:p></o:p></pre>
<pre>identifiers defined in MemMap.h [...] For instance:<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>#define EEP_START_SEC_VAR_16BIT<o:p></o:p></pre>
<pre>#include "MemMap.h"<o:p></o:p></pre>
<pre>static uint16 EepTimer;<o:p></o:p></pre>
<pre>static uint16 EepRemainingBytes;<o:p></o:p></pre>
<pre>#define EEP_STOP_SEC_VAR_16BIT<o:p></o:p></pre>
<pre>#include "MemMap.h""<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>Section 8.2.2, MEMMAP005:<o:p></o:p></pre>
<pre>"The file MemMap.h shall provide a mechanism to select different code, variable<o:p></o:p></pre>
<pre>or constant sections by checking the definition of the module specific memory<o:p></o:p></pre>
<pre>allocation key words for starting a section [...]"<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>In practice MemMap.h can reach several MBs and can be included several thousand<o:p></o:p></pre>
<pre>times causing an overflow in the source location space.<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>The problem does not occur with GCC because it tracks line numbers rather than<o:p></o:p></pre>
<pre>file offsets. Column numbers are tracked separately and are optional. I.e., in<o:p></o:p></pre>
<pre>GCC a source location can be either a (line+column) tuple packed into 32 bits or<o:p></o:p></pre>
<pre>(when the line number exceeds a certain threshold) a 32-bit line number.<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>We are looking for an acceptable way of resolving the problem and propose the<o:p></o:p></pre>
<pre>following approaches for discussion:<o:p></o:p></pre>
<pre>1. Use 64 bits for source location tracking.<o:p></o:p></pre>
<pre>2. Track until an overflow occurs after that make the lexer output<o:p></o:p></pre>
<pre> the <invalid location> special value for all subsequent tokens.<o:p></o:p></pre>
<pre>3. Implement an approach similar to the one used by GCC and start tracking line<o:p></o:p></pre>
<pre> numbers instead of file offsets after a certain threshold. Resort to (2)<o:p></o:p></pre>
<pre> when even line numbers overflow.<o:p></o:p></pre>
<pre>4. (?) Detect the multiple inclusion pattern and track it differently (for now<o:p></o:p></pre>
<pre> we don't have specific ideas on how to implement this)<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>Is any of these approaches viable? What caveats should we expect? (we already<o:p></o:p></pre>
<pre>know about static_asserts guarding the sizes of certain class fields which start<o:p></o:p></pre>
<pre>failing in the first approach).<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>Other suggestions are welcome.<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>I don't think any of the above approaches are reasonable; they would all require fundamental restructuring of major parts of Clang, an efficiency or memory size hit for all other users of Clang, or some combination of those.<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>Your code pattern seems unreasonable; including a multi-megabyte file thousands of times is not a good idea. Can you split out parts of MemMap.h into a separate header that is only included once, and keep only the parts that actually change on repeated inclusion in MemMap.h itself?<o:p></o:p></pre>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>cfe-dev mailing list<o:p></o:p></pre>
<pre><a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">cfe-dev at lists.llvm.org</span></a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">cfe-dev at lists.llvm.org</span></a>><o:p></o:p></pre>
<pre><a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" moz-do-not-send="true"><span style="color:#0563C1">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</span></a><o:p></o:p></pre>
<pre>-------------- next part --------------<o:p></o:p></pre>
<pre>An HTML attachment was scrubbed...<o:p></o:p></pre>
<pre>URL: <<a href="http://lists.llvm.org/pipermail/cfe-dev/attachments/20191010/bc5ff8cd/attachment.html" moz-do-not-send="true"><span style="color:#0563C1">http://lists.llvm.org/pipermail/cfe-dev/attachments/20191010/bc5ff8cd/attachment.html</span></a>><o:p></o:p></pre>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:9.0pt;font-family:"Helvetica",sans-serif">_______________________________________________<br>
cfe-dev mailing list<br>
</span><a href="mailto:cfe-dev@lists.llvm.org"
moz-do-not-send="true"><span
style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#0563C1">cfe-dev@lists.llvm.org</span></a><span
style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
</span><a
href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev"
moz-do-not-send="true"><span
style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#0563C1">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</span></a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
cfe-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>
<a class="moz-txt-link-freetext" href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a>
</pre>
</blockquote>
</body>
</html>