<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">A possible solution not mentioned: reserve a bit in SourceLocation for "IsBigLoc". When IsBigLoc = true, the remaining 30 bits of the "ID" field is not an offset but an index into a vector of uintptr_ts in SourceManager, each holding 64-bit offset data.</div><div style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">Only a few changes to private methods and isolated details would be needed I think: e.g. method `unsigned SourceLocation::getOffset()` would become `uintptr_t SourceLocation::getOffset(const SourceManager &SM)`, and would dereference and return the larger data when IsBigLoc. And more generally `unsigned` types should be changed to `uintptr_t` everywhere 32-bit encodings are currently assumed (mostly in SourceManager).</div><div style="margin: 0px; font-stretch: normal; line-height: normal; min-height: 14px;" class=""><br class=""></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class="">This might be easier and less intrusive than allowing 64-bit SourceLocations depending on a build flag, if I understand that proposal correctly, since that would require templating virtually everything with the SourceLocation type.</div><div><br class=""><blockquote type="cite" class=""><div class="">On Feb 2, 2021, at 9:21 AM, Keane, Erich via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" class="">cfe-dev@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Original thread here:<span class="Apple-converted-space"> </span><a href="https://lists.llvm.org/pipermail/cfe-dev/2019-October/063459.html" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">https://lists.llvm.org/pipermail/cfe-dev/2019-October/063459.html</a><o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I’m bringing this back up since we have a reproduction of this in Boost now. We haven’t finished analyzing what boost is doing, but simply doing an include of:<o:p class=""></o:p></div><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">#include <libs/vmd/test/test_doc_modifiers_return_type.cxx><o:p class=""></o:p></pre><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Now causes us to run out of source locations, hitting:<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">/llvm/clang/lib/Basic/SourceManager.cpp:680: clang::SourceLocation clang::SourceManager::createExpansionLocImpl(const clang::SrcMgr::ExpansionInfo &, unsigned int, int, unsigned int): Assertion `NextLocalOffset + TokLength + 1 > NextLocalOffset && NextLocalOffset + TokLength + 1 <= CurrentLoadedOffset && "Ran out of source locations!"' failed.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">From the last discussion, it seems that increasing our source-location size isn’t really acceptable due to how much it is stored in the AST( Multiple times per node), and giving up on location isn’t viable either. Additionally, the source location/source manager layout is quite complex and I don’t quite understand it yet, so I don’t have a good way of suggesting an alternative.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">SO, I’d like to re-start the discussion into this, we need to find a way to make our compiler able to support more source-locations, as I can’t imagine this is going to be the only time we run into this.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">-Erihc<span class="Apple-converted-space"> </span><o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">>>>>>>>>>>>>>>>>>>>><o:p class=""> </o:p></div><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Hi Matt.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Thanks for the offer. Whenever you’re back and have a moment is fine by me, I don’t think we’re in a massive hurry.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Thanks,<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Christof<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">From: Matt Asplund <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">mwasplund at gmail.com</a>><o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Sent: 10 October 2019 14:09<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">To: Christof Douma <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">Christof.Douma at arm.com</a>><o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Cc: Richard Smith <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">richard at metafoo.co.uk</a>>; Mikhail Maltsev <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">Mikhail.Maltsev at arm.com</a>>; Clang Dev <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">cfe-dev at lists.llvm.org</a>>; nd <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">nd at arm.com</a>><o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Subject: Re: [cfe-dev] [RFC] Clang SourceLocation overflow<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Sure, I can share my changes. In general my changes are very localized to the code paths I was hitting oveflow issues and tried to keep as much as possible using 32bit encodings. Is there an ETA for when you will start investigating, I am out of town for the next week but if needed can get access to my desktop.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">-Matt<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">On Thu, Oct 10, 2019, 2:13 AM Christof Douma <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">Christof.Douma at arm.com</a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">Christof.Douma at arm.com</a>>> wrote:<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Hi Richard.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Thanks for the clarification, I certainly had not realized that location tracking was needed for correctness of clang. Decoupling this sounds like a painful processes, and the only thing we get is errors without any location information. Sounds like not a great trade-off. We’ll go experiment a bit with the size impact of using 64-bits and will attempt to take the route of a cmake configuration option.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">If there are people that have already done some experiments on the size impact of 64-bits location pointers, we’re welcome your insights. @Matt Asplund<mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">mwasplund at gmail.com</a>> I believe you said that you’ve done some prototyping, is there something you can share?<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Thanks,<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Christof<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">From: Richard Smith <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">richard at metafoo.co.uk</a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">richard at metafoo.co.uk</a>>><o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Sent: 08 October 2019 19:53<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">To: Christof Douma <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">Christof.Douma at arm.com</a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">Christof.Douma at arm.com</a>>><o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Cc: Mikhail Maltsev <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">Mikhail.Maltsev at arm.com</a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">Mikhail.Maltsev at arm.com</a>>>; Clang Dev <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">cfe-dev at lists.llvm.org</a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">cfe-dev at lists.llvm.org</a>>>; nd <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">nd at arm.com</a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">nd at arm.com</a>>><o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Subject: Re: [cfe-dev] [RFC] Clang SourceLocation overflow<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">On Tue, 8 Oct 2019, 10:42 Christof Douma via cfe-dev, <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">cfe-dev at lists.llvm.org</a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">cfe-dev at lists.llvm.org</a>>> wrote:<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Hi Richard, Paul and other.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Thanks for the input so far. I wanted to point out that it’s not our code-base. Rather, we’re seeing more use of the LLVM technology in the automotive market and as usual we’re faced with existing code bases that are tried and tested with other toolchains (gcc or others) and when LLVM comes along things don’t always work directly.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">We’ve suggested better ways of structuring their code and your suggestions are certainly good input. However, legacy code is especially sticky in any market that has to handle ‘safety’ concerns, like automotive, aerospace and medical markets. Code changes are pretty expensive in those fields. So while I hope that over time we see more sensible coding structures, I don’t expect that to happen any time soon. In the mean time, we’re searching for a solution for this coding pattern that doesn’t play well with clang.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Hope that gave some more background of where this question comes from.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Do all options that were suggested by Mikhail really require fundamental restructuring of major parts of clang? This surprised me, I had expected that the option 2 to be possible without a complete overhaul. (2 is “Track until an overflow occurs after that make the lexer output the <invalid location> special value for all subsequent tokens.”)<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Clang uses source locations as part of the semantic representation of the AST in some cases (simple example: some forms of initialization might use parens or not, with different semantics, and we distinguish between them based on whether we have paren locations; there are also some checks that look at which of two source locations came first when determining which warnings or errors to produce, and so on). Maybe we could go through and fix all of those, but that's still a very large task and it'd be hard to check we got them all.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Not nice user experience but maybe doable? I was hoping there was something slightly better that still works without a major restructuring (maybe something that at least gives a rough location or something that only gives the location of the error and not the include stack under an option or using some kind of heuristic to detect that things go haywire).<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">As an alternative, I was curious if it would be possible and acceptable to make the switch between 32-bit and 64-bit location tracking a build-time/cmake decision? I’ve not done any estimation on the memory size growth, so maybe this is a dead end.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">If someone is prepared to do the work to add and maintain this build mode, I think this might be a feasible option.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Thanks,<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Christof<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">From: cfe-dev <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">cfe-dev-bounces at lists.llvm.org</a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">cfe-dev-bounces at lists.llvm.org</a>>> On Behalf Of Richard Smith via cfe-dev<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Sent: 07 October 2019 20:36<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">To: Mikhail Maltsev <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">Mikhail.Maltsev at arm.com</a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">Mikhail.Maltsev at arm.com</a>>><o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Cc: nd <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">nd at arm.com</a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">nd at arm.com</a>>>; <a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">cfe-dev at lists.llvm.org</a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">cfe-dev at lists.llvm.org</a>><o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Subject: Re: [cfe-dev] [RFC] Clang SourceLocation overflow<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">On Wed, 2 Oct 2019 at 09:26, Mikhail Maltsev via cfe-dev <<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">cfe-dev at lists.llvm.org</a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">cfe-dev at lists.llvm.org</a>>> wrote:<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Hi all,<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">We are experiencing a problem with Clang SourceLocation overflow.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Currently source locations are 32-bit values, one bit is a flag, which gives<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">a source location space of 2^31 characters.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">When the Clang lexer processes an #include directive it reserves the total size<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">of the file being included in the source location space. An overflow can occur<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">if a large file (which does not have include guards by design) is included many<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">times into a single TU.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">The pattern of including a file multiple times is for example required by<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">the AUTOSAR standard [1], which is widely used in the automotive industry.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Specifically the pattern is described in the Specification of Memory Mapping [2]:<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Section 8.2.1, MEMMAP003:<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">"The start and stop symbols for section control are configured with section<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">identifiers defined in MemMap.h [...] For instance:<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">#define EEP_START_SEC_VAR_16BIT<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">#include "MemMap.h"<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">static uint16 EepTimer;<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">static uint16 EepRemainingBytes;<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">#define EEP_STOP_SEC_VAR_16BIT<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">#include "MemMap.h""<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Section 8.2.2, MEMMAP005:<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">"The file MemMap.h shall provide a mechanism to select different code, variable<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">or constant sections by checking the definition of the module specific memory<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">allocation key words for starting a section [...]"<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">In practice MemMap.h can reach several MBs and can be included several thousand<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">times causing an overflow in the source location space.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">The problem does not occur with GCC because it tracks line numbers rather than<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">file offsets. Column numbers are tracked separately and are optional. I.e., in<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">GCC a source location can be either a (line+column) tuple packed into 32 bits or<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">(when the line number exceeds a certain threshold) a 32-bit line number.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">We are looking for an acceptable way of resolving the problem and propose the<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">following approaches for discussion:<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">1. Use 64 bits for source location tracking.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">2. Track until an overflow occurs after that make the lexer output<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""> the <invalid location> special value for all subsequent tokens.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">3. Implement an approach similar to the one used by GCC and start tracking line<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""> numbers instead of file offsets after a certain threshold. Resort to (2)<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""> when even line numbers overflow.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">4. (?) Detect the multiple inclusion pattern and track it differently (for now<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""> we don't have specific ideas on how to implement this)<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Is any of these approaches viable? What caveats should we expect? (we already<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">know about static_asserts guarding the sizes of certain class fields which start<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">failing in the first approach).<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Other suggestions are welcome.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">I don't think any of the above approaches are reasonable; they would all require fundamental restructuring of major parts of Clang, an efficiency or memory size hit for all other users of Clang, or some combination of those.<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">Your code pattern seems unreasonable; including a multi-megabyte file thousands of times is not a good idea. Can you split out parts of MemMap.h into a separate header that is only included once, and keep only the parts that actually change on repeated inclusion in MemMap.h itself?<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">_______________________________________________<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">cfe-dev mailing list<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">cfe-dev at lists.llvm.org</a><mailto:<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">cfe-dev at lists.llvm.org</a>><o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class=""><a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">-------------- next part --------------<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">An HTML attachment was scrubbed...<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: "Courier New";" class="">URL: <<a href="http://lists.llvm.org/pipermail/cfe-dev/attachments/20191010/bc5ff8cd/attachment.html" style="color: rgb(5, 99, 193); text-decoration: underline;" class="">http://lists.llvm.org/pipermail/cfe-dev/attachments/20191010/bc5ff8cd/attachment.html</a>><o:p class=""></o:p></pre><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">cfe-dev mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><a href="mailto:cfe-dev@lists.llvm.org" style="color: rgb(5, 99, 193); text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">cfe-dev@lists.llvm.org</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" style="color: rgb(5, 99, 193); text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a></div></blockquote></div><br class=""></body></html>