<div dir="ltr">The increase in memory usage would be a darn shame. Most of the complexity of the source location machinery exists just to save these four bytes. If we used 64-bit values, source locations could easily be pointers into mapped files or something simple.<div><br></div><div>Can we detect files with pathologically many source locations, and collapse them to one source location, or something like that?<br>For a 2+GB file, it's not unreasonable to say "error, somewhere in this file, you figure it out, it's your own fault for generating source code this large".</div><div>I'm sure this would break some invariants, but it's worth exploring before increasing memory usage.<br><br></div><div>Also, what is the current failure mode when source locations are exhausted? Does clang print a useful error message instructing the user to reduce the size of their pre-processed input? That seems like a good place to start.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Oct 7, 2019 at 2:54 AM Oliver Stannard via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I think that we can rule out option 2, given that you've found a pattern used in real-world code which hits this limit. We'd like to avoid giving <invalid location> to any real users.<br><br>I'm not familiar with how GCC does source locations, but this sounds like it's just delaying the problem by a factor of the average line length? If so, adding two different thresholds at which line number accuracy is degraded doesn't seem worth it.<br><br>I also don't think option 4 is viable, as it would increase complexity a lot, and would only work for this exact pattern. I've also seen bug reports from customers trying to pre-process (mechanically generated) assembly files >2GB, which this solution wouldn't help with.<br><br>That leaves option 1, which looks like the best solution to me. The obvious concern with this one is the increase in memory consumption, since there will be a large number of these objects (one per token/AST node?). @Matt: do you have any memory consumption numbers from your prototype whoch could help here?<br><div><br></div><div>Oliver</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 3 Oct 2019 at 17:23, Matt Asplund via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">I don't want to distract from your question, but wanted to add that I have been seeing source location overflow issues for many months when using clangs implementation of c++20 modules. I have a personal branch where I have made a partial conversion over to 64 bit source locations for test purposes.<div dir="auto"><br></div><div dir="auto">-Matt</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Oct 2, 2019, 9:26 AM Mikhail Maltsev via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi all,<br>
<br>
We are experiencing a problem with Clang SourceLocation overflow.<br>
Currently source locations are 32-bit values, one bit is a flag, which gives<br>
a source location space of 2^31 characters.<br>
<br>
When the Clang lexer processes an #include directive it reserves the total size<br>
of the file being included in the source location space. An overflow can occur<br>
if a large file (which does not have include guards by design) is included many<br>
times into a single TU.<br>
<br>
The pattern of including a file multiple times is for example required by<br>
the AUTOSAR standard [1], which is widely used in the automotive industry.<br>
Specifically the pattern is described in the Specification of Memory Mapping [2]:<br>
<br>
Section 8.2.1, MEMMAP003:<br>
"The start and stop symbols for section control are configured with section<br>
identifiers defined in MemMap.h [...] For instance:<br>
<br>
#define EEP_START_SEC_VAR_16BIT<br>
#include "MemMap.h"<br>
static uint16 EepTimer;<br>
static uint16 EepRemainingBytes;<br>
#define EEP_STOP_SEC_VAR_16BIT<br>
#include "MemMap.h""<br>
<br>
Section 8.2.2, MEMMAP005:<br>
"The file MemMap.h shall provide a mechanism to select different code, variable<br>
or constant sections by checking the definition of the module specific memory<br>
allocation key words for starting a section [...]"<br>
<br>
In practice MemMap.h can reach several MBs and can be included several thousand<br>
times causing an overflow in the source location space.<br>
<br>
The problem does not occur with GCC because it tracks line numbers rather than<br>
file offsets. Column numbers are tracked separately and are optional. I.e., in<br>
GCC a source location can be either a (line+column) tuple packed into 32 bits or<br>
(when the line number exceeds a certain threshold) a 32-bit line number.<br>
<br>
We are looking for an acceptable way of resolving the problem and propose the<br>
following approaches for discussion:<br>
1. Use 64 bits for source location tracking.<br>
2. Track until an overflow occurs after that make the lexer output<br>
the <invalid location> special value for all subsequent tokens.<br>
3. Implement an approach similar to the one used by GCC and start tracking line<br>
numbers instead of file offsets after a certain threshold. Resort to (2)<br>
when even line numbers overflow.<br>
4. (?) Detect the multiple inclusion pattern and track it differently (for now<br>
we don't have specific ideas on how to implement this)<br>
<br>
Is any of these approaches viable? What caveats should we expect? (we already<br>
know about static_asserts guarding the sizes of certain class fields which start<br>
failing in the first approach).<br>
<br>
Other suggestions are welcome.<br>
<br>
[1]. <a href="https://www.autosar.org" rel="noreferrer noreferrer" target="_blank">https://www.autosar.org</a><br>
[2].<br>
<a href="https://www.autosar.org/fileadmin/user_upload/standards/classic/3-0/AUTOSAR_SWS_MemoryMapping.pdf" rel="noreferrer noreferrer" target="_blank">https://www.autosar.org/fileadmin/user_upload/standards/classic/3-0/AUTOSAR_SWS_MemoryMapping.pdf</a><br>
<br>
-- <br>
Regards,<br>
Mikhail Maltsev<br>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" rel="noreferrer" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
</blockquote></div>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
</blockquote></div>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank">cfe-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
</blockquote></div>