<div dir="auto">I don't want to distract from your question, but wanted to add that I have been seeing source location overflow issues for many months when using clangs implementation of c++20 modules. I have a personal branch where I have made a partial conversion over to 64 bit source locations for test purposes.<div dir="auto"><br></div><div dir="auto">-Matt</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Oct 2, 2019, 9:26 AM Mikhail Maltsev via cfe-dev <<a href="mailto:cfe-dev@lists.llvm.org">cfe-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br>
<br>
We are experiencing a problem with Clang SourceLocation overflow.<br>
Currently source locations are 32-bit values, one bit is a flag, which gives<br>
a source location space of 2^31 characters.<br>
<br>
When the Clang lexer processes an #include directive it reserves the total size<br>
of the file being included in the source location space. An overflow can occur<br>
if a large file (which does not have include guards by design) is included many<br>
times into a single TU.<br>
<br>
The pattern of including a file multiple times is for example required by<br>
the AUTOSAR standard [1], which is widely used in the automotive industry.<br>
Specifically the pattern is described in the Specification of Memory Mapping [2]:<br>
<br>
Section 8.2.1, MEMMAP003:<br>
"The start and stop symbols for section control are configured with section<br>
identifiers defined in MemMap.h [...] For instance:<br>
<br>
#define EEP_START_SEC_VAR_16BIT<br>
#include "MemMap.h"<br>
static uint16 EepTimer;<br>
static uint16 EepRemainingBytes;<br>
#define EEP_STOP_SEC_VAR_16BIT<br>
#include "MemMap.h""<br>
<br>
Section 8.2.2, MEMMAP005:<br>
"The file MemMap.h shall provide a mechanism to select different code, variable<br>
or constant sections by checking the definition of the module specific memory<br>
allocation key words for starting a section [...]"<br>
<br>
In practice MemMap.h can reach several MBs and can be included several thousand<br>
times causing an overflow in the source location space.<br>
<br>
The problem does not occur with GCC because it tracks line numbers rather than<br>
file offsets. Column numbers are tracked separately and are optional. I.e., in<br>
GCC a source location can be either a (line+column) tuple packed into 32 bits or<br>
(when the line number exceeds a certain threshold) a 32-bit line number.<br>
<br>
We are looking for an acceptable way of resolving the problem and propose the<br>
following approaches for discussion:<br>
1. Use 64 bits for source location tracking.<br>
2. Track until an overflow occurs after that make the lexer output<br>
the <invalid location> special value for all subsequent tokens.<br>
3. Implement an approach similar to the one used by GCC and start tracking line<br>
numbers instead of file offsets after a certain threshold. Resort to (2)<br>
when even line numbers overflow.<br>
4. (?) Detect the multiple inclusion pattern and track it differently (for now<br>
we don't have specific ideas on how to implement this)<br>
<br>
Is any of these approaches viable? What caveats should we expect? (we already<br>
know about static_asserts guarding the sizes of certain class fields which start<br>
failing in the first approach).<br>
<br>
Other suggestions are welcome.<br>
<br>
[1]. <a href="https://www.autosar.org" rel="noreferrer noreferrer" target="_blank">https://www.autosar.org</a><br>
[2].<br>
<a href="https://www.autosar.org/fileadmin/user_upload/standards/classic/3-0/AUTOSAR_SWS_MemoryMapping.pdf" rel="noreferrer noreferrer" target="_blank">https://www.autosar.org/fileadmin/user_upload/standards/classic/3-0/AUTOSAR_SWS_MemoryMapping.pdf</a><br>
<br>
-- <br>
Regards,<br>
Mikhail Maltsev<br>
_______________________________________________<br>
cfe-dev mailing list<br>
<a href="mailto:cfe-dev@lists.llvm.org" target="_blank" rel="noreferrer">cfe-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev" rel="noreferrer noreferrer" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</a><br>
</blockquote></div>