[cfe-dev] PCH/preamble files that exceeds 512M

[Dmitry Polukhin via cfe-dev]

Hi all,

I worked on crashes and asserts on clangd and found that Clang doesn't
support PCH files that exceed 512M. It happens because PCH uses *bit*
offsets from the beginning of the file for some indexes and other data
structures. The simplest example is SOURCE_LOCATION_OFFSETS in source
manager block. But there are other similar cases. I identified some cases
and made them 64bit to double check my hypotheses. It indeed helped in my
case but I see other places where uint32_t is used for storing bit offsets
in the file. I see two possible approaches to fix this issue:

1. Use uint64_t in all places where Clang needs bit offset from the
beginning of the files. It is relatively straight forwarded approach but it
increases file size even if uint32_t is enough. In my case I saw about 4%
increase for 700Mb preamble file. Such increase sounds modest to me and I
implemented it in https://reviews.llvm.org/D76295

2. Store offsets from the beginning of corresponding data structure i.e. it
will give 512M size limit on individual blocks instead of whole file. It
won't increase file size much bit will add some complication of
loading/storing logic and may still require 64 bit offsets if it is not
possible to find good anchors for relative offsets.

The question is which approach do you think is the best?

Dmitry
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20200317/c24819d2/attachment.html>