[cfe-dev] PCH/preamble files that exceeds 512M

[David Blaikie via cfe-dev]

Subscribing because I'm interested and adding Richard for visibility.

On Tue, Mar 17, 2020 at 9:46 AM Dmitry Polukhin via cfe-dev <
cfe-dev at lists.llvm.org> wrote:

> Hi all,
>
> I worked on crashes and asserts on clangd and found that Clang doesn't
> support PCH files that exceed 512M. It happens because PCH uses *bit*
> offsets from the beginning of the file for some indexes and other data
> structures. The simplest example is SOURCE_LOCATION_OFFSETS in source
> manager block. But there are other similar cases. I identified some cases
> and made them 64bit to double check my hypotheses. It indeed helped in my
> case but I see other places where uint32_t is used for storing bit offsets
> in the file. I see two possible approaches to fix this issue:
>
> 1. Use uint64_t in all places where Clang needs bit offset from the
> beginning of the files. It is relatively straight forwarded approach but it
> increases file size even if uint32_t is enough. In my case I saw about 4%
> increase for 700Mb preamble file. Such increase sounds modest to me and I
> implemented it in https://reviews.llvm.org/D76295
>
> 2. Store offsets from the beginning of corresponding data structure i.e.
> it will give 512M size limit on individual blocks instead of whole file. It
> won't increase file size much bit will add some complication of
> loading/storing logic and may still require 64 bit offsets if it is not
> possible to find good anchors for relative offsets.
>
> The question is which approach do you think is the best?
>
> Dmitry
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20200317/2a10e82e/attachment-0001.html>