[cfe-dev] PCH/preamble files that exceeds 512M

[Sam McCall via cfe-dev]

+clangd-dev, this is definitely interesting! We hadn't seen such large
preambles before.

I have to confess from the clangd side we've mostly been reusing the
modules/preamble format without deeply probing into it, so I don't have an
informed opinion (both your options seem reasonable to me, with 2 sounding
nicer but more work).

On Tue, Mar 17, 2020 at 6:18 PM David Blaikie via cfe-dev <
cfe-dev at lists.llvm.org> wrote:

> Subscribing because I'm interested and adding Richard for visibility.
>
> On Tue, Mar 17, 2020 at 9:46 AM Dmitry Polukhin via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
>
>> Hi all,
>>
>> I worked on crashes and asserts on clangd and found that Clang doesn't
>> support PCH files that exceed 512M. It happens because PCH uses *bit*
>> offsets from the beginning of the file for some indexes and other data
>> structures. The simplest example is SOURCE_LOCATION_OFFSETS in source
>> manager block. But there are other similar cases. I identified some cases
>> and made them 64bit to double check my hypotheses. It indeed helped in my
>> case but I see other places where uint32_t is used for storing bit offsets
>> in the file. I see two possible approaches to fix this issue:
>>
>> 1. Use uint64_t in all places where Clang needs bit offset from the
>> beginning of the files. It is relatively straight forwarded approach but it
>> increases file size even if uint32_t is enough. In my case I saw about 4%
>> increase for 700Mb preamble file. Such increase sounds modest to me and I
>> implemented it in https://reviews.llvm.org/D76295
>>
>> 2. Store offsets from the beginning of corresponding data structure i.e.
>> it will give 512M size limit on individual blocks instead of whole file. It
>> won't increase file size much bit will add some complication of
>> loading/storing logic and may still require 64 bit offsets if it is not
>> possible to find good anchors for relative offsets.
>>
>> The question is which approach do you think is the best?
>>
>> Dmitry
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20200317/27973351/attachment.html>