[llvm] Solve llvm-dwp overflow problem, skipped over 4GB '.dwo' files (PR #71902)

[David Blaikie via llvm-commits]

dwblaikie wrote:

> I see. But according to my current test result, gdb(10.1) can't recognise '.dwp' files emitted after the debug_info section exceeds 4GB, and gdb reports an error about the dwarf version.I'm not sure if it's because GDB10.1 can't rebuild the correct column. But it looks like there's a problem with processing the dwarf header, which makes debugging completely impossible.

Right - if you're using gdb the recommendation would be not to pass this flag, and to make changes to reduce debug info to fit into the current DWARF DWP limit of 4GB per section.

We could add another flag that's "give up and provide a valid but truncated dwp at 4GB" which seems a bit awkward, but perhaps no more/less awkward than the current "create a corrupted index that we know can be recovered from".

Alternatively you could go and work with gdb to do some similar recovery.

(I did just have the thought that our recovery, especially in DWARFv5, could be better - we could tombstone the .debug_info column values in the index (set them to `0xffffffff` at dwp-generation time), then, rather than rescanning the /whole/ .debug_info section, we could skim the .debug_info column in the index, find the highest offset that's not tombstoned, and start scanning from there to only rebuild the overflowed part of the column - would require another coordination between consumer and producer, whereas the current solution (discarding the whole column and rebuilding) only requires the consumer to do something different and works with the old gold-dwp that overflows naturally, and with llvm-dwp that can be explicitly requested to overflow)

> I'm wondering if it's better to change the logic of 'ContinueOnCuIndexOverflow' in this case, or if it's better for me to add a extra flag like 'ContinueWithBestEffort'.

Right, the `ContinueOnCUIndexOverflow` is intentionally doing what it's doing, and doesn't seem like what you want.

What you want is less a `Continue` and more a 'create valid DWP even if it can't include all the inputs'... (which is 'continue' instead of 'error' I guess). Not sure what we'd call that flag.



https://github.com/llvm/llvm-project/pull/71902