[PATCH] D131247: [ELF] Parallelize writes of different OutputSections

[Fangrui Song via Phabricator via llvm-commits]

MaskRay added a comment.

In D131247#3723722 <https://reviews.llvm.org/D131247#3723722>, @andrewng wrote:

> I've looked a bit more at the Windows performance degradation and have come up with the following code for `taskSize` and `asyncParallelFor`:
>
>   size_t tasks = size / (4 * 1024 * 1024);
>   size_t taskSize = tasks ? sections.size() / tasks : sections.size();
>   asyncParallelFor(tg, std::max<size_t>(1, taskSize), 0, sections.size(), fn);
>
> The `4MB` is somewhat arbitrary but this appears to work OK on my Windows PC and mostly eliminates the performance degradation that I've seen so far. In fact there's a ~3% improvement for `mozilla` from `lld-speed-test.tar.xz`.
>
> I've also tried to test on Linux, although only with an Ubuntu 22.04.1 VM on my Windows PC. I seem to see a similar performance degradation for `scylla` and `mozilla` (and the UE4 based link too). @MaskRay, could you please try testing `scylla` and `mozilla` to see if you can reproduce the performance degradation? The above patch also improves the situation for my setup and actually results in performance improvements for the problematic test cases.
>
> Not really too sure what the next steps should be for this review. Parallel optimisations of this nature are always going to be somewhat tricky across platforms.

I confirm the scylladb regression on a Debian testing machine, using the previous sharding strategy. Thanks for suggesting the 4MiB task size strategy. I have switched to the strategy and it fixes the regression.

For the new Parallel API, I use `std::function<void()>` and place it in `.cpp` to avoid template code size bloat. I have tested that there is no observable slowdown compared with `FuncTy Fn`.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D131247/new/

https://reviews.llvm.org/D131247