[PATCH] D155579: [Windows] Avoid using FileIndex for unique IDs on network mounts

Mon Aug 21 12:43:58 PDT 2023

mstorsjo added a comment.

In D155579#4604241 <https://reviews.llvm.org/D155579#4604241>, @aganea wrote:

> Unfortunately in our case the situation is even worse than what Martin was suggesting. My timings are 2x slower after this patch than without. Tested with a stage2 clang-cl built with ThinLTO and all optimizations on.
>
> Build times for our game project, which is based on UE 5.2, on a 32c/64t Threadripper:
>
> - before patch: 8 min 13 sec
> - after patch: 15 min 20 sec
>
> Same project, clang-scan-deps:
>
> - before patch: 1 min 48 sec
> - after patch: 3 min 47 sec

Ouch, that's quite seriously bad. I had expected that the overhead would be noticeable in something like clang-scan-deps, but for actual compilation where most effort is spent on other, actually compute intensive things, that's really quite spectacular.

> It seems one of root issues is the call to `GetVolumePathNameW` inside `is_local_internal`, which is absolutely terrible. It generates 12 (!!!) separate kernel calls for each folder component in the path. The deeper your files are, the longer it takes to execute.

Oh, wow, that explains things...

> One option like I was suggesting would be to hide this new behavior behind a (disabled) option, and tell users about it. But I'm not sure how useful it would be.

Yeah that's probably not too useful in the long run. When you hit this issue, you get extremely confusing error behaviours, to the point that I believe we can't diagnose and suggest the option to the user. I wouldn't expect any regular user to figure out to enable the option really. So whatever we do, it should probably work pretty much automatically.

You who might have more use of nontrivial Windows build scenarios - how much impact would it be to go all in on the new path canonicalization + hash approach, i.e. ditching the file index entirely? Performance wise I would believe that it would be no significant change to before. The only thing that matters probably is how it behaves wrt symlinks/hardlinks, if those are present and in use.

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D155579/new/

https://reviews.llvm.org/D155579