<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/60947>60947</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            ClangScanDeps make a lot of stat() calls when they could be cached.
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
            clang:tooling
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          tru
      </td>
    </tr>
</table>

<pre>
    We use clang-scan-deps in Make mode to output C++ files include dependencies for our build system. Recently we debugged why it was slow on specific files and figured out that it makes many `stat` calls which is very expensive on Windows.

There is already a cache in `DependencyScanningFilesystem` to avoid doing more stats than necessary and without this cache it would probably be even slower. But we found that system header misses was never cached and led to many extra stat calls.

This is because of this code here: https://github.com/llvm/llvm-project/blob/219ba2fb7b0ae89101f3c81a47fe4fc4aa80dea4/clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp#L180

Which tries to avoid cache stat failures on files without an extension (assuming them to be directories for the modules in this case).

Unfortunately, this means that C++ std library headers are not cached here. I added some statistics to `DependencyScanningFilesystem` to observe cache misses/hits and saw this on a heavy file:

`Hits: 414274, Misses: 163058`

That's a lot of misses.

If I short-cut the method `shouldCacheStatFailures` to always return `true` we saw the following stats:

`Hits: 537638, Misses: 40345`

Much better. This has a considerable performance impact on Windows.

One approach that I am thinking of is to only avoid caching the misses if Mode == Full, but cache all the misses if Mode == Make. But that doesn't entirely solve the problem because it would still make full mode pretty slow.

Any thoughts on how to improve the current situation?

cc @Bigcheese @benlangmuir @ChuanqiXu9 
</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJyUVUtv47YW_jX05iCGRCl-LLzIA8YNcIML3Jliuj0ijyx2KFLlwx79--JQdiaZdop2k1g2RZ7vSYzRnBzRQdw_ivvnFeY0-HBIIa86r-fDF4IcCZRFd7qLCt2dpimCcfCKXwlGrwmSB5_TlBM8Cfko5CP0xhIvUjZrAk0TOU1OGYrQ-wA-B-iysRriHBONa_g_KXLJznDh5V0-nUjDZZjBJLhghGj9BbyDOJEyvVHXE9Bp6M0pB9I8AqQBE78y4leKMKKbQWyqmDCJTQUKrY1wGYwawEQ4U5iBvk3kojkT7_7FOO0vcS2qZ1E9LH8_DxSIl6MNhHoGBIVqIKZAbKrnG7b5k0LnjDsdebICi89MHvDsjQbtjTvB6AMBzxN5VgeOFMWIYS5QLiYNCwwTb6ckuPhsNUzBd9jZGToCOpMrlFBYw2NOzFrvs9MLAcvpMBBqCjCaGCkWFh2dKSwb63KgJc0TFqLoWwpYZluI-oEFE5mEjhSyH3x_HZL1Z4ZE8wBDSlMUzYOQRyGPJ5OG3K2VH4U8Wnu-_bubgv-NVBLy2FnfCXmU9b5D2XfbrkLa7euq7hu1q7Hd9tT2qkXcVZqwFfJYfMgbGX7xs_fWlOc_y_CXX37XZq2mScjmv_Wueo_zSzFHCuzUN-kWJQozPRqbA0V2y2LBm2bomEH2kncg5A5jzCNLngYaea-OQJtAKvlwy0EaSoLyEpab7JGE3H9g_xfX-5Cyw0R2FvJpWTkSurgofstdTBqs6QIbapE_AgYC59NNdhZrDS-AWpOG6McFmYnJqIL5n5nad5HCma7cLB4T8jiYtKQy4mWZ0jtAnuU8F8LYHu-QiU31H5PYNNDWrdy2jO512a15gHrTVPc7sak-ehGTkNsICNYntuJy_AfKXnp4gTj4kO5USRTBSGnwuhTCwJF64tE_JUzHq6i3vNoLzhECpRxKxlPIxL9d6AqL02atv7C8Jcw_R3XfbDfN7iOqtmra-x9AvWY1QEcpcaRL2gZkhMq7aDQF7CzBRKH3YUSnCMw4oUo_a63_OQKcpuCR7cwWeQEcWRH3laf2PaeZdXR2fmfzq2FvpWF6eOWEi-ZZNM9wzNYylC5f7QRo7d-s5ytiKagygvYUnZDbBOSSCWRniN6eqezA_WZpfGuYt-KLyVhbGh36zJ_4gClQSnPpwA-4H9wMnMfTkIr1Bn9hlGacgr8epHII5BJEkzIm451oju-3UApEWz2akxqIIvFDR457Z8wm8OPTkNH9bn7Ne1jpQ6P3zR5XdKg32219v6v31Wo4tJK2qLBp9E7v93rTtPtmV29w26m66Te4MgdZyaaSsqkbtvl6jxVq2lbthup-X7eirWhEY9dcmmsfTisTY6bDptq325XFjmwst7aUSys2D-nWh5Kv8nAoddvlUxRtZU1M8fteySRLhyd-kTP-zJd64fgtVOXWlDsh928XJ3FH0QyqCNNd06_Xqxzs4V-3f4HDpVEQ_REAAP__K23eCw">