<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/58509>58509</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            check-cxx uses clang-tidy from $PATH instead of built-from-source, which can make the test fail
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          yabinc
      </td>
    </tr>
</table>

<pre>
    I was running check-cxx from df7606a066b75ce55ae4a186c785f996e0985db1 from 2022-09-30. check-cxx fails as below:

Command Output (stdout):
--
$ ":" "RUN: at line 12"
$ "clang-tidy" "/ssd/android/llvm-toolchain-testing/out/llvm-project/libcxx/test/libcxx/clang_tidy.sh.cpp" "--warnings-as-errors=*" "-header-filter=.*" "--config-file=/ssd/android/llvm-toolchain-testing/out/llvm-project/libcxx/test/libcxx/../../.clang-tidy" "--" "-Wweak-vtables" "-Wno-unknown-warning-option" "-nostdinc++" "-I" "/ssd/android/llvm-toolchain-testing/out/stage2/include/c++/v1" "-I" "/ssd/android/llvm-toolchain-testing/out/stage2/include/x86_64-unknown-linux-gnu/c++/v1" "-I" "/ssd/android/llvm-toolchain-testing/out/llvm-project/libcxx/test/support" "-std=c++2b" "-Werror" "-Wall" "-Wextra" "-Wshadow" "-Wundef" "-Wno-unused-command-line-argument" "-Wno-attributes" "-Wno-pessimizing-move" "-Wno-c++11-extensions" "-Wno-noexcept-type" "-Wno-atomic-alignment" "-Wno-user-defined-literals" "-Wno-tautological-compare" "-Wsign-compare" "-Wunused-variable" "-Wunused-parameter" "-Wunreachable-code" "-Wno-unused-local-typedef" "-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER" "-D_LIBCPP_ENABLE_EXPERIMENTAL" "-D_LIBCPP_DISABLE_AVAILABILITY" "-fcoroutines-ts" "-Werror=thread-safety" "-Wuser-defined-warnings" "-fno-modules"
# command output:
/ssd/android/llvm-toolchain-testing/out/stage2/include/c++/v1/__type_traits/decay.h:30:8: error: keyword '__decay' will be made available as an identifier for the remainder of the translation unit [clang-diagnostic-keyword-compat,-warnings-as-errors]
struct __decay {
       ^
/ssd/android/llvm-toolchain-testing/out/stage2/include/c++/v1/__type_traits/is_referenceable.h:28:8: error: keyword '__is_referenceable' will be made available as an identifier for the remainder of the translation unit [clang-diagnostic-keyword-compat,-warnings-as-errors]
struct __is_referenceable : integral_constant<bool,
       ^
/ssd/android/llvm-toolchain-testing/out/stage2/include/c++/v1/__type_traits/make_signed.h:42:8: error: keyword '__make_signed' will be made available as an identifier for the remainder of the translation unit [clang-diagnostic-keyword-compat,-warnings-as-errors]
struct __make_signed {};
       ^
/ssd/android/llvm-toolchain-testing/out/stage2/include/c++/v1/__type_traits/make_unsigned.h:44:8: error: keyword '__make_unsigned' will be made available as an identifier for the remainder of the translation unit [clang-diagnostic-keyword-compat,-warnings-as-errors]
struct __make_unsigned {};
       ^

I found it failed because I had a clang-tidy built from newer source in $PATH. check-cxx seems to only search for clang-tidy from $PATH instead of using the one built from source. It causes two pains:
1) If it finds a clang-tidy built from different source, the test may fail just because of version mismatch.
2) If there is no clang-tidy in $PATH, the test using clang-tidy is skipped without using clang-tidy built from source.

The behavior also surprises me. Because in https://llvm.org/docs/TestingGuide.html, it says:

> lit performs substitution on each RUN line to replace LLVM tool names with the full paths to the executable built for each tool (in $(LLVM_OBJ_ROOT)/$(BuildMode)/bin). This ensures that lit does not invoke any stray LLVM tools in the user’s path during testing.



</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJzVWFuPmzgU_jXkxQIRAgl5yEMyybRZzU0z2e72CRkwwR2wkS_JZH_9HhuSkJluW63abXcUIbCPz-U73zm2J-X5YbZGeyyR0IxRtkVZSbJnN3t5QYXgNcqLydgfY388TidRRqIIkxAP43E2iaNiOh0TfxpHeTpspQM_CFx_6o58r68I00oisJGSiu-d0dzxl47fPa94XWOWo3utGq2QE8RS5VwrJ5ieJF23WxGEIBCY8SAwb4-_38EHwgpVlBE0DMxsXzSrMNu6iuaHboUTXEuZwxNsCk7NW1XtaldxXmUlpsxVRCoAAiasF-10I_gnktlPmkJM8GLk-t_WVGJMebL0sqbpLLruHgsDrXSxdIkQXEhntHSCYxBuSXBOhFvQShEBU15vzs04K-jWTBK76ke573mnxxvQAP_u5Y89wc_uTuG0IvI0yLir2TPje3aM1eWNopwdJRiHpFKWOcHC_trR9b_OiVR4S8w60FnpnBj4j7qvd8MfYuAlHifj8BQoEE6_uFumv7vtr2RM6qbhQh3tALDAi86DID3lxBLt9IWr6jzzogQ-fckS51CUx0_NclJcJlZLkgMNbZmasImLxVbXhKm-HFZK0FSrS1o0REpa078MJWq-I_25zufh0AWPCJPAl4u1jJOXjDTKVYeGXJriNc1cXNEte-0GOCtcCAHcNM5CQeHqQqvCWvGKb2mGKxNVg8VZuQSNbwY7BHZYUEP71-Mgi2tiKvc8IQiG3IIwKMvJZ-CsuDFvAuvBvUxu1ourh4fk_fwpubtPHh7n727nydPHp83qNnm_mi9Xj29kV3fzxc0qWf35sHpc367uNvObNzLL9ZMVmn-Yr2_mi_XNevPxKFRkXADvAC_pqjNSLX1GS1VCLLkrcUHU4RxhH-VjeztphDBrnuu2Qxzb8Qh1FELcdvrzNvD9q_86SQy0CfCcmpiuc5Lhg1eCzZEPj9hsGl2Ec_RMDnsucvB9kiRWEt7QnlYVbFioht6M8A62MJNPs4thhmgOtKMFJQIVXCBVEiRIDb5CH0e8sANgnMkKmzaINKOwtUWLtrPmFG9NRwQOd7ZbzkFsV5_bLKJli5RUQmcKdU4iZ7Jox1H750Sr_w5RKhNBCiIIy4gBxoIbxF8G9_WiXx3n1_4iExFlimyhqySwMwNu0H5GVykgC0p_Xjpq_EwS071IbjMRBl_ORE_-V09Cz1VL-cnSGf1E4lt3NOtjHX4D1scV_wu0j85-FW_7XIOjcGxA4Ic56cOyFPoTbBFojeBwgTA6HyhRquGU214WGNlDSJJrkRGoKgArfJhv3vdvDpKQWiLFEWfVAb6wyEoLS0-j1dWtBTVSwX5lgNLSXGcMXByuBj27rUUPrRWyboKBPUcNQCxPmxJkforWhY0JoJf_GEROC9sgVKcWcG5TBESDBB8sJOiThq8jKuDbjghz3EE1lTVWWem1VoPOKigQAIlEjPfNnjG6sNIG2peTSD7TBs4WQDVVAtPfyrzFo5_SDehOSYl3FLCGAxRHUotGUANWDdAtulDAo1KpxuIWXHfl5nFhCiznmamXTVtx7zQQ2ytVbfqkgVXig3x1FXRGK7jHKdQQATmGxEudwmKlLdHhZ05VCO587W0PaCFIU2Fgz83Nh1tkqhwxOItJG7YFqNBQakD_0rLIjJAXkml7fzliACFaxXY93D9bmOHFaE3uF78lj_f3G3MfNSGaiQUszG_tyc4MphQuOlMPbUpAHg6yWhhOlfZWqlDOiUmkArB2_BnKnAGToYIPZ6-lAdI4Z85VzipwYt-ZTqV1HOVaWCK3OF6kqX0OyGw4Ho-D6XA0jAb5bJRPR1M8AOAqMjvXkmX6N9SNRcU1s-6Z0vuSAkIZ9CfTIM7cM-weaFHNLmmwBfx16kHr6Tjx9lpDpdTmgHgdxZE_HZSzOJ6EUYbjNAzGkyyKJ9EwSElKhtNpFk4KfwBdklRyBr0O2taAzsw_G4Z-4A_jcBhFXpzFo3iEw8LPwzQYh07om75ZeUdODsTM-pBqOKuGfkWlkudJLG3PI0f9cE0ouZgdMGQ3G1hvZ9bVvwE-4JlY">