<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/72511>72511</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            check-all misses 3200+ compiler-rt tests in runtimes build
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
            cmake,
            compiler-rt
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
            vitalybuka
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          rorth
      </td>
    </tr>
</table>

<pre>
    As has been reported for a long time (probably since the LLVM 14 release cycle when the runtimes build was first made the `reltest.sh` default), building with `-DLLVM_ENABLE_RUNTIMES=compiler-rt` (which is the default these days) misses a large number of `compiler-rt` tests that *are* run with `-DLLVM_ENABLE_PROJECTS=compiler-rt`.  The issue came up again in Issue #71971 and has apparently never been noticed during the development of the runtimes build (which doesn't speak particularly highly for the diligence of the testing at that point) and completely ignored despite the release tester reports since, massively reducing `compiler-rt` test coverage and missing all sorts of issues in new patches that would be noticed immedidately otherwise.

To provide detail, I've now run two `x86_64-pc-linux-gnu` builds, one projects build and one runtimes build.  Here are the differences in overall test results from just that change (which should be a no-op for test results):
```
-Total Discovered Tests: 110511
-  Skipped :     84 (0.08%)
-  Unsupported      :   3723 (3.37%)
-  Passed : 106411 (96.29%)
-  Expectedly Failed:    292 (0.26%)
+Total Discovered Tests: 107217
+  Skipped          :     48 (0.04%)
+ Unsupported      :   3591 (3.35%)
+  Passed           : 103287 (96.33%)
+  Expectedly Failed:    290 (0.27%)
   Failed           : 1 (0.00%)
```
As one can see, more than 3200 tests are no longer run in a runtimes build.  I've then extracted the names of the tests run from both build's `ninja check-all` output.  Here are the test categrories that are missing in the runtimes build:
```
    836 AddressSanitizer-Unit
      3 Builtins
    215 Builtins-i386-linux
 215 Builtins-x86_64-linux
     70 GwpAsan-Unittest
     12 Interception-Unit
     58 LLVMFuzzer-Unittest
    194 SanitizerCommon-Unit
    275 ScudoStandalone-Unit
    275 ScudoStandalone-Unit-GwpAsanTorture
     79 ThreadSanitizer-Unit
 15 XRay-x86_64-linux           15 out of 29 tests turn UNSUPPORTED
```
In most cases, all tests from a category are missing, while for the `xray` tests about half of the tests changed from `PASS` to `UNSUPPORTED`.

I then looked for the missing tests: in the case of `builtins`, the `lit.*` files that controlled the tests used to live in
```
-test/builtins/lit.site.cfg.py
-test/builtins/Unit/I386LinuxConfig/lit.site.cfg.py
-test/builtins/Unit/X86_64LinuxConfig/lit.site.cfg.py
```
while they are now missing completely: they are neither present in `runtimes-bins` nor in `builtins-bins` where they should be after splitting the object directories for `builtins` from the rest of `compiler-rt`.  Without the `lit.*` files, it's no wonder the `builtins` tests aren't run at all.

For the other missing tests, the `lit.*` files are present, but I don't yet see why the tests aren't run.
</pre>
<img width="1px" height="1px" alt="" src="http://email.email.llvm.org/o/eJyUV1tv2zgW_jXKy4ENibIufvCD28S7WXQ6QePszltBSccWG4oUSMqu59cvDkU5tpsMZoSircXDc_u-cxG3VuwV4irKPkWMHYTj8lQNrzxiLMru7_jgWm1WRhvX3lW6Oa3WFlpuoUJUYLDXxmEDO22Ag9RqD050CBEre6MrXskTWKFqBNcifPny398gWYBBidwi1KdaIhxbVP7YDIouW6gGIRs4cgs7YayDjjejgiiPDUqH1s1tG-UxNLjjg3QRW0bs83hPqD0chWtJeHZPJr8_fF1_-vLw_dvL1-3jbw_PUXpf664XEs3MOFITsfLYiroFYb2doJb-bxEafrIRW0InrEVLcXKzR1BDV6EBvSNLNwrJRVLFHURszQ1GbE3xfeDZ07ff__PwefurZ3OAbYsgrB0Qat4hDD3wPRcKhIJH_zpiaZEsiwS4ajw0vO-5QeXkCRQe0IxYKe1EjQ00g6EUjWEeUOq-Q-UojHcwOCem0WhVxAoHtkf-Cj03TtSD5EaeoBX7Vp48CbxaIcUeCfSglLJBNrkbU9JroQgz7zEFLNGhPIHYK23IRbS9cCPkE1dIB5rAODuSijDviMEHum2wGWoy8z4cUOsDGr5Hb5Ww9C5JCdZr1LsxzZYyq_AIPXd1iwHFox5kAxWe0yi6DhvRcO-4di2ao7A4j-L7KF6Pf2819EYfREOZdlxI8vcxYsWB1Bw9H9xRk78_y_x7vpj19UwKNfyc7dVAfnsQLF3TCknZD6zdhA2FQa-vIZsD_BsNAjcYwNjt0BAaPjCfAynHjBi0g3QWdkZ38GOwAZ665WqPb9jbdgqeg9Iz3Y9IX2ig-ktD1FEehz_-52yrHZdwL6zPPzawpdqI0jUkSZwlSRADeH4VfY8N0BE95YJciOdxGbGMLEyCL8oOfWg8_hlvpAVL6UY6T4ubG0_c2qA5ifNFkpDcMp9T27gSfPjZY-2wkSfYcCGxCc6wJRudYfnljYh9-ovw4oIlxVnwLcDzM0W6KEOkixvlH4aaLZMQanZ7ZQr22kwSp6wsQthp-suljwOPQ-BXOQUIYrdmQiDxlYFrRqytp23NFVgca1h7snIFKYun5kkEVtrPFKr7wXc8_ivbQ0E5miL40xlOYXjqK05yFz3IejWe7ZV27agiYoWlElRC_eBQt1i_zriUVH56cP3gbitqbCbc4d5oI6YGQadTUxHvzbOPCsRTPc1h3TQGrX3mSjjxJ5rZixLuTQQghU-DkE4o-_aWJdn57UykZT72jyBwdRpazOU5PUUM_zr2a8uVN0jBXZwmDB6VQ1Nj74RWtz5lpZ_pm-HPyeHr-8lyAeeAPuuu-0UFKzJ4rodGPzuuGi61wr8pMQtub7Vxg8HLkJawbQ3y5t1cJhn88Y2fbvKxjOJlkhHiRBi2nCb4YBS8fH1-eXr6_dv24f5dBB8VdNpzwqJv1lODDZ2Vj2zR5nTJEhI8tkLieW7SHDD89LY_8Ir8abncXbN4bNDNqD3K46f187O_5WfJpbt5fDWRHscykVq_hqWNlE60dVPnCvyleMJ6U03My2PyO3grhZtHjNIBOyGnSqi1ckZLGapwdHmgluQ0SHFAEOr9UeHJwzZnY2xDFqxwOK93-3l_-lDO48s2j2mZfyFAP2u1E_t_ruAPT4q_oeLG9RFI1-Ip9K3jOatv-w2l9k0EBS0N0Bu0tH8J5bfb0DJm1ZhsUNqEo8nZ89GxRRNsXkzoHW1JtpfCuWnN0xWtDdAIg7UbWxZBfw3rSKZx47Lu3a12DvA_4Vri5AcEIG5QHgtLrfuoVYNnal_aOnf4caWkrkwtVMortm4CP_12dcPSvyAhZTdkdfwocPAIjR5NndDR1IFje7pg56Un87tmlTbLdMnvcJUUcZykyyzO79pVGses2i0ajJcxplnC8nRXFmVcxmVRFAt2J1YsZmmSJHnCspLl8zKL2Y5lizwry6rI4mgRY8eFnEt56Oba7O_8yrkqWJYkd5JXKG34Eqs7_or0EcY-068LIMYPM7MiHbNq2NtoEUthnX3T6oSTuDrPsunLhaYrTfsLbSEBQt1Mq7vByFXrXE8NIWKbiG32wrVDNa91R0UhD9M_s7CXRmwzLtAR2_iA_h8AAP__Ud2Omg">