<table border="1" cellspacing="0" cellpadding="8">
    <tr>
        <th>Issue</th>
        <td>
            <a href=https://github.com/llvm/llvm-project/issues/149993>149993</a>
        </td>
    </tr>

    <tr>
        <th>Summary</th>
        <td>
            [libc] Flaky builds on CI
        </td>
    </tr>

    <tr>
      <th>Labels</th>
      <td>
            libc
      </td>
    </tr>

    <tr>
      <th>Assignees</th>
      <td>
      </td>
    </tr>

    <tr>
      <th>Reporter</th>
      <td>
          saturn691
      </td>
    </tr>
</table>

<pre>
    ## What is this problem?

Compiler built in header files such as `<stdint.h>` are sometimes not built before files that include it such as `a64l.cpp`.

## Why is this problem caused?

Courtesy of @statham-arm

> The basic problem is that when build step A creates stdint.h, and build step B uses it, ninja needs to know not to attempt B before A has finished.
>  
> If you write #include <stdint.h> inside B's source file, then the effect is that when step B runs, it outputs a dependency file listing all the headers it included, and the next run of ninja will know that step B depends on stdint.h.
> 
> But that doesn't help the first time a build happens inside a given build directory. Before ninja has run step B at all, it hasn't seen the #include, and doesn't know stdint.h is needed. So the first run of ninja might run step B before A, or concurrently with A, and it won't know any reason not to.
> 
> This kind of thing is likely to go wrong in CI more than in developer builds, because developers build over and over in the same directory, so mostly the depfile will already exist from a previous build, and nothing will go wrong. But CI usually starts a build from scratch in a fresh directory.
> 
> And yes, that's what ninja -t missingdeps is for: it detects situations in which
> - build step A creates a file
> - build step B turned out to use the file, based on dependencies discovered during the build (e.g. clang printing out all the files that were #included)
> - but build step B had no explicit dependency on the file, so that in the first run of step B there's a risk of a problem.

## How could this problem be solved?

One way is to build all builtin header files first, before anything else gets built.

## When is this problem observed?

Mostly in CI. See https://lab.llvm.org/buildbot/#/builders/11/builds/20066

```
[203/2518] Building CXX object libc/src/stdlib/CMakeFiles/libc.src.stdlib.a64l.dir/a64l.cpp.obj
FAILED: libc/src/stdlib/CMakeFiles/libc.src.stdlib.a64l.dir/a64l.cpp.obj 
/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-ubfz36i3/./bin/clang++ --target=armv7em-none-eabi -DLIBC_NAMESPACE=__llvm_libc_22_0_0_git -I/var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/libc -isystem /var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-ubfz36i3/include/armv7em-unknown-none-eabi --target=armv7em-none-eabi -Wno-atomic-alignment "-Dvfprintf(stream, format, vlist)=vprintf(format, vlist)" "-Dfprintf(stream, format, ...)=printf(format)" -D_LIBCPP_PRINT=1 -mthumb -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -ffunction-sections -fdata-sections -ffile-prefix-map=/var/lib/buildbot/fuchsia-x86_64-linux/build/llvm-build-ubfz36i3/runtimes/runtimes-armv7em-none-eabi-bins=../../../../llvm-project -ffile-prefix-map=/var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/= -no-canonical-prefixes -Os -DNDEBUG --target=armv7em-none-eabi -DLIBC_QSORT_IMPL=LIBC_QSORT_HEAP_SORT -DLIBC_TYPES_TIME_T_IS_32_BIT -DLIBC_ADD_NULL_CHECKS "-DLIBC_MATH=(LIBC_MATH_SKIP_ACCURATE_PASS | LIBC_MATH_SMALL_TABLES)" -DLIBC_ERRNO_MODE=LIBC_ERRNO_MODE_EXTERNAL -fpie -ffreestanding -DLIBC_FULL_BUILD -nostdlibinc -ffixed-point -fno-builtin -fno-exceptions -fno-lax-vector-conversions -fno-unwind-tables -fno-asynchronous-unwind-tables -fno-rtti -ftrivial-auto-var-init=pattern -fno-omit-frame-pointer -Wall -Wextra -Werror -Wconversion -Wno-sign-conversion -Wdeprecated -Wno-c99-extensions -Wno-gnu-imaginary-constant -Wno-pedantic -Wimplicit-fallthrough -Wwrite-strings -Wextra-semi -Wnewline-eof -Wnonportable-system-include-path -Wstrict-prototypes -Wthread-safety -Wglobal-constructors -DLIBC_COPT_PUBLIC_PACKAGING -MD -MT libc/src/stdlib/CMakeFiles/libc.src.stdlib.a64l.dir/a64l.cpp.obj -MF libc/src/stdlib/CMakeFiles/libc.src.stdlib.a64l.dir/a64l.cpp.obj.d -o libc/src/stdlib/CMakeFiles/libc.src.stdlib.a64l.dir/a64l.cpp.obj -c /var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/libc/src/stdlib/a64l.cpp
/var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/libc/src/stdlib/a64l.cpp:22:18: error: unknown type name 'int32_t'
   22 | constexpr static int32_t b64_char_to_int(char ch) {
      | ^
/var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/libc/src/stdlib/a64l.cpp:47:3: error: unknown type name 'int32_t'
   47 |   int32_t result = 0;
      | ^
/var/lib/buildbot/fuchsia-x86_64-linux/llvm-project/libc/src/stdlib/a64l.cpp:50:5: error: unknown type name 'int32_t'
   50 |     int32_t cur_val = b64_char_to_int(s[i]);
      |     ^
3 errors generated.
```
</pre>
<img width="1" height="1" alt="" src="http://email.email.llvm.org/o/eJzEWN1u4zjSfRrmpkBDkWI7vsiF_DdtTJLO10kj810JlFSyOE2RWpKy43n6RVFyYie9g-1FsAsEsSWSxVOnqk6RFs7JrUa8YeM5Gy8vROdrY2-c8J3Vk9nlRW7Kww2LExYn8FwLD9KBr6WD1ppcYcOSNYtSFqUL07RSoYW8k8qD1FCjKNFCJRU6cF1Rg3DAJhFLFs6XUvtRzZIVm0QgLIIzDXrZoANt_GAkx8pYHCz4sLsuVFciSH9qUUyu1KhoWzaJRj2aV8SH94ChEJ3D8gR3Zz26A5gK2FXkvPC1aLiwzWApWcFTjZALJ4tXI3LAs69RB7AlOI8tpFBYFJ4cProYL0Do8nTSHDqHDqSnMS31nwI0YunAG_ihzT4w4A0I77FpPcyPRKRQCweV1NLVWI4GdDB8bio4mA72VnoEFidHrs75BqmdLBHmLJ46cKazRc8wgfHkjq8RsKqw8OduDthtpx3NlR5M59vOOxBQYou6RF0cgi1Q0nmptyCUCvb6XCCfjyEsj8TQsMYXT4YpCD0he6lUT0YAMOzdb-PA6Fd-jywMH_PO9ytKg06zeOqhRtWGXSppnQdKMhBDPGrRtqjdkRQBW7l7jWgpLRbe2MMI5n0AemwUBAI7gBKe3BwoqcWwq8OByrdIHD1-gxYcPHpCbFMeYDmCR3OC-IyYRm5rf7r9MTfIurFQGF101qL26gB76et-hPaVHvbmZGOhD2BROKOHjHvH5ROVzQ-pS9rd1xRP6UDJH6gOlJ9bA3tr6K2GxQYaguFroem5xB0q0w56UIaMyTHU3tuYG4g2O7QBYfgie96caPAtBLTeGWiMI79ovMQ2pFrIFKEsivIA-CKdh8qaBgS0FnfSdMMuRxa06V0J644ujELiLDbQuU4odQDnhQ2Z3SMMFl1hhS9qAiigsujqkxQ5py7VJRzQ9TUlfKi1PaVlH0TuoZHOSb0tsXXEamUsS1IKUYkeC-_ASd8JL03ITtjXsqgH4_zniiP6Mv7JnDmQnGNJBUuBoyD06dWXfS4cDeq3MpbooJSuoIBgCWVniTJa05tl8TWOtiMolNBbaK3UodzJ_rHkT1R7j_a0DkoWz05g-nOotaAYAb60ShaBj1dpMfoMtjPHpvCxWI5-12gxsC_ASveDhsRRxc97xRezh8J0qjxvFzm1JrU76RhfNcJe9I3FDNjJ6dCz3ve9AKpP_lCmQh_69EPlELbo--z07_sW6g-Ny-QO7SmQu74YQvWN4BERau9bx5KUxWsWr5XIR0rtmpGxWxavA9Lc-DCYHF-gdSxeX14en-kpjqLJZAA0iYa_KGXjeRzRwnh8ec3GS5jTAnJm8ccfYPI_qWUomRcsXjsb_vtSyZzF68Wd-IFrYoRwybwYOVuM-uFRaN-ltCxeHzv5yOR_sihdp5vb1ZIK4_PMhhKN1zth-zXnzFRdUTsp-Mv1JJtccSV193KcQfPVruHhgXd59VcykcTHiGZIzeJ1qAcWz1k8B869sFv0LFkK2-ym2HBtNHIUuQS-vN3MF9l9erd6fEgXK5Yss4ysZ-RGFsdZlEXZVnrgm19GG1C21lBABmKAS3dwHhv4ZN9fe9v66GSnqbvoU2f_lolnbbjwppEFF0pudYPaA4tjvtxVQVkqFl87b1E0VEiVsY0IJbWjUwZpSbLcvU78yXAc9-b-3tpoNOptfTAVLPBlRhF7eMgevm3un1iyvATe-LprcuDVw2YBvNKGO2yE9rLgUnu0rXGSJBx4tZNO5lJJf-BSK6nR8VqWJWrgz2gtyf-yFB55OJ-8vev0thO2xJKLnZBKDDY07oE_k-zwZ3zxVvQ8dpoOt7wVVjTo0QJ_DudB7jwpuAP-XAjn-T86QSuHJsQriarkUksvhZJ_UWfmz7LpFZhXQilfW9Nta1rf9wTu9tIXNS-xEp3y_fba4EuBref-0GJ4pflOWt8JxUtvCE6JCj3yn4y4brtF5znZt3Qa4889al4YvUPrApGEWaEg2eGS-oIXPcPPhRclb8QhR95p17WtsR5L4FXV6YLmcIdF31B5VQovTp9JrXlrsZIvvBEtS5afXCe20-F6c_KVfygGnkvtWLKkVFy_-3da1J8D-J1MsGQJXBteCG20LIQazKMD_tUBX94vV_Pvv_1buvZ_j1-_PWWbu4dblpy--bJKHzL6dpz49P8Pq8fsaXO3yp6yzWOWxNl88zqaLpfZ_ffb22zxZbX4_bEv4zBylz59CT5fvz5mj79vHrJ0sfj-LX1aZQ_p4yOw6QJOJtylt7fZUzq_XT2-lnUYXn37dv81u_u6XB3xvr3JVn88rb7dp7fAq1YicW8RnRc69L7Bwppgzr9vbgOJfQuSugiReqGSNFL7XiOOB4Xw0BfMkIXacCVe-C6cKU_Sfhjr9F7qknuR09EivBLuoIvaGm0697Nx670EXnkrd1IoLjpv-E7YUOukdXTHtAMU01Cxk3L0aIN8nEtMUKUgAicVSbInt_q8TEtsLRYiVCDNKGYzji8e9eAPvdvqjstGbKUW9kDLidRBSlosg5D-ayF6L2wBIwlw6Cm4J5HlaKpeh0gNiBfet0E-9C3eCl8PSlN4qgZvSLzIoK_pUsGdqNAfgD9vlcmF6nHajkLkjtFffH14yh6-z283i-whXfye_ra5_w343RL43dNnHl743foTzY1K4OZT4RW_fr74yWnlI5rXn3j-g8Pbr22QpHHMkvTymo6dQxdOYTjUQGhsmq6mLJ5K7ZM4o9sdi1IAoIPGdAEhQfCltXSLpBQe5kE-ucqKWtjMm0xqz-JreoKiZvEM2HTeWwEIVth49d9w9mrKkjT5dV-vpgElvDpn0dE5gLpIxJL_iS_jiP79ui_jaPDlzZuis9lOqODOx7A5Np5LNl6GA-O5q-EzuJv0KBxsUaMlJRydXqguypuknCUzcYE3l9NxcnkdxZPLi_rmepLk0dV0NpnFyfRazGblZDzFcTKeTLGMxexC3sRRPI6mcXwZJZfR5Ugkk6LKK5FUVZHksxm7irARUr3e_i6kcx3eXF7NZrPkQokclQs_-cZxT2rMxssLezMcXbaOXUV0enZvFrz0KvxMHBaMl7BW4sdh-HWHbuaLzUVn1c35FXQrfd3lo8I0Qzg_RjVAC3fQHt3uJv5nAAAA__9qhYZj">