[llvm-dev] RFC: Using GitHub Actions for CI testing on the release/* branches

Thu Feb 6 08:41:23 PST 2020

Hi Tom,

Thank you for setting this up. It's very useful.

One question about what this builds. It only builds "ninja check-all", not
"ninja all"[1]. check-all isn't a strict superset of all so while this
covers most things, this does miss building a few things such as:
bin/clang-offload-wrapper
bin/llvm-itanium-demangle-fuzzer
bin/llvm-microsoft-demangle-fuzzer
bin/llvm-PerfectShuffle
...
lib/libclang.so
...

For completeness, should this do "ninja all", "ninja check-all" as most
buildbot builders do?

Thanks
Russ

[1]
https://github.com/llvm/actions/blob/master/build-test-llvm-project/main.js

On Fri, 13 Dec 2019 at 22:07, John Byrd via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> > I think my concern is that LLVM could prove to be too big and require
> too many resources for github's infrastructure. How many patches go into
> LLVM a day, and how many build and test jobs does GitHub allow users to run
> concurrently before being throttled?
>
> Please, no one confuse "what should be LLVM's official CIs" with "what can
> we do to make it easier for individuals and small companies to experiment
> with small changes in their own forks."
>
> Current usage limits for the free Github actions tier are:
>
> - You can execute up to 20 workflows concurrently per repository.
> - You can execute up to 1000 API requests in an hour across all actions
> within a repository.
> - Each job in a workflow can run for up to 6 hours of execution time.
> - The number of jobs you can run concurrently across all repositories in
> your account depends on your GitHub plan.
> - You can have a maximum concurrent set of 5 maximum concurrent MacOS jobs.
>
> So, this isn't enough juice to replace all the LLVM buildbots, but it is
> enough to have CI on *your personal fork* of LLVM.  And having CI on
> everyone's personal fork, makes it that much more likely that your
> Phabricator patches will be correct the first time.
>
> Best of all, if you don't like Github's CI, you can completely ignore it
> and proceed with your current workflow.
>
> On Thu, Dec 12, 2019 at 10:53 AM Reid Kleckner <rnk at google.com> wrote:
>
>> I think everyone agrees that LLVM needs better CI, automatic pre-commit
>> testing of various platforms, etc. This is not rocket science, it is
>> standard practice for 2019 software engineering.
>>
>> I think my concern is that LLVM could prove to be too big and require too
>> many resources for github's infrastructure. How many patches go into LLVM a
>> day, and how many build and test jobs does GitHub allow users to run
>> concurrently before being throttled?
>>
>> You may have seen that Christian Kuhnel has been working on a pre-commit
>> testing bot that integrates with Phab:
>> https://reviews.llvm.org/p/merge_guards_bot/
>>
>> https://github.com/google/llvm-premerge-checks/blob/master/docs/user_doc.md
>> I hope that ends up being the way forward and suits Tom's original
>> release testing goals.
>>
>> On Wed, Dec 11, 2019 at 11:43 PM John Byrd via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>>> Please forgive the incorrect threading on this reply to Tom
>>> Stellard's RFC.
>>>
>>> > I would like to start using GitHub Actions[1] for CI testing on the
>>> release/*
>>> > branches.  As far as I know we don't have any buildbots listening to
>>> the
>>> > release branches, and I think GitHub Actions are a good way for us to
>>> > quickly bring-up some CI jobs there.
>>>
>>> Personally, I feel that Tom's proof of concept, is more important than
>>> we seem to be giving him credit for.
>>>
>>> As of this writing, the Github actions system permits all comers, six
>>> hours of CPU time per build platform.  Due to this long CPU allotment,
>>> AFAIK, Github is one of the few CIs in town that lets anyone build and
>>> smoke llvm for free.
>>>
>>> Consider the workflow of someone who has never worked on llvm before.
>>> They will probably fork the monorepo on Github, in order to fix bugs or add
>>> a feature or such.  At the moment they do this, they get a built-in
>>> workflow that will sanity-check their builds on several important targets.
>>> Zero braining involved.
>>>
>>> Giving Joe Programmer a CI system that magically smoke tests llvm, out
>>> of the box, after he forks the repo, is a compelling reason to make
>>> something like Tom's system a standard part of llvm master.
>>>
>>> Concerns might be raised that llvm is "preferring" one CI system over
>>> another.  Some thoughts about that.  First, because the monorepo's on
>>> Github, you'll end up going to github.com anyway to get your first
>>> pull.  Second, nothing about Github actions precludes supporting other CI
>>> systems in the future.
>>>
>>> Thanks for your kind consideration.
>>>
>>> --
>>> ---
>>>
>>> John Byrd
>>> Gigantic Software
>>> 2321 E 4th Street
>>> Suite C #429
>>> Santa Ana, CA  92705-3862
>>> http://www.giganticsoftware.com
>>> T: (949) 892-3526 F: (206) 309-0850
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>
>>
>
> --
> ---
>
> John Byrd
> Gigantic Software
> 2321 E 4th Street
> Suite C #429
> Santa Ana, CA  92705-3862
> http://www.giganticsoftware.com
> T: (949) 892-3526 F: (206) 309-0850
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200206/ac9d9ba7/attachment.html>