[cfe-dev] [llvm-dev] [Openmp-dev] RFC: Using GitHub Actions for CI testing on the release/* branches

[Christian Kühnel via cfe-dev]

Hi Tom,

First of: I'm a big fan of adding more automatic tests to find bugs. Great
work!

On the other hand, I think we should consider creating some sort of test
strategy for the LLVM project:

   - What tests do we expect users to run before uploading patches?
   - What tests do we expect users to run before merging?
   - What tests do we run after merging?
   - What failures must be fixed, what failures can be ignored?
   - What do we check for on the build bots?
   - What do we check for on the release branches?
   - What do we check for on the pre-merge tests?
   - Which CI tools do we want to use (github, Jenkins, bulidbot, ...)?
   - What about running clang-tidy and clang-format?
   - What CMake configurations should we check? (Release/Debug, assertions,
   ...)
   - Do we want to run tests with Sanitizers?
   - Which of these systems do we expect users to monitor?

I suppose it would be good to have a document that summarizes all of this
so that we

   1. do not test the same thing twice and
   2. do not miss any important checks.

Does something like this exist?
Is anyone working on this?

Best,
Christian

On Wed, Nov 20, 2019 at 7:26 AM Tom Stellard via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> > Hi Tom,
> >
> > This sounds very interesting! +1 from me.
> > What does this imply for the release testers?
> > Would it be possible / desirable for us to add self-hosted runners for
> > other architectures? I'm assuming GitHub only provides x86, right? I
> > totally missed any details about that in their docs, but it seems to
> > be implied here [1]. I'm not saying we should go all-possible-arches
> > from the start, we can definitely try it out only for x86 this time
> > around if you think that would be the best approach.
> >
>
> Nothing changes for release testers at this point.  The main goal here is
> to get some post-commit testing so we can catch issues in the branch early.
>
> Longer term I think have self-hosted runners would be great, because it
> would
> allow us to fully automate the testing and uploading we do when a release
> gets
> tagged.
>
> You are correct that GitHub only provides x86 machines, however, they don't
> have enough disk space (14GB) to be able to run the test-release.sh
> script, so
> adding x86 self-hosted builders with more disk space would still be very
> useful.
>
> -Tom
>
> >
> > [1]
> https://github.blog/2019-11-05-self-hosted-runners-for-github-actions-is-now-in-beta/
> >
> > Thanks,
> > Diana
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>


-- 
Best,
Christian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20191121/02172ea9/attachment.html>