[libcxx-commits] [libcxx] Rehome libcxx-builder docker image & attempt gentler termination. (PR #71604)
via libcxx-commits
libcxx-commits at lists.llvm.org
Wed Nov 8 10:08:11 PST 2023
https://github.com/EricWF updated https://github.com/llvm/llvm-project/pull/71604
>From e2007524c306aad4adf3f867b145060e5bd47b99 Mon Sep 17 00:00:00 2001
From: eric <eric at efcs.ca>
Date: Tue, 7 Nov 2023 18:39:00 -0500
Subject: [PATCH 1/2] Rehome libcxx-builder docker image & attempt gentler
termination.
There are three changes present in this PR.
1. Use github packages for libcxx-builder rather than dockerhub.
The ldionne/libcxx-builder image will now be hosted at
ghcr.io/libcxx/libcxx-builder. This has the benefit of allowing members
of the github org to push new versions.
In the future I hope to add github actions to rebuild the image as
needed.
2. Add docker-compose file
The compose file allows to to specify the package repository, so that
users can simply write 'docker compose build' and 'docker compose push'.
It also gives us a centralized place to manage version arguments which
change frequently.
3. Use non-shell CMD form.
This may help the google libcxx builders disconnect more gracefully as the shell form of CMD may eat the
shutdown signal. I'm hoping this corrects inaccurate agent counts from
the buildkite API, since when the VM's terminate, they do so without
signaling it to buildkite, which hangs around waiting for them to
reconnect. It's likely more changes will be needed though.
---
libcxx/utils/ci/Dockerfile | 18 +++++++++---------
libcxx/utils/ci/docker-compose.yml | 11 +++++++++++
libcxx/utils/ci/run-buildbot-container | 4 ++--
libcxx/utils/ci/vendor/android/Dockerfile | 4 ++--
4 files changed, 24 insertions(+), 13 deletions(-)
create mode 100644 libcxx/utils/ci/docker-compose.yml
diff --git a/libcxx/utils/ci/Dockerfile b/libcxx/utils/ci/Dockerfile
index 962de84e852ce59..53267fd1121e079 100644
--- a/libcxx/utils/ci/Dockerfile
+++ b/libcxx/utils/ci/Dockerfile
@@ -22,12 +22,11 @@
# If you're only looking to run the Docker image locally for debugging a
# build bot, see the `run-buildbot-container` script located in this directory.
#
-# A pre-built version of this image is maintained on DockerHub as ldionne/libcxx-builder.
-# To update the image, rebuild it and push it to ldionne/libcxx-builder (which
-# will obviously only work if you have permission to do so).
+# A pre-built version of this image is maintained on Github under the libc++ organization, as ghcr.io/libcxx/libcxx-builder.
+# To update the image, rebuild it and push it to github (all members of the libc++ organization should be able to do this).
#
-# $ docker build -t ldionne/libcxx-builder libcxx/utils/ci
-# $ docker push ldionne/libcxx-builder
+# $ docker compose build
+# $ docker compose push
#
FROM ubuntu:jammy
@@ -66,8 +65,8 @@ RUN locale-gen
# LLVM 15, we still need to have Clang 12 in this Docker image because the LLVM
# 14 release branch CI uses it. The tip-of-trunk CI will never use Clang 12,
# though.
-# LLVM POST-BRANCH bump version
-ENV LLVM_HEAD_VERSION=18
+ARG LLVM_HEAD_VERSION
+ENV LLVM_HEAD_VERSION=${LLVM_HEAD_VERSION}
RUN apt-get update && apt-get install -y lsb-release wget software-properties-common
RUN wget https://apt.llvm.org/llvm.sh -O /tmp/llvm.sh
RUN bash /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 3)) # for CI transitions
@@ -97,8 +96,9 @@ RUN apt-get update && apt-get install -y llvm-$(($LLVM_HEAD_VERSION - 2))-dev ll
libomp5-$LLVM_HEAD_VERSION
# Install the most recent GCC, like clang install the previous version as a transition.
-ENV GCC_LATEST_VERSION=13
RUN add-apt-repository ppa:ubuntu-toolchain-r/test
+ARG GCC_LATEST_VERSION # populated in the docker-compose file
+ENV GCC_LATEST_VERSION=${GCC_LATEST_VERSION}
RUN apt-get update && apt install -y gcc-$((GCC_LATEST_VERSION - 1)) g++-$((GCC_LATEST_VERSION - 1))
RUN apt-get update && apt install -y gcc-$GCC_LATEST_VERSION g++-$GCC_LATEST_VERSION
@@ -131,4 +131,4 @@ ENV PATH="${PATH}:/home/libcxx-builder/.buildkite-agent/bin"
RUN echo "tags=\"queue=libcxx-builders,arch=$(uname -m),os=linux\"" >> "/home/libcxx-builder/.buildkite-agent/buildkite-agent.cfg"
# By default, start the Buildkite agent (this requires a token).
-CMD buildkite-agent start
+CMD ["buildkite-agent", "start"]
diff --git a/libcxx/utils/ci/docker-compose.yml b/libcxx/utils/ci/docker-compose.yml
new file mode 100644
index 000000000000000..7f03d8f3af0f293
--- /dev/null
+++ b/libcxx/utils/ci/docker-compose.yml
@@ -0,0 +1,11 @@
+version: '3.7'
+services:
+ libcxx-builder:
+ image: ghcr.io/libcxx/libcxx-builder
+ build:
+ context: .
+ dockerfile: Dockerfile
+ args:
+ GCC_LATEST_VERSION: 13
+ # LLVM POST-BRANCH bump version
+ LLVM_HEAD_VERSION: 18
diff --git a/libcxx/utils/ci/run-buildbot-container b/libcxx/utils/ci/run-buildbot-container
index 07fb2706beb20e3..7c00b88097f17b8 100755
--- a/libcxx/utils/ci/run-buildbot-container
+++ b/libcxx/utils/ci/run-buildbot-container
@@ -26,6 +26,6 @@ if [[ ! -d "${MONOREPO_ROOT}/libcxx/utils/ci" ]]; then
echo "Was unable to find the root of the LLVM monorepo; are you running from within the monorepo?"
exit 1
fi
-docker pull ldionne/libcxx-builder
-docker run -it --volume "${MONOREPO_ROOT}:/llvm" --workdir "/llvm" --cap-add=SYS_PTRACE ldionne/libcxx-builder \
+docker pull ghcr.io/libcxx/libcxx-builder
+docker run -it --volume "${MONOREPO_ROOT}:/llvm" --workdir "/llvm" --cap-add=SYS_PTRACE ghcr.io/libcxx/libcxx-builder \
bash -c 'git config --global --add safe.directory /llvm ; exec bash'
diff --git a/libcxx/utils/ci/vendor/android/Dockerfile b/libcxx/utils/ci/vendor/android/Dockerfile
index 0acfff8e031dc81..4d35334f89bb220 100644
--- a/libcxx/utils/ci/vendor/android/Dockerfile
+++ b/libcxx/utils/ci/vendor/android/Dockerfile
@@ -7,13 +7,13 @@
#===----------------------------------------------------------------------===##
#
-# This Dockerfile extends ldionne/libcxx-builder with Android support, including
+# This Dockerfile extends ghcr.io/libcxx/libcxx-builder with Android support, including
# Android Clang and sysroot, Android platform-tools, and the Docker client.
#
# $ docker build -t libcxx-builder-android libcxx/utils/ci/vendor/android
#
-FROM ldionne/libcxx-builder
+FROM ghcr.io/libcxx/libcxx-builder
# Switch back to the root user to install things into /opt and /usr.
USER root
>From 4d8d285c9335f979b582463674aa1cfdc88e0417 Mon Sep 17 00:00:00 2001
From: Eric <eric at efcs.ca>
Date: Wed, 8 Nov 2023 13:08:04 -0500
Subject: [PATCH 2/2] Update libcxx/utils/ci/Dockerfile
Co-authored-by: Mark de Wever <zar-rpg at xs4all.nl>
---
libcxx/utils/ci/Dockerfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/libcxx/utils/ci/Dockerfile b/libcxx/utils/ci/Dockerfile
index 53267fd1121e079..0dcde11c44d5670 100644
--- a/libcxx/utils/ci/Dockerfile
+++ b/libcxx/utils/ci/Dockerfile
@@ -65,7 +65,7 @@ RUN locale-gen
# LLVM 15, we still need to have Clang 12 in this Docker image because the LLVM
# 14 release branch CI uses it. The tip-of-trunk CI will never use Clang 12,
# though.
-ARG LLVM_HEAD_VERSION
+ARG LLVM_HEAD_VERSION # populated in the docker-compose file
ENV LLVM_HEAD_VERSION=${LLVM_HEAD_VERSION}
RUN apt-get update && apt-get install -y lsb-release wget software-properties-common
RUN wget https://apt.llvm.org/llvm.sh -O /tmp/llvm.sh
More information about the libcxx-commits
mailing list