[libcxx-commits] [libcxx] [libc++][Android] Add libcxx-builder-android Docker image (PR #69273)
Ryan Prichard via libcxx-commits
libcxx-commits at lists.llvm.org
Wed Oct 18 00:38:34 PDT 2023
https://github.com/rprichard updated https://github.com/llvm/llvm-project/pull/69273
>From dcde8e8dc4957d84f9246505f586ec2bd2c7cf83 Mon Sep 17 00:00:00 2001
From: Ryan Prichard <rprichard at google.com>
Date: Tue, 10 Oct 2023 00:06:19 -0700
Subject: [PATCH 1/2] [libc++][Android] Add libcxx-builder-android Docker image
Add a Dockerfile for a new Docker image, libcxx-builder-android, that
extends libcxx-builder with support for testing Android.
The image includes these things:
* An Android Clang compiler and sysroot.
* The Android platform-tools (e.g. adb), so that an Android buildbot
can run programs on an Android device. At container startup, copy
these platform tools to an "android-platform-tools" Docker volume to
share them with an emulator container. This copying ensures that the
emulator and libcxx-builder containers avoid mismatched adb
versions.
* Docker, so that an Android buildbot can manage a sibling Docker
container that runs the Android emulator.
Add an Android-specific run-buildbot-container script for local
development. Currently using this script requires building
libcxx-build-android and an emulator image locally.
Fixes: https://github.com/llvm/llvm-project/issues/69270
Differential Revision: https://reviews.llvm.org/D155271
---
libcxx/utils/ci/vendor/android/Dockerfile | 83 +++++++++++++++++++
.../ci/vendor/android/container-setup.sh | 19 +++++
.../ci/vendor/android/run-buildbot-container | 31 +++++++
3 files changed, 133 insertions(+)
create mode 100644 libcxx/utils/ci/vendor/android/Dockerfile
create mode 100755 libcxx/utils/ci/vendor/android/container-setup.sh
create mode 100755 libcxx/utils/ci/vendor/android/run-buildbot-container
diff --git a/libcxx/utils/ci/vendor/android/Dockerfile b/libcxx/utils/ci/vendor/android/Dockerfile
new file mode 100644
index 000000000000000..0acfff8e031dc81
--- /dev/null
+++ b/libcxx/utils/ci/vendor/android/Dockerfile
@@ -0,0 +1,83 @@
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+#
+# This Dockerfile extends ldionne/libcxx-builder with Android support, including
+# Android Clang and sysroot, Android platform-tools, and the Docker client.
+#
+# $ docker build -t libcxx-builder-android libcxx/utils/ci/vendor/android
+#
+
+FROM ldionne/libcxx-builder
+
+# Switch back to the root user to install things into /opt and /usr.
+USER root
+WORKDIR /
+
+# Install the Android platform tools (e.g. adb) into /opt/android/sdk.
+RUN apt-get update && apt-get install -y unzip
+RUN mkdir -p /opt/android/sdk && cd /opt/android/sdk && \
+ curl -LO https://dl.google.com/android/repository/platform-tools-latest-linux.zip && \
+ unzip platform-tools-latest-linux.zip && \
+ rm platform-tools-latest-linux.zip
+
+# Install the current Android compiler. Specify the prebuilts commit to retrieve
+# this compiler version even after it's removed from HEAD.
+ENV ANDROID_CLANG_VERSION=r498229b
+ENV ANDROID_CLANG_PREBUILTS_COMMIT=5186d132c99aa75dc25207c392e3ea5b93d0107e
+RUN git clone --filter=blob:none --sparse \
+ https://android.googlesource.com/platform/prebuilts/clang/host/linux-x86 \
+ /opt/android/clang && \
+ git -C /opt/android/clang checkout ${ANDROID_CLANG_PREBUILTS_COMMIT} && \
+ git -C /opt/android/clang sparse-checkout add clang-${ANDROID_CLANG_VERSION} && \
+ rm -fr /opt/android/clang/.git && \
+ ln -sf /opt/android/clang/clang-${ANDROID_CLANG_VERSION} /opt/android/clang/clang-current && \
+ # The "git sparse-checkout" and "ln" commands succeed even if nothing was
+ # checked out, so use this "ls" command to fix that.
+ ls /opt/android/clang/clang-current/bin/clang
+
+# Install an Android sysroot. New AOSP sysroots are available at
+# https://ci.android.com/builds/branches/aosp-main/grid, the "ndk" target. The
+# NDK also makes its sysroot prebuilt available at
+# https://android.googlesource.com/platform/prebuilts/ndk/+/refs/heads/dev/platform/sysroot.
+ENV ANDROID_SYSROOT_BID=10957860
+RUN cd /opt/android && \
+ curl -L -o ndk_platform.tar.bz2 \
+ https://androidbuildinternal.googleapis.com/android/internal/build/v3/builds/${ANDROID_SYSROOT_BID}/ndk/attempts/latest/artifacts/ndk_platform.tar.bz2/url && \
+ tar xf ndk_platform.tar.bz2 && \
+ rm ndk_platform.tar.bz2
+
+# Install Docker. Mark the binary setuid so it can be run without prefixing it
+# with sudo. Adding the container user to the docker group doesn't work because
+# /var/run/docker.sock is owned by the host's docker GID, not the container's
+# docker GID.
+RUN apt-get update && apt-get install -y gpg && \
+ install -m 0755 -d /etc/apt/keyrings && \
+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \
+ chmod a+r /etc/apt/keyrings/docker.gpg && \
+ echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
+ "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
+ tee /etc/apt/sources.list.d/docker.list > /dev/null && \
+ apt-get update && apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin && \
+ chmod u+s /usr/bin/docker
+
+COPY ./container-setup.sh /opt/android/container-setup.sh
+
+USER libcxx-builder
+WORKDIR /home/libcxx-builder
+
+# Add Android platform-tools to the PATH.
+ENV PATH="/opt/android/sdk/platform-tools:${PATH}"
+
+# Reset the buildkite-agent.cfg file. The tags are provided by an environment
+# variable instead.
+RUN cp /home/libcxx-builder/.buildkite-agent/buildkite-agent.dist.cfg \
+ /home/libcxx-builder/.buildkite-agent/buildkite-agent.cfg
+
+# Modify the Buildkite agent cmdline to do Android setup stuff first.
+CMD /opt/android/container-setup.sh && buildkite-agent start
diff --git a/libcxx/utils/ci/vendor/android/container-setup.sh b/libcxx/utils/ci/vendor/android/container-setup.sh
new file mode 100755
index 000000000000000..56bc232fefa1e6b
--- /dev/null
+++ b/libcxx/utils/ci/vendor/android/container-setup.sh
@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+set -e
+
+# Different versions of adb can sometimes be incompatible (i.e. "adb server
+# version (nn) doesn't match this client (mm); killing..."). Ensure that the adb
+# in the main builder image matches that in the emulator by sharing the
+# platform-tools from the main image.
+if [ -d /mnt/android-platform-tools ]; then
+ sudo rm -fr /mnt/android-platform-tools/platform-tools
+ sudo cp -r /opt/android/sdk/platform-tools /mnt/android-platform-tools
+fi
diff --git a/libcxx/utils/ci/vendor/android/run-buildbot-container b/libcxx/utils/ci/vendor/android/run-buildbot-container
new file mode 100755
index 000000000000000..4ab83194c05d5c4
--- /dev/null
+++ b/libcxx/utils/ci/vendor/android/run-buildbot-container
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+
+# Similar to libcxx/utils/ci/run-buildbot-container, but adds additional options
+# needed for running Android tests.
+
+set -e
+
+MONOREPO_ROOT="$(git rev-parse --show-toplevel)"
+if [[ ! -d "${MONOREPO_ROOT}/libcxx/utils/ci/vendor/android" ]]; then
+ echo "Was unable to find the root of the LLVM monorepo; are you running from within the monorepo?"
+ exit 1
+fi
+
+DOCKER_OPTIONS=(-it)
+DOCKER_OPTIONS+=(--volume "${MONOREPO_ROOT}:/llvm")
+DOCKER_OPTIONS+=(--workdir "/llvm")
+DOCKER_OPTIONS+=(--cap-add=SYS_PTRACE)
+
+# Mount this volume to allow the main image to share its copy of the Android
+# platform tools with the emulator image, ensuring that the adb versions match.
+# This argument will create a new volume if it doesn't already exist.
+DOCKER_OPTIONS+=(--volume android-platform-tools:/mnt/android-platform-tools)
+
+# Pass through the Docker socket so that the buildbot can start a sibling
+# container running an Android emulator.
+if [ -S /var/run/docker.sock ]; then
+ DOCKER_OPTIONS+=(--volume /var/run/docker.sock:/var/run/docker.sock)
+fi
+
+docker run "${DOCKER_OPTIONS[@]}" libcxx-builder-android \
+ bash -c 'git config --global --add safe.directory /llvm; (/opt/android/container-setup.sh && exec bash)'
>From 55107010cfdfd9b59aaa7eac757d395675308d7e Mon Sep 17 00:00:00 2001
From: Ryan Prichard <rprichard at google.com>
Date: Tue, 10 Oct 2023 00:06:19 -0700
Subject: [PATCH 2/2] Add missing files to ignore_format.txt
---
libcxx/utils/data/ignore_format.txt | 2 ++
1 file changed, 2 insertions(+)
diff --git a/libcxx/utils/data/ignore_format.txt b/libcxx/utils/data/ignore_format.txt
index e4f56b00c758343..14b37185c364784 100644
--- a/libcxx/utils/data/ignore_format.txt
+++ b/libcxx/utils/data/ignore_format.txt
@@ -7301,4 +7301,6 @@ libcxx/utils/ci/Dockerfile
libcxx/utils/ci/macos-ci-setup
libcxx/utils/ci/run-buildbot
libcxx/utils/ci/run-buildbot-container
+libcxx/utils/ci/vendor/android/Dockerfile
+libcxx/utils/ci/vendor/android/run-buildbot-container
libcxx/utils/libcxx-lit
More information about the libcxx-commits
mailing list