[libcxx-commits] [libcxx] [libc++][Android] Add libcxx-builder-android Docker image (PR #69273)

Ryan Prichard via libcxx-commits libcxx-commits at lists.llvm.org
Mon Oct 16 18:42:39 PDT 2023 

https://github.com/rprichard created https://github.com/llvm/llvm-project/pull/69273

Add a Dockerfile for a new Docker image, libcxx-builder-android, that extends libcxx-builder with support for testing Android.

The image includes these things:

 * An Android Clang compiler and sysroot.

 * The Android platform-tools (e.g. adb), so that an Android buildbot can run programs on an Android device. At container startup, copy these platform tools to an "android-platform-tools" Docker volume to share them with an emulator container. This copying ensures that the emulator and libcxx-builder containers avoid mismatched adb versions.

 * Docker, so that an Android buildbot can manage a sibling Docker container that runs the Android emulator.

Add an Android-specific run-buildbot-container script for local development. Currently using this script requires building libcxx-build-android and an emulator image locally.

Fixes: https://github.com/llvm/llvm-project/issues/69270
Differential Revision: https://reviews.llvm.org/D155271

>From dcde8e8dc4957d84f9246505f586ec2bd2c7cf83 Mon Sep 17 00:00:00 2001
From: Ryan Prichard <rprichard at google.com>
Date: Tue, 10 Oct 2023 00:06:19 -0700
Subject: [PATCH] [libc++][Android] Add libcxx-builder-android Docker image

Add a Dockerfile for a new Docker image, libcxx-builder-android, that
extends libcxx-builder with support for testing Android.

The image includes these things:

 * An Android Clang compiler and sysroot.

 * The Android platform-tools (e.g. adb), so that an Android buildbot
   can run programs on an Android device. At container startup, copy
   these platform tools to an "android-platform-tools" Docker volume to
   share them with an emulator container. This copying ensures that the
   emulator and libcxx-builder containers avoid mismatched adb
   versions.

 * Docker, so that an Android buildbot can manage a sibling Docker
   container that runs the Android emulator.

Add an Android-specific run-buildbot-container script for local
development. Currently using this script requires building
libcxx-build-android and an emulator image locally.

Fixes: https://github.com/llvm/llvm-project/issues/69270
Differential Revision: https://reviews.llvm.org/D155271
---
 libcxx/utils/ci/vendor/android/Dockerfile     | 83 +++++++++++++++++++
 .../ci/vendor/android/container-setup.sh      | 19 +++++
 .../ci/vendor/android/run-buildbot-container  | 31 +++++++
 3 files changed, 133 insertions(+)
 create mode 100644 libcxx/utils/ci/vendor/android/Dockerfile
 create mode 100755 libcxx/utils/ci/vendor/android/container-setup.sh
 create mode 100755 libcxx/utils/ci/vendor/android/run-buildbot-container

diff --git a/libcxx/utils/ci/vendor/android/Dockerfile b/libcxx/utils/ci/vendor/android/Dockerfile
new file mode 100644
index 000000000000000..0acfff8e031dc81
--- /dev/null
+++ b/libcxx/utils/ci/vendor/android/Dockerfile
@@ -0,0 +1,83 @@
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+#
+# This Dockerfile extends ldionne/libcxx-builder with Android support, including
+# Android Clang and sysroot, Android platform-tools, and the Docker client.
+#
+#   $ docker build -t libcxx-builder-android libcxx/utils/ci/vendor/android
+#
+
+FROM ldionne/libcxx-builder
+
+# Switch back to the root user to install things into /opt and /usr.
+USER root
+WORKDIR /
+
+# Install the Android platform tools (e.g. adb) into /opt/android/sdk.
+RUN apt-get update && apt-get install -y unzip
+RUN mkdir -p /opt/android/sdk && cd /opt/android/sdk && \
+    curl -LO https://dl.google.com/android/repository/platform-tools-latest-linux.zip && \
+    unzip platform-tools-latest-linux.zip && \
+    rm platform-tools-latest-linux.zip
+
+# Install the current Android compiler. Specify the prebuilts commit to retrieve
+# this compiler version even after it's removed from HEAD.
+ENV ANDROID_CLANG_VERSION=r498229b
+ENV ANDROID_CLANG_PREBUILTS_COMMIT=5186d132c99aa75dc25207c392e3ea5b93d0107e
+RUN git clone --filter=blob:none --sparse \
+        https://android.googlesource.com/platform/prebuilts/clang/host/linux-x86 \
+        /opt/android/clang && \
+    git -C /opt/android/clang checkout ${ANDROID_CLANG_PREBUILTS_COMMIT} && \
+    git -C /opt/android/clang sparse-checkout add clang-${ANDROID_CLANG_VERSION} && \
+    rm -fr /opt/android/clang/.git && \
+    ln -sf /opt/android/clang/clang-${ANDROID_CLANG_VERSION} /opt/android/clang/clang-current && \
+    # The "git sparse-checkout" and "ln" commands succeed even if nothing was
+    # checked out, so use this "ls" command to fix that.
+    ls /opt/android/clang/clang-current/bin/clang
+
+# Install an Android sysroot. New AOSP sysroots are available at
+# https://ci.android.com/builds/branches/aosp-main/grid, the "ndk" target. The
+# NDK also makes its sysroot prebuilt available at
+# https://android.googlesource.com/platform/prebuilts/ndk/+/refs/heads/dev/platform/sysroot.
+ENV ANDROID_SYSROOT_BID=10957860
+RUN cd /opt/android && \
+    curl -L -o ndk_platform.tar.bz2 \
+        https://androidbuildinternal.googleapis.com/android/internal/build/v3/builds/${ANDROID_SYSROOT_BID}/ndk/attempts/latest/artifacts/ndk_platform.tar.bz2/url && \
+    tar xf ndk_platform.tar.bz2 && \
+    rm ndk_platform.tar.bz2
+
+# Install Docker. Mark the binary setuid so it can be run without prefixing it
+# with sudo. Adding the container user to the docker group doesn't work because
+# /var/run/docker.sock is owned by the host's docker GID, not the container's
+# docker GID.
+RUN apt-get update && apt-get install -y gpg && \
+    install -m 0755 -d /etc/apt/keyrings && \
+    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \
+    chmod a+r /etc/apt/keyrings/docker.gpg && \
+    echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
+         "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
+         tee /etc/apt/sources.list.d/docker.list > /dev/null && \
+    apt-get update && apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin && \
+    chmod u+s /usr/bin/docker
+
+COPY ./container-setup.sh /opt/android/container-setup.sh
+
+USER libcxx-builder
+WORKDIR /home/libcxx-builder
+
+# Add Android platform-tools to the PATH.
+ENV PATH="/opt/android/sdk/platform-tools:${PATH}"
+
+# Reset the buildkite-agent.cfg file. The tags are provided by an environment
+# variable instead.
+RUN cp /home/libcxx-builder/.buildkite-agent/buildkite-agent.dist.cfg \
+       /home/libcxx-builder/.buildkite-agent/buildkite-agent.cfg
+
+# Modify the Buildkite agent cmdline to do Android setup stuff first.
+CMD /opt/android/container-setup.sh && buildkite-agent start
diff --git a/libcxx/utils/ci/vendor/android/container-setup.sh b/libcxx/utils/ci/vendor/android/container-setup.sh
new file mode 100755
index 000000000000000..56bc232fefa1e6b
--- /dev/null
+++ b/libcxx/utils/ci/vendor/android/container-setup.sh
@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+set -e
+
+# Different versions of adb can sometimes be incompatible (i.e. "adb server
+# version (nn) doesn't match this client (mm); killing..."). Ensure that the adb
+# in the main builder image matches that in the emulator by sharing the
+# platform-tools from the main image.
+if [ -d /mnt/android-platform-tools ]; then
+    sudo rm -fr /mnt/android-platform-tools/platform-tools
+    sudo cp -r /opt/android/sdk/platform-tools /mnt/android-platform-tools
+fi
diff --git a/libcxx/utils/ci/vendor/android/run-buildbot-container b/libcxx/utils/ci/vendor/android/run-buildbot-container
new file mode 100755
index 000000000000000..4ab83194c05d5c4
--- /dev/null
+++ b/libcxx/utils/ci/vendor/android/run-buildbot-container
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+
+# Similar to libcxx/utils/ci/run-buildbot-container, but adds additional options
+# needed for running Android tests.
+
+set -e
+
+MONOREPO_ROOT="$(git rev-parse --show-toplevel)"
+if [[ ! -d "${MONOREPO_ROOT}/libcxx/utils/ci/vendor/android" ]]; then
+    echo "Was unable to find the root of the LLVM monorepo; are you running from within the monorepo?"
+    exit 1
+fi
+
+DOCKER_OPTIONS=(-it)
+DOCKER_OPTIONS+=(--volume "${MONOREPO_ROOT}:/llvm")
+DOCKER_OPTIONS+=(--workdir "/llvm")
+DOCKER_OPTIONS+=(--cap-add=SYS_PTRACE)
+
+# Mount this volume to allow the main image to share its copy of the Android
+# platform tools with the emulator image, ensuring that the adb versions match.
+# This argument will create a new volume if it doesn't already exist.
+DOCKER_OPTIONS+=(--volume android-platform-tools:/mnt/android-platform-tools)
+
+# Pass through the Docker socket so that the buildbot can start a sibling
+# container running an Android emulator.
+if [ -S /var/run/docker.sock ]; then
+    DOCKER_OPTIONS+=(--volume /var/run/docker.sock:/var/run/docker.sock)
+fi
+
+docker run "${DOCKER_OPTIONS[@]}" libcxx-builder-android \
+    bash -c 'git config --global --add safe.directory /llvm; (/opt/android/container-setup.sh && exec bash)'

More information about the libcxx-commits mailing list