[libcxx-commits] [libcxx] [libc++][CI] Refactor creating inline files (PR #88250)
Marius Brehler via libcxx-commits
libcxx-commits at lists.llvm.org
Wed Apr 10 03:06:52 PDT 2024
https://github.com/marbre created https://github.com/llvm/llvm-project/pull/88250
When creating inline files, spaces cannot be used for indention. This refactors to follow the here-doc syntax and replaces spaces with tabs, see [1]. The usage of spaces, breaks the OpenSSF Scorecard `Pinned-Dependencies` check, see current scanning results at [2]:
```
Reason
internal error: error parsing shell code: libcxx/utils/ci/Dockerfile:1:1: unclosed here-document 'EOF'
```
[1] https://docs.docker.com/reference/dockerfile/#example-creating-inline-files
[2] https://securityscorecards.dev/viewer/?uri=github.com/llvm/llvm-project
>From afb60e2f96f367d1bf8adecef6f78bf2442c4443 Mon Sep 17 00:00:00 2001
From: Marius Brehler <marius.brehler at iml.fraunhofer.de>
Date: Wed, 10 Apr 2024 09:46:31 +0000
Subject: [PATCH] [libc++][CI] Refactor creating inline files
When creating inline files, spaces cannot be used for indention. This
refactors to follow the here-doc syntax and replaces spaces with tabs,
see [1]. The usage of spaces, breaks the OpenSSF Scorecard
`Pinned-Dependencies` check, see current scanning results at [2]:
```
Reason
internal error: error parsing shell code: libcxx/utils/ci/Dockerfile:1:1: unclosed here-document 'EOF'
```
[1] https://docs.docker.com/reference/dockerfile/#example-creating-inline-files
[2] https://securityscorecards.dev/viewer/?uri=github.com/llvm/llvm-project
---
libcxx/utils/ci/Dockerfile | 156 ++++++++++++++++++-------------------
1 file changed, 78 insertions(+), 78 deletions(-)
diff --git a/libcxx/utils/ci/Dockerfile b/libcxx/utils/ci/Dockerfile
index c77f6c435baf4d..3feb32a271a7bc 100644
--- a/libcxx/utils/ci/Dockerfile
+++ b/libcxx/utils/ci/Dockerfile
@@ -58,10 +58,10 @@ ENV LLVM_HEAD_VERSION=${LLVM_HEAD_VERSION}
# HACK: The github actions runner image already has sudo and requires its use. The buildkite base image does not.
# Reconcile this.
-RUN <<EOF
- apt-get update || true
- apt-get install -y sudo || true
- echo "ALL ALL = (ALL) NOPASSWD: ALL" | tee /etc/sudoers || true
+RUN <<-EOF
+ apt-get update || true
+ apt-get install -y sudo || true
+ echo "ALL ALL = (ALL) NOPASSWD: ALL" | tee /etc/sudoers || true
EOF
# Installing tzdata before other packages avoids the time zone prompts.
@@ -104,20 +104,20 @@ RUN sudo apt-get update \
# Install various tools used by the build or the test suite
#RUN apt-get update && apt-get install -y ninja-build python3 python3-distutils python3-psutil git gdb ccache
# TODO add ninja-build once 1.11 is available in Ubuntu, also remove the manual installation.
-RUN <<EOF
- wget -qO /tmp/ninja.gz https://github.com/ninja-build/ninja/releases/latest/download/ninja-linux.zip
- gunzip /tmp/ninja.gz
- chmod a+x /tmp/ninja
- sudo mv /tmp/ninja /usr/local/bin/ninja
+RUN <<-EOF
+ wget -qO /tmp/ninja.gz https://github.com/ninja-build/ninja/releases/latest/download/ninja-linux.zip
+ gunzip /tmp/ninja.gz
+ chmod a+x /tmp/ninja
+ sudo mv /tmp/ninja /usr/local/bin/ninja
EOF
# These two locales are not enabled by default so generate them
-RUN <<EOF
- printf "fr_CA ISO-8859-1\ncs_CZ ISO-8859-2" | sudo tee -a /etc/locale.gen
- sudo mkdir /usr/local/share/i1en/
- printf "fr_CA ISO-8859-1\ncs_CZ ISO-8859-2" | sudo tee -a /usr/local/share/i1en/SUPPORTED
- sudo locale-gen
+RUN <<-EOF
+ printf "fr_CA ISO-8859-1\ncs_CZ ISO-8859-2" | sudo tee -a /etc/locale.gen
+ sudo mkdir /usr/local/share/i1en/
+ printf "fr_CA ISO-8859-1\ncs_CZ ISO-8859-2" | sudo tee -a /usr/local/share/i1en/SUPPORTED
+ sudo locale-gen
EOF
# Install Clang <latest>, <latest-1> and ToT, which are the ones we support.
@@ -127,35 +127,35 @@ EOF
# LLVM 15, we still need to have Clang 12 in this Docker image because the LLVM
# 14 release branch CI uses it. The tip-of-trunk CI will never use Clang 12,
# though.
-RUN <<EOF
- sudo apt-get update
- wget https://apt.llvm.org/llvm.sh -O /tmp/llvm.sh
- chmod +x /tmp/llvm.sh
- sudo /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 3)) all # for CI transitions
- sudo /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 2)) all # previous release
- sudo /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 1)) all # latest release
- sudo /tmp/llvm.sh $LLVM_HEAD_VERSION all # current ToT
- sudo apt-get install -y libomp5-$LLVM_HEAD_VERSION
- sudo rm -rf /var/lib/apt/lists/*
+RUN <<-EOF
+ sudo apt-get update
+ wget https://apt.llvm.org/llvm.sh -O /tmp/llvm.sh
+ chmod +x /tmp/llvm.sh
+ sudo /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 3)) all # for CI transitions
+ sudo /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 2)) all # previous release
+ sudo /tmp/llvm.sh $(($LLVM_HEAD_VERSION - 1)) all # latest release
+ sudo /tmp/llvm.sh $LLVM_HEAD_VERSION all # current ToT
+ sudo apt-get install -y libomp5-$LLVM_HEAD_VERSION
+ sudo rm -rf /var/lib/apt/lists/*
EOF
# Install the most recent GCC, like clang install the previous version as a transition.
-RUN <<EOF
- sudo add-apt-repository ppa:ubuntu-toolchain-r/test
- sudo apt-get update
- sudo apt-get install -y \
- gcc-$((GCC_LATEST_VERSION - 1)) \
- g++-$((GCC_LATEST_VERSION - 1)) \
- gcc-$GCC_LATEST_VERSION \
- g++-$GCC_LATEST_VERSION
- sudo rm -rf /var/lib/apt/lists/*
+RUN <<-EOF
+ sudo add-apt-repository ppa:ubuntu-toolchain-r/test
+ sudo apt-get update
+ sudo apt-get install -y \
+ gcc-$((GCC_LATEST_VERSION - 1)) \
+ g++-$((GCC_LATEST_VERSION - 1)) \
+ gcc-$GCC_LATEST_VERSION \
+ g++-$GCC_LATEST_VERSION
+ sudo rm -rf /var/lib/apt/lists/*
EOF
-RUN <<EOF
- # Install a recent CMake
- wget https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1-linux-x86_64.sh -O /tmp/install-cmake.sh
- sudo bash /tmp/install-cmake.sh --prefix=/usr --exclude-subdir --skip-license
- rm /tmp/install-cmake.sh
+RUN <<-EOF
+ # Install a recent CMake
+ wget https://github.com/Kitware/CMake/releases/download/v3.21.1/cmake-3.21.1-linux-x86_64.sh -O /tmp/install-cmake.sh
+ sudo bash /tmp/install-cmake.sh --prefix=/usr --exclude-subdir --skip-license
+ rm /tmp/install-cmake.sh
EOF
# ===----------------------------------------------------------------------===##
@@ -171,12 +171,12 @@ ARG ANDROID_SYSROOT_BID
RUN apt-get update && apt-get install -y curl unzip git
# Install the Android platform tools (e.g. adb) into /opt/android/sdk.
-RUN <<EOF
- mkdir -p /opt/android/sdk
- cd /opt/android/sdk
- curl -LO https://dl.google.com/android/repository/platform-tools-latest-linux.zip
- unzip platform-tools-latest-linux.zip
- rm platform-tools-latest-linux.zip
+RUN <<-EOF
+ mkdir -p /opt/android/sdk
+ cd /opt/android/sdk
+ curl -LO https://dl.google.com/android/repository/platform-tools-latest-linux.zip
+ unzip platform-tools-latest-linux.zip
+ rm platform-tools-latest-linux.zip
EOF
# Install the current Android compiler. Specify the prebuilts commit to retrieve
@@ -184,17 +184,17 @@ EOF
ENV ANDROID_CLANG_VERSION=$ANDROID_CLANG_VERSION
ENV ANDROID_CLANG_PREBUILTS_COMMIT=$ANDROID_CLANG_PREBUILTS_COMMIT
-RUN <<EOF
- git clone --filter=blob:none --sparse \
- https://android.googlesource.com/platform/prebuilts/clang/host/linux-x86 \
- /opt/android/clang
- git -C /opt/android/clang checkout ${ANDROID_CLANG_PREBUILTS_COMMIT}
- git -C /opt/android/clang sparse-checkout add clang-${ANDROID_CLANG_VERSION}
- rm -fr /opt/android/clang/.git
- ln -sf /opt/android/clang/clang-${ANDROID_CLANG_VERSION} /opt/android/clang/clang-current
- # The "git sparse-checkout" and "ln" commands succeed even if nothing was
- # checked out, so use this "ls" command to fix that.
- ls /opt/android/clang/clang-current/bin/clang
+RUN <<-EOF
+ git clone --filter=blob:none --sparse \
+ https://android.googlesource.com/platform/prebuilts/clang/host/linux-x86 \
+ /opt/android/clang
+ git -C /opt/android/clang checkout ${ANDROID_CLANG_PREBUILTS_COMMIT}
+ git -C /opt/android/clang sparse-checkout add clang-${ANDROID_CLANG_VERSION}
+ rm -fr /opt/android/clang/.git
+ ln -sf /opt/android/clang/clang-${ANDROID_CLANG_VERSION} /opt/android/clang/clang-current
+ # The "git sparse-checkout" and "ln" commands succeed even if nothing was
+ # checked out, so use this "ls" command to fix that.
+ ls /opt/android/clang/clang-current/bin/clang
EOF
# Install an Android sysroot. New AOSP sysroots are available at
@@ -203,25 +203,25 @@ EOF
# https://android.googlesource.com/platform/prebuilts/ndk/+/refs/heads/dev/platform/sysroot.
ENV ANDROID_SYSROOT_BID=$ANDROID_SYSROOT_BID
-RUN <<EOF
- cd /opt/android
- curl -L -o ndk_platform.tar.bz2 \
- https://androidbuildinternal.googleapis.com/android/internal/build/v3/builds/${ANDROID_SYSROOT_BID}/ndk/attempts/latest/artifacts/ndk_platform.tar.bz2/url
- tar xf ndk_platform.tar.bz2
- rm ndk_platform.tar.bz2
+RUN <<-EOF
+ cd /opt/android
+ curl -L -o ndk_platform.tar.bz2 \
+ https://androidbuildinternal.googleapis.com/android/internal/build/v3/builds/${ANDROID_SYSROOT_BID}/ndk/attempts/latest/artifacts/ndk_platform.tar.bz2/url
+ tar xf ndk_platform.tar.bz2
+ rm ndk_platform.tar.bz2
EOF
# Install Docker
-RUN <<EOF
- curl -fsSL https://get.docker.com -o /tmp/get-docker.sh
- sh /tmp/get-docker.sh
- rm /tmp/get-docker.sh
-
- # Install Docker. Mark the binary setuid so it can be run without prefixing it
- # with sudo. Adding the container user to the docker group doesn't work because
- # /var/run/docker.sock is owned by the host's docker GID, not the container's
- # docker GID.
- chmod u+s /usr/bin/docker
+RUN <<-EOF
+ curl -fsSL https://get.docker.com -o /tmp/get-docker.sh
+ sh /tmp/get-docker.sh
+ rm /tmp/get-docker.sh
+
+ # Install Docker. Mark the binary setuid so it can be run without prefixing it
+ # with sudo. Adding the container user to the docker group doesn't work because
+ # /var/run/docker.sock is owned by the host's docker GID, not the container's
+ # docker GID.
+ chmod u+s /usr/bin/docker
EOF
# ===----------------------------------------------------------------------===##
@@ -240,13 +240,13 @@ WORKDIR /home/libcxx-builder
# Install the Buildkite agent and dependencies. This must be done as non-root
# for the Buildkite agent to be installed in a path where we can find it.
-RUN <<EOF
- cd /home/libcxx-builder
- curl -sL https://raw.githubusercontent.com/buildkite/agent/main/install.sh -o /tmp/install-agent.sh
- bash /tmp/install-agent.sh
- rm /tmp/install-agent.sh
- echo "tags=\"queue=libcxx-builders,arch=$(uname -m),os=linux\"" \
- >> /home/libcxx-builder/.buildkite-agent/buildkite-agent.cfg
+RUN <<-EOF
+ cd /home/libcxx-builder
+ curl -sL https://raw.githubusercontent.com/buildkite/agent/main/install.sh -o /tmp/install-agent.sh
+ bash /tmp/install-agent.sh
+ rm /tmp/install-agent.sh
+ echo "tags=\"queue=libcxx-builders,arch=$(uname -m),os=linux\"" \
+ >> /home/libcxx-builder/.buildkite-agent/buildkite-agent.cfg
EOF
USER libcxx-builder
More information about the libcxx-commits
mailing list