[PATCH] D91494: Build reproducible tarballs for releases

[Aaron Puchert via Phabricator via llvm-commits]

aaronpuchert created this revision.
aaronpuchert added reviewers: hans, tstellar.
Herald added a reviewer: sscalpone.
Herald added a project: LLVM.
Herald added a subscriber: llvm-commits.
aaronpuchert requested review of this revision.
Herald added a reviewer: jdoerfert.
Herald added a subscriber: sstefan1.

Currently the tarballs contain superfluous metadata, like the user name
of the packager and via Pax headers even the PID of the tar process that
packaged the files. We build the monorepo projects directly from the git
repo using "git archive" and for the test-suite we add some flags as
recommended by https://reproducible-builds.org/docs/archives/. We don't
use numeric owners though to be compatible with "git archive".

The advantage of "git archive" is that the releaser doesn't have to
download the tar ball and extract it, rather the archive is built
directly from the repository. This is probably what GitHub uses
internally to produce the tarballs, so I wouldn't expect a difference.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D91494

Files:
  llvm/utils/release/export.sh


Index: llvm/utils/release/export.sh
===================================================================
--- llvm/utils/release/export.sh
+++ llvm/utils/release/export.sh
@@ -13,7 +13,7 @@
 
 set -e
 
-projects="llvm clang test-suite compiler-rt libcxx libcxxabi libclc clang-tools-extra polly lldb lld openmp libunwind flang"
+projects="llvm clang compiler-rt libcxx libcxxabi libclc clang-tools-extra polly lldb lld openmp libunwind flang"
 
 release=""
 rc=""
@@ -37,26 +37,34 @@
         tag="$tag-$rc"
     fi
 
-    llvm_src_dir=llvm-project-$release$rc
-    mkdir -p $llvm_src_dir
+    llvm_src_dir=$(readlink -f $(dirname "$(readlink -f "$0")")/../../..)
+    [ -d $llvm_src_dir/.git ] || ( echo "No git repository at $llvm_src_dir" ; exit 1 )
 
     echo $tag
-    echo "Fetching LLVM project source ..."
-    curl -L https://github.com/llvm/llvm-project/archive/$tag.tar.gz | \
-        tar -C $llvm_src_dir --strip-components=1 -xzf -
+    target_dir=$(pwd)
 
     echo "Creating tarball for llvm-project ..."
-    tar -cJf llvm-project-$release$rc.tar.xz $llvm_src_dir
+    pushd $llvm_src_dir/
+    git archive --prefix=llvm-project-$release$rc.src/ $tag . | xz >$target_dir/llvm-project-$release$rc.src.tar.xz
+    popd
 
-    echo "Fetching LLVM test-suite source ..."
-    mkdir -p $llvm_src_dir/test-suite
-    curl -L https://github.com/llvm/test-suite/archive/$tag.tar.gz | \
-        tar -C $llvm_src_dir/test-suite --strip-components=1 -xzf -
+    if [ ! -d test-suite-$release$rc.src ]
+    then
+      echo "Fetching LLVM test-suite source ..."
+      mkdir -p test-suite-$release$rc.src
+      curl -L https://github.com/llvm/test-suite/archive/$tag.tar.gz | \
+          tar -C test-suite-$release$rc.src --strip-components=1 -xzf -
+    fi
+    echo "Creating tarball for test-suite ..."
+    tar --sort=name --owner=0 --group=0 \
+        --pax-option=exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime \
+        -cJf test-suite-$release$rc.src.tar.xz test-suite-$release$rc.src
 
     for proj in $projects; do
         echo "Creating tarball for $proj ..."
-        mv $llvm_src_dir/$proj $llvm_src_dir/$proj-$release$rc.src
-        tar -C $llvm_src_dir -cJf $proj-$release$rc.src.tar.xz $proj-$release$rc.src
+        pushd $llvm_src_dir/$proj
+        git archive --prefix=$proj-$release$rc.src/ $tag . | xz >$target_dir/$proj-$release$rc.src.tar.xz
+        popd
     done
 }
 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: D91494.305357.patch
Type: text/x-patch
Size: 2415 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20201115/5ba86a46/attachment.bin>