[PATCH] D101446: [llvm] Improve export.sh with help and snapshot

Thu May 6 15:23:43 PDT 2021

kwk added a comment.

In D101446#2738274 <https://reviews.llvm.org/D101446#2738274>, @hans wrote:

> Is having <yyyymmdd> in the filename really the most useful thing. I'd imagine having the git ref directly might be more useful, or perhaps the output of "git-describe" on it. Or maybe it should have both the git-ref and date in the filename?
> And when using the date, perhaps it should use the date of the commit (commitdate, not authordate i guess) rather than the date when the script is run?
> Or maybe the script should take a template argument for the filenames?

Okay, the question is valid but I have a perfect reason for why you want the date of the script being run. As a package maintainer I already generate daily source tarballs and keep the last three days in this pre-release of my fork: https://github.com/kwk/llvm-project/releases/tag/source-snapshot.

Now, when I want to build a package today, I can create the download URL simply by putting the current date into this URL: `https://github.com/kwk/llvm-project/releases/download/source-snapshot/llvm-<YYYYYMMDD>.src.tar.xz`. And if I want to know which git revision or LLVM version this was, I can query these URLs respectively:

- https://github.com/kwk/llvm-project/releases/download/source-snapshot/llvm-git-revision-<YYYYYMMDD>.txt
- https://github.com/kwk/llvm-project/releases/download/source-snapshot/llvm-version-<YYYYYMMDD>.txt

If the date in here was anything else than the current date, I cannot construct the download URL. Having the date seems a bit old-school, yes, but it most certainly is easier to consume than any date of a commit which you cannot know upfront.

In D101446#2740503 <https://reviews.llvm.org/D101446#2740503>, @aaronpuchert wrote:

>> This is useful for creating daily snapshot tarballs that can easily be consumed by packagers who want to build a daily snapshot.
>
> My understanding (derived from the containing directory's name) is that this script is used for creating the official release tarballs to be published on `releases.llvm.org` or GitHub. Are there plans to publish snapshots?

Like I mentioned to @hans above, I think we could utilize github to generate daily source tarballs for us and stored them in a pre-release. The advantage of a pre-release in this case is that it doesn't show up on the repo's front-page. I'm using my own fork's pre-release: https://github.com/kwk/llvm-project/releases/tag/source-snapshot and I'm using my own github workflow to generate this on a schedule: https://github.com/kwk/llvm-project/blob/snapshot/.github/workflows/re-build-source-snapshot.yml. Here you can see the individual runs of this action: https://github.com/kwk/llvm-project/actions/workflows/re-build-source-snapshot.yml.

> Short of that it's just a "create tarball for a given git hash", which isn't really LLVM-specific. Packagers will often have their own preferred way of creating snapshots, e.g. in Open Build Service <https://openbuildservice.org/> I'd use a source service to create the snapshots automatically.
>
> But that's not up to me. If @hans and @tstellar are fine with this, it's good to go.
>
> In D101446#2738274 <https://reviews.llvm.org/D101446#2738274>, @hans wrote:
>
>> Is having <yyyymmdd> in the filename really the most useful thing. I'd imagine having the git ref directly might be more useful, or perhaps the output of "git-describe" on it. Or maybe it should have both the git-ref and date in the filename?
>
> Makes sense to me, after all the result should ideally be reproducible. (Prior to compression at least.)

I agree that it's a bit of a sacrifice. When you run the export script twice a day you get two different source tarballs for the same day that carry the same name. But I'd say that by the extra or additional files, you can always know which revision is being tar-ed: `https://github.com/kwk/llvm-project/releases/download/source-snapshot/llvm-git-revision-<YYYYYMMDD>.txt`.

I've played with both solutions, i.e. having the git checkout and tarball creation in my snapshot pipeline or outsourcing it. I must say that I liked the outsourced version better, simply because it makes packaging a bliss. All I have to do is take our regular RPM spec files and prefix the content with something like this, which is easily generated, given the `llvm-git-revision-<YYYYMMDD>.txt` and `llvm-version-<YYYYYMMDD>.txt` files:

  ################################################################################
  # BEGIN SNAPSHOT PREFIX
  ################################################################################

  %global _with_snapshot_build 1
  %bcond_with snapshot_build

  %if %{with snapshot_build}
  %global llvm_snapshot_yyyymmdd 20210506
  %global llvm_snapshot_version 13.0.0
  %global llvm_snapshot_git_revision 207b08a9130bd8167851f9e053f5d67bfd1969c8

  # Split version
  %global llvm_snapshot_version_major %{lua: print(string.match(rpm.expand("%{llvm_snapshot_version}"), "[0-9]+"));}
  %global llvm_snapshot_version_minor %{lua: print(string.match(rpm.expand("%{llvm_snapshot_version}"), "%p([0-9]+)%p"));}
  %global llvm_snapshot_version_patch %{lua: print(string.match(rpm.expand("%{llvm_snapshot_version}"), "%p([0-9]+)$"));}

  # Shorten git revision
  %global llvm_snapshot_git_revision_short %{lua: print(string.sub(rpm.expand("%llvm_snapshot_git_revision"), 0, 14));}
  %endif

  ################################################################################
  # END SNAPSHOT PREFIX
  ################################################################################

All we needed to do was to have special snapshot treatment fenced with `%if %{with snapshot_build}`:

  %if %{with snapshot_build}
  Source0:        https://github.com/kwk/llvm-project/releases/download/source-snapshot/llvm-%{llvm_snapshot_yyyymmdd}.src.tar.xz
  %else
  Source0:        https://github.com/llvm/llvm-project/releases/download/llvmorg-%{version}%{?rc_ver:-rc%{rc_ver}}/%{llvm_srcdir}.tar.xz
  Source1:        https://github.com/llvm/llvm-project/releases/download/llvmorg-%{version}%{?rc_ver:-rc%{rc_ver}}/%{llvm_srcdir}.tar.xz.sig
  Source2:        tstellar-gpg-key.asc
  %endif

Sorry to bore you with these RPM stuff. But I think it proofs a lot of benefit.

================
Comment at: llvm/utils/release/export.sh:37
+                      [-final|--final]
+                      [-snapshot|--snapshot <git-ref>] 
+
----------------
hans wrote:
> The test-release.sh script (and maybe others?) have a "-git-ref" option. Maybe that's a better name than "-snapshot", for consistency?
Agreed. Renamed.

================
Comment at: llvm/utils/release/export.sh:57
+
+Additional files being generated:
+
----------------
hans wrote:
> Additional to what?
Additional files to the source tarballs.

================
Comment at: llvm/utils/release/export.sh:91-97
+    tag="llvmorg-$release"
+
+    if [ "$rc" = "final" ]; then
+        rc=""
+    else
+        tag="$tag-$rc"
+    fi
----------------
aaronpuchert wrote:
> Seems like this duplicates 77-83, maybe remove those lines?
Oh, sure. I must have missed that. Done.

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D101446/new/

https://reviews.llvm.org/D101446