[llvm-dev] [cfe-dev] [Openmp-dev] Bugzilla migration is stopped again

[Anton Korobeynikov via llvm-dev]

Dear Arthur,

> Respectfully, yet frustrated with the never-ending email thread,
I understand your frustration and please rest assured that my own
frustration is certainly not less than yours. I'm also very exhausted
at the moment as the things are beyond my control. The constant
pushing from this and similar emails does not help in resolving the
situation. I certainly have to note that your accusations in "we'll do
it live section" are not quite accurate in many aspects - if you have
not seen the outcome of test imports, then it does not mean that there
were none. I would say even more, this means that they were successful
as nothing triggered excessive notifications (we made such a mistake
once – you could even find the reports of this in the MLs) . For your
information: the last "dry run import" which gated the migration was
14th full (52k issues) try. In all previous runs issues were found and
either a workaround was prepared or they were reported to GitHub.

Now let's proceed from the emotions to the real things. I do
appreciate your willingness to provide the help. Please see my notes
below.

> I'll test it out this week on a blank repo, with the goal of mirroring a 100-bug subset of the LLVM Bugzilla publicly visible in https://github.com/Quuxplusone/LLVMBugzillaTest/ by EOW.
First of all, proof-of-concept should certainly be 10k issues on
non-blank repo. It should already have closed issues / pull requests
in order to represent the real llvm-project repo.

Now down to details:
0. I would suggest you not to use the GitHub API. YMMV, but from our
experience: API is rate limited, and many things are outside your
control including:
  - ids
  - timestamps
  - notifications
1. The real migration starts from a local gitlab instance, where you
import all bugzilla issues. You can certainly skip this step in your
own experiments and proceed directly to step 2, but this will allow
you to check the outcome of the import. The script we used could be
found at https://github.com/llvm/bugzilla2gitlab/tree/llvm

2. Then you need to prepare the dump which could be consumed by GitHub
Enterprise Migration API:
https://docs.github.com/en/rest/reference/migrations We are using
gitlab-to-github scripts provided by GitHub. I'm not sure I can share
them as they are not public – I will ask GH support engineers on
Monday and will return to you.

3. After the dump is prepared you need to upload it via GitHub
Enterprise Migration API. Note that import is only possible into empty
repo (it is essentially created). If the import failed you'd need to
ask GitHub engineers whether the error is real or whether it could be
ignored. If the error is real, then you'd likely need to restart from
scratch – it is possible to resume, but practice shows that this might
create duplicate comments.

4. After the import finished check the results: number of objects
(issues, comments, attachments) that were imported. If there are any
objects that failed import, then you need to figure out which ones and
what to do. Your options are: ignore or restart the import. Here is
the checklist I'm using for the content:
https://docs.google.com/document/d/1G6DZ6AxzSaOlrtTxoxtqYKnD4Myv40QfKK4wj54y8ms/edit

5. At this point one should have something similar to
https://github.com/llvm/llvm-bugzilla-archive

6. In order to transfer issues from the archive to the live repo there
are two options:
  - Use GitHub rate-limited API
  - Ask GitHub folks
The first variant triggers notifications to everyone mentioned,
assigned or commented on the issue. There is no way to silence these
notifications.
In our case here we are relying on GitHub support engineers that do
this migration step for us. There is no API, no script, nothing that
is within our control. We did several test migrations from dry-run
repo to another repo (and this is how we found all bugs wrt issue
transfer in the past). As I already said, the circular reference
rewriting was not included into my original checklist - I expected
that this feature "just works" and was only spotted later.

Hope this helps. Should you have more questions, I will certainly be
happy to help you. I'm interested in finishing this 2+ year project
more than anyone else.

-- 
With best regards, Anton Korobeynikov
Department of Statistical Modelling, Saint Petersburg State University