[llvm-dev] [RFC] Python 2 / Python 3 status, final step(s)

Tue Jan 5 05:32:52 PST 2021

Hi folks,

>From my experience with running the mlir-nvidia band clangd buildbot
workers and setting up pre-merge testing I can really recommend setting up
the workers using Docker (or a similar technology):

   - easy versioning/diffing/reverting of your changes
   - you can run two different images in production and staging
   - Easy to copy-and-past to run a different config
   - You can test your exact changes locally, before deploying them. This
   gives you much faster iteration times.

Downsides:

   - Need to get familiar with Docker
   - Need to workaround infrastructure issues if you have some special
   usages (e.g. GPU access)
   - Bumpy experience on Windows

My Dockerfiles are in llvm-zorg
<https://github.com/llvm/llvm-zorg/tree/main/buildbot/google/docker>, feel
free to reuse these.

Best,
Christian

On Fri, Dec 18, 2020 at 9:11 AM Serge Guelton via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> Thanks Reid for taking this risky step. As someone who just lost one day
> upgrading/downgrading/messing up its laptop, I fully appreciate the risk
> you took, and I'm glad everything looks fine.
>
> The two slaves I had issues whith were:
>
>     http://lab.llvm.org:8011/#/builders/127 (now green)
>     http://lab.llvm.org:8011/#/workers/14 (now green)
>
> Thanks a bunch!
>
> > As a long suffering maintainer
>
> And extra thanks for the maintenance task too o/
>
> On Thu, Dec 17, 2020 at 9:19 PM Reid Kleckner <rnk at google.com> wrote:
>
>> Nevermind. I went ahead and took the risk of installing Python 3 and
>> putting it on PATH, and the buildbot service code seems to be working as
>> is. Both bots have made it past cmake, so they're probably on track to
>> recover.
>>
>> On Thu, Dec 17, 2020 at 11:38 AM Reid Kleckner <rnk at google.com> wrote:
>>
>>> Can we delay this? I maintain two Windows buildbots, and I'm about to go
>>> on holiday break, and I'd really rather not scramble to get them up to date:
>>> http://lab.llvm.org:8011/#/builders/clang-x64-windows-msvc
>>> http://lab.llvm.org:8011/#/builders/sanitizer-windows
>>>
>>> Looking at them now, it seems you broke them five hours ago.
>>>
>>> There is risk involved in simply installing Python 3, since it might
>>> necessitate upgrading the buildbot worker version, which I have
>>> successfully avoided doing for almost five years now.
>>>
>>> This has not been a good month for Windows bots. As a long suffering
>>> maintainer, I'm asking for a reprieve.
>>>
>>> On Mon, Dec 14, 2020 at 10:30 PM Serge Guelton via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> Hi Folks,
>>>>
>>>> unless someone has strong arguments against it, the transition to
>>>> Python 3 will happen *this week*, on Thursday morning, european time.
>>>> The associated review is https://reviews.llvm.org/D93097, feel free to
>>>> drop in.
>>>> The impact should be low, Python3 is already picked by CMake as
>>>> default, but a Python 2 fallback still exists, and it will be removed.
>>>>
>>>> **If you're a bot owner** please make sure Python3 is available on your
>>>> system ! Now, is the time :-)
>>>>
>>>> On Mon, Oct 19, 2020 at 9:34 PM Jonas Devlieghere via llvm-dev <
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> Sounds good. In LLDB we have a slightly different timeline for the
>>>>> test suite (outlined in
>>>>> http://lists.llvm.org/pipermail/lldb-dev/2020-August/016388.html) but
>>>>> everything else aligns with the plan here. I already removed the fallback
>>>>> to Python 2 in preparation for the 12.0 release.
>>>>>
>>>>> On Tue, Oct 13, 2020 at 11:49 AM David Blaikie via llvm-dev <
>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>
>>>>>> *thumbs up*
>>>>>>
>>>>>> Re: out of date buildbots: maybe stage the shebang changes in a
>>>>>> time/way that's easy to rollback (or even intended to rollback after a
>>>>>> small time window (few hours) off-peak) to flush out the buildbots. Then
>>>>>> after that rollout, go over the buildbot failures and follow-up with
>>>>>> buildbot owners to get them cleaned up before a more intended-to-stick
>>>>>> rollout is done.
>>>>>>
>>>>>> On Tue, Oct 13, 2020 at 7:18 AM James Y Knight via llvm-dev <
>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>
>>>>>>> Yay, thanks for persevering with this migration!
>>>>>>>
>>>>>>> On Tue, Oct 13, 2020 at 8:04 AM Serge Guelton via llvm-dev <
>>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>>
>>>>>>>> Hi Folks,
>>>>>>>>
>>>>>>>> Now that LLVM 11.0.0 has been released, it's time to prepare for
>>>>>>>> the final step
>>>>>>>> envisionned in the previous RFC named *[RFC] Python 2 / Python 3
>>>>>>>> status* [0],
>>>>>>>> ie. requiring Python3.6 for LLVM 12.0.0, to be released in 2021.
>>>>>>>> At least Fedora already only ships Python3 and we didn't have much
>>>>>>>> bugs reported
>>>>>>>> wrt. Python compatibility for the LLVM toolchain.
>>>>>>>>
>>>>>>>> Indeed, all Python scripts should now be at least compatible with
>>>>>>>> both python2 (py2)
>>>>>>>> and python3 (py3), some of them already are already py3 only.
>>>>>>>>
>>>>>>>> The build system still depends on Python in a few place, but it
>>>>>>>> explicitly
>>>>>>>> mentions Python3_EXECUTABLE, and the main dependency (llvm-build)
>>>>>>>> is currently
>>>>>>>> being removed  in D89142.
>>>>>>>>
>>>>>>>> The shebangs have already been harmonized in D83857: some mention
>>>>>>>> /usr/bin/env
>>>>>>>> python, some mention /usr/bin/env python3, and none mention python2
>>>>>>>> anymore. It
>>>>>>>> would be great to have all script use the same shebang, PEP394 [1]
>>>>>>>> makes it
>>>>>>>> (relatively) clear that in our case, explicitly requiring python3
>>>>>>>> is the way to
>>>>>>>> go.
>>>>>>>>
>>>>>>>> So basically, next step would be to update the documentation,
>>>>>>>> remove the Python2
>>>>>>>> fallback in root CMakeLists.txt and update shebangs.
>>>>>>>>
>>>>>>>> Anyone seeing an issue with that approach? I'm still a bit worried
>>>>>>>> about
>>>>>>>> buildbots still running on py2 (?)
>>>>>>>>
>>>>>>>> Serge
>>>>>>>>
>>>>>>>> [0]
>>>>>>>> http://lists.llvm.org/pipermail/llvm-dev/2020-January/138730.html
>>>>>>>> [1] https://www.python.org/dev/peps/pep-0394/
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> LLVM Developers mailing list
>>>>>>>> llvm-dev at lists.llvm.org
>>>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> LLVM Developers mailing list
>>>>>>> llvm-dev at lists.llvm.org
>>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>
>>>>>> _______________________________________________
>>>>>> LLVM Developers mailing list
>>>>>> llvm-dev at lists.llvm.org
>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>
>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210105/04af5e08/attachment-0001.html>