[lldb-dev] Python 3 and dotest

Zachary Turner via lldb-dev lldb-dev at lists.llvm.org
Mon Oct 12 19:31:49 PDT 2015

Moving this to the public list because it seems useful to see what other
members of the community have to say as well.

On Mon, Oct 12, 2015 at 4:22 PM Zachary Turner <zturner at google.com> wrote:

> It's worth also pointing out that if we go the route of supporting both
> then everyone running the test suite will need to test their changes with
> both Python 2 and Python 3 anyway.  For testing LLDB on different OSes we
> don't really require this because not everyone has access to different
> hardware so we can't mandate that they test their patches on every
> supported hardware configuration.  But realistically speaking, everyone has
> access to Python 2.x and Python 3.x, so the burden should be on the person
> submitting patches to ensure that it runs on both.
> On Mon, Oct 12, 2015 at 3:24 PM Todd Fiala <todd.fiala at gmail.com> wrote:
>> On Mon, Oct 12, 2015 at 3:14 PM, Zachary Turner <zturner at google.com>
>> wrote:
>>> Is installing python 3 not as simple as just running a package manager
>>> and selecting python 3?  I didn't think anyone would need to be building
>>> their own, but I don't know how it works in the OSX world.
>> I may fiddle with this on my off time just to see.  OS X has some pathing
>> mechanisms (like user-directory Library, plain Library, and system Library
>> directory, I think they may call them domains) that stack.  So if something
>> (like a python) installs a python framework in /Library, it can hide
>> /System/Library.  Same with $HOME/Library vs. /Library and /System/Library,
>> IIRC.  (Greg can correct me or I'll double check later).  So it is quite
>> possible that installing a new python just essentially hides the other one.
>> I kinda need to see what happens though to know the real point of
>> conflict.
Currently LLDB won't compile when using Python 3 headers.  I've fixed most
of the issues, but there are two remaining.  First is that FILE* APIs have
been removed in Python 3, and second is that some of the SWIG files use
functions like PyString_FromStringAndSize which is also removed in Python
3).  Once I fix those and get those committed, you should be able to test
on OSX and let me know what issues you run into.

>>> I'm only really talking about requiring python 3 for running dotest,
>>> which is not something that many people do, so it doesn't seem like that
>>> big of a burden.
>>> The thing I'm concerned about it is that it's going to be a bigger
>>> burden for everyone, including Apple, if we *don't* do this.  I mean time
>>> will tell, obviously, but I don't expect this to be painless.  And it's
>>> goign to be the type of pain that doesn't go away over time, either, it's
>>> going to reoccur every time someone checks something in to the test suite.
>>> All that time is going to add up over the long run and be worse than taking
>>> the initial hit of installing python 3 on the internal build systems.
>>> There's also the issue that Todd was saying he could make some
>>> improvements to the test suite if it could use Python 3, but I'll defer to
>>> him for that since I don't know the details.
>> The async IO handling (via asyncore) would be easier to write in a
>> platform-neutral way with Python 3.5.  (Might be Python 3.3 but I seem to
>> recall they really went gung-ho on the Windows/POSIX parity side in 3.5).
>> That would benefit.  There are a few other minor places where we do things
>> to avoid incompatibility with Windows, or do something slightly more
>> awkward because we don't have a good answer on Windows.  (Depending on
>> subprocess.Popen.communicate() is one example - that could be replaced with
>> asyncore if we had Windows file handle support there).  Probably the bigger
>> benefit is almost all new work and newer ideas are getting implemented in
>> Python 3.x, where Python 2.x is just getting security fixes.  To the extent
>> that we care about that for the test suite, that's what we're missing out
>> on.
>> -Todd
>>> On Mon, Oct 12, 2015 at 1:33 PM Greg Clayton <gclayton at apple.com> wrote:
>>>> > On Oct 12, 2015, at 10:41 AM, Zachary Turner <zturner at google.com>
>>>> wrote:
>>>> >
>>>> > Hi Greg,
>>>> >
>>>> > I talked with Todd a little bit about this while you were on
>>>> vacation, but now that you're back I want to ask you as well.
>>>> >
>>>> > Is it possile to require Python 3 just for running the test suite?
>>>> You can see from my previous patch that gets `ScriptInterpreterPython`
>>>> ready for Python 3 that I did it in a way such that LLDB will support both
>>>> Python 2 and Python 3, so I don't intend to change that.  But probably
>>>> sometime this week I'm going to start poking at dotest.  There's a lot of
>>>> code in there, and I expect it's goign to be a challenge to support both
>>>> Python 2 and Python 3.  Not only just in terms of writing the code in a way
>>>> that the version specific differences are cleanly expressed, but also in
>>>> terms of being able to actually maintain the code so that nobody breaks
>>>> anyone else when they make changes to the test runner or add new tests.  I
>>>> can already foresee that if both 2 and 3 are supported in dotest that will
>>>> have multiple breakages a week on both sides of the fence and waste a lot
>>>> of time for everyone.
>>>> >
>>>> > Also, from what Todd told me in previous discussions, there are some
>>>> features in Python 3 that he thinks would make the test suite faster and
>>>> more robust (ask him for more details), but we can't really use them unless
>>>> everyone is on board.
>>>> >
>>>> > I know Apple has a lot of code internally that uses Python 2 and that
>>>> isn't going to change any time soon, but I guess the question is whether
>>>> that code is in the form of tests or depends on dotest in any way.  I
>>>> suspect the answer is no, but I could be wrong.  If I'm right, then it
>>>> seems like we could make everyone's life a lot easier by saying that if
>>>> you're a developer of LLDB, you need to install Python 3.  If you're a user
>>>> of LLDB's SB API for some purpose other than writing tests, you can use 2
>>>> or 3.
>>>> >
>>>> > AFAICT, swig will generate bindings that are compatible with both 2
>>>> and 3, so no issues there either.
>>>> >
>>>> > Thoughts?
>>>> I believe the answer is as you suspect. Installing an extra python is a
>>>> large undertaking as you already know and we constantly see people hosing
>>>> up their system by installing an extra python. WWDC is full of people that
>>>> have done this and they have problems and we usually say "sorry, you are on
>>>> your own if you installed your own python". So I am not sure what people do
>>>> to hose things up or if people select the wrong arguments when
>>>> building/installing python, but it isn't going to be anything that anyone
>>>> at Apple will spend any time on as we have our hands quite full and windows
>>>> is the only thing that requires python 3 due to reasons you are all too
>>>> familiar with. So I would rather MacOS stick with the installed and fully
>>>> supported python if at all possible.
>>>> Greg
>> --
>> -Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151013/1cdb98b3/attachment-0001.html>

More information about the lldb-dev mailing list