[lldb-dev] Python 3 and dotest

[Zachary Turner via lldb-dev]

Moving this to the public list because it seems useful to see what other
members of the community have to say as well.

On Mon, Oct 12, 2015 at 4:22 PM Zachary Turner <zturner at google.com> wrote:

> It's worth also pointing out that if we go the route of supporting both
> then everyone running the test suite will need to test their changes with
> both Python 2 and Python 3 anyway.  For testing LLDB on different OSes we
> don't really require this because not everyone has access to different
> hardware so we can't mandate that they test their patches on every
> supported hardware configuration.  But realistically speaking, everyone has
> access to Python 2.x and Python 3.x, so the burden should be on the person
> submitting patches to ensure that it runs on both.
>
>
> On Mon, Oct 12, 2015 at 3:24 PM Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> On Mon, Oct 12, 2015 at 3:14 PM, Zachary Turner <zturner at google.com>
>> wrote:
>>
>>> Is installing python 3 not as simple as just running a package manager
>>> and selecting python 3?  I didn't think anyone would need to be building
>>> their own, but I don't know how it works in the OSX world.
>>>
>>
>> I may fiddle with this on my off time just to see.  OS X has some pathing
>> mechanisms (like user-directory Library, plain Library, and system Library
>> directory, I think they may call them domains) that stack.  So if something
>> (like a python) installs a python framework in /Library, it can hide
>> /System/Library.  Same with $HOME/Library vs. /Library and /System/Library,
>> IIRC.  (Greg can correct me or I'll double check later).  So it is quite
>> possible that installing a new python just essentially hides the other one.
>>
>> I kinda need to see what happens though to know the real point of
>> conflict.
>>
>
Currently LLDB won't compile when using Python 3 headers.  I've fixed most
of the issues, but there are two remaining.  First is that FILE* APIs have
been removed in Python 3, and second is that some of the SWIG files use
functions like PyString_FromStringAndSize which is also removed in Python
3).  Once I fix those and get those committed, you should be able to test
on OSX and let me know what issues you run into.



>
>>
>>>
>>> I'm only really talking about requiring python 3 for running dotest,
>>> which is not something that many people do, so it doesn't seem like that
>>> big of a burden.
>>>
>>> The thing I'm concerned about it is that it's going to be a bigger
>>> burden for everyone, including Apple, if we *don't* do this.  I mean time
>>> will tell, obviously, but I don't expect this to be painless.  And it's
>>> goign to be the type of pain that doesn't go away over time, either, it's
>>> going to reoccur every time someone checks something in to the test suite.
>>> All that time is going to add up over the long run and be worse than taking
>>> the initial hit of installing python 3 on the internal build systems.
>>>
>>> There's also the issue that Todd was saying he could make some
>>> improvements to the test suite if it could use Python 3, but I'll defer to
>>> him for that since I don't know the details.
>>>
>>>
>> The async IO handling (via asyncore) would be easier to write in a
>> platform-neutral way with Python 3.5.  (Might be Python 3.3 but I seem to
>> recall they really went gung-ho on the Windows/POSIX parity side in 3.5).
>> That would benefit.  There are a few other minor places where we do things
>> to avoid incompatibility with Windows, or do something slightly more
>> awkward because we don't have a good answer on Windows.  (Depending on
>> subprocess.Popen.communicate() is one example - that could be replaced with
>> asyncore if we had Windows file handle support there).  Probably the bigger
>> benefit is almost all new work and newer ideas are getting implemented in
>> Python 3.x, where Python 2.x is just getting security fixes.  To the extent
>> that we care about that for the test suite, that's what we're missing out
>> on.
>>
>> -Todd
>>
>>
>>> On Mon, Oct 12, 2015 at 1:33 PM Greg Clayton <gclayton at apple.com> wrote:
>>>
>>>>
>>>> > On Oct 12, 2015, at 10:41 AM, Zachary Turner <zturner at google.com>
>>>> wrote:
>>>> >
>>>> > Hi Greg,
>>>> >
>>>> > I talked with Todd a little bit about this while you were on
>>>> vacation, but now that you're back I want to ask you as well.
>>>> >
>>>> > Is it possile to require Python 3 just for running the test suite?
>>>> You can see from my previous patch that gets `ScriptInterpreterPython`
>>>> ready for Python 3 that I did it in a way such that LLDB will support both
>>>> Python 2 and Python 3, so I don't intend to change that.  But probably
>>>> sometime this week I'm going to start poking at dotest.  There's a lot of
>>>> code in there, and I expect it's goign to be a challenge to support both
>>>> Python 2 and Python 3.  Not only just in terms of writing the code in a way
>>>> that the version specific differences are cleanly expressed, but also in
>>>> terms of being able to actually maintain the code so that nobody breaks
>>>> anyone else when they make changes to the test runner or add new tests.  I
>>>> can already foresee that if both 2 and 3 are supported in dotest that will
>>>> have multiple breakages a week on both sides of the fence and waste a lot
>>>> of time for everyone.
>>>> >
>>>> > Also, from what Todd told me in previous discussions, there are some
>>>> features in Python 3 that he thinks would make the test suite faster and
>>>> more robust (ask him for more details), but we can't really use them unless
>>>> everyone is on board.
>>>> >
>>>> > I know Apple has a lot of code internally that uses Python 2 and that
>>>> isn't going to change any time soon, but I guess the question is whether
>>>> that code is in the form of tests or depends on dotest in any way.  I
>>>> suspect the answer is no, but I could be wrong.  If I'm right, then it
>>>> seems like we could make everyone's life a lot easier by saying that if
>>>> you're a developer of LLDB, you need to install Python 3.  If you're a user
>>>> of LLDB's SB API for some purpose other than writing tests, you can use 2
>>>> or 3.
>>>> >
>>>> > AFAICT, swig will generate bindings that are compatible with both 2
>>>> and 3, so no issues there either.
>>>> >
>>>> > Thoughts?
>>>>
>>>> I believe the answer is as you suspect. Installing an extra python is a
>>>> large undertaking as you already know and we constantly see people hosing
>>>> up their system by installing an extra python. WWDC is full of people that
>>>> have done this and they have problems and we usually say "sorry, you are on
>>>> your own if you installed your own python". So I am not sure what people do
>>>> to hose things up or if people select the wrong arguments when
>>>> building/installing python, but it isn't going to be anything that anyone
>>>> at Apple will spend any time on as we have our hands quite full and windows
>>>> is the only thing that requires python 3 due to reasons you are all too
>>>> familiar with. So I would rather MacOS stick with the installed and fully
>>>> supported python if at all possible.
>>>>
>>>> Greg
>>>
>>>
>>
>>
>> --
>> -Todd
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151013/1cdb98b3/attachment-0001.html>