[lldb-dev] Python 3 and dotest

[Todd Fiala via lldb-dev]

On Mon, Oct 12, 2015 at 7:31 PM, Zachary Turner <zturner at google.com> wrote:

> Moving this to the public list because it seems useful to see what other
> members of the community have to say as well.
>
> On Mon, Oct 12, 2015 at 4:22 PM Zachary Turner <zturner at google.com> wrote:
>
>> It's worth also pointing out that if we go the route of supporting both
>> then everyone running the test suite will need to test their changes with
>> both Python 2 and Python 3 anyway.  For testing LLDB on different OSes we
>> don't really require this because not everyone has access to different
>> hardware so we can't mandate that they test their patches on every
>> supported hardware configuration.  But realistically speaking, everyone has
>> access to Python 2.x and Python 3.x, so the burden should be on the person
>> submitting patches to ensure that it runs on both.
>>
>>
>> On Mon, Oct 12, 2015 at 3:24 PM Todd Fiala <todd.fiala at gmail.com> wrote:
>>
>>> On Mon, Oct 12, 2015 at 3:14 PM, Zachary Turner <zturner at google.com>
>>> wrote:
>>>
>>>> Is installing python 3 not as simple as just running a package manager
>>>> and selecting python 3?  I didn't think anyone would need to be building
>>>> their own, but I don't know how it works in the OSX world.
>>>>
>>>
>>> I may fiddle with this on my off time just to see.  OS X has some
>>> pathing mechanisms (like user-directory Library, plain Library, and system
>>> Library directory, I think they may call them domains) that stack.  So if
>>> something (like a python) installs a python framework in /Library, it can
>>> hide /System/Library.  Same with $HOME/Library vs. /Library and
>>> /System/Library, IIRC.  (Greg can correct me or I'll double check later).
>>> So it is quite possible that installing a new python just essentially hides
>>> the other one.
>>>
>>> I kinda need to see what happens though to know the real point of
>>> conflict.
>>>
>>
> Currently LLDB won't compile when using Python 3 headers.  I've fixed most
> of the issues, but there are two remaining.  First is that FILE* APIs have
> been removed in Python 3, and second is that some of the SWIG files use
> functions like PyString_FromStringAndSize which is also removed in Python
> 3).  Once I fix those and get those committed, you should be able to test
> on OSX and let me know what issues you run into.
>
>
Great, looking forward to something to test!


>
>
>>
>>>
>>>>
>>>> I'm only really talking about requiring python 3 for running dotest,
>>>> which is not something that many people do, so it doesn't seem like that
>>>> big of a burden.
>>>>
>>>> The thing I'm concerned about it is that it's going to be a bigger
>>>> burden for everyone, including Apple, if we *don't* do this.  I mean time
>>>> will tell, obviously, but I don't expect this to be painless.  And it's
>>>> goign to be the type of pain that doesn't go away over time, either, it's
>>>> going to reoccur every time someone checks something in to the test suite.
>>>> All that time is going to add up over the long run and be worse than taking
>>>> the initial hit of installing python 3 on the internal build systems.
>>>>
>>>> There's also the issue that Todd was saying he could make some
>>>> improvements to the test suite if it could use Python 3, but I'll defer to
>>>> him for that since I don't know the details.
>>>>
>>>>
>>> The async IO handling (via asyncore) would be easier to write in a
>>> platform-neutral way with Python 3.5.  (Might be Python 3.3 but I seem to
>>> recall they really went gung-ho on the Windows/POSIX parity side in 3.5).
>>> That would benefit.  There are a few other minor places where we do things
>>> to avoid incompatibility with Windows, or do something slightly more
>>> awkward because we don't have a good answer on Windows.  (Depending on
>>> subprocess.Popen.communicate() is one example - that could be replaced with
>>> asyncore if we had Windows file handle support there).  Probably the bigger
>>> benefit is almost all new work and newer ideas are getting implemented in
>>> Python 3.x, where Python 2.x is just getting security fixes.  To the extent
>>> that we care about that for the test suite, that's what we're missing out
>>> on.
>>>
>>> -Todd
>>>
>>>
>>>> On Mon, Oct 12, 2015 at 1:33 PM Greg Clayton <gclayton at apple.com>
>>>> wrote:
>>>>
>>>>>
>>>>> > On Oct 12, 2015, at 10:41 AM, Zachary Turner <zturner at google.com>
>>>>> wrote:
>>>>> >
>>>>> > Hi Greg,
>>>>> >
>>>>> > I talked with Todd a little bit about this while you were on
>>>>> vacation, but now that you're back I want to ask you as well.
>>>>> >
>>>>> > Is it possile to require Python 3 just for running the test suite?
>>>>> You can see from my previous patch that gets `ScriptInterpreterPython`
>>>>> ready for Python 3 that I did it in a way such that LLDB will support both
>>>>> Python 2 and Python 3, so I don't intend to change that.  But probably
>>>>> sometime this week I'm going to start poking at dotest.  There's a lot of
>>>>> code in there, and I expect it's goign to be a challenge to support both
>>>>> Python 2 and Python 3.  Not only just in terms of writing the code in a way
>>>>> that the version specific differences are cleanly expressed, but also in
>>>>> terms of being able to actually maintain the code so that nobody breaks
>>>>> anyone else when they make changes to the test runner or add new tests.  I
>>>>> can already foresee that if both 2 and 3 are supported in dotest that will
>>>>> have multiple breakages a week on both sides of the fence and waste a lot
>>>>> of time for everyone.
>>>>> >
>>>>> > Also, from what Todd told me in previous discussions, there are some
>>>>> features in Python 3 that he thinks would make the test suite faster and
>>>>> more robust (ask him for more details), but we can't really use them unless
>>>>> everyone is on board.
>>>>> >
>>>>> > I know Apple has a lot of code internally that uses Python 2 and
>>>>> that isn't going to change any time soon, but I guess the question is
>>>>> whether that code is in the form of tests or depends on dotest in any way.
>>>>> I suspect the answer is no, but I could be wrong.  If I'm right, then it
>>>>> seems like we could make everyone's life a lot easier by saying that if
>>>>> you're a developer of LLDB, you need to install Python 3.  If you're a user
>>>>> of LLDB's SB API for some purpose other than writing tests, you can use 2
>>>>> or 3.
>>>>> >
>>>>> > AFAICT, swig will generate bindings that are compatible with both 2
>>>>> and 3, so no issues there either.
>>>>> >
>>>>> > Thoughts?
>>>>>
>>>>> I believe the answer is as you suspect. Installing an extra python is
>>>>> a large undertaking as you already know and we constantly see people hosing
>>>>> up their system by installing an extra python. WWDC is full of people that
>>>>> have done this and they have problems and we usually say "sorry, you are on
>>>>> your own if you installed your own python". So I am not sure what people do
>>>>> to hose things up or if people select the wrong arguments when
>>>>> building/installing python, but it isn't going to be anything that anyone
>>>>> at Apple will spend any time on as we have our hands quite full and windows
>>>>> is the only thing that requires python 3 due to reasons you are all too
>>>>> familiar with. So I would rather MacOS stick with the installed and fully
>>>>> supported python if at all possible.
>>>>>
>>>>> Greg
>>>>
>>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>


-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151013/c643bb1e/attachment.html>