[lldb-dev] RFC: Separation of embedded Python from the rest of LLDB.

[Zachary Turner]

Sorry, hit send too soon.  The big thing that will change is people who do:

#include "lldb-python.h"

and

#if !defined(LLDB_DISABLE_PYTHON)

There are currently a lot of instances of this in the code.  The path
forward for this would be to gradually change #if
defined(LLDB_DISABLE_PYTHON) to simply checking if GetScriptInterpreter()
returns null.  I scanned about 10% of the occurrences of this and there's
nothing in any of them that are python specific.

I don't know how much of this kind of thing is out of tree, but either way
it's a purely mechanical change.  If there are places that are actually
relying on it being a ScriptInterpreterPython, then that's an error in the
code, because as you said it was designed to be pluggable.

On Wed, Feb 25, 2015 at 1:33 PM Zachary Turner <zturner at google.com> wrote:

> Also, I don't think the fixups necessary for out of tree code associated
> with separating the library will be anything larger than what would need to
> be done for someone adding another language anyway.  Some header and source
> files will move around, and the build dependency structure will change.
> The big thing this will break is people who do
>
> #include "lldb/lldb-python.h"
>
> On Wed, Feb 25, 2015 at 11:47 AM Zachary Turner <zturner at google.com>
> wrote:
>
>> Most Python stuff is in ScriptInterpreterPython (see
>> Include/Utility/PythonPointer.h for an example of code that isn't).  But
>> ScriptInterpreterPython is just compiled straight into Interpreter, instead
>> of into its own library.  That's the part that isn't working.  If someone
>> wants to add support for another language, it would be nice to do it in a
>> way that's upstreamable.  The way to do that in my opinion isn't to just
>> keep sticking new implementations into Interpreter, it's to separate the
>> python interpreter code into its own library that can be conditionally
>> linked in or not.
>>
>> If we separate the library, then there is a little bit of upfront pain to
>> fix out of tree code, and after that initial hurdle is overcome, everything
>> is back to normal.  If we fragment the testing environments into one
>> subset of people that only cares about python 2.7 and another subset of
>> people that only cares about 3.5, then there will be contiuous burden on
>> everyone to not break the other person.  So I don't think we should do the
>> python 3.5 solution unless *everyone* is on board to start moving to
>> 3.5.  Anwyay, python 3.5 isn't even released yet, nor is the C++ compiler.
>> The compiler is targeting summer of this year and python 3.5 is targeting
>> september of this year (I believe).
>>
>>
>> On Wed, Feb 25, 2015 at 11:31 AM Greg Clayton <gclayton at apple.com> wrote:
>>
>>> We had planned on python being able to be replaced by another language
>>> and abstracted all python stuff into the pure virtual ScriptInterpreter and
>>> the one and only scripting subclass ScriptInterpreterPython. What part of
>>> this abstraction isn't working for people? I don't see the need for heroic
>>> efforts that are proposed without a specific reason. All python specific
>>> stuff is in ScriptInterpreterPython and a few swig class. Adding support
>>> for another language should be as simple as configuring swig and making it
>>> generate all of the classes and subclassing ScriptInterpreter. Python can
>>> be disabled for people that don't want/need python support, as any language
>>> that we add should be.
>>>
>>> So I like solution 2 where only Windows uses Python 3.5 and we allow the
>>> test suite to be updated so that it can run with 3.5, yet it still runs on
>>> 2.7.
>>>
>>> On Feb 24, 2015, at 8:56 PM, Zachary Turner <zturner at google.com> wrote:
>>> >
>>> > A little background: The single biggest painpoint for working with
>>> LLDB on Windows currently is Python.  There is a long documented history of
>>> the problems with python extension modules on Windows.  Part of it is
>>> Microsoft's fault, part of it is Python's fault (PEP 384 attempts to solve
>>> this, but it appears stalled), but the end result is that it's really
>>> terrible and there's nothing anyone can do about it.
>>> >
>>> > The implications of this for LLDB on Windows are the following:
>>> > 1) Anyone building LLDB on Windows needs to compile Python from source
>>> > 2) Even after doing so, configuring the LLDB build is difficult and
>>> requires a lot of manual steps
>>> > 3) You can't use any other extension modules in your custom built
>>> version of python, including any of the over 50,000 from PyPI, unless you
>>> also build them from source, which isn't even always possible.
>>> >
>>> > If you want to be compatible with the binary release of Python and
>>> interoperate with the version of Python that probably 99% of Windows people
>>> have on their machine, you must compile your extension module with a very
>>> old version of the compiler.  So old, in fact, that it's almost
>>> end-of-lifed.  If it weren't for Python actually, it would have been
>>> end-of-lifed already, and Microsoft ships a free version of this toolchain
>>> for the express purpose of compiling python extension modules.
>>> >
>>> > I've been thinking about this for many months, and I believe I finally
>>> have a solution (2 solutions actually, although I think we should do both.
>>> I'll get to that later).  Both will probably be painful, but in the end for
>>> the better on all platforms.
>>> >
>>> > Solution 1: Decouple embedded python code from LLDB
>>> > Rationale: Currently calls to embedded python code live in a few
>>> different places.  Primarily in source/Interpreter (e.g.
>>> ScriptInterpreterPython) and the generated SWIG code, but there's a few
>>> utility headers scattered around.
>>> >
>>> > I'm proposing moving all of this code to a single shared library with
>>> nothing else and creating some heavy restrictions on what kind of code can
>>> go into this library, primarily to satisfy the requirement that it be
>>> compilable with the old version of the compiler in question.  These
>>> restrictions would be:
>>> > 1) Can't use fancy C++.  It's hard to say exactly what subset of C++
>>> we could use here, but a good rough approximation is to say C++98.
>>> > 2) Can't depend on any LLVM headers or even other LLDB libraries.
>>> Other LLDB projects could depend on it, but it has to stand alone to
>>> guarantee that it doesn't pick up C++ that won't compile under the old MSVC
>>> compiler.
>>> >
>>> > I understand that the first restriction is very annoying, but it would
>>> only be imposed upon a very small amount of source code.  About 2 or 3
>>> source files.
>>> >
>>> > The second restriction, and the decoupling as a whole will probably
>>> cause some pain and breakage for out-of-tree code, but I believe it's a
>>> positive in the long run.  In particular, it opens the door to replacing
>>> the embedded interpreter with that of a different language.  By separating
>>> the code in this fashion, all one has to do is write a new module for their
>>> own language and initialize it instead of ScriptInterpreterPython.  I've
>>> already seen people asking on the list about generating bindings for other
>>> languages and replacing the interpreter, and i believe a separation of this
>>> kind is a pre-requisite anyway.
>>> >
>>> > Solution 2: Upgrade to Python 3.5
>>> > Rationale: Hopefully I didn't send you running for the hills.  This is
>>> going to have to happen someday anyway.  Python 2.7 end of life is set to
>>> 2020.  Seems like a long time, but it'll be here before we know it, and if
>>> we aren't prepared, it's going to be a world of hurt.
>>> >
>>> > Why does this help us on Windows?  Visual Studio 2015, which releases
>>> sometime this year, is finally set to have a stable ABI for it's CRT.  This
>>> means that any application compiled against VS2015 or later will be forward
>>> compatible with future versions of the CRT forever.  Python 3.5 is not yet
>>> released, but the current proposal is for Python 3.5 to ship its binary
>>> release compiled with VC++ 2015, effectively killing this problem.
>>> >
>>> > I understand that there is a lot of out-of-tree code that is written
>>> against Python 2.7.  I would encourage the people who own this code to
>>> begin thinking about migrating to Python 3 soon.  In the meantime, I
>>> believe we can begin to address this in-tree in 3 main phases.
>>> >
>>> > 1) Port the test suite to Python 3.5, using a subset of Python 3.5
>>> that is also compatible with 2.7.  This ensures no out of tree code is
>>> broken.
>>> > 2) Upgrading ScriptInterpreterPython with preprocessor flags to select
>>> whether you want to link against Python 2.7 and 3.5, and expose these
>>> options from CMake and Xcode so you can choose what you link against.
>>> > 3) Making multiple buildbots with different configurations and
>>> different versions of Python to get better code coverage to ensure that
>>> tests work under both language versions.
>>> >
>>> > I realize that the impact of both of these changes is high, but we
>>> have a very strong desire to solve this problem on Windows and so we need
>>> to push some kind of solution forward.  I think both solutions actually
>>> contribute to the long term benefit of LLDB as a whole, and so I think it's
>>> worth seriously considering both of these and trying to come up with a path
>>> forward.
>>> > _______________________________________________
>>> > lldb-dev mailing list
>>> > lldb-dev at cs.uiuc.edu
>>> > http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150225/f8df5965/attachment.html>