[lldb-dev] RFC: Separation of embedded Python from the rest of LLDB.

Wed Feb 25 13:33:35 PST 2015

Also, I don't think the fixups necessary for out of tree code associated
with separating the library will be anything larger than what would need to
be done for someone adding another language anyway.  Some header and source
files will move around, and the build dependency structure will change.
The big thing this will break is people who do

#include "lldb/lldb-python.h"

On Wed, Feb 25, 2015 at 11:47 AM Zachary Turner <zturner at google.com> wrote:

> Most Python stuff is in ScriptInterpreterPython (see
> Include/Utility/PythonPointer.h for an example of code that isn't).  But
> ScriptInterpreterPython is just compiled straight into Interpreter, instead
> of into its own library.  That's the part that isn't working.  If someone
> wants to add support for another language, it would be nice to do it in a
> way that's upstreamable.  The way to do that in my opinion isn't to just
> keep sticking new implementations into Interpreter, it's to separate the
> python interpreter code into its own library that can be conditionally
> linked in or not.
>
> If we separate the library, then there is a little bit of upfront pain to
> fix out of tree code, and after that initial hurdle is overcome, everything
> is back to normal.  If we fragment the testing environments into one
> subset of people that only cares about python 2.7 and another subset of
> people that only cares about 3.5, then there will be contiuous burden on
> everyone to not break the other person.  So I don't think we should do the
> python 3.5 solution unless *everyone* is on board to start moving to
> 3.5.  Anwyay, python 3.5 isn't even released yet, nor is the C++ compiler.
> The compiler is targeting summer of this year and python 3.5 is targeting
> september of this year (I believe).
>
>
> On Wed, Feb 25, 2015 at 11:31 AM Greg Clayton <gclayton at apple.com> wrote:
>
>> We had planned on python being able to be replaced by another language
>> and abstracted all python stuff into the pure virtual ScriptInterpreter and
>> the one and only scripting subclass ScriptInterpreterPython. What part of
>> this abstraction isn't working for people? I don't see the need for heroic
>> efforts that are proposed without a specific reason. All python specific
>> stuff is in ScriptInterpreterPython and a few swig class. Adding support
>> for another language should be as simple as configuring swig and making it
>> generate all of the classes and subclassing ScriptInterpreter. Python can
>> be disabled for people that don't want/need python support, as any language
>> that we add should be.
>>
>> So I like solution 2 where only Windows uses Python 3.5 and we allow the
>> test suite to be updated so that it can run with 3.5, yet it still runs on
>> 2.7.
>>
>> On Feb 24, 2015, at 8:56 PM, Zachary Turner <zturner at google.com> wrote:
>> >
>> > A little background: The single biggest painpoint for working with LLDB
>> on Windows currently is Python.  There is a long documented history of the
>> problems with python extension modules on Windows.  Part of it is
>> Microsoft's fault, part of it is Python's fault (PEP 384 attempts to solve
>> this, but it appears stalled), but the end result is that it's really
>> terrible and there's nothing anyone can do about it.
>> >
>> > The implications of this for LLDB on Windows are the following:
>> > 1) Anyone building LLDB on Windows needs to compile Python from source
>> > 2) Even after doing so, configuring the LLDB build is difficult and
>> requires a lot of manual steps
>> > 3) You can't use any other extension modules in your custom built
>> version of python, including any of the over 50,000 from PyPI, unless you
>> also build them from source, which isn't even always possible.
>> >
>> > If you want to be compatible with the binary release of Python and
>> interoperate with the version of Python that probably 99% of Windows people
>> have on their machine, you must compile your extension module with a very
>> old version of the compiler.  So old, in fact, that it's almost
>> end-of-lifed.  If it weren't for Python actually, it would have been
>> end-of-lifed already, and Microsoft ships a free version of this toolchain
>> for the express purpose of compiling python extension modules.
>> >
>> > I've been thinking about this for many months, and I believe I finally
>> have a solution (2 solutions actually, although I think we should do both.
>> I'll get to that later).  Both will probably be painful, but in the end for
>> the better on all platforms.
>> >
>> > Solution 1: Decouple embedded python code from LLDB
>> > Rationale: Currently calls to embedded python code live in a few
>> different places.  Primarily in source/Interpreter (e.g.
>> ScriptInterpreterPython) and the generated SWIG code, but there's a few
>> utility headers scattered around.
>> >
>> > I'm proposing moving all of this code to a single shared library with
>> nothing else and creating some heavy restrictions on what kind of code can
>> go into this library, primarily to satisfy the requirement that it be
>> compilable with the old version of the compiler in question.  These
>> restrictions would be:
>> > 1) Can't use fancy C++.  It's hard to say exactly what subset of C++ we
>> could use here, but a good rough approximation is to say C++98.
>> > 2) Can't depend on any LLVM headers or even other LLDB libraries.
>> Other LLDB projects could depend on it, but it has to stand alone to
>> guarantee that it doesn't pick up C++ that won't compile under the old MSVC
>> compiler.
>> >
>> > I understand that the first restriction is very annoying, but it would
>> only be imposed upon a very small amount of source code.  About 2 or 3
>> source files.
>> >
>> > The second restriction, and the decoupling as a whole will probably
>> cause some pain and breakage for out-of-tree code, but I believe it's a
>> positive in the long run.  In particular, it opens the door to replacing
>> the embedded interpreter with that of a different language.  By separating
>> the code in this fashion, all one has to do is write a new module for their
>> own language and initialize it instead of ScriptInterpreterPython.  I've
>> already seen people asking on the list about generating bindings for other
>> languages and replacing the interpreter, and i believe a separation of this
>> kind is a pre-requisite anyway.
>> >
>> > Solution 2: Upgrade to Python 3.5
>> > Rationale: Hopefully I didn't send you running for the hills.  This is
>> going to have to happen someday anyway.  Python 2.7 end of life is set to
>> 2020.  Seems like a long time, but it'll be here before we know it, and if
>> we aren't prepared, it's going to be a world of hurt.
>> >
>> > Why does this help us on Windows?  Visual Studio 2015, which releases
>> sometime this year, is finally set to have a stable ABI for it's CRT.  This
>> means that any application compiled against VS2015 or later will be forward
>> compatible with future versions of the CRT forever.  Python 3.5 is not yet
>> released, but the current proposal is for Python 3.5 to ship its binary
>> release compiled with VC++ 2015, effectively killing this problem.
>> >
>> > I understand that there is a lot of out-of-tree code that is written
>> against Python 2.7.  I would encourage the people who own this code to
>> begin thinking about migrating to Python 3 soon.  In the meantime, I
>> believe we can begin to address this in-tree in 3 main phases.
>> >
>> > 1) Port the test suite to Python 3.5, using a subset of Python 3.5 that
>> is also compatible with 2.7.  This ensures no out of tree code is broken.
>> > 2) Upgrading ScriptInterpreterPython with preprocessor flags to select
>> whether you want to link against Python 2.7 and 3.5, and expose these
>> options from CMake and Xcode so you can choose what you link against.
>> > 3) Making multiple buildbots with different configurations and
>> different versions of Python to get better code coverage to ensure that
>> tests work under both language versions.
>> >
>> > I realize that the impact of both of these changes is high, but we have
>> a very strong desire to solve this problem on Windows and so we need to
>> push some kind of solution forward.  I think both solutions actually
>> contribute to the long term benefit of LLDB as a whole, and so I think it's
>> worth seriously considering both of these and trying to come up with a path
>> forward.
>> > _______________________________________________
>> > lldb-dev mailing list
>> > lldb-dev at cs.uiuc.edu
>> > http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150225/b8f0926b/attachment.html>