[lldb-dev] RFC: Separation of embedded Python from the rest of LLDB.

[Vince Harron]

This seems like a lot of work to support Windows users who might want to
use pre-compiled python modules.

I think we should distribute a VS2015 based version of Python 2.7 binaries
and call it a day.

We can worry about 2020 problems in 2020.  =)

Reasonable people may disagree.


On Tue, Feb 24, 2015 at 11:56 PM, Zachary Turner <zturner at google.com> wrote:

> A little background: The single biggest painpoint for working with LLDB on
> Windows currently is Python.  There is a long
> <https://mail.python.org/pipermail/distutils-sig/2013-February/020006.html>
> documented <https://docs.python.org/2/extending/windows.html> history of
> the problems with python extension modules on Windows.  Part
> <https://msdn.microsoft.com/en-us/library/ms235460.aspx> of it is
> Microsoft's <https://msdn.microsoft.com/en-us/library/bb531344.aspx>
> fault, part of it is Python's fault (PEP 384
> <https://www.python.org/dev/peps/pep-0384/> attempts to solve this, but
> it appears stalled), but the end result is that it's really terrible and
> there's nothing anyone can do about it.
>
> The implications of this for LLDB on Windows are the following:
> 1) Anyone building LLDB on Windows needs to compile Python from source
> 2) Even after doing so, configuring the LLDB build is difficult and
> requires a lot of manual steps
> 3) You can't use any other extension modules in your custom built version
> of python, including any of the over 50,000 from PyPI, unless you also
> build them from source, which isn't even always possible.
>
> If you want to be compatible with the binary release of Python and
> interoperate with the version of Python that probably 99% of Windows people
> have on their machine, you *must* compile your extension module with a
> very old version of the compiler.  So old, in fact, that it's almost
> end-of-lifed.  If it weren't for Python actually, it would have been
> end-of-lifed already, and Microsoft ships a free version
> <http://www.microsoft.com/en-us/download/details.aspx?id=44266> of this
> toolchain for the express purpose of compiling python extension modules.
>
> I've been thinking about this for many months, and I believe I finally
> have a solution (2 solutions actually, although I think we should do both.
> I'll get to that later).  Both will probably be painful, but in the end for
> the better on all platforms.
>
> *Solution 1:* *Decouple embedded python code from LLDB*
> *Rationale:* Currently calls to embedded python code live in a few
> different places.  Primarily in source/Interpreter (e.g.
> ScriptInterpreterPython) and the generated SWIG code, but there's a few
> utility headers scattered around.
>
> I'm proposing moving all of this code to a single shared library with
> nothing else and creating some heavy restrictions on what kind of code can
> go into this library, primarily to satisfy the requirement that it be
> compilable with the old version of the compiler in question.  These
> restrictions would be:
> 1) Can't use fancy C++.  It's hard to say exactly what subset of C++ we
> could use here, but a good rough approximation is to say C++98.
> 2) Can't depend on any LLVM headers or even other LLDB libraries.  Other
> LLDB projects could depend on it, but it has to stand alone to guarantee
> that it doesn't pick up C++ that won't compile under the old MSVC compiler.
>
> I understand that the first restriction is very annoying, but it would
> only be imposed upon a very small amount of source code.  About 2 or 3
> source files.
>
> The second restriction, and the decoupling as a whole will probably cause
> some pain and breakage for out-of-tree code, but I believe it's a positive
> in the long run.  In particular, it opens the door to replacing the
> embedded interpreter with that of a different language.  By separating the
> code in this fashion, all one has to do is write a new module for their own
> language and initialize it instead of ScriptInterpreterPython.  I've
> already seen people asking on the list about generating bindings for other
> languages and replacing the interpreter, and i believe a separation of this
> kind is a pre-requisite anyway.
>
> *Solution 2:* *Upgrade to Python 3.5*
> *Rationale:* Hopefully I didn't send you running for the hills.  This is
> going to have to happen someday anyway.  Python 2.7 end of life is set to
> 2020 <https://hg.python.org/peps/rev/76d43e52d978>.  Seems like a long
> time, but it'll be here before we know it, and if we aren't prepared, it's
> going to be a world of hurt.
>
> Why does this help us on Windows?  Visual Studio 2015, which releases
> sometime this year, is finally set to have a stable ABI
> <http://blogs.msdn.com/b/vcblog/archive/2014/06/10/the-great-crt-refactoring.aspx>
> for it's CRT.  This means that any application compiled against VS2015 or
> later will be forward compatible with future versions of the CRT forever
> <https://mail.python.org/pipermail/python-dev/2014-June/134888.html>.
> Python 3.5 is not yet released, but the current proposal is for Python 3.5
> <https://mail.python.org/pipermail/python-dev/2014-June/134866.html> to
> ship its binary release compiled with VC++ 2015, effectively killing this
> problem.
>
> I understand that there is a lot of out-of-tree code that is written
> against Python 2.7.  I would encourage the people who own this code to
> begin thinking about migrating to Python 3 soon.  In the meantime, I
> believe we can begin to address this in-tree in 3 main phases.
>
> 1) Port the test suite to Python 3.5, using a subset of Python 3.5 that is
> also compatible with 2.7.  This ensures no out of tree code is broken.
> 2) Upgrading ScriptInterpreterPython with preprocessor flags to select
> whether you want to link against Python 2.7 and 3.5, and expose these
> options from CMake and Xcode so you can choose what you link against.
> 3) Making multiple buildbots with different configurations and different
> versions of Python to get better code coverage to ensure that tests work
> under both language versions.
>
> I realize that the impact of both of these changes is high, but we have a
> very strong desire to solve this problem on Windows and so we need to push
> some kind of solution forward.  I think both solutions actually contribute
> to the long term benefit of LLDB as a whole, and so I think it's worth
> seriously considering both of these and trying to come up with a path
> forward.
>
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
>
>


-- 

Vince Harron | Technical Lead Manager | vharron at google.com | 858-442-0868
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150225/847b9ac4/attachment.html>