[lldb-dev] RFC: Separation of embedded Python from the rest of LLDB.

[Zachary Turner]

IANAL, but I floated this idea once and people weren't super thrilled about
the idea of checking in binaries or headers. That reminds me though that I
should ask again.

It also doesn't solve the problem for users of lldb on windows who need
support for using other extension modules in their scripts. While not me, I
believe there are some people who care about that and would like to make
that happen.
On Wed, Feb 25, 2015 at 5:58 AM Vince Harron <vharron at google.com> wrote:

> This seems like a lot of work to support Windows users who might want to
> use pre-compiled python modules.
>
> I think we should distribute a VS2015 based version of Python 2.7 binaries
> and call it a day.
>
> We can worry about 2020 problems in 2020.  =)
>
> Reasonable people may disagree.
>
>
> On Tue, Feb 24, 2015 at 11:56 PM, Zachary Turner <zturner at google.com>
> wrote:
>
>> A little background: The single biggest painpoint for working with LLDB
>> on Windows currently is Python.  There is a long
>> <https://mail.python.org/pipermail/distutils-sig/2013-February/020006.html>
>> documented <https://docs.python.org/2/extending/windows.html> history of
>> the problems with python extension modules on Windows.  Part
>> <https://msdn.microsoft.com/en-us/library/ms235460.aspx> of it is
>> Microsoft's <https://msdn.microsoft.com/en-us/library/bb531344.aspx>
>> fault, part of it is Python's fault (PEP 384
>> <https://www.python.org/dev/peps/pep-0384/> attempts to solve this, but
>> it appears stalled), but the end result is that it's really terrible and
>> there's nothing anyone can do about it.
>>
>> The implications of this for LLDB on Windows are the following:
>> 1) Anyone building LLDB on Windows needs to compile Python from source
>> 2) Even after doing so, configuring the LLDB build is difficult and
>> requires a lot of manual steps
>> 3) You can't use any other extension modules in your custom built version
>> of python, including any of the over 50,000 from PyPI, unless you also
>> build them from source, which isn't even always possible.
>>
>> If you want to be compatible with the binary release of Python and
>> interoperate with the version of Python that probably 99% of Windows people
>> have on their machine, you *must* compile your extension module with a
>> very old version of the compiler.  So old, in fact, that it's almost
>> end-of-lifed.  If it weren't for Python actually, it would have been
>> end-of-lifed already, and Microsoft ships a free version
>> <http://www.microsoft.com/en-us/download/details.aspx?id=44266> of this
>> toolchain for the express purpose of compiling python extension modules.
>>
>> I've been thinking about this for many months, and I believe I finally
>> have a solution (2 solutions actually, although I think we should do both.
>> I'll get to that later).  Both will probably be painful, but in the end for
>> the better on all platforms.
>>
>> *Solution 1:* *Decouple embedded python code from LLDB*
>> *Rationale:* Currently calls to embedded python code live in a few
>> different places.  Primarily in source/Interpreter (e.g.
>> ScriptInterpreterPython) and the generated SWIG code, but there's a few
>> utility headers scattered around.
>>
>> I'm proposing moving all of this code to a single shared library with
>> nothing else and creating some heavy restrictions on what kind of code can
>> go into this library, primarily to satisfy the requirement that it be
>> compilable with the old version of the compiler in question.  These
>> restrictions would be:
>> 1) Can't use fancy C++.  It's hard to say exactly what subset of C++ we
>> could use here, but a good rough approximation is to say C++98.
>> 2) Can't depend on any LLVM headers or even other LLDB libraries.  Other
>> LLDB projects could depend on it, but it has to stand alone to guarantee
>> that it doesn't pick up C++ that won't compile under the old MSVC compiler.
>>
>> I understand that the first restriction is very annoying, but it would
>> only be imposed upon a very small amount of source code.  About 2 or 3
>> source files.
>>
>> The second restriction, and the decoupling as a whole will probably cause
>> some pain and breakage for out-of-tree code, but I believe it's a positive
>> in the long run.  In particular, it opens the door to replacing the
>> embedded interpreter with that of a different language.  By separating the
>> code in this fashion, all one has to do is write a new module for their own
>> language and initialize it instead of ScriptInterpreterPython.  I've
>> already seen people asking on the list about generating bindings for other
>> languages and replacing the interpreter, and i believe a separation of this
>> kind is a pre-requisite anyway.
>>
>> *Solution 2:* *Upgrade to Python 3.5*
>> *Rationale:* Hopefully I didn't send you running for the hills.  This is
>> going to have to happen someday anyway.  Python 2.7 end of life is set to
>> 2020 <https://hg.python.org/peps/rev/76d43e52d978>.  Seems like a long
>> time, but it'll be here before we know it, and if we aren't prepared, it's
>> going to be a world of hurt.
>>
>> Why does this help us on Windows?  Visual Studio 2015, which releases
>> sometime this year, is finally set to have a stable ABI
>> <http://blogs.msdn.com/b/vcblog/archive/2014/06/10/the-great-crt-refactoring.aspx>
>> for it's CRT.  This means that any application compiled against VS2015 or
>> later will be forward compatible with future versions of the CRT forever
>> <https://mail.python.org/pipermail/python-dev/2014-June/134888.html>.
>> Python 3.5 is not yet released, but the current proposal is for Python
>> 3.5 <https://mail.python.org/pipermail/python-dev/2014-June/134866.html>
>> to ship its binary release compiled with VC++ 2015, effectively killing
>> this problem.
>>
>> I understand that there is a lot of out-of-tree code that is written
>> against Python 2.7.  I would encourage the people who own this code to
>> begin thinking about migrating to Python 3 soon.  In the meantime, I
>> believe we can begin to address this in-tree in 3 main phases.
>>
>> 1) Port the test suite to Python 3.5, using a subset of Python 3.5 that
>> is also compatible with 2.7.  This ensures no out of tree code is broken.
>> 2) Upgrading ScriptInterpreterPython with preprocessor flags to select
>> whether you want to link against Python 2.7 and 3.5, and expose these
>> options from CMake and Xcode so you can choose what you link against.
>> 3) Making multiple buildbots with different configurations and different
>> versions of Python to get better code coverage to ensure that tests work
>> under both language versions.
>>
>> I realize that the impact of both of these changes is high, but we have a
>> very strong desire to solve this problem on Windows and so we need to push
>> some kind of solution forward.  I think both solutions actually contribute
>> to the long term benefit of LLDB as a whole, and so I think it's worth
>> seriously considering both of these and trying to come up with a path
>> forward.
>>
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
>>
>>
>
>
> --
>
> Vince Harron | Technical Lead Manager | vharron at google.com | 858-442-0868
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150225/801bb9c2/attachment.html>