[lldb-dev] RFC: Separation of embedded Python from the rest of LLDB.

[Zachary Turner]

It won't be loaded into a

On Wed, Feb 25, 2015 at 11:05 AM Ted Woodward <ted.woodward at codeaurora.org>
wrote:

> The icon for pcbuild.sln has a 9 in it; I assumed that meant VS 2009.
> Silly me J
>
>
>
> I’m not confident that a DLL built with VS 2008 that has any C++ code will
> not crash when loaded into a program built with VS2013, because of the
> incompatible C++ library issue.
>
This is actually a fairly easy problem to solve.  The interface boundary
between the VS 2008 DLL and the VS2013 (or whatever version) stuff simply
needs to have some rules on it.  C++ classes are fine, the only thing that
isn't fine is allocating memory one one side which is freed on the other
side.  So you can have all the C++ you want, you just have to keep that C++
in the DLL, and not give the user access to any of its internal state that
would cause code on the other side to trigger a free.

This sounds scary, but in practice it's really not that difficult.  For
example, suppose you have ScriptInterpreterPython defined in the DLL. It
derives from a ScriptInterpreter base interface.  You don't expose
ScriptInterpreterPython outside the DLL, only ScriptInterpreter.  Anyone
who uses a pointer to ScriptInterpreter is fine, doesn't matter what side
of the boundary they're on, because the actual code for the implementation
of these methods lives in the DLL.  If you expose the full object though,
you run the risk of someone passing it by value, then have news on one side
and deletes on the other.  Another thing you can't do is pass std::strings
or similar by value (which also means you can't return them from
dllexported functions).

Since a side benefit of this refactor is to open the door for someone
dropping in a new embedded interpreter though, we would want to hide the
implementation details of the interpreter anyway, so this problem is solved
as a natural consequence of the design.  All we expose is a pointer to the
base interface, hide the rest, and pass the pointer across boundaries,
making it clear that the pointer is always owned by the DLL.

Lots of Microsoft system DLLs have C++ code in them, so this isn't really
an issue in practice as long as you define your interface.

How do we get around the debug-vs-release issue that makes us have
> python27_d.dll on a debug build right now?
>
The reason we need python27_d.dll on a debug build right now is because all
of LLDB is compiled into the a single massive DLL that is basically
lldb.exe but as a dll that python can load.  So that's actually the point
of the split.  Python only needs just enough that it can call into LLDB's
public API.  The C++ implementation of the public API classes are already
dllexported, so the SWIG generated code that wraps those will just call
into some dllexported methods, across the boundary, and into code that is
compiled with VS2013.

If a developer is hacking on the extension module itself (e.g. the
implementation of ScriptInterpreterPython), he/she will need to compile
python himself with VS2013 in order to step through the code.  But that
code is is low-traffic code and relatively stable.

Think about it like kernel32.dll and yourdll.dll.  Who knows what version
of the compiler kernel32.dll is built with.  But they can call into each
other no problem, and you can still debug your program with no problem,
even though kernel32 is a release binary.


>
> --
>
> Qualcomm Innovation Center, Inc.
>
> The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a
> Linux Foundation Collaborative Project
>
>
>
> *From:* Zachary Turner [mailto:zturner at google.com]
> *Sent:* Wednesday, February 25, 2015 12:35 PM
>
>
> *To:* Ted Woodward; Reid Kleckner; Vince Harron
> *Cc:* lldb-dev at cs.uiuc.edu
> *Subject:* Re: [lldb-dev] RFC: Separation of embedded Python from the
> rest of LLDB.
>
>
>
> Just to be clear, with solution #1 proposed in the original message, then
> your option #3 is guaranteed to work.  (also, minor pedantic nit - it's VC
> 2008, not 2009).  You won't have to build python, it will literally just
> work.  The reason for this is that I will have the CMake build
> automatically compile the separate shared library with Microsoft's
> toolchain whose sole purpose in life is to build python extension modules
> <https://www.microsoft.com/en-us/download/details.aspx?id=44266> that
> interoperate with the binary release of python 2.7
>
>
>
> On Wed, Feb 25, 2015 at 10:19 AM Ted Woodward <ted.woodward at codeaurora.org>
> wrote:
>
> I’m all for making it simpler. Believe me, coming up with my solution was
> a pain. But once I built Python, I was done with that part. We save off the
> binaries/modules, build against them, then copy them into the lib
> directory. I’ve got a patch that lets the builder set a default PYTHONHOME
> and PYTHONPATH in cmake, and if they don’t exist when LLDB is run it sets
> them in the environment based on the defaults.
>
>
>
> We need to build Python on Windows with VS2013 because the binaries from
> python.org are built with VS 2009, and will cause crashes in LLDB built
> with VS2013 because of incompatibilities between the 2009 C++ library and
> the 2012 C++ library. Our options seem to be:
>
> 1)      Build Python and save an artifact
>
> 2)      Require everyone build Python
>
> 3)      Separate the Python interface into its own shared library and
> hope that when it calls the 2009 C++RT it won’t crash
>
>
>
>
>
> I went for #1 for Hexagon LLDB because I couldn’t rely on the user having
> the correct Python installed, on Windows or Linux.
>
>
>
> --
>
> Qualcomm Innovation Center, Inc.
>
> The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a
> Linux Foundation Collaborative Project
>
>
>
> *From:* Zachary Turner [mailto:zturner at google.com]
> *Sent:* Wednesday, February 25, 2015 11:47 AM
> *To:* Ted Woodward; Reid Kleckner; Vince Harron
>
>
> *Cc:* lldb-dev at cs.uiuc.edu
> *Subject:* Re: [lldb-dev] RFC: Separation of embedded Python from the
> rest of LLDB.
>
>
>
>
>
> On Wed, Feb 25, 2015 at 9:39 AM Ted Woodward <ted.woodward at codeaurora.org>
> wrote:
>
>
>
> Saying you can’t build Python with VS2013 isn’t true. I did. I’m having
> issues with some of the bindings (specifically “print lldb.debugger” gives
> “No value”, even though it’s there), but for the most part it works.
>
> Well sure, I did too.  As it's the only way to run tests.  I think his
> point is just that it's a big barrier to entry.  I've got 3 guys who sit
> around me.  One works on the windows linker, and the other two are
> responsible for much of clang-cl.  None of them feel like going through
> these hoops to build LLDB, even though there's bugs in LLDB on Windows that
> their expertise would be a great asset in making some progress on.
>
>
>
> I mean yes I could do it for them, and yes they could ultimately do it
> themselves, but the point is that it shouldn't be this difficult.  What
> about people without the same amount of technical background, but who still
> want to hack around on the debugger?  I dont' want to discourage anyone
> from being able to work on the project.  Barriers close doors. I want the
> doors to be open.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150225/4cf3ca3d/attachment.html>