[lldb-dev] LLDB_PYTHON_HOME

Mon May 11 16:34:19 PDT 2020

Ug.  After a rebase, this problem has again resurfaced for me.

* PATH has only C:\Python36
* cmake version 3.15.19101501-MSVC_2
* cmake -GNinja -DLLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN=ON
-DCMAKE_BUILD_TYPE=Debug -DLLDB_TEST_DEBUG_TEST_CRASHES=1
-DPYTHON_HOME=C:\Python36
-DLLDB_PYTHON_HOME=C:\Python36 -DPython3_ROOT_DIR=C:\Python36
-DLLDB_TEST_COMPILER=D:\src\llvm\build\ninja_dbg\bin\clang.exe
..\..\llvm-project\llvm -DLLVM_ENABLE_ZLIB=OFF
-DLLVM_ENABLE_PROJECTS="clang;lld;lldb;debuginfo-tests"

Yet when linking LLDB, it goes looking at the Python 3.7 installation that
comes with Visual Studio.  That wouldn't be much of a problem unless you're
trying to build debug, which I am.  The VS version, doesn't come with debug
versions of the interpreter libraries, so the link fails:

[1/7] Linking CXX shared library bin\liblldb.dll
FAILED: bin/liblldb.dll lib/liblldb.lib
cmd.exe /C "cd . && "C:\Program Files (x86)\Microsoft Visual
Studio\2019\Professional\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe"
-E vs_link_dll --intdir=tools\lldb\source\API\CMakeFiles\liblldb.dir
--rc=C:\PROGRA~2\WI3CF2~1\10\bin\100183~1.0\x64\rc.exe
--mt=C:\PROGRA~2\WI3CF2~1\10\bin\100183~1.0\x64\mt.exe --manifests  --
C:\PROGRA~2\MICROS~1\2019\PROFES~1\VC\Tools\MSVC\1424~1.283\bin\Hostx64\x64\link.exe
/nologo @CMakeFiles\liblldb.rsp  /out:bin\liblldb.dll
/implib:lib\liblldb.lib /pdb:bin\liblldb.pdb /dll /version:11.0
/machine:x64 /debug /INCREMENTAL   && cd ."
LINK Pass 1: command
"C:\PROGRA~2\MICROS~1\2019\PROFES~1\VC\Tools\MSVC\1424~1.283\bin\Hostx64\x64\link.exe
/nologo @CMakeFiles\liblldb.rsp /out:bin\liblldb.dll
/implib:lib\liblldb.lib /pdb:bin\liblldb.pdb /dll /version:11.0
/machine:x64 /debug /INCREMENTAL /MANIFEST
/MANIFESTFILE:tools\lldb\source\API\CMakeFiles\liblldb.dir/intermediate.manifest
tools\lldb\source\API\CMakeFiles\liblldb.dir/manifest.res" failed (exit
code 1104) with the following output:
LINK : fatal error LNK1104: cannot open file 'python37_d.lib'
ninja: build stopped: subcommand failed.

It looks like Cmake has added several more hints for finding Python
<https://cmake.org/cmake/help/v3.15/module/FindPython3.html#hints>.  I'm
trying now with -DPython3_FIND_REGISTRY=LAST.

I miss hermetic builds.

On Thu, Feb 27, 2020 at 11:33 AM Adrian McCarthy <amccarth at google.com>
wrote:

> >  This was all fixed in CMake 3.12.
>
> For some definitions of "all fixed." ;-)
>
> It seems weird to me that FindPython3 found the VS-distributed Python,
> which isn't mentioned anywhere in the environment block, and that it chose
> that one over the Python 3 installation that was in the path and that
> included the interpreters and all of the corresponding libraries.
>
> >  With FindPython{2,3} you know you'll have a matching interpreter and
> library.
>
> The build failure was that the Python distributed in VS does not have a
> debug version of the library (python37_d.lib), so you don't actually get a
> matching interpreter and library for debug builds.
>
> It's also not clear whether LLVM was consistently using the Python found
> the old way.  My generated build.ninja file seemed to be using both
> versions inconsistently to run lit tests and the like.
>
> Anyway, thanks for the explanations.  I've LGTMed your patch, which should
> eliminate future surprises when something else smuggles yet another version
> of Python onto a machine.
>
> On Thu, Feb 27, 2020 at 11:14 AM Jonas Devlieghere <jonas at devlieghere.com>
> wrote:
>
>> So to make my previous explanation more concrete:
>>
>> On Thu, Feb 27, 2020 at 11:05 AM Jonas Devlieghere <jonas at devlieghere.com>
>> wrote:
>>
>>>
>>>
>>> On Thu, Feb 27, 2020 at 10:53 AM Adrian McCarthy <amccarth at google.com>
>>> wrote:
>>>
>>>> Thanks for the info.  Setting Python3_ROOT_DIR solves the problem.
>>>>
>>>> Looking at the cmake output from before setting Python3_ROOT_DIR, cmake
>>>> looks for Python twice and finds it at the two different locations.
>>>>
>>>> Early on:
>>>>
>>>> -- Found PythonInterp: C:/Python36/python.exe (found version "3.6.8")
>>>>
>>>
>> ^ This is using the "old" (CMake < 3.12) way of finding the Python
>> interpreter.
>>
>>
>>>
>>>> Which looks good (modulo the incorrect slash direction).  But later:
>>>>
>>>> -- Found Python3: C:/Program Files (x86)/Microsoft Visual
>>>> Studio/Shared/Python37_64/python.exe (found version "3.7.5") found
>>>> components:  Interpreter Development
>>>> -- Found PythonInterpAndLibs: C:/Program Files (x86)/Microsoft Visual
>>>> Studio/Shared/Python37_64/libs/python37.lib
>>>>
>>>
>> ^ This is using the "new" (CMake > 3.12) way of finding the Python
>> interpreter and libraries.
>>
>>
>>>
>>>> Which is where the discrepancy comes in.  Note that only C:\Python36 is
>>>> in my PATH.
>>>>
>>>> It's frustrating that this keeps breaking.  Last time, I had to purge
>>>> all but one Python installation from my machine to get it to make a
>>>> consistent choice.  But I just upgraded to VS 2019, and it smuggled in its
>>>> own version.
>>>>
>>>> So why are there two searches anyway?  And why do they have different
>>>> algorithms that lead to different results?  (I'm not sure _how_ it ever
>>>> found the Microsoft copy, since there's nothing in the process environment
>>>> that points that way.)
>>>>
>>>
>>> The reason there's two searches is because LLVM and LLDB have different
>>> requirements. LLVM just needs a python interpreter to run some scripts.
>>> LLDB on the other hand needs an interpreter and a matching Python library
>>> to link against. Before CMake 3.12, finding the interpreter and the
>>> libraries are two separate calls to find with no guarantees that they
>>> match. This lead to all kinds of issues, where you're linking against one
>>> version of Python and then trying to run the test suite with a totally
>>> different interpreter. There were other problems on Windows, which meant
>>> that we had our own hand-rolled implementation to find Python.
>>>
>>> This was all fixed in CMake 3.12. With FindPython{2,3} you know you'll
>>> have a matching interpreter and library. It also fixed all the problems we
>>> had to work around for Windows. Unfortunately, LLVM's minimum CMake version
>>> is 3.4, so we can't use it yet. For LLDB on Windows we agreed that the
>>> benefits of using FindPython3 are worth bumping the minimum required CMake
>>> version (see lldb/CMakeLists.txt, line 2-4). Once LLVM moves to CMake 3.12
>>> or later, all these problems should be fixed. We can then call FindPython3
>>> once and rely on everything being consistent.
>>>
>>>
>>>>
>>>> On Thu, Feb 27, 2020 at 10:23 AM Jonas Devlieghere <
>>>> jonas at devlieghere.com> wrote:
>>>>
>>>>> Hey Adrian,
>>>>>
>>>>> Config.h gets generated by expanding the corresponding CMake
>>>>> variables. If you look at LLDBConfig.cmake, you can see that
>>>>> LLDB_PYTHON_HOME is computed from PYTHON_EXECUTABLE. The problem appears
>>>>> that somehow CMake ignored your specified PYTHON_HOME and decided to pick a
>>>>> different Python. I'm not sure why though, because I use a similar CMake
>>>>> invocation on Windows.
>>>>>
>>>>> > cmake ..\llvm-project\llvm -G Ninja
>>>>> -DCMAKE_BUILD_TYPE=RelWithDebInfo
>>>>> -DLLVM_ENABLE_PROJECTS="llvm;clang;lldb;lld" -DLLVM_ENABLE_ASSERTIONS=OFF
>>>>> -DLLVM_ENABLE_ZLIB=FALSE -DLLDB_ENABLE_PYTHON=TRUE
>>>>> -DPYTHON_HOME="C:/Program Files/Python36/"
>>>>>
>>>>> According to FindPython3 (
>>>>> https://cmake.org/cmake/help/v3.12/module/FindPython3.html), you can
>>>>> set Python3_ROOT_DIR as a hint. Can you give that a try? If that works we
>>>>> should populate that variable from PYTHON_HOME in
>>>>> FindPythonInterpAndLibs.cmake.
>>>>>
>>>>> Cheers,
>>>>> Jonas
>>>>>
>>>>> On Thu, Feb 27, 2020 at 10:10 AM Adrian McCarthy via lldb-dev <
>>>>> lldb-dev at lists.llvm.org> wrote:
>>>>>
>>>>>> Is there documentation on how lldb\include\lldb\host\config.h is
>>>>>> generated?  I'm again having the problem of the config trying to point to
>>>>>> the wrong Python installation.
>>>>>>
>>>>>> When I run cmake, I explicitly point PYTHON_HOME to C:\Python36 like
>>>>>> this:
>>>>>>
>>>>>> cmake -GNinja -DLLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN=ON
>>>>>> -DCMAKE_BUILD_TYPE=Debug -DLLDB_TEST_DEBUG_TEST_CRASHES=1
>>>>>> -DPYTHON_HOME=C:\Python36
>>>>>> -DLLDB_TEST_COMPILER=D:\src\llvm\build\ninja\bin\clang.exe
>>>>>> ..\..\llvm-project\llvm -DLLVM_ENABLE_ZLIB=OFF
>>>>>> -DLLVM_ENABLE_PROJECTS="clang;lld;lldb"
>>>>>>
>>>>>> But the generated Config.h contains:
>>>>>>
>>>>>> #define LLDB_PYTHON_HOME "C:/Program Files (x86)/Microsoft Visual
>>>>>> Studio/Shared/Python37_64"
>>>>>>
>>>>>>
>>>>>> And the mismatch causes my build to fail because it goes looking for
>>>>>> python37_d.dll, which is apparently not part of the Microsoft distribution.
>>>>>> _______________________________________________
>>>>>> lldb-dev mailing list
>>>>>> lldb-dev at lists.llvm.org
>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>
>>>>>
>> I've put up a patch to use PYTHON_HOME as the hint:
>> https://reviews.llvm.org/D75275
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20200511/d72097ba/attachment.html>