[lldb-dev] LLDB_PYTHON_HOME

Tue May 12 11:55:57 PDT 2020

Setting extra Python3 strategy variables in the Cmake call didn't solve the
problem.  They may have been necessary, but they weren't sufficient.

In the end, I installed Python 3.8.2, and pointed everything at that.  For
reasons I haven't discovered, Cmake seems to choose that one over the
incomplete Python 3.7 distribution that's bundled with Visual Studio.  (The
obvious guess is that it's a higher version number, but the Cmake
documentation denies that.)

"Listen, strange women lying in ponds *Cmake* distributing swords *selecting
Python installations* is no basis for a system of government *cross-platform
build configuration.*"

On Mon, May 11, 2020 at 4:34 PM Adrian McCarthy <amccarth at google.com> wrote:

> Ug.  After a rebase, this problem has again resurfaced for me.
>
> * PATH has only C:\Python36
> * cmake version 3.15.19101501-MSVC_2
> * cmake -GNinja -DLLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN=ON
> -DCMAKE_BUILD_TYPE=Debug -DLLDB_TEST_DEBUG_TEST_CRASHES=1 -DPYTHON_HOME=C:\Python36
> -DLLDB_PYTHON_HOME=C:\Python36 -DPython3_ROOT_DIR=C:\Python36
> -DLLDB_TEST_COMPILER=D:\src\llvm\build\ninja_dbg\bin\clang.exe
> ..\..\llvm-project\llvm -DLLVM_ENABLE_ZLIB=OFF
> -DLLVM_ENABLE_PROJECTS="clang;lld;lldb;debuginfo-tests"
>
> Yet when linking LLDB, it goes looking at the Python 3.7 installation that
> comes with Visual Studio.  That wouldn't be much of a problem unless you're
> trying to build debug, which I am.  The VS version, doesn't come with debug
> versions of the interpreter libraries, so the link fails:
>
> [1/7] Linking CXX shared library bin\liblldb.dll
> FAILED: bin/liblldb.dll lib/liblldb.lib
> cmd.exe /C "cd . && "C:\Program Files (x86)\Microsoft Visual
> Studio\2019\Professional\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe"
> -E vs_link_dll --intdir=tools\lldb\source\API\CMakeFiles\liblldb.dir
> --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100183~1.0\x64\rc.exe
> --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100183~1.0\x64\mt.exe --manifests  --
> C:\PROGRA~2\MICROS~1\2019\PROFES~1\VC\Tools\MSVC\1424~1.283\bin\Hostx64\x64\link.exe
> /nologo @CMakeFiles\liblldb.rsp  /out:bin\liblldb.dll
> /implib:lib\liblldb.lib /pdb:bin\liblldb.pdb /dll /version:11.0
> /machine:x64 /debug /INCREMENTAL   && cd ."
> LINK Pass 1: command
> "C:\PROGRA~2\MICROS~1\2019\PROFES~1\VC\Tools\MSVC\1424~1.283\bin\Hostx64\x64\link.exe
> /nologo @CMakeFiles\liblldb.rsp /out:bin\liblldb.dll
> /implib:lib\liblldb.lib /pdb:bin\liblldb.pdb /dll /version:11.0
> /machine:x64 /debug /INCREMENTAL /MANIFEST
> /MANIFESTFILE:tools\lldb\source\API\CMakeFiles\liblldb.dir/intermediate.manifest
> tools\lldb\source\API\CMakeFiles\liblldb.dir/manifest.res" failed (exit
> code 1104) with the following output:
> LINK : fatal error LNK1104: cannot open file 'python37_d.lib'
> ninja: build stopped: subcommand failed.
>
> It looks like Cmake has added several more hints for finding Python
> <https://cmake.org/cmake/help/v3.15/module/FindPython3.html#hints>.  I'm
> trying now with -DPython3_FIND_REGISTRY=LAST.
>
> I miss hermetic builds.
>
> On Thu, Feb 27, 2020 at 11:33 AM Adrian McCarthy <amccarth at google.com>
> wrote:
>
>> >  This was all fixed in CMake 3.12.
>>
>> For some definitions of "all fixed." ;-)
>>
>> It seems weird to me that FindPython3 found the VS-distributed Python,
>> which isn't mentioned anywhere in the environment block, and that it chose
>> that one over the Python 3 installation that was in the path and that
>> included the interpreters and all of the corresponding libraries.
>>
>> >  With FindPython{2,3} you know you'll have a matching interpreter and
>> library.
>>
>> The build failure was that the Python distributed in VS does not have a
>> debug version of the library (python37_d.lib), so you don't actually get a
>> matching interpreter and library for debug builds.
>>
>> It's also not clear whether LLVM was consistently using the Python found
>> the old way.  My generated build.ninja file seemed to be using both
>> versions inconsistently to run lit tests and the like.
>>
>> Anyway, thanks for the explanations.  I've LGTMed your patch, which
>> should eliminate future surprises when something else smuggles yet another
>> version of Python onto a machine.
>>
>> On Thu, Feb 27, 2020 at 11:14 AM Jonas Devlieghere <jonas at devlieghere.com>
>> wrote:
>>
>>> So to make my previous explanation more concrete:
>>>
>>> On Thu, Feb 27, 2020 at 11:05 AM Jonas Devlieghere <
>>> jonas at devlieghere.com> wrote:
>>>
>>>>
>>>>
>>>> On Thu, Feb 27, 2020 at 10:53 AM Adrian McCarthy <amccarth at google.com>
>>>> wrote:
>>>>
>>>>> Thanks for the info.  Setting Python3_ROOT_DIR solves the problem.
>>>>>
>>>>> Looking at the cmake output from before setting Python3_ROOT_DIR,
>>>>> cmake looks for Python twice and finds it at the two different locations.
>>>>>
>>>>> Early on:
>>>>>
>>>>> -- Found PythonInterp: C:/Python36/python.exe (found version "3.6.8")
>>>>>
>>>>
>>> ^ This is using the "old" (CMake < 3.12) way of finding the Python
>>> interpreter.
>>>
>>>
>>>>
>>>>> Which looks good (modulo the incorrect slash direction).  But later:
>>>>>
>>>>> -- Found Python3: C:/Program Files (x86)/Microsoft Visual
>>>>> Studio/Shared/Python37_64/python.exe (found version "3.7.5") found
>>>>> components:  Interpreter Development
>>>>> -- Found PythonInterpAndLibs: C:/Program Files (x86)/Microsoft Visual
>>>>> Studio/Shared/Python37_64/libs/python37.lib
>>>>>
>>>>
>>> ^ This is using the "new" (CMake > 3.12) way of finding the Python
>>> interpreter and libraries.
>>>
>>>
>>>>
>>>>> Which is where the discrepancy comes in.  Note that only C:\Python36
>>>>> is in my PATH.
>>>>>
>>>>> It's frustrating that this keeps breaking.  Last time, I had to purge
>>>>> all but one Python installation from my machine to get it to make a
>>>>> consistent choice.  But I just upgraded to VS 2019, and it smuggled in its
>>>>> own version.
>>>>>
>>>>> So why are there two searches anyway?  And why do they have different
>>>>> algorithms that lead to different results?  (I'm not sure _how_ it ever
>>>>> found the Microsoft copy, since there's nothing in the process environment
>>>>> that points that way.)
>>>>>
>>>>
>>>> The reason there's two searches is because LLVM and LLDB have different
>>>> requirements. LLVM just needs a python interpreter to run some scripts.
>>>> LLDB on the other hand needs an interpreter and a matching Python library
>>>> to link against. Before CMake 3.12, finding the interpreter and the
>>>> libraries are two separate calls to find with no guarantees that they
>>>> match. This lead to all kinds of issues, where you're linking against one
>>>> version of Python and then trying to run the test suite with a totally
>>>> different interpreter. There were other problems on Windows, which meant
>>>> that we had our own hand-rolled implementation to find Python.
>>>>
>>>> This was all fixed in CMake 3.12. With FindPython{2,3} you know you'll
>>>> have a matching interpreter and library. It also fixed all the problems we
>>>> had to work around for Windows. Unfortunately, LLVM's minimum CMake version
>>>> is 3.4, so we can't use it yet. For LLDB on Windows we agreed that the
>>>> benefits of using FindPython3 are worth bumping the minimum required CMake
>>>> version (see lldb/CMakeLists.txt, line 2-4). Once LLVM moves to CMake 3.12
>>>> or later, all these problems should be fixed. We can then call FindPython3
>>>> once and rely on everything being consistent.
>>>>
>>>>
>>>>>
>>>>> On Thu, Feb 27, 2020 at 10:23 AM Jonas Devlieghere <
>>>>> jonas at devlieghere.com> wrote:
>>>>>
>>>>>> Hey Adrian,
>>>>>>
>>>>>> Config.h gets generated by expanding the corresponding CMake
>>>>>> variables. If you look at LLDBConfig.cmake, you can see that
>>>>>> LLDB_PYTHON_HOME is computed from PYTHON_EXECUTABLE. The problem appears
>>>>>> that somehow CMake ignored your specified PYTHON_HOME and decided to pick a
>>>>>> different Python. I'm not sure why though, because I use a similar CMake
>>>>>> invocation on Windows.
>>>>>>
>>>>>> > cmake ..\llvm-project\llvm -G Ninja
>>>>>> -DCMAKE_BUILD_TYPE=RelWithDebInfo
>>>>>> -DLLVM_ENABLE_PROJECTS="llvm;clang;lldb;lld" -DLLVM_ENABLE_ASSERTIONS=OFF
>>>>>> -DLLVM_ENABLE_ZLIB=FALSE -DLLDB_ENABLE_PYTHON=TRUE
>>>>>> -DPYTHON_HOME="C:/Program Files/Python36/"
>>>>>>
>>>>>> According to FindPython3 (
>>>>>> https://cmake.org/cmake/help/v3.12/module/FindPython3.html), you can
>>>>>> set Python3_ROOT_DIR as a hint. Can you give that a try? If that works we
>>>>>> should populate that variable from PYTHON_HOME in
>>>>>> FindPythonInterpAndLibs.cmake.
>>>>>>
>>>>>> Cheers,
>>>>>> Jonas
>>>>>>
>>>>>> On Thu, Feb 27, 2020 at 10:10 AM Adrian McCarthy via lldb-dev <
>>>>>> lldb-dev at lists.llvm.org> wrote:
>>>>>>
>>>>>>> Is there documentation on how lldb\include\lldb\host\config.h is
>>>>>>> generated?  I'm again having the problem of the config trying to point to
>>>>>>> the wrong Python installation.
>>>>>>>
>>>>>>> When I run cmake, I explicitly point PYTHON_HOME to C:\Python36 like
>>>>>>> this:
>>>>>>>
>>>>>>> cmake -GNinja -DLLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN=ON
>>>>>>> -DCMAKE_BUILD_TYPE=Debug -DLLDB_TEST_DEBUG_TEST_CRASHES=1
>>>>>>> -DPYTHON_HOME=C:\Python36
>>>>>>> -DLLDB_TEST_COMPILER=D:\src\llvm\build\ninja\bin\clang.exe
>>>>>>> ..\..\llvm-project\llvm -DLLVM_ENABLE_ZLIB=OFF
>>>>>>> -DLLVM_ENABLE_PROJECTS="clang;lld;lldb"
>>>>>>>
>>>>>>> But the generated Config.h contains:
>>>>>>>
>>>>>>> #define LLDB_PYTHON_HOME "C:/Program Files (x86)/Microsoft Visual
>>>>>>> Studio/Shared/Python37_64"
>>>>>>>
>>>>>>>
>>>>>>> And the mismatch causes my build to fail because it goes looking for
>>>>>>> python37_d.dll, which is apparently not part of the Microsoft distribution.
>>>>>>> _______________________________________________
>>>>>>> lldb-dev mailing list
>>>>>>> lldb-dev at lists.llvm.org
>>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>
>>>>>>
>>> I've put up a patch to use PYTHON_HOME as the hint:
>>> https://reviews.llvm.org/D75275
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20200512/44ad9e3e/attachment-0001.html>