Adrian McCarthy via lldb-dev lldb-dev at lists.llvm.org
Mon May 11 16:34:19 PDT 2020

Ug.  After a rebase, this problem has again resurfaced for me.

* PATH has only C:\Python36
* cmake version 3.15.19101501-MSVC_2
-DLLDB_PYTHON_HOME=C:\Python36 -DPython3_ROOT_DIR=C:\Python36
..\..\llvm-project\llvm -DLLVM_ENABLE_ZLIB=OFF

Yet when linking LLDB, it goes looking at the Python 3.7 installation that
comes with Visual Studio.  That wouldn't be much of a problem unless you're
trying to build debug, which I am.  The VS version, doesn't come with debug
versions of the interpreter libraries, so the link fails:

[1/7] Linking CXX shared library bin\liblldb.dll
FAILED: bin/liblldb.dll lib/liblldb.lib
cmd.exe /C "cd . && "C:\Program Files (x86)\Microsoft Visual
-E vs_link_dll --intdir=tools\lldb\source\API\CMakeFiles\liblldb.dir
--mt=C:\PROGRA~2\WI3CF2~1\10\bin\100183~1.0\x64\mt.exe --manifests  --
/nologo @CMakeFiles\liblldb.rsp  /out:bin\liblldb.dll
/implib:lib\liblldb.lib /pdb:bin\liblldb.pdb /dll /version:11.0
/machine:x64 /debug /INCREMENTAL   && cd ."
LINK Pass 1: command
/nologo @CMakeFiles\liblldb.rsp /out:bin\liblldb.dll
/implib:lib\liblldb.lib /pdb:bin\liblldb.pdb /dll /version:11.0
/machine:x64 /debug /INCREMENTAL /MANIFEST
tools\lldb\source\API\CMakeFiles\liblldb.dir/manifest.res" failed (exit
code 1104) with the following output:
LINK : fatal error LNK1104: cannot open file 'python37_d.lib'
ninja: build stopped: subcommand failed.

It looks like Cmake has added several more hints for finding Python
<https://cmake.org/cmake/help/v3.15/module/FindPython3.html#hints>.  I'm
trying now with -DPython3_FIND_REGISTRY=LAST.

I miss hermetic builds.

On Thu, Feb 27, 2020 at 11:33 AM Adrian McCarthy <amccarth at google.com>

> >  This was all fixed in CMake 3.12.
> For some definitions of "all fixed." ;-)
> It seems weird to me that FindPython3 found the VS-distributed Python,
> which isn't mentioned anywhere in the environment block, and that it chose
> that one over the Python 3 installation that was in the path and that
> included the interpreters and all of the corresponding libraries.
> >  With FindPython{2,3} you know you'll have a matching interpreter and
> library.
> The build failure was that the Python distributed in VS does not have a
> debug version of the library (python37_d.lib), so you don't actually get a
> matching interpreter and library for debug builds.
> It's also not clear whether LLVM was consistently using the Python found
> the old way.  My generated build.ninja file seemed to be using both
> versions inconsistently to run lit tests and the like.
> Anyway, thanks for the explanations.  I've LGTMed your patch, which should
> eliminate future surprises when something else smuggles yet another version
> of Python onto a machine.
> On Thu, Feb 27, 2020 at 11:14 AM Jonas Devlieghere <jonas at devlieghere.com>
> wrote:
>> So to make my previous explanation more concrete:
>> On Thu, Feb 27, 2020 at 11:05 AM Jonas Devlieghere <jonas at devlieghere.com>
>> wrote:
>>> On Thu, Feb 27, 2020 at 10:53 AM Adrian McCarthy <amccarth at google.com>
>>> wrote:
>>>> Thanks for the info.  Setting Python3_ROOT_DIR solves the problem.
>>>> Looking at the cmake output from before setting Python3_ROOT_DIR, cmake
>>>> looks for Python twice and finds it at the two different locations.
>>>> Early on:
>>>> -- Found PythonInterp: C:/Python36/python.exe (found version "3.6.8")
>> ^ This is using the "old" (CMake < 3.12) way of finding the Python
>> interpreter.
>>>> Which looks good (modulo the incorrect slash direction).  But later:
>>>> -- Found Python3: C:/Program Files (x86)/Microsoft Visual
>>>> Studio/Shared/Python37_64/python.exe (found version "3.7.5") found
>>>> components:  Interpreter Development
>>>> -- Found PythonInterpAndLibs: C:/Program Files (x86)/Microsoft Visual
>>>> Studio/Shared/Python37_64/libs/python37.lib
>> ^ This is using the "new" (CMake > 3.12) way of finding the Python
>> interpreter and libraries.
>>>> Which is where the discrepancy comes in.  Note that only C:\Python36 is
>>>> in my PATH.
>>>> It's frustrating that this keeps breaking.  Last time, I had to purge
>>>> all but one Python installation from my machine to get it to make a
>>>> consistent choice.  But I just upgraded to VS 2019, and it smuggled in its
>>>> own version.
>>>> So why are there two searches anyway?  And why do they have different
>>>> algorithms that lead to different results?  (I'm not sure _how_ it ever
>>>> found the Microsoft copy, since there's nothing in the process environment
>>>> that points that way.)
>>> The reason there's two searches is because LLVM and LLDB have different
>>> requirements. LLVM just needs a python interpreter to run some scripts.
>>> LLDB on the other hand needs an interpreter and a matching Python library
>>> to link against. Before CMake 3.12, finding the interpreter and the
>>> libraries are two separate calls to find with no guarantees that they
>>> match. This lead to all kinds of issues, where you're linking against one
>>> version of Python and then trying to run the test suite with a totally
>>> different interpreter. There were other problems on Windows, which meant
>>> that we had our own hand-rolled implementation to find Python.
>>> This was all fixed in CMake 3.12. With FindPython{2,3} you know you'll
>>> have a matching interpreter and library. It also fixed all the problems we
>>> had to work around for Windows. Unfortunately, LLVM's minimum CMake version
>>> is 3.4, so we can't use it yet. For LLDB on Windows we agreed that the
>>> benefits of using FindPython3 are worth bumping the minimum required CMake
>>> version (see lldb/CMakeLists.txt, line 2-4). Once LLVM moves to CMake 3.12
>>> or later, all these problems should be fixed. We can then call FindPython3
>>> once and rely on everything being consistent.
>>>> On Thu, Feb 27, 2020 at 10:23 AM Jonas Devlieghere <
>>>> jonas at devlieghere.com> wrote:
>>>>> Hey Adrian,
>>>>> Config.h gets generated by expanding the corresponding CMake
>>>>> variables. If you look at LLDBConfig.cmake, you can see that
>>>>> LLDB_PYTHON_HOME is computed from PYTHON_EXECUTABLE. The problem appears
>>>>> that somehow CMake ignored your specified PYTHON_HOME and decided to pick a
>>>>> different Python. I'm not sure why though, because I use a similar CMake
>>>>> invocation on Windows.
>>>>> > cmake ..\llvm-project\llvm -G Ninja
>>>>> -DCMAKE_BUILD_TYPE=RelWithDebInfo
>>>>> -DPYTHON_HOME="C:/Program Files/Python36/"
>>>>> According to FindPython3 (
>>>>> https://cmake.org/cmake/help/v3.12/module/FindPython3.html), you can
>>>>> set Python3_ROOT_DIR as a hint. Can you give that a try? If that works we
>>>>> should populate that variable from PYTHON_HOME in
>>>>> FindPythonInterpAndLibs.cmake.
>>>>> Cheers,
>>>>> Jonas
>>>>> On Thu, Feb 27, 2020 at 10:10 AM Adrian McCarthy via lldb-dev <
>>>>> lldb-dev at lists.llvm.org> wrote:
>>>>>> Is there documentation on how lldb\include\lldb\host\config.h is
>>>>>> generated?  I'm again having the problem of the config trying to point to
>>>>>> the wrong Python installation.
>>>>>> When I run cmake, I explicitly point PYTHON_HOME to C:\Python36 like
>>>>>> this:
>>>>>> -DPYTHON_HOME=C:\Python36
>>>>>> -DLLDB_TEST_COMPILER=D:\src\llvm\build\ninja\bin\clang.exe
>>>>>> ..\..\llvm-project\llvm -DLLVM_ENABLE_ZLIB=OFF
>>>>>> -DLLVM_ENABLE_PROJECTS="clang;lld;lldb"
>>>>>> But the generated Config.h contains:
>>>>>> #define LLDB_PYTHON_HOME "C:/Program Files (x86)/Microsoft Visual
>>>>>> Studio/Shared/Python37_64"
>>>>>> And the mismatch causes my build to fail because it goes looking for
>>>>>> python37_d.dll, which is apparently not part of the Microsoft distribution.
>>>>>> _______________________________________________
>>>>>> lldb-dev mailing list
>>>>>> lldb-dev at lists.llvm.org
>>>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>> I've put up a patch to use PYTHON_HOME as the hint:
>> https://reviews.llvm.org/D75275
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20200511/d72097ba/attachment.html>

More information about the lldb-dev mailing list