[llvm-dev] lit errors when relying on PYTHONPATH for modules

[Nagurne, James via llvm-dev]

I apologize for the second message, but I've created a revision with my proposed solution that works on our end.
I'd appreciate any feedback or guidance if this change is not the right way to fix my problem.

https://reviews.llvm.org/D78229/new/

Thanks,
J.B.

From: Nagurne, James
Sent: Wednesday, April 15, 2020 1:52 PM
To: llvm-dev at lists.llvm.org
Cc: llvm_cgt at list.ti.com - LLVM Compiler Development (May contain non-TIers) (llvm_cgt at list.ti.com)
Subject: lit errors when relying on PYTHONPATH for modules

Our team utilizes a local directory for most of our non-standard python modules. An example of this is 'pygments'.
We put this directory into our PYTHONPATH when we provision our systems for builds and validation.

When LLVM configuration begins, there's a section of code in llvm/cmake/config-ix.cmake that goes and looks for 'pygments', among other modules.
It correctly finds these modules and sets LLVM_HAVE_OPT_VIEWER_MODULES.

This variable is used to initialize a feature checked by the opt-viewer lit tests. Since we had the modules during LLVM configuration, we will therefore run the tests.

lit creates a test script that looks something like this:
set -o pipefail;set -x;{ : 'RUN: at line 5';   /usr/bin/python3 /<LLVM_ROOT>/llvm/tools/opt-viewer/opt-viewer.py <options>
...
}

And executes that script with:
/bin/bash <BUILD_DIR>/test/tools/opt-viewer/Output/basic.test.script

Starting a new shell via /bin/bash will not propagate the PYTHONPATH with which we initially configured LLVM, and the test fails because it can't find 'pygments'. The function which executes this command, executeCommand, does have an 'env' parameter, but the argument sent in is usually 'test.config.environment', which does not include PYTHONPATH.

Are we missing something fundamental in our setup?
There's a list of host variables that get propagated into the test suite (with_system_environment) in lit.cfg.py. Should we add "PYTHONPATH" to the list of variables propagated in llvm/test/lit.cfg.py?

Thanks,
J.B.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200415/906519a0/attachment.html>