[Lldb-commits] [PATCH] D54567: Fix LLDB's lit files

Wed Nov 14 21:55:35 PST 2018

zturner created this revision.
zturner added reviewers: stella.stamenova, labath, aleksandr.urakov, rnk.
Herald added subscribers: jfb, arphaman, delcypher, JDevlieghere, aheejin, aprantl, mgorny.
Herald added a reviewer: alexshap.

Recently I tried to port LLDB's lit configuration files over to use a lot of the common lit infrastructure down in LLVM.  This appeared to work on the surface, but broke some cases that weren't broken before and also exposed some additional problems with the old approach that we were just getting lucky with.

This is a big patch, so I'll try to give a high level overview of the problem as best I can to make reviewing easier.

When we set up a lit environment, the goal is to make it as hermetic as possible.  We should not be relying on PATH and enabling the use of arbitrary shell commands.  Instead, only whitelisted commands should be allowed.  These are, generally speaking, the lit builtins such as `echo`, `cd`, etc, as well as anything for which substitutions have been explicitly set up for.  These substitutions should map to the build output directory, but in some cases it's useful to be able to override this (for example to point to an installed tools directory).

This is, of course, how it's //supposed// to work.  What was //actually// happening is that we were bringing in `PATH` and `LD_LIBRARY_PATH` and then just running the given run line as a shell command.  This led to problems such as finding the wrong version of `clang-cl` on `PATH` since it wasn't even a substitution, and flakiness / non-determinism since the environment the tests were running in would change per-machine.  On the other hand, it also made other things possible.  For example, we had some tests that were explicitly running `cl.exe` and `link.exe` instead of `clang-cl` and `lld-link` and the only reason it worked at all is because it was finding them on `PATH`.  Unfortunately we can't entirely get rid of these tests, because they support a few things in debug info that `clang-cl` and `lld-link` don't (notably, the `LF_UDT_MOD_SRC_LINE` record which makes some of the tests fail.

This patch attempts to fix all of this.  It's a large patch, but I'll try to summarize the changes:

1. **Removal of functionality** - The lit test suite no longer respects `LLDB_TEST_C_COMPILER` and `LLDB_TEST_CXX_COMPILER`.  This means there is no more support for gcc, but nobody was using this anyway (note: The functionality is still there for the dotest suite, just not the lit test suite).  There is no longer a single substitution `%cxx` and `%cc` which maps to <arbitrary-compiler>, you now explicitly specify the compiler with a substitution like `%clang` or `%clangxx` or `%clang_cl`.  We can revisit this in the future when someone needs gcc.

2. Introduction of the `LLDB_LIT_TOOLS_DIR` directory.  This does in spirit what `LLDB_TEST_C_COMPILER` and `LLDB_TEST_CXX_COMPILER` used to do, but now more friendly.  If this is not specified, all tools are expected to be the just-built tools.  If it is specified, the tools which are not themselves being tested but are being used to construct and run checks (e.g. `clang`, `FileCheck`, `llvm-mc`, etc) will be searched for in this directory first, then the build output directory.

3. Changes to core llvm lit files.  The `use_lld()` and `use_clang()` functions were introduced long ago in anticipation of using them in lldb, but since they were never actually used anywhere but their respective problems, there were some issues to be resolved regarding generality and ability to use them outside their project.

4. Changes to .test files - These are all just replacing things like `clang-cl` with `%clang_cl` and `%cxx` with `%clangxx`, etc.

5. Changes to `lit.cfg.py` - Previously we would load up some system environment variables and then add some new things to them.  Then do a bunch of work building out our own substitutions.  First, we delete the system environment variable code, making the environment hermetic.  Then, we refactor the substitution logic into two separate helper functions, one which sets up substitutions for the tools we want to test (which must come from the build output directory), and another which sets up substitutions for support tools (like compilers, etc).

6. New substitutions for MSVC -- Previously we relied on location of MSVC by bringing in the entire parent's `PATH` and letting `subprocess.Popen` just run the command line.  Now we set up real substitutions that should have the same effect.  We use `PATH` to find them, and then look for `INCLUDE` and `LIB` to construct a substitution command line with appropriate `/I` and `/LIBPATH:` arguments.  The nice thing about this is that it opens the door to having separate `%msvc-cl32` and `%msvc-cl64` substitutions, rather than only requiring the user to run vcvars first.  Because we can deduce the path to 32-bit libraries from 64-bit library directories, and vice versa.  Without these substitutions this would have been impossible.

Stella and Aleksandr, can you apply this patch locally and see if it fixes the issues you wer encountering?

https://reviews.llvm.org/D54567

Files:
  clang/test/lit.cfg.py
  lldb/lit/Breakpoint/case-insensitive.test
  lldb/lit/Breakpoint/case-sensitive.test
  lldb/lit/Expr/TestIRMemoryMap.test
  lldb/lit/Expr/TestIRMemoryMapWindows.test
  lldb/lit/Quit/TestQuitExitCode-30.test
  lldb/lit/Quit/TestQuitExitCode30.test
  lldb/lit/Quit/TestQuitExitCodeHexA.test
  lldb/lit/SymbolFile/DWARF/apple-index-is-used.cpp
  lldb/lit/SymbolFile/DWARF/debug-names-compressed.cpp
  lldb/lit/SymbolFile/DWARF/dwarf5-index-is-used.cpp
  lldb/lit/SymbolFile/DWARF/dwarf5-partial-index.cpp
  lldb/lit/SymbolFile/DWARF/find-basic-function.cpp
  lldb/lit/SymbolFile/DWARF/find-basic-namespace.cpp
  lldb/lit/SymbolFile/DWARF/find-basic-type.cpp
  lldb/lit/SymbolFile/DWARF/find-basic-variable.cpp
  lldb/lit/SymbolFile/DWARF/find-function-regex.cpp
  lldb/lit/SymbolFile/DWARF/find-method-local-struct.cpp
  lldb/lit/SymbolFile/DWARF/find-method.cpp
  lldb/lit/SymbolFile/DWARF/find-qualified-variable.cpp
  lldb/lit/SymbolFile/DWARF/find-type-in-function.cpp
  lldb/lit/SymbolFile/DWARF/find-variable-dwo.cpp
  lldb/lit/SymbolFile/DWARF/find-variable-file.cpp
  lldb/lit/SymbolFile/NativePDB/ast-reconstruction.cpp
  lldb/lit/SymbolFile/NativePDB/bitfields.cpp
  lldb/lit/SymbolFile/NativePDB/disassembly.cpp
  lldb/lit/SymbolFile/NativePDB/function-types-builtins.cpp
  lldb/lit/SymbolFile/NativePDB/function-types-calling-conv.cpp
  lldb/lit/SymbolFile/NativePDB/function-types-classes.cpp
  lldb/lit/SymbolFile/NativePDB/global-classes.cpp
  lldb/lit/SymbolFile/NativePDB/globals-bss.cpp
  lldb/lit/SymbolFile/NativePDB/globals-fundamental.cpp
  lldb/lit/SymbolFile/NativePDB/nested-types.cpp
  lldb/lit/SymbolFile/NativePDB/s_constant.cpp
  lldb/lit/SymbolFile/NativePDB/simple-breakpoints.cpp
  lldb/lit/SymbolFile/NativePDB/source-list.cpp
  lldb/lit/SymbolFile/NativePDB/tag-types.cpp
  lldb/lit/SymbolFile/PDB/ast-restore.test
  lldb/lit/SymbolFile/PDB/calling-conventions.test
  lldb/lit/SymbolFile/PDB/class-layout.test
  lldb/lit/SymbolFile/PDB/compilands.test
  lldb/lit/SymbolFile/PDB/enums-layout.test
  lldb/lit/SymbolFile/PDB/func-symbols.test
  lldb/lit/SymbolFile/PDB/function-level-linking.test
  lldb/lit/SymbolFile/PDB/function-nested-block.test
  lldb/lit/SymbolFile/PDB/pointers.test
  lldb/lit/SymbolFile/PDB/type-quals.test
  lldb/lit/SymbolFile/PDB/typedefs.test
  lldb/lit/SymbolFile/PDB/udt-layout.test
  lldb/lit/SymbolFile/PDB/variables-locations.test
  lldb/lit/SymbolFile/PDB/variables.test
  lldb/lit/SymbolFile/PDB/vbases.test
  lldb/lit/helper/toolchain.py
  lldb/lit/lit.cfg.py
  lldb/lit/lit.site.cfg.py.in
  lldb/test/CMakeLists.txt
  lldb/unittests/Process/gdb-remote/GDBRemoteTestUtils.cpp
  llvm/utils/lit/lit/llvm/config.py
  llvm/utils/lit/lit/llvm/subst.py

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D54567.174155.patch
Type: text/x-patch
Size: 60089 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/lldb-commits/attachments/20181115/f02b385e/attachment-0001.bin>