[lldb-dev] How do I use lit to only run the lldb test suite, now that dotest multiprocessing capabilities have been removed?

[Ted Woodward via lldb-dev]

Hi Jonas,

What I need is a way to run the test suite with arbitrary command line arguments. Sometimes I want to run one or more tests with -f, sometimes I want to run one or more test files with -p, and sometimes I want to run the entire suite, either in parallel or 1 at a time (--no-multiprocess). I might be running from a directory where I've built lldb (but not clang, using clang from an arbitrary location), a directory where I've built everything (but in this case we set everything up with cmake), or a directory where I've checkout out just the sources and have copied the binaries from a distribution, so no cmake.

dotest used to handle these cases; how do I handle them now?

In our environment, lldb will launch and connect to the hexagon simulator, much like how it launches and connects to debugserver or lldb-server. But it has to use a specific version of clang, because we've found that if we have mismatches in clang/simulator/RTOS, bad things can happen. User code is in a shared library that gets loaded by a wrapper run under the RTOS. The RTOS, wrapper and user code all need to be built with the same complier and c/c++ libraries.

Ted

> -----Original Message-----
> From: Jonas Devlieghere <jonas at devlieghere.com>
> Sent: Friday, August 9, 2019 3:18 PM
> To: Ted Woodward <tedwood at quicinc.com>
> Cc: LLDB <lldb-dev at lists.llvm.org>
> Subject: [EXT] Re: How do I use lit to only run the lldb test suite, now that
> dotest multiprocessing capabilities have been removed?
> 
> Hey Ted,
> 
> On Thu, Aug 8, 2019 at 2:08 PM Ted Woodward <tedwood at quicinc.com>
> wrote:
> >
> > Thanks Jonas.
> >
> > Is full support for --param fairly recent? I tried it with a version of our
> master, based on top-of-tree from about a month ago, and it didn't work quite
> right. It's passing the dotest args, but it's also generating some args, so I'm
> seeing odd effects.
> 
> It's not something I touched recently, but it's always possible they made some
> changes in LLVM.
> 
> > Here is my run line:
> > bin/python bin/llvm-lit /local/mnt/ted/8.4/llvm/lldb/lit/Suite --param
> 'dotest-args=-A v66 -C /prj/dsp/qdsp6/release/internal/HEXAGON/branch-
> 8.4/linux64/latest/Tools/bin/hexagon-clang --executable
> /local/scratch/ted/8.4/build/bin/lldb -t -v -f
> RecursiveTypesTestCase.test_recursive_type_1_dwarf'
> >
> > I only want to run RecursiveTyepsTestCase.test_recursive_type_1_dwarf,
> but it's running the whole test suite.
> 
> Do you know about the lldb-dotest binary? You can still use it to invoke a
> single test, similar to how lit does it. You should be able to just pass your
> arguments to that.
> 
> Here's a dotest line from the run:
> >
> > /local/mnt/ted/8.4/build/bin/python
> > /local/mnt/ted/8.4/llvm/lldb/test/dotest.py -q --arch=v66 -s
> > /local/mnt/ted/8.4/build/lldb-test-traces --build-dir
> > /local/mnt/ted/8.4/build/lldb-test-build.noindex -S nm -u CXXFLAGS -u
> > CFLAGS --executable /local/mnt/ted/8.4/build/./bin/lldb --dsymutil
> > /local/mnt/ted/8.4/build/./bin/dsymutil --filecheck
> > /local/mnt/ted/8.4/build/./bin/FileCheck -C
> > /local/mnt/ted/8.4/build/./bin/clang --env ARCHIVER=/usr/bin/ar --env
> > OBJCOPY=/usr/bin/objcopy -A v66 -C
> > /prj/dsp/qdsp6/release/internal/HEXAGON/branch-8.4/linux64/latest/Tool
> > s/bin/hexagon-clang --executable /local/scratch/ted/8.4/build/bin/lldb
> > -t -v -f RecursiveTypesTestCase.test_recursive_type_1_dwarf --env
> > LLVM_LIBS_DIR=/local/mnt/ted/8.4/build/./lib
> > /local/mnt/ted/8.4/llvm/lldb/packages/Python/lldbsuite/test/functional
> > ities/breakpoint/debugbreak -p TestDebugBreak.py
> >
> >
> > It's got both --arch= and -A, -C is set to my build directory clang as well as the
> clang I told it to use, --executable is set twice, and it's got -f
> RecursiveTypesTestCase.test_recursive_type_1_dwarf and -p
> TestDebugBreak.py .
> 
> Both lit and lldb-dotest are configured using the dotest arguments that we can
> configure at CMake configuration time. That would explain where the extra
> options come from. If those are not the ones you want, you can still invoke
> dotest.py directly.
> 
> >
> > These tests that do a "process launch" (which is most of them) invoke the
> hexagon simulator, but it was never launched. There was also only 1 testcase
> built in /local/mnt/ted/8.4/build/lldb-test-build.noindex -
> types/TestRecursiveTypes.test_recursive_type_1_dwarf .
> 
> Other than the extra arguments I can't think of any reason why this would
> behave differently. Lit is just a simple wrapper that invokes dotest.py with the
> right arguments.
> 
> > This does not have the patch that removes multiprocess support from
> dotest.
> >
> > Ted
> >
> > > -----Original Message-----
> > > From: Jonas Devlieghere <jonas at devlieghere.com>
> > > Sent: Thursday, August 8, 2019 2:50 PM
> > > To: Ted Woodward <tedwood at quicinc.com>
> > > Cc: LLDB <lldb-dev at lists.llvm.org>
> > > Subject: [EXT] Re: How do I use lit to only run the lldb test suite,
> > > now that dotest multiprocessing capabilities have been removed?
> > >
> > > Hey Ted,
> > >
> > > 1. You can run just the dotest-tests by pointing lit at the `lit/Suite`
> directory.
> > > 2. You can pass arguments to dotest by passing `dotest-args` in --param.
> > >
> > > The invocation would look something like this:
> > >
> > > /path/to/llvm/bin/llvm-lit /path/to/lldb/lit/Suite --param
> > > 'dotest-args=--foo -- bar'
> > >
> > > Hope that helps,
> > > Jonas
> > >
> > > On Thu, Aug 8, 2019 at 9:31 AM Ted Woodward <tedwood at quicinc.com>
> > > wrote:
> > > >
> > > > RE: https://reviews.llvm.org/D65311
> > > >
> > > >
> > > >
> > > > Internally we use dotest to run the lldb test suite with various
> > > > RTOS
> > > configurations for the test binaries. In these runs we don’t care
> > > about the lit tests or the unit tests, because they are OS agnostic.
> > > We do this by specifying the compiler, lldb, and test flavor (static
> > > testcase + os, dynamic library testcase loaded by an OS image,
> > > dynamic library testcase loaded by an OS image running the OS’ debug
> stub).
> > > >
> > > >
> > > >
> > > > With the multiprocess testrunner removed, how do I have lit:
> > > >
> > > > Only run the lldb test suite
> > > > Run dotest with specific arguments
> > > >
> > > >
> > > >
> > > > We’re not running cmake; we’re taking an existing tools build and
> > > > running
> > > the tests from the source directory using the toolset.