[lldb-dev] How do I use lit to only run the lldb test suite, now that dotest multiprocessing capabilities have been removed?

Fri Aug 9 14:56:31 PDT 2019

So far the only thing that changed by removing multiprocess is that
`--no-multiprocess` is always enabled. Everything else you describe is
still possible, and will continue to be possible.

Even when we remove the driver functionality from dotest.py this will
all continue to work. The only difference is that dotest.py will
operate on a single file after being invoked by some other driver. The
problem you describe with the dual arguments seems relatively easy to
fix by loading a different configuration in lit. It's not something
I've looked at yet, because everything we care about is configurable
from CMake or overridable by passing --params to lit.

On Fri, Aug 9, 2019 at 1:30 PM Ted Woodward <tedwood at quicinc.com> wrote:
>
> Hi Jonas,
>
> What I need is a way to run the test suite with arbitrary command line arguments. Sometimes I want to run one or more tests with -f, sometimes I want to run one or more test files with -p, and sometimes I want to run the entire suite, either in parallel or 1 at a time (--no-multiprocess). I might be running from a directory where I've built lldb (but not clang, using clang from an arbitrary location), a directory where I've built everything (but in this case we set everything up with cmake), or a directory where I've checkout out just the sources and have copied the binaries from a distribution, so no cmake.
>
> dotest used to handle these cases; how do I handle them now?
>
> In our environment, lldb will launch and connect to the hexagon simulator, much like how it launches and connects to debugserver or lldb-server. But it has to use a specific version of clang, because we've found that if we have mismatches in clang/simulator/RTOS, bad things can happen. User code is in a shared library that gets loaded by a wrapper run under the RTOS. The RTOS, wrapper and user code all need to be built with the same complier and c/c++ libraries.
>
> Ted
>
> > -----Original Message-----
> > From: Jonas Devlieghere <jonas at devlieghere.com>
> > Sent: Friday, August 9, 2019 3:18 PM
> > To: Ted Woodward <tedwood at quicinc.com>
> > Cc: LLDB <lldb-dev at lists.llvm.org>
> > Subject: [EXT] Re: How do I use lit to only run the lldb test suite, now that
> > dotest multiprocessing capabilities have been removed?
> >
> > Hey Ted,
> >
> > On Thu, Aug 8, 2019 at 2:08 PM Ted Woodward <tedwood at quicinc.com>
> > wrote:
> > >
> > > Thanks Jonas.
> > >
> > > Is full support for --param fairly recent? I tried it with a version of our
> > master, based on top-of-tree from about a month ago, and it didn't work quite
> > right. It's passing the dotest args, but it's also generating some args, so I'm
> > seeing odd effects.
> >
> > It's not something I touched recently, but it's always possible they made some
> > changes in LLVM.
> >
> > > Here is my run line:
> > > bin/python bin/llvm-lit /local/mnt/ted/8.4/llvm/lldb/lit/Suite --param
> > 'dotest-args=-A v66 -C /prj/dsp/qdsp6/release/internal/HEXAGON/branch-
> > 8.4/linux64/latest/Tools/bin/hexagon-clang --executable
> > /local/scratch/ted/8.4/build/bin/lldb -t -v -f
> > RecursiveTypesTestCase.test_recursive_type_1_dwarf'
> > >
> > > I only want to run RecursiveTyepsTestCase.test_recursive_type_1_dwarf,
> > but it's running the whole test suite.
> >
> > Do you know about the lldb-dotest binary? You can still use it to invoke a
> > single test, similar to how lit does it. You should be able to just pass your
> > arguments to that.
> >
> > Here's a dotest line from the run:
> > >
> > > /local/mnt/ted/8.4/build/bin/python
> > > /local/mnt/ted/8.4/llvm/lldb/test/dotest.py -q --arch=v66 -s
> > > /local/mnt/ted/8.4/build/lldb-test-traces --build-dir
> > > /local/mnt/ted/8.4/build/lldb-test-build.noindex -S nm -u CXXFLAGS -u
> > > CFLAGS --executable /local/mnt/ted/8.4/build/./bin/lldb --dsymutil
> > > /local/mnt/ted/8.4/build/./bin/dsymutil --filecheck
> > > /local/mnt/ted/8.4/build/./bin/FileCheck -C
> > > /local/mnt/ted/8.4/build/./bin/clang --env ARCHIVER=/usr/bin/ar --env
> > > OBJCOPY=/usr/bin/objcopy -A v66 -C
> > > /prj/dsp/qdsp6/release/internal/HEXAGON/branch-8.4/linux64/latest/Tool
> > > s/bin/hexagon-clang --executable /local/scratch/ted/8.4/build/bin/lldb
> > > -t -v -f RecursiveTypesTestCase.test_recursive_type_1_dwarf --env
> > > LLVM_LIBS_DIR=/local/mnt/ted/8.4/build/./lib
> > > /local/mnt/ted/8.4/llvm/lldb/packages/Python/lldbsuite/test/functional
> > > ities/breakpoint/debugbreak -p TestDebugBreak.py
> > >
> > >
> > > It's got both --arch= and -A, -C is set to my build directory clang as well as the
> > clang I told it to use, --executable is set twice, and it's got -f
> > RecursiveTypesTestCase.test_recursive_type_1_dwarf and -p
> > TestDebugBreak.py .
> >
> > Both lit and lldb-dotest are configured using the dotest arguments that we can
> > configure at CMake configuration time. That would explain where the extra
> > options come from. If those are not the ones you want, you can still invoke
> > dotest.py directly.
> >
> > >
> > > These tests that do a "process launch" (which is most of them) invoke the
> > hexagon simulator, but it was never launched. There was also only 1 testcase
> > built in /local/mnt/ted/8.4/build/lldb-test-build.noindex -
> > types/TestRecursiveTypes.test_recursive_type_1_dwarf .
> >
> > Other than the extra arguments I can't think of any reason why this would
> > behave differently. Lit is just a simple wrapper that invokes dotest.py with the
> > right arguments.
> >
> > > This does not have the patch that removes multiprocess support from
> > dotest.
> > >
> > > Ted
> > >
> > > > -----Original Message-----
> > > > From: Jonas Devlieghere <jonas at devlieghere.com>
> > > > Sent: Thursday, August 8, 2019 2:50 PM
> > > > To: Ted Woodward <tedwood at quicinc.com>
> > > > Cc: LLDB <lldb-dev at lists.llvm.org>
> > > > Subject: [EXT] Re: How do I use lit to only run the lldb test suite,
> > > > now that dotest multiprocessing capabilities have been removed?
> > > >
> > > > Hey Ted,
> > > >
> > > > 1. You can run just the dotest-tests by pointing lit at the `lit/Suite`
> > directory.
> > > > 2. You can pass arguments to dotest by passing `dotest-args` in --param.
> > > >
> > > > The invocation would look something like this:
> > > >
> > > > /path/to/llvm/bin/llvm-lit /path/to/lldb/lit/Suite --param
> > > > 'dotest-args=--foo -- bar'
> > > >
> > > > Hope that helps,
> > > > Jonas
> > > >
> > > > On Thu, Aug 8, 2019 at 9:31 AM Ted Woodward <tedwood at quicinc.com>
> > > > wrote:
> > > > >
> > > > > RE: https://reviews.llvm.org/D65311
> > > > >
> > > > >
> > > > >
> > > > > Internally we use dotest to run the lldb test suite with various
> > > > > RTOS
> > > > configurations for the test binaries. In these runs we don’t care
> > > > about the lit tests or the unit tests, because they are OS agnostic.
> > > > We do this by specifying the compiler, lldb, and test flavor (static
> > > > testcase + os, dynamic library testcase loaded by an OS image,
> > > > dynamic library testcase loaded by an OS image running the OS’ debug
> > stub).
> > > > >
> > > > >
> > > > >
> > > > > With the multiprocess testrunner removed, how do I have lit:
> > > > >
> > > > > Only run the lldb test suite
> > > > > Run dotest with specific arguments
> > > > >
> > > > >
> > > > >
> > > > > We’re not running cmake; we’re taking an existing tools build and
> > > > > running
> > > > the tests from the source directory using the toolset.