[lldb-dev] How do I use lit to only run the lldb test suite, now that dotest multiprocessing capabilities have been removed?

[Ted Woodward via lldb-dev]

Thanks for the pointers, Jonas.

I cloned a new repo, then ran cmake, setting up arbitrary test compiler and lldb locations. Then ran llvm-lit tools/lldb/lit/Suite , and the test suite ran as expected. There were a few failures, but I've got a plan now.

Ted

> -----Original Message-----
> From: Jonas Devlieghere <jonas at devlieghere.com>
> Sent: Friday, August 9, 2019 4:57 PM
> To: Ted Woodward <tedwood at quicinc.com>
> Cc: LLDB <lldb-dev at lists.llvm.org>
> Subject: [EXT] Re: How do I use lit to only run the lldb test suite, now that
> dotest multiprocessing capabilities have been removed?
> 
> So far the only thing that changed by removing multiprocess is that `--no-
> multiprocess` is always enabled. Everything else you describe is still possible,
> and will continue to be possible.
> 
> Even when we remove the driver functionality from dotest.py this will all
> continue to work. The only difference is that dotest.py will operate on a single
> file after being invoked by some other driver. The problem you describe with
> the dual arguments seems relatively easy to fix by loading a different
> configuration in lit. It's not something I've looked at yet, because everything
> we care about is configurable from CMake or overridable by passing --params
> to lit.
> 
> 
> On Fri, Aug 9, 2019 at 1:30 PM Ted Woodward <tedwood at quicinc.com>
> wrote:
> >
> > Hi Jonas,
> >
> > What I need is a way to run the test suite with arbitrary command line
> arguments. Sometimes I want to run one or more tests with -f, sometimes I
> want to run one or more test files with -p, and sometimes I want to run the
> entire suite, either in parallel or 1 at a time (--no-multiprocess). I might be
> running from a directory where I've built lldb (but not clang, using clang from
> an arbitrary location), a directory where I've built everything (but in this case
> we set everything up with cmake), or a directory where I've checkout out just
> the sources and have copied the binaries from a distribution, so no cmake.
> >
> > dotest used to handle these cases; how do I handle them now?
> >
> > In our environment, lldb will launch and connect to the hexagon simulator,
> much like how it launches and connects to debugserver or lldb-server. But it
> has to use a specific version of clang, because we've found that if we have
> mismatches in clang/simulator/RTOS, bad things can happen. User code is in a
> shared library that gets loaded by a wrapper run under the RTOS. The RTOS,
> wrapper and user code all need to be built with the same complier and c/c++
> libraries.
> >
> > Ted
> >
> > > -----Original Message-----
> > > From: Jonas Devlieghere <jonas at devlieghere.com>
> > > Sent: Friday, August 9, 2019 3:18 PM
> > > To: Ted Woodward <tedwood at quicinc.com>
> > > Cc: LLDB <lldb-dev at lists.llvm.org>
> > > Subject: [EXT] Re: How do I use lit to only run the lldb test suite,
> > > now that dotest multiprocessing capabilities have been removed?
> > >
> > > Hey Ted,
> > >
> > > On Thu, Aug 8, 2019 at 2:08 PM Ted Woodward <tedwood at quicinc.com>
> > > wrote:
> > > >
> > > > Thanks Jonas.
> > > >
> > > > Is full support for --param fairly recent? I tried it with a
> > > > version of our
> > > master, based on top-of-tree from about a month ago, and it didn't
> > > work quite right. It's passing the dotest args, but it's also
> > > generating some args, so I'm seeing odd effects.
> > >
> > > It's not something I touched recently, but it's always possible they
> > > made some changes in LLVM.
> > >
> > > > Here is my run line:
> > > > bin/python bin/llvm-lit /local/mnt/ted/8.4/llvm/lldb/lit/Suite
> > > > --param
> > > 'dotest-args=-A v66 -C
> > > /prj/dsp/qdsp6/release/internal/HEXAGON/branch-
> > > 8.4/linux64/latest/Tools/bin/hexagon-clang --executable
> > > /local/scratch/ted/8.4/build/bin/lldb -t -v -f
> > > RecursiveTypesTestCase.test_recursive_type_1_dwarf'
> > > >
> > > > I only want to run
> > > > RecursiveTyepsTestCase.test_recursive_type_1_dwarf,
> > > but it's running the whole test suite.
> > >
> > > Do you know about the lldb-dotest binary? You can still use it to
> > > invoke a single test, similar to how lit does it. You should be able
> > > to just pass your arguments to that.
> > >
> > > Here's a dotest line from the run:
> > > >
> > > > /local/mnt/ted/8.4/build/bin/python
> > > > /local/mnt/ted/8.4/llvm/lldb/test/dotest.py -q --arch=v66 -s
> > > > /local/mnt/ted/8.4/build/lldb-test-traces --build-dir
> > > > /local/mnt/ted/8.4/build/lldb-test-build.noindex -S nm -u CXXFLAGS
> > > > -u CFLAGS --executable /local/mnt/ted/8.4/build/./bin/lldb
> > > > --dsymutil /local/mnt/ted/8.4/build/./bin/dsymutil --filecheck
> > > > /local/mnt/ted/8.4/build/./bin/FileCheck -C
> > > > /local/mnt/ted/8.4/build/./bin/clang --env ARCHIVER=/usr/bin/ar
> > > > --env OBJCOPY=/usr/bin/objcopy -A v66 -C
> > > > /prj/dsp/qdsp6/release/internal/HEXAGON/branch-8.4/linux64/latest/
> > > > Tool s/bin/hexagon-clang --executable
> > > > /local/scratch/ted/8.4/build/bin/lldb
> > > > -t -v -f RecursiveTypesTestCase.test_recursive_type_1_dwarf --env
> > > > LLVM_LIBS_DIR=/local/mnt/ted/8.4/build/./lib
> > > > /local/mnt/ted/8.4/llvm/lldb/packages/Python/lldbsuite/test/functi
> > > > onal ities/breakpoint/debugbreak -p TestDebugBreak.py
> > > >
> > > >
> > > > It's got both --arch= and -A, -C is set to my build directory
> > > > clang as well as the
> > > clang I told it to use, --executable is set twice, and it's got -f
> > > RecursiveTypesTestCase.test_recursive_type_1_dwarf and -p
> > > TestDebugBreak.py .
> > >
> > > Both lit and lldb-dotest are configured using the dotest arguments
> > > that we can configure at CMake configuration time. That would
> > > explain where the extra options come from. If those are not the ones
> > > you want, you can still invoke dotest.py directly.
> > >
> > > >
> > > > These tests that do a "process launch" (which is most of them)
> > > > invoke the
> > > hexagon simulator, but it was never launched. There was also only 1
> > > testcase built in /local/mnt/ted/8.4/build/lldb-test-build.noindex -
> > > types/TestRecursiveTypes.test_recursive_type_1_dwarf .
> > >
> > > Other than the extra arguments I can't think of any reason why this
> > > would behave differently. Lit is just a simple wrapper that invokes
> > > dotest.py with the right arguments.
> > >
> > > > This does not have the patch that removes multiprocess support
> > > > from
> > > dotest.
> > > >
> > > > Ted
> > > >
> > > > > -----Original Message-----
> > > > > From: Jonas Devlieghere <jonas at devlieghere.com>
> > > > > Sent: Thursday, August 8, 2019 2:50 PM
> > > > > To: Ted Woodward <tedwood at quicinc.com>
> > > > > Cc: LLDB <lldb-dev at lists.llvm.org>
> > > > > Subject: [EXT] Re: How do I use lit to only run the lldb test
> > > > > suite, now that dotest multiprocessing capabilities have been
> removed?
> > > > >
> > > > > Hey Ted,
> > > > >
> > > > > 1. You can run just the dotest-tests by pointing lit at the
> > > > > `lit/Suite`
> > > directory.
> > > > > 2. You can pass arguments to dotest by passing `dotest-args` in --
> param.
> > > > >
> > > > > The invocation would look something like this:
> > > > >
> > > > > /path/to/llvm/bin/llvm-lit /path/to/lldb/lit/Suite --param
> > > > > 'dotest-args=--foo -- bar'
> > > > >
> > > > > Hope that helps,
> > > > > Jonas
> > > > >
> > > > > On Thu, Aug 8, 2019 at 9:31 AM Ted Woodward
> > > > > <tedwood at quicinc.com>
> > > > > wrote:
> > > > > >
> > > > > > RE: https://reviews.llvm.org/D65311
> > > > > >
> > > > > >
> > > > > >
> > > > > > Internally we use dotest to run the lldb test suite with
> > > > > > various RTOS
> > > > > configurations for the test binaries. In these runs we don’t
> > > > > care about the lit tests or the unit tests, because they are OS agnostic.
> > > > > We do this by specifying the compiler, lldb, and test flavor
> > > > > (static testcase + os, dynamic library testcase loaded by an OS
> > > > > image, dynamic library testcase loaded by an OS image running
> > > > > the OS’ debug
> > > stub).
> > > > > >
> > > > > >
> > > > > >
> > > > > > With the multiprocess testrunner removed, how do I have lit:
> > > > > >
> > > > > > Only run the lldb test suite
> > > > > > Run dotest with specific arguments
> > > > > >
> > > > > >
> > > > > >
> > > > > > We’re not running cmake; we’re taking an existing tools build
> > > > > > and running
> > > > > the tests from the source directory using the toolset.