[lldb-dev] How do I use lit to only run the lldb test suite, now that dotest multiprocessing capabilities have been removed?

[Jonas Devlieghere via lldb-dev]

Hey Ted,

On Thu, Aug 8, 2019 at 2:08 PM Ted Woodward <tedwood at quicinc.com> wrote:
>
> Thanks Jonas.
>
> Is full support for --param fairly recent? I tried it with a version of our master, based on top-of-tree from about a month ago, and it didn't work quite right. It's passing the dotest args, but it's also generating some args, so I'm seeing odd effects.

It's not something I touched recently, but it's always possible they
made some changes in LLVM.

> Here is my run line:
> bin/python bin/llvm-lit /local/mnt/ted/8.4/llvm/lldb/lit/Suite --param 'dotest-args=-A v66 -C /prj/dsp/qdsp6/release/internal/HEXAGON/branch-8.4/linux64/latest/Tools/bin/hexagon-clang --executable /local/scratch/ted/8.4/build/bin/lldb -t -v -f RecursiveTypesTestCase.test_recursive_type_1_dwarf'
>
> I only want to run RecursiveTyepsTestCase.test_recursive_type_1_dwarf, but it's running the whole test suite.

Do you know about the lldb-dotest binary? You can still use it to
invoke a single test, similar to how lit does it. You should be able
to just pass your arguments to that.

Here's a dotest line from the run:
>
> /local/mnt/ted/8.4/build/bin/python /local/mnt/ted/8.4/llvm/lldb/test/dotest.py -q --arch=v66 -s /local/mnt/ted/8.4/build/lldb-test-traces --build-dir /local/mnt/ted/8.4/build/lldb-test-build.noindex -S nm -u CXXFLAGS -u CFLAGS --executable /local/mnt/ted/8.4/build/./bin/lldb --dsymutil /local/mnt/ted/8.4/build/./bin/dsymutil --filecheck /local/mnt/ted/8.4/build/./bin/FileCheck -C /local/mnt/ted/8.4/build/./bin/clang --env ARCHIVER=/usr/bin/ar --env OBJCOPY=/usr/bin/objcopy -A v66 -C /prj/dsp/qdsp6/release/internal/HEXAGON/branch-8.4/linux64/latest/Tools/bin/hexagon-clang --executable /local/scratch/ted/8.4/build/bin/lldb -t -v -f RecursiveTypesTestCase.test_recursive_type_1_dwarf --env LLVM_LIBS_DIR=/local/mnt/ted/8.4/build/./lib /local/mnt/ted/8.4/llvm/lldb/packages/Python/lldbsuite/test/functionalities/breakpoint/debugbreak -p TestDebugBreak.py
>
>
> It's got both --arch= and -A, -C is set to my build directory clang as well as the clang I told it to use, --executable is set twice, and it's got -f RecursiveTypesTestCase.test_recursive_type_1_dwarf and -p TestDebugBreak.py .

Both lit and lldb-dotest are configured using the dotest arguments
that we can configure at CMake configuration time. That would explain
where the extra options come from. If those are not the ones you want,
you can still invoke dotest.py directly.

>
> These tests that do a "process launch" (which is most of them) invoke the hexagon simulator, but it was never launched. There was also only 1 testcase built in /local/mnt/ted/8.4/build/lldb-test-build.noindex - types/TestRecursiveTypes.test_recursive_type_1_dwarf .

Other than the extra arguments I can't think of any reason why this
would behave differently. Lit is just a simple wrapper that invokes
dotest.py with the right arguments.

> This does not have the patch that removes multiprocess support from dotest.
>
> Ted
>
> > -----Original Message-----
> > From: Jonas Devlieghere <jonas at devlieghere.com>
> > Sent: Thursday, August 8, 2019 2:50 PM
> > To: Ted Woodward <tedwood at quicinc.com>
> > Cc: LLDB <lldb-dev at lists.llvm.org>
> > Subject: [EXT] Re: How do I use lit to only run the lldb test suite, now that
> > dotest multiprocessing capabilities have been removed?
> >
> > Hey Ted,
> >
> > 1. You can run just the dotest-tests by pointing lit at the `lit/Suite` directory.
> > 2. You can pass arguments to dotest by passing `dotest-args` in --param.
> >
> > The invocation would look something like this:
> >
> > /path/to/llvm/bin/llvm-lit /path/to/lldb/lit/Suite --param 'dotest-args=--foo --
> > bar'
> >
> > Hope that helps,
> > Jonas
> >
> > On Thu, Aug 8, 2019 at 9:31 AM Ted Woodward <tedwood at quicinc.com>
> > wrote:
> > >
> > > RE: https://reviews.llvm.org/D65311
> > >
> > >
> > >
> > > Internally we use dotest to run the lldb test suite with various RTOS
> > configurations for the test binaries. In these runs we don’t care about the lit
> > tests or the unit tests, because they are OS agnostic. We do this by specifying
> > the compiler, lldb, and test flavor (static testcase + os, dynamic library testcase
> > loaded by an OS image, dynamic library testcase loaded by an OS image
> > running the OS’ debug stub).
> > >
> > >
> > >
> > > With the multiprocess testrunner removed, how do I have lit:
> > >
> > > Only run the lldb test suite
> > > Run dotest with specific arguments
> > >
> > >
> > >
> > > We’re not running cmake; we’re taking an existing tools build and running
> > the tests from the source directory using the toolset.