[lldb-dev] [PATCH] D24591: [LIT] First pass of LLDB LIT support

[Pavel Labath via lldb-dev]

On 15 September 2016 at 16:15, Zachary Turner <zturner at google.com> wrote:
> lldb-commits to bcc, lldb-dev to cc.
>
>> The biggest feature I see missing here is the ability to run tests
>> remotely. Remote debugging is the most important use case for our team, and
>> now we have a number of (experimental) bots running the remote test suite.
>> We want to make sure we can debug correctly when the host and target have
>> different architectures
>
> That is a good point.  I thought about this a little bit and I believe I
> have a solution to most of the issues you would face when running a remote
> test suite, but perhaps it would be better to take that to a different
> thread on lldb-dev rather than on this commit.  So I'm taking lldb-commits
> off of this thread and adding lldb-dev.   Responses inline.
>
> On Thu, Sep 15, 2016 at 4:02 AM Pavel Labath <labath at google.com> wrote:
>
>
>>
>>  - the test suite needs to know how to connect to the remote target
>> (mostly it's just executing three commands: `platform select XXX`, `platform
>> connect` and `platform settings --working-dir YYY`), but it can get a bit
>> more complicated in some cases (if we need to install shared libraries along
>> with the main executable, if it involves the test executable writing
>> something to a file, etc.)
>
> This seems like the easiest of the problems to solve, so I'm mentioning it
> first.  I can imagine a couple of command line options to the lldb
> executable itself that allow one to configure remote debugging from the
> command line.  Then, we could provide one CMake target for each remote
> platform.  check-lldb-remote-android, check-lldb-remote-fizzbuzz, whatever.
> And you could configure all of these at CMake time with one command.  Like
> cmake -DLLDB_TEST_REMOTE_TARGETS=android,fizzbuzz,etc.  Internally, each
> target's CMake logic could configure some appropriate variables that would
> be picked up by lit so that the lit substitution for running lldb would
> supply the appropriate command line to configure the remote target.

I am happy to hear that you are thinking about the remote case. Yes,
this is definitely solvable, we just need to make sure the solution is
there, and it fits our needs.

>
>  - this includes the need to pass funny compiler switches based on the
> target (some android targets can only run `-pie` executables, some cannot
> run `-pie` executables at all).
>  This should be handled the same way as above.  Configuring the lit
> substitutions for the compiler invocation can depend on which cmake target
> you're running lit through.  ninja check-lldb-remote-android could define
> one set of lit substitutions, and another target could define another.  Is
> this sufficient?  Is it easy to define a mapping between the set of all
> remote targets and the set of all command line flag combinations?
The set of flags should be pretty easy to define, but I am a bit
worried about the idea of doing it in cmake. See below.

>
>> - we need to be able to fully specify the toolchain used to compile the
>> target executables (specifically, we should not assume the host compiler)
>
> Will you want a single run of the test suite to use multiple compilers, or
> can we assume there is only 1 compiler for a given cmake output directory?
> I think we can assume the latter since even now we only have 1 cmake
> variable `LLDB_TEST_COMPILER`, but let me know if this is a wrong assumption
Having one run of a test suite with a just a single compiler is fine.
dotest.py used to support more compilers, but that has been a long
time ago (I just checked it last week, it is not working anymore).
However, we do run the tests with multiple compilers from a single
output directory.

We do this by bypassing cmake and invoking dotest.py directly. Our
buildbot does that as well. Check out
<http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/19561>
you will see it does 6 test runs from a single cmake output directory
(if you click on the "stdio" link you can see the dotest.py commands
it invokes).

In my normal workflow I use "ninja check-lldb" to validate my changes,
but when I am working on a compiler-specific issue I do try it out
with the non-standard compiler by invoking dotest.py directly. I would
not want to need a fresh build of lldb for that as nothing within lldb
changes - I only need to the change compiler used to compile the test
executables. The same is true for android. Right now I am able to run
android x86 and android arm64 from the same build directory just by
specifying options to dotest.py. (Obviously I need to have a separate
build folder for the server component, but the python tests are run
against the client and the server is used only indirectly. Potentially
we could change that to run the test suite from the android build
folder, but then we would need a way to specify a path to the client
host build somewhere).

So, yes there is only one LLDB_TEST_COMPILER, but that is not the only
way to run the test suite. I think it's fine to have one test compiler
as the official blessed way to run the tests, but it's also important
to have a way to run non-standard configurations of the test suite
without rebuilding.

I hope this makes things a bit clearer.
pl