[Lldb-commits] [PATCH] D46005: [test] Add a utility for dumping all tests in the dotest suite

[Adrian Prantl via Phabricator via lldb-commits]

aprantl added a comment.

In https://reviews.llvm.org/D46005#1109920, @zturner wrote:

> In https://reviews.llvm.org/D46005#1109911, @aprantl wrote:
>
> > In https://reviews.llvm.org/D46005#1109817, @zturner wrote:
> >
> > > In https://reviews.llvm.org/D46005#1109761, @JDevlieghere wrote:
> > >
> > > > In https://reviews.llvm.org/D46005#1109693, @zturner wrote:
> > > >
> > > > > FWIW, I think the single biggest improvement one could make to the LLDB test suite runtime is to compile all inferiors up front as part of the CMake step.  If you run the test suite twice every inferior is unnecessarily compiled twice.
> > > >
> > > >
> > > > I'm a big proponent of moving as much logic as possible into the configuration stage, but a common use case (at least for us) is testing a single build with a different compiler/configuration.
> > >
> > >
> > > Then the configure / CMake stage could build all inferiors with every possible configuration you want to test, each one writing to a different output directory.  Sorta like how in LLVM you can specify `LLVM_TARGETS_TO_BUILD=X86,ARM,...` you could specify something like `LLDB_TEST_INFERIOR_CONFIGS=gcc-x64;gcc-x86;clang-arm64`.
> > >
> > > Obviously this is a big change and a lot of work, but I think it would make the test suite run in under 30 seconds
> >
> >
> > I don't think that moving this to the configure stage is the right choice. I'm also skeptical about your claim about the saved time (are you talking about Windows?).
> >
> > In my opinion the configuration should configure what is being built, not what is being tested. I don't want to have to lock down at configure time which tests I will be running. As an analogy, this is similar to how LLVM has the in-tree regression tests but also the external test-suite that can be run with endless permutations of options, to test one specific build of LLVM, without having to reconfigure/rebuild.
>
>
> This is a bit of a digression from this patch,


Agreed :-)

but no I wasn't talking about specific to Windows.  The amount of time required to compile several thousand inferiors is quite significant, especially since it is completely redundant and shouldn't be changing between each run of the test suite.  So it's unnecessary.

> 

I suppose you are looking at this from the perspective of an LLDB developer, where rebuilding the tests after changing something in LLDB seems redundant. I was thinking about this from the perspective of our bots, and there LLVM changes much much faster than LLDB, so between two bot runs the only thing that changes is the compiler, which forces you to rebuild all the source-based tests anyway.

... but let's postpone this discussion for another day. As you said it's a diversion for this review.

> Also just because it could be done during the CMake configure step doesn't mean it couldn't *also* be done via some external script.  Similar to how if you run `ninja check-lldb` it runs with the default set of options, but you can still run `dotest.py` manually to pass additional flags.  So building inferiors as part of the configure step does not necessitate locking down at configure time which tests you will be running.




https://reviews.llvm.org/D46005