[Lldb-commits] [PATCH] D46005: [test] Add a utility for dumping all tests in the dotest suite

[Adrian Prantl via Phabricator via lldb-commits]

aprantl added a comment.

In https://reviews.llvm.org/D46005#1109817, @zturner wrote:

> In https://reviews.llvm.org/D46005#1109761, @JDevlieghere wrote:
>
> > In https://reviews.llvm.org/D46005#1109693, @zturner wrote:
> >
> > > FWIW, I think the single biggest improvement one could make to the LLDB test suite runtime is to compile all inferiors up front as part of the CMake step.  If you run the test suite twice every inferior is unnecessarily compiled twice.
> >
> >
> > I'm a big proponent of moving as much logic as possible into the configuration stage, but a common use case (at least for us) is testing a single build with a different compiler/configuration.
>
>
> Then the configure / CMake stage could build all inferiors with every possible configuration you want to test, each one writing to a different output directory.  Sorta like how in LLVM you can specify `LLVM_TARGETS_TO_BUILD=X86,ARM,...` you could specify something like `LLDB_TEST_INFERIOR_CONFIGS=gcc-x64;gcc-x86;clang-arm64`.
>
> Obviously this is a big change and a lot of work, but I think it would make the test suite run in under 30 seconds


I don't think that moving this to the configure stage is the right choice. I'm also skeptical about your claim about the saved time (are you talking about Windows?).

In my opinion the configuration should configure what is being built, not what is being tested. I don't want to have to lock down at configure time which tests I will be running. As an analogy, this is similar to how LLVM has the in-tree regression tests but also the external test-suite that can be run with endless permutations of options, to test one specific build of LLVM, without having to reconfigure/rebuild.


https://reviews.llvm.org/D46005