[libcxx-commits] [libcxx] [llvm] [openmp] [lit] Add a flag to disable lit time tests (PR #98270)

[Vincent Lee via libcxx-commits]

thevinster wrote:

> Orthogonal to the real issue here, but have you considered the additional overhead of spinning up lit per test? I would expect (but it's always possible I'm wrong) that spinning up lit 10 times to run 10 tests individually would be slower than spinning it up once to run them all.

It's possible that the lit test, itself, could be slower by having to spin it up N times for N tests, but the net result for us by using a distributed build system like buck should still be faster since we would've distributed these tests across multiple machines and not be limited by the cores on a single machine. If we had modeled our testing approach similar to lit's model of taking in a test suite, then we would be sacrificing flexibility of being able to run a single lit test (for those who may only want to debug the single test during development)

> One random idea that I haven't really given much thought to how effective it would be (or even whether it would actually work) would be to avoid a single centralised file and instead have a file per test, that are read in following test discovery to form a combined database of sorts for determining order within that set of tests.

Yep this would be a potential solution. The problem I see here with this approach is the lit overhead for doing I/O just to write a timestamp for each file. To better illustrate, LLVM has ~60000 lit tests. Having to do I/O for ~60000 files might not scale and could end up further slowing down the current behavior (which wouldn't be ideal); whereas, disabling it via a flag would maintain the status quo. 

I'm open to other potential forms of solution. And currently, disabling via a flag is the best compromise (even though it may seem like a workaround). 

https://github.com/llvm/llvm-project/pull/98270