[Lldb-commits] [PATCH] D12416: Parse dotest.py/dosep.py output, providing general stats and skip reason breakdown by counts.

[Todd Fiala via lldb-commits]

tfiala added a comment.

In http://reviews.llvm.org/D12416#235191, @clayborg wrote:

> I would like to see a few things as long as we are chaning things:
>
> - get rid of dosep.py and just put the functionality into dotest.py if possible, it can just spawn itself in different modes


I'm all for doing that.  It's a bit of a bigger job, though.  I'd like to tackle that separate from getting the skip reason tabulation in (and I'm happy to get skip tabulation into dosep.py now).

> - if we can't get rid of dosep.py lets make all options that work for dotest.py work in dosep.py


That makes sense.  We should be able to build the options for dotest.py without requiring them to be passed through, and would aid in a transition to a dotest.py-only world.

> - we should never have to launch dosep.py with any arguments in order for it to run correctly


Hmm, would you say that you can currently run dotest.py without any arguments and it does an admirable job?  I think at the very least we need to specify the architecture(s) we're testing, and the compiler specification on Linux is pretty critical per some other bugs we've discussed (not resolving symlinks before making decisions based on compiler name, for example).  I'd shoot for saying dosep.py should match dotest.py in terms of what it needs as arguments, and independently tackling the items in dotest.py that currently require options to make them smarter.  (i.e. the goal is command-line parity on dosep.py/dotest.py, only adding args to dosep.py for additional functionality that is related to the parallel test running).

> - add options for formatted (JSON) output and have dosep.py (or dotest.py if we move dosep.py functionality over there) enable that option when spawning subprocesses. We can then parse the output easily in the tools that spawns the sub processes and we can expand the format as needed. This would also allow for buildbots to translate the results of the testing into their preferred format for correct display for the buildbot web interfaces.


I like that.  This could be done as a separate task.

> - all functionality should exist in dotest.py (no external reporting python scripts...)


I'm on board with that.

For now I'd like to just tackle the part of getting the reporting in dosep.py.  I'm happy to take a crack at both (1) merging the command line options so dosep.py no longer requires the pass-through dotest.py options, and instead takes the options directly on the dosep.py command line, and (2) trying to eliminate dosep.py altogether.

(And I'd like to see our test harness get some more tests for itself!()

These are great longer-term roadmap items for the test infrastructure IMHO.

-Todd


http://reviews.llvm.org/D12416