[Lldb-commits] [PATCH] D12416: Parse dotest.py/dosep.py output, providing general stats and skip reason breakdown by counts.

[Greg Clayton via lldb-commits]

> On Aug 28, 2015, at 9:36 AM, Zachary Turner <zturner at google.com> wrote:
> 
> 
> 
> On Fri, Aug 28, 2015 at 8:41 AM Todd Fiala <todd.fiala at gmail.com> wrote:
> tfiala added a comment.
> 
> In http://reviews.llvm.org/D12416#235112, @labath wrote:
> 
> > Same question as Zachary. This sounds like a very useful feature and it would be nice to have it integrated into the current test system,
> 
> 
> Well, I'd be happy to do that in dosep.py.  There are a couple of challenges:
> 
> 1. It requires information that is only provided when parsable output mode is invoked in dotest.py.  So it mandates a particular output style that is not necessarily what everyone wants to see.
> One solution to this that I've thought of is to have a way to launch dotest in IPC mode.  Piping stdin and stdout is slow because it results in having to parse text which was not printed with fast consumption in mind.  If dosep could could launch dotest with an option like --pipe=dotest_pipe_instance_1 or something, then they could communicate over that pipe in a structured format like json that is easy to parse.

On unix pipes and files are about the same speed so I don't see the need to do any fancy piping. Is it different on windows? I would like to avoid any complicated mechanisms like pipes or other special things so that we don't need any special python modules. I would like to stick to STDIO unless there is a really compelling reason.

>  
> > Parsing the output like this is likely to break the script due to random changes in the other parts.
> 
> 
> That probably cuts both ways.  Changing output of a test script (the output I'm adding) seems quite plausible to break other scripts that parse the output of the test infrastructure.
> The pipe issue would solve that, but another way to at least alleviate the pain associated with that is to agree that we print all summary information at the end, and we try to keep the summary information as stable as possible.  We still might have to change stuff sometimes, but the benefit of doing things this way is that we can then agree that everything EXCEPT the summary information can change for any reason at all, or no reason at all, and it won't break anyone.  The summary informatino could be printed in a nice structured format with 1 result on each line, so that new information could simply be appended, and the only time you'd break something is if you deleted a statistic.

With JSON, as long as you don't remove anything, everything will still work. Adding new key/value pairs is additive and won't affect things. Text scraping is definitely not what we want even if we try to keep it consistent.
>  
> Thoughts?  It might be more work to do things this way, but I kind of feel like the complexity of dosep and dotest is getting to the point where it might be worth considering putting some extra time in to think about things like this.

The one reason the IPC would be nice is to communicate individual packets, but we could still do the same thing with STDIO. 

It would be nice to have individual packets that would:
1 - communicate when a dotest.py invocation is started and what tests it plans to run
2 - communicate status on each test as it starts
3 - communicate status on each test as it finishes with the status
4 - if a dotest.py crashes, we should know which test was running and know that it crashed and should be able to run the remaining tests that we didn't get to since we know what was supposed to run from step 1 and we know what started and finished due to step 2 and 3

I think right know if dotest crashes, we just lose any tests that weren't executed before it crashed...