[Lldb-commits] [PATCH] D12416: Parse dotest.py/dosep.py output, providing general stats and skip reason breakdown by counts.

[Zachary Turner via lldb-commits]

It doesn't necessarily have to be through fancy pipes, I guess the point is
just that right now we use stdio as a way to print stuff that a person will
read, so naturally it's formatted differently than if we were trying to
produce something that would be consumed by a machine.

But you're probably right, that instead of having an option to print
machine-consumable output to a special pipe, the option could just print
machine consumable output to stdio.  No problems with that on Windows.

On Mon, Aug 31, 2015 at 11:03 AM Greg Clayton <clayborg at gmail.com> wrote:

>
> > On Aug 28, 2015, at 9:36 AM, Zachary Turner <zturner at google.com> wrote:
> >
> >
> >
> > On Fri, Aug 28, 2015 at 8:41 AM Todd Fiala <todd.fiala at gmail.com> wrote:
> > tfiala added a comment.
> >
> > In http://reviews.llvm.org/D12416#235112, @labath wrote:
> >
> > > Same question as Zachary. This sounds like a very useful feature and
> it would be nice to have it integrated into the current test system,
> >
> >
> > Well, I'd be happy to do that in dosep.py.  There are a couple of
> challenges:
> >
> > 1. It requires information that is only provided when parsable output
> mode is invoked in dotest.py.  So it mandates a particular output style
> that is not necessarily what everyone wants to see.
> > One solution to this that I've thought of is to have a way to launch
> dotest in IPC mode.  Piping stdin and stdout is slow because it results in
> having to parse text which was not printed with fast consumption in mind.
> If dosep could could launch dotest with an option like
> --pipe=dotest_pipe_instance_1 or something, then they could communicate
> over that pipe in a structured format like json that is easy to parse.
>
> On unix pipes and files are about the same speed so I don't see the need
> to do any fancy piping. Is it different on windows? I would like to avoid
> any complicated mechanisms like pipes or other special things so that we
> don't need any special python modules. I would like to stick to STDIO
> unless there is a really compelling reason.
>
> >
> > > Parsing the output like this is likely to break the script due to
> random changes in the other parts.
> >
> >
> > That probably cuts both ways.  Changing output of a test script (the
> output I'm adding) seems quite plausible to break other scripts that parse
> the output of the test infrastructure.
> > The pipe issue would solve that, but another way to at least alleviate
> the pain associated with that is to agree that we print all summary
> information at the end, and we try to keep the summary information as
> stable as possible.  We still might have to change stuff sometimes, but the
> benefit of doing things this way is that we can then agree that everything
> EXCEPT the summary information can change for any reason at all, or no
> reason at all, and it won't break anyone.  The summary informatino could be
> printed in a nice structured format with 1 result on each line, so that new
> information could simply be appended, and the only time you'd break
> something is if you deleted a statistic.
>
> With JSON, as long as you don't remove anything, everything will still
> work. Adding new key/value pairs is additive and won't affect things. Text
> scraping is definitely not what we want even if we try to keep it
> consistent.
> >
> > Thoughts?  It might be more work to do things this way, but I kind of
> feel like the complexity of dosep and dotest is getting to the point where
> it might be worth considering putting some extra time in to think about
> things like this.
>
> The one reason the IPC would be nice is to communicate individual packets,
> but we could still do the same thing with STDIO.
>
> It would be nice to have individual packets that would:
> 1 - communicate when a dotest.py invocation is started and what tests it
> plans to run
> 2 - communicate status on each test as it starts
> 3 - communicate status on each test as it finishes with the status
> 4 - if a dotest.py crashes, we should know which test was running and know
> that it crashed and should be able to run the remaining tests that we
> didn't get to since we know what was supposed to run from step 1 and we
> know what started and finished due to step 2 and 3
>
> I think right know if dotest crashes, we just lose any tests that weren't
> executed before it crashed...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-commits/attachments/20150831/5852932f/attachment.html>