[lldb-dev] [RFC]The future of pexpect

[Jim Ingham via lldb-dev]

I don't think anybody uses these tests.  They are all time based benchmarks, and in the end there was just too much variability for them to be really useful.  We really need to do more work tracking performance, but I think a better approach is to focus on how much work we do (how many DIE's did you have to parse to do X, how many lookups did it take to compile an expression, or how many memory requests did a task take, things like that.  Those seem to me likely to be ore stable.

That said, I'm unclear why any of the benchmark tests would need pexpect to function.  The Driver used to have a lot more functionality in it that has since been moved into the SB API's.  Maybe there was good reason for doing performance testing through the driver but I can't think of any good reason for doing that nowadays. 

Jim

> On Jan 31, 2019, at 10:23 AM, Davide Italiano via lldb-dev <lldb-dev at lists.llvm.org> wrote:
> 
> On Thu, Jan 31, 2019 at 10:09 AM Pavel Labath <pavel at labath.sk> wrote:
>> 
>> On 31/01/2019 02:32, Davide Italiano via lldb-dev wrote:
>>> As you probably know (I didn’t), lldb embeds its own version of
>>> `pexpect-2.4`, which doesn’t support python3.
>>> This is the (relatively short) list of tests relying on pyexpect:
>>> 
>>> testcases/tools/lldb-mi/syntax/TestMiSyntax.py:        import pexpect
>>>                         # 7 (EOF)
>>> testcases/tools/lldb-mi/lldbmi_testcase.py:        import pexpect
>>> testcases/tools/lldb-mi/signal/TestMiSignal.py:        import pexpect
>>> testcases/tools/lldb-mi/signal/TestMiSignal.py:        import pexpect
>>> testcases/lldbtest.py:            import pexpect
>>> testcases/driver/batch_mode/TestBatchMode.py:        import pexpect
>>> testcases/driver/batch_mode/TestBatchMode.py:        import pexpect
>>> testcases/driver/batch_mode/TestBatchMode.py:        import pexpect
>>> testcases/driver/batch_mode/TestBatchMode.py:        import pexpect
>>> testcases/lldbpexpect.py:    import pexpect
>>> testcases/terminal/TestSTTYBeforeAndAfter.py:        import pexpect
>>> testcases/darwin_log.py:        import pexpect
>>> testcases/macosx/nslog/TestDarwinNSLogOutput.py:        import pexpect
>>> testcases/benchmarks/stepping/TestSteppingSpeed.py:        import pexpect
>>> testcases/benchmarks/frame_variable/TestFrameVariableResponse.py:
>>>   import pexpect
>>> testcases/benchmarks/turnaround/TestCompileRunToBreakpointTurnaround.py:
>>>        import pexpect
>>> testcases/benchmarks/turnaround/TestCompileRunToBreakpointTurnaround.py:
>>>        import pexpect
>>> testcases/benchmarks/expression/TestExpressionCmd.py:        import pexpect
>>> testcases/benchmarks/expression/TestRepeatedExprs.py:        import pexpect
>>> testcases/benchmarks/expression/TestRepeatedExprs.py:        import pexpect
>>> testcases/benchmarks/startup/TestStartupDelays.py:        import pexpect
>>> testcases/functionalities/command_regex/TestCommandRegex.py:
>>> import pexpect
>>> testcases/functionalities/single-quote-in-filename-to-lldb/TestSingleQuoteInFilename.py:
>>>        import pexpect
>>> testcases/functionalities/format/TestFormats.py:        import pexpect
>>> 
>>> (I count 14, but there might be something else).
>>> 
>>> I audited all of them and from what I see they’re almost all testing the driver.
>>> I had a chat with my coworkers and we agreed it's reasonable to
>>> replace them with lit tests (as they're just running commands).
>>> This would allow us to get rid of an external dependency, which
>>> happened to be cause of trouble in the past.
>>> 
>>> Are there any objections?
>>> 
>>> Thanks,
>>> 
>> 
>> I'm not a fan of pexpect, and if these tests can be converted to lit,
>> then I'm all for it. But I do have a question.
>> 
>> There is a class of tests that cannot be written in the current lit
>> framework, but they can with pexpect. A couple of weeks ago we had a
>> patch fixing a bug where pressing up arrow while searching through the
>> command history caused a crash. In the end a test for this was not
>> included because it was hard for a reason unrelated to pexpect, but
>> without pexpect (or something equivalent) writing a test for this would
>> be impossible.
>> 
> 
> I don't know about this, to be honest. Maybe lit should grow an
> interactive mode somehow to accomodate for this functionality?
> I'm not an expert in how it's implemented so that could be hard to achieve.
> FWIW, I haven't seen anything that really requires interactivity, but
> I have to admit I haven't looked really deeply.
> 
>> What's our story for testing interactive command-line functionalities?
>> The way I see it, if we don't use pexpect, we'll either have to use some
>> other tool which simulates a realistic terminal, or write our own. (We
>> already have one attempt for this in
>> unittests/Editline/EditlineTest.cpp, but this would need more work to be
>> fully functional.)
>> 
>> pl
>> 
>> 
>> PS: Does anyone actually use the benchmark tests? Can we just delete them?
> 
> I don't know. Maybe Jim knows. I personally don't use them.
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev