[Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support

[Jim Ingham via lldb-commits]

> On Sep 14, 2016, at 4:00 PM, Zachary Turner <zturner at google.com> wrote:
> 
> On Wed, Sep 14, 2016 at 3:51 PM Sean Callanan <scallanan at apple.com> wrote:
> How different is that really from
> 
> (lldb) script lldb.frame.FindVariable("argc").GetValue()
> '1'
> (lldb) script lldb.process.GetNumThreads()
> 1
> (lldb) script lldb.process.GetThreadAtIndex(0).GetThreadID()
> 3514809
> 
> ?  If it's developer-only, then this is even fairly well-documented using e.g. "script dir(lldb.process)"
> 
> Admittedly I hadn't thought of doing that.  That said I can think of a couple of differences:
> 
> 1. Writing it using script commands means you have to be familiar with the python api, somewhat raising the barrier to entry for writing new tests.

The lldb api is pretty simple, especially for basic stuff like checking variable values.  Plus, if you wanted to write tests like "I have a funky new type and I want to check that all its children are right" the bit of time you spent looking at how SBValues works will pay off many-fold in how much more accurate you can make these tests, and how much more easily you can get exactly what you want to test, when you aren't trying to parse text output of the value printing.  I think this is really a false tradeoff.

And if you are testing something simple, you can just use Sean's inline tests, they are not really any more difficult to write than the lit tests (except that using the SB API's are the natural language, but I see that as a positive, not a negative.)

> 
> 2. It depends on proper behavior of the script command, which means that if the script command itself is broken, every single test will be broken.

> 3. It means that building the lldb python module is required to run tests.  Wouldn't it be awesome if there was a way to run tests even with LLDB_DISABLE_PYTHON=1?  Then script could just be a "feature" (as in, you could limit tests which use the script command to tests whose purpose is explicitly to test where the scripting system works, and for those you could just add "REQUIRES: python" at the top of the lit file.  And I imagine that not invoking python during the running of the test suite would drastically speed up its execution.
> 

I would find that very not awesome, because it would mean ALL our tests are scraping text output, and I lived through that once already and would rather not do it again.

> 4. You can test a LOT more things when you are able to use an api that doesn't have to be stable.

When I mentioned that API, I had in mind an internal Python module that you could use to grub into internals.  I'm all for that.  I'm not so much in favor of "I have an ad hoc command that prints out various bits of texts, and we'll use that for testing."

Jim