[PATCH] D158954: WIP: [lit] Enable debugging lit tests with python's pdb

[Joel E. Denny via Phabricator via llvm-commits]

jdenny created this revision.
jdenny added reviewers: Endill, jhenderson, yln, ldionne, aaron.ballman, awarzynski, MaskRay, rnk, mstorsjo, asavonic.
Herald added a subscriber: delcypher.
Herald added a project: All.
jdenny requested review of this revision.
Herald added a project: LLVM.

This patch enables using python's default debugger, pdb, to interactively step through and debug directives in lit tests, even existing tests that do *not* use PYTHON directives (proposed in D154987 <https://reviews.llvm.org/D154987>).  For example, let's say we have a test `llvm/test/example.txt` (that contains no PYTHON directives despite what lit seems to claim in the output):

  $ LIT_OPTS=--pdb LIT_FILTER=example.txt ninja check-llvm
  [0/1] Running the LLVM regression tests
  -- Testing: 1 of 48226 tests, 0 workers --
  ******************** STARTED DEBUGGING 'LLVM :: example.txt' ********************
  # started executing 'PYTHON:' directive from line 1 to 8
  > /home/jdenny/llvm-project/llvm/test/example.txt(1)<module>()
  -> ; DEFINE: %{msg} =
  (Pdb) l
    1  -> ; DEFINE: %{msg} =
    2     ; DEFINE: %{greet} = echo %{msg}
    3
    4     ; REDEFINE: %{msg} = hello world
    5     ; RUN: %{greet}
    6
    7     ; REDEFINE: %{msg} = goodby world
    8     ; RUN: %{greet}
  [EOF]
  (Pdb) n
  > /home/jdenny/llvm-project/llvm/test/example.txt(2)<module>()
  -> ; DEFINE: %{greet} = echo %{msg}
  (Pdb) n
  > /home/jdenny/llvm-project/llvm/test/example.txt(4)<module>()
  -> ; REDEFINE: %{msg} = hello world
  (Pdb) n
  > /home/jdenny/llvm-project/llvm/test/example.txt(5)<module>()
  -> ; RUN: %{greet}
  (Pdb) p lit.expand("%{greet}")
  'echo hello world'
  (Pdb) n
  # lit.run called from /home/jdenny/llvm-project/llvm/test/example.txt:5
  #         called from /usr/lib/python3.8/bdb.py:587
  #         called from /usr/lib/python3.8/pdb.py:1595
  echo hello world
  # executed command: echo hello world
  # .---command stdout------------
  # | hello world
  # `-----------------------------
  > /home/jdenny/llvm-project/llvm/test/example.txt(7)<module>()
  -> ; REDEFINE: %{msg} = goodby world
  (Pdb) n
  > /home/jdenny/llvm-project/llvm/test/example.txt(8)<module>()
  -> ; RUN: %{greet}
  (Pdb) p lit.expand("%{greet}")
  'echo goodby world'
  (Pdb) p lit.expand("%{msg}")
  'goodby world'
  (Pdb) p lit.redefine("%{msg}", "goodbye world")
  None
  (Pdb) p lit.expand("%{greet}")
  'echo goodbye world'
  (Pdb) n
  # lit.run called from /home/jdenny/llvm-project/llvm/test/example.txt:8
  #         called from /usr/lib/python3.8/bdb.py:587
  #         called from /usr/lib/python3.8/pdb.py:1595
  echo goodbye world
  # executed command: echo goodbye world
  # .---command stdout------------
  # | goodbye world
  # `-----------------------------
  --Return--
  > /home/jdenny/llvm-project/llvm/test/example.txt(8)<module>()->None
  -> ; RUN: %{greet}
  (Pdb) n
  # finished executing 'PYTHON:' directive from line 1 to 8
  ******************** FINISHED DEBUGGING 'LLVM :: example.txt' ********************
  
  Testing Time: 72.12s
    Excluded: 47642
    Passed  :     1   

A fundamental component of this patch is that, when lit is passed the `--pdb` command-line option, lit treats RUN, DEFINE, and REDEFINE directives as if they were PYTHON directives with corresponding calls to the `lit` object.  That's why lit claims it's executing a PYTHON directive.  Of course, `--pdb` also works fine for tests that actually use PYTHON directives or `config.prologue`.

(For the above example, I configured the test suite to use lit's internal shell in order to improve the execution trace, but I have found bash works fine here too.)

This patch also introduces `lit.define(name, value)` and `lit.redefine(name, value)` functions in order to support `DEFINE` and `REDEFINE` directives.  Those are likely useful on their own, and I'll probably extract them into a parent patch later and improve them.

This patch is a WIP and is not ready for a detailed review.  First, I still need to extend the documentation and lit's test suite.  Second, I wonder if the patch should be generalized to work with other python debuggers besides pdb, but I haven't given that idea enough thought yet.  I am sharing this patch early to demonstrate another benefit of extending lit with PYTHON directives, as proposed in D154987 <https://reviews.llvm.org/D154987>.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D158954

Files:
  llvm/utils/lit/lit/LitConfig.py
  llvm/utils/lit/lit/TestRunner.py
  llvm/utils/lit/lit/cl_arguments.py
  llvm/utils/lit/lit/display.py
  llvm/utils/lit/lit/main.py
  llvm/utils/lit/lit/run.py
  llvm/utils/lit/lit/worker.py

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D158954.553796.patch
Type: text/x-patch
Size: 22530 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20230827/0fa04c95/attachment.bin>