Pausing on raised exceptions also stops inside pygments since IPython 9.17 #1596

Description

With the raised exception filter on, the debugger also stops in the code that IPython runs to show a traceback. For json.loads("x"), the expected stop in the cell is followed by 244 stops in pygments, executing and asttokens. A JupyterLab user has to press Continue for each of them. This happens with ipykernel 7.2.0 or later and IPython 9.17.0 or later. With IPython 9.16.1, the same cell stops once.

Reproduce

import collections

from jupyter_client.manager import start_new_kernel

km, kc = start_new_kernel(kernel_name="python3")
seq = 0


def debug(command, arguments=None):
    global seq
    seq += 1
    msg = kc.session.msg(
        "debug_request",
        {"type": "request", "seq": seq, "command": command, "arguments": arguments or {}},
    )
    kc.control_channel.send(msg)
    while True:
        reply = kc.control_channel.get_msg(timeout=30)
        if reply["parent_header"]["msg_id"] == msg["header"]["msg_id"]:
            return reply["content"]


debug("initialize", {"adapterID": "python", "pathFormat": "path"})
debug("attach")
debug("setExceptionBreakpoints", {"filters": ["raised"]})
debug("configurationDone")

msg_id = kc.execute('import json\njson.loads("x")')
stops = collections.Counter()
while True:
    msg = kc.get_iopub_msg(timeout=30)
    if msg["msg_type"] == "debug_event" and msg["content"]["event"] == "stopped":
        thread = msg["content"]["body"]["threadId"]
        frame = debug("stackTrace", {"threadId": thread})["body"]["stackFrames"][0]
        path = frame["source"]["path"].split("site-packages/")[-1]
        stops[f"{frame['name']} ({path}:{frame['line']})"] += 1
        debug("continue", {"threadId": thread})
    elif msg["parent_header"].get("msg_id") == msg_id and msg["content"].get("execution_state") == "idle":
        break

for where, count in stops.items():
    print(f"{count:3d} x {where}")
kc.stop_channels()
km.shutdown_kernel(now=True)

With IPython 9.18.1, the script prints:

  1 x <module> (/tmp/ipykernel_2100496/2406433274.py:2)
  1 x <module> (asttokens/util.py:203)
  2 x _for_filename_and_lines (executing/executing.py:204)
  1 x verify (executing/_position_node_finder.py:1127)
  1 x __init__ (executing/_position_node_finder.py:228)
  1 x executing (executing/executing.py:260)
  1 x <module> (pygments/formatters/html.py:23)
231 x get_tokendefs (pygments/lexer.py:630)
  6 x get_tokens_unprocessed (pygments/lexer.py:751)

With IPython 9.16.1, it prints:

  1 x <module> (/tmp/ipykernel_2101655/2406433274.py:2)

Cause

Since #1481, the debugger can step into libraries, because debug_just_my_code is False by default. To keep the debugger out of its own code, the kernel tells debugpy to skip the files of the modules that are loaded when it starts. It takes that list before it creates the shell:

Since ipython/ipython#15333, IPython imports stack_data, executing, asttokens and the pygments lexers when it formats the first traceback, so their files are not in the list. These packages raise and catch exceptions in normal use. For example, each time stack_data highlights a frame, it creates a new lexer class, and pygments raises and catches a ValueError 33 times while it sets up that class.

CC @Carreau since this is in part a fallout from ipython/ipython#15333, though a fix may belong in ipykernel.

Expected behavior

• The raised filter does not stop in the code that the kernel runs to show a traceback, also when that code is imported after the kernel starts.
• It still stops where the user's code, or a library that it calls, raises an exception.

Context

• ipykernel 7.4.0, 7.2.0 and main (6.29.5 is not affected)
• IPython 9.17.0, 9.17.1 and 9.18.1 (9.16.1 is not affected)
• debugpy 1.8.22, pygments 2.21.0, stack_data 0.6.3, Python 3.12
• Ubuntu 26.04, and Ubuntu 24.04 on GitHub Actions

Found while updating the JupyterLab documentation screenshots to IPython 9.18 (jupyterlab/jupyterlab#20316), with help of Opus 5.5.