Shut down Jupyter kernels when their kernel scope ends (bd-hxhnnlzs)

[cscheid]

Summary

Fixes the Jupyter kernel process leak (bd-hxhnnlzs): every process that executed a Jupyter document — the test suite and q2 render alike — leaked one ipykernel process per (kernel, working dir). 2338 orphans had accumulated on one dev machine, each reparented to PID 1 and holding ~6 listening sockets, plus 4932 stale kernel-*.json connection files.

Root cause: kernel sessions live in a process-global static DAEMON, statics never drop, so the kernel Child's kill_on_drop never fired and nothing ever called shutdown_all() (the idle reaper had zero callers).

Fix — refcounted KernelScope:

• The jupyter engine's execute_qmd holds an inner scope around each engine run, so no caller can leak by accident.
• Outer scopes in q2 render (around pipeline.run(), dropped before any process::exit), q2 preview, and q2 provide-hub (server lifetime) keep kernels warm across documents / re-renders.
• When the last scope drops, all sessions are shut down: shutdown_request on the control channel (when a private runtime is possible), SIGKILL backstop, bounded reap, connection-file removal.

Two adjacent latent bugs surfaced by the new e2e test, both fixed:

• Startup TOCTOU: concurrent renders sharing a session key both spawned kernels (read-check → multi-second spawn → insert; the loser was evicted and killed). Every two-doc project render hit this. Now serialized behind a start_lock with a re-check.
• Cross-call session reuse never actually worked: sessions held ZeroMQ sockets and the kernel Child bound to a per-call throwaway tokio runtime, so genuine reuse failed with "Tokio context … being shutdown" (masked by the TOCTOU race always re-spawning). Sessions now live on a shared long-lived ENGINE_RUNTIME.

Also deletes the never-called idle-timeout machinery and adds panic-safety scopes to the direct-daemon-API tests.

Tests (TDD — written first, verified failing pre-fix)

• quarto-core jupyter_kernel_cleanup.rs: drives record_capture under an isolated JUPYTER_RUNTIME_DIR, records each spawned kernel's ports from its connection file, asserts ports are dead + files removed when it returns.
• quarto jupyter_kernel_cleanup_e2e.rs: drives the real q2 binary on a two-python-doc website; asserts exactly one kernel served both docs (pinning warm reuse) and nothing survives process exit.
• Both skip when jupyter isn't installed, same gating as the existing engine tests.

Verification

• cargo nextest run --workspace: 11273/11273 passed, zero orphaned kernels after the run (previously ~15 per run).
• Full cargo xtask verify (incl. WASM leg) and cargo xtask lint: green.
• Manual end-to-end: cargo run --bin q2 -- render <two-python-doc website> → "Rendered 2 of 2 files", executed outputs present in both HTML files, zero orphan processes, zero stale connection files (output inspected).
• The two #[ignore]d jupyter_integration pipeline tests fail on main with a pre-existing nested-runtime panic — filed as bd-yaccefzk, not a regression from this change.

Plan/findings: claude-notes/plans/2026-08-10-jupyter-kernel-leak.md

🤖 Generated with Claude Code

[posit-snyk-bot]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.