Skip to content

Stop losing AI messages when a deploy interrupts them - #5069

Open
elias-ba wants to merge 1 commit into
mainfrom
timeout-oban-lifecycle
Open

Stop losing AI messages when a deploy interrupts them#5069
elias-ba wants to merge 1 commit into
mainfrom
timeout-oban-lifecycle

Conversation

@elias-ba

@elias-ba elias-ba commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Description

Oban's [:oban, :job, :stop] event had no handler attached, so a job that finished without raising was never seen by ObanManager. An AI chat message whose job was killed mid-run stayed :processing forever, and the panel stayed spinning.

Both events are now attached under their own handler ids. That separation matters: :telemetry detaches a handler from every event in an attach_many the first time it raises, so sharing one id meant a single bad :exception payload could silently take :stop down with it.

The Oban shutdown grace period also went up to 6 minutes, which is longer than the longest an AI job can now run. A deploy that lands mid-answer waits for the answer instead of severing it. There's a startup warning if the two ever drift back out of order.

Closes #__

Validation steps

  1. Start the app and check both events have a handler:
    :telemetry.list_handlers([:oban, :job, :stop])
    :telemetry.list_handlers([:oban, :job, :exception])
    Each should include Lightning.ObanManager.handle_event/4.
  2. mix test test/lightning/oban_manager_test.exs

AI Usage

  • I have used Claude Code

Pre-submission checklist

  • I have performed an AI review of my code
  • I have updated the changelog. (for the whole stack, in #5074)
  • I have ticked a box in "AI usage" in this PR

@github-project-automation github-project-automation Bot moved this to New Issues in Core Aug 15, 2026
@github-actions

Copy link
Copy Markdown

Security Review ✅

  • S0 (project scoping): N/A — telemetry handlers look up ChatMessage by a message_id from trusted Oban.Job.args set at enqueue time, not from a spoofable HTTP/LiveView param, and no new project-scoped query surface is introduced.
  • S1 (authorization): N/A — no new controllers, LiveView events, or CRUD actions; changes are limited to telemetry attachment, Oban shutdown_grace_period, and internal Logger-level tweaks.
  • S2 (audit trail): N/A — the only mutation is an internal ChatMessage status transition to :error for interrupted AI messages, which is not a project/instance configuration resource in the audit scope.

@elias-ba
elias-ba force-pushed the timeout-oban-lifecycle branch 2 times, most recently from f742165 to 4cff9e9 Compare August 15, 2026 23:39
@codecov

codecov Bot commented Aug 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.50000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 90.6%. Comparing base (44aeb4a) to head (fe286cd).

Files with missing lines Patch % Lines
lib/lightning/ai_assistant/message_processor.ex 87.5% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##            main   #5069     +/-   ##
=======================================
- Coverage   90.6%   90.6%   -0.0%     
=======================================
  Files        421     421             
  Lines      20012   20014      +2     
=======================================
- Hits       18137   18134      -3     
- Misses      1875    1880      +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

The handler for Oban's :stop event was written but never attached, so a
job cancelled mid-flight told nobody. Its message stayed :processing
forever, the panel stayed locked for everyone in that session, and no
error was raised. Attaching it needed three other things to be true.

It gets its own handler id rather than joining the exception one.
:telemetry drops a handler from every event it registered for the first
time it raises, so sharing an id would let one bad :stop take Oban
exception reporting down with it until the next restart.

The lookups become get rather than get!, because :stop fires for every
successful job in every queue - the busiest path we have - and a message
deleted while its job ran would raise there and trigger exactly that.
Both paths are now covered by a test that deletes the message first,
since returning at all is what proves the clause is there.

The two :circuit events go: Oban has not emitted them since 2.6.

The drain window moves from two minutes to six. It was shorter than an
AI job's own ceiling, so a deploy landing on a running job killed it
after Oban had already stopped the producer that would have reported it -
no telemetry at all, which is the one case attaching :stop cannot rescue.
Application start now warns if that inverts again.

Two log lines drop to warning. Both sat next to a Sentry capture
deliberately set to warning, so the error level was raising a second and
louder event for something the code had already judged not to be a fault.

The new test emits the event instead of calling the handler directly, the
way the others do. That is why this was covered and still broken.
@elias-ba
elias-ba force-pushed the timeout-oban-lifecycle branch from 4cff9e9 to fe286cd Compare August 17, 2026 04:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: New Issues

Development

Successfully merging this pull request may close these issues.

1 participant