Clear AI messages abandoned in processing - #5071
Open
elias-ba wants to merge 1 commit into
Open
Conversation
Security Review ✅
|
elias-ba
force-pushed
the
timeout-failure-taxonomy
branch
from
August 15, 2026 22:50
3683332 to
5a28e60
Compare
elias-ba
force-pushed
the
timeout-stuck-message-reaper
branch
from
August 15, 2026 22:50
01753c3 to
55042f3
Compare
processing_started_at has been written on every message since it started processing and read by nothing. This reads it. A message is left processing forever when its job dies without emitting telemetry. Attaching Oban's stop event covers most of that, but not the case it cannot reach: when a job outlives the drain window, Oban stops the producer first and kills the task after, so there is nothing left to report it. Until something clears the row the panel stays locked for everyone in that session, and nothing raises. Two guards decide what to reap, because neither is sound alone. Age is vulnerable to clock skew and to a queue that was paused. A live Oban job for the same message is the cross-node authority. The live set is read after the candidates, never before - a job starting in between then appears in it and is skipped, where the other order could reap a message whose job had only just begun. The write is a guarded update rather than a read-modify-write, so a job that finishes between the select and the write keeps its own result. The grace period sits well clear of the longest a run can legitimately take, so a slow answer is never mistaken for an abandoned one, and a partial index keeps the every-five-minutes scan off the table.
elias-ba
force-pushed
the
timeout-failure-taxonomy
branch
from
August 15, 2026 23:40
5a28e60 to
1813938
Compare
elias-ba
force-pushed
the
timeout-stuck-message-reaper
branch
from
August 15, 2026 23:40
55042f3 to
3abf115
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Even with the lifecycle fix, a message can still be stranded: the node goes away mid-answer, or a job dies in a way nothing gets to observe. The row stays
:processingand the panel spins until someone reloads.A cron worker sweeps them. It reads the candidate messages first and the live Oban job ids second, so a job that starts between the two reads is treated as alive rather than reaped. The update is guarded on
status == :processing, so a message that finished in the gap is left alone.There's a partial index on the processing rows, since that's the only slice the sweep looks at.
Closes #__
Validation steps
mix test test/lightning/ai_assistant/stuck_message_reaper_test.exs:processingwith an oldprocessing_started_atand no Oban job, runLightning.AiAssistant.StuckMessageReaper.perform(%Oban.Job{}), and check it moves to:errorand the session gets told.AI Usage
Pre-submission checklist