Skip to content

Commit b4d9d41

Browse files
committed
fix(scheduler): revive dead brain-init executions in ~6m instead of 30m
The window before db-scheduler reclaims an execution whose owner died is heartbeat-interval × missed-heartbeats-limit. At 5m × 6 that was thirty minutes. During it a dead brain-init stage is invisible rather than failed: the init-status endpoint keeps returning the last stage and percentage it reached, with completedAt null and errorMessage null, so a frozen run is indistinguishable from a slow one. smoke-test.sh waits 1200s (20m) — less than the revival window — so any run in which a stage died failed the smoke test even though initialization would have finished normally once revived. Observed exactly that: a stage stopped heartbeating, the smoke test timed out at 74%, and brain init then completed at 100% roughly 37 minutes after the scheduler revived it. 1m × 6 keeps the same six-missed-heartbeat tolerance and brings revival to about six minutes, comfortably inside the smoke-test window. Shortening the interval does not risk reclaiming healthy work: heartbeats are sent from the execution's own thread, so a long-running LLM call keeps heartbeating and is never mistaken for a dead owner. Only the heartbeat interval changes. application-test.properties overrides polling-interval and immediate-execution-enabled, not these keys, so tests are unaffected.
1 parent ac233ee commit b4d9d41

1 file changed

Lines changed: 13 additions & 1 deletion

File tree

backend/src/main/resources/application.properties

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,19 @@ spring.batch.job.enabled=false
7171

7272
# db-scheduler Configuration (distributed task scheduling)
7373
db-scheduler.enabled=true
74-
db-scheduler.heartbeat-interval=5m
74+
# The revival window for an execution whose owner died is
75+
# heartbeat-interval × missed-heartbeats-limit. At the old 5m × 6 that was THIRTY
76+
# MINUTES, during which a dead brain-init stage sits at a frozen percentage with no
77+
# error: the status endpoint keeps reporting the last stage it reached, so it is
78+
# indistinguishable from slow progress. smoke-test.sh gives up after 1200s (20m),
79+
# so any run where a stage died failed the smoke test even though init would have
80+
# completed fine once revived — observed exactly that, init finishing ~37m after
81+
# revival.
82+
#
83+
# 1m × 6 keeps the same six-missed-heartbeat tolerance — a live-but-slow execution
84+
# heartbeats from its own thread, so a long LLM call is never mistaken for death —
85+
# while bringing revival to ~6m, comfortably inside the smoke-test window.
86+
db-scheduler.heartbeat-interval=1m
7587
db-scheduler.missed-heartbeats-limit=6
7688
db-scheduler.polling-interval=10s
7789
db-scheduler.polling-strategy=fetch

0 commit comments

Comments
 (0)