Commit b4d9d41
committed
fix(scheduler): revive dead brain-init executions in ~6m instead of 30m
The window before db-scheduler reclaims an execution whose owner died is
heartbeat-interval × missed-heartbeats-limit. At 5m × 6 that was thirty minutes.
During it a dead brain-init stage is invisible rather than failed: the
init-status endpoint keeps returning the last stage and percentage it reached,
with completedAt null and errorMessage null, so a frozen run is indistinguishable
from a slow one. smoke-test.sh waits 1200s (20m) — less than the revival window —
so any run in which a stage died failed the smoke test even though initialization
would have finished normally once revived. Observed exactly that: a stage stopped
heartbeating, the smoke test timed out at 74%, and brain init then completed at
100% roughly 37 minutes after the scheduler revived it.
1m × 6 keeps the same six-missed-heartbeat tolerance and brings revival to about
six minutes, comfortably inside the smoke-test window. Shortening the interval
does not risk reclaiming healthy work: heartbeats are sent from the execution's
own thread, so a long-running LLM call keeps heartbeating and is never mistaken
for a dead owner.
Only the heartbeat interval changes. application-test.properties overrides
polling-interval and immediate-execution-enabled, not these keys, so tests are
unaffected.1 parent ac233ee commit b4d9d41
1 file changed
Lines changed: 13 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
71 | 71 | | |
72 | 72 | | |
73 | 73 | | |
74 | | - | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
75 | 87 | | |
76 | 88 | | |
77 | 89 | | |
| |||
0 commit comments