Skip to content

fix(rivetkit): refuse actions after onSleep with actor.restarting - #5876

Open
eersnington wants to merge 2 commits into
fix/drain-going-away-close-reasonfrom
fix/rivetkit-restarting-after-on-sleep
Open

eersnington wants to merge 2 commits into
fix/drain-going-away-close-reasonfrom
fix/rivetkit-restarting-after-on-sleep

Conversation

@eersnington

@eersnington eersnington commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Sleep grace accepted new actions for its whole duration. The TypeScript sleep hook runs the app's onSleep, then closes SQLite and clears runtime state such as vars. So an action dispatched during that window ran against an actor that onSleep was tearing down. SleepFinalize refused with actor.stopping, which does not tell sleep from destroy.

Now dispatch stays open only while the sleep hooks still run. After they finish, it refuses with the new actor.restarting error. The action did not run, and the error is retryable.

Lifecycle state Before After
Started run run
SleepGrace, onSleep or connection disconnects running run run
SleepGrace, after those hooks run against a torn-down actor actor.restarting (503), not run
SleepFinalize actor.stopping actor.restarting (503), not run
DestroyGrace actor.stopping actor.stopping

The check reuses the existing core_dispatched_hooks counter, which counts the graceful cleanup and connection disconnects that emit_grace_events starts. No new state. #5860 moves the refusal point to the start of the hooks.

// rivetkit-core/src/actor/task.rs
LifecycleState::Started => None,
LifecycleState::SleepGrace if self.ctx.core_dispatched_hook_count() > 0 => None,
LifecycleState::SleepGrace | LifecycleState::SleepFinalize => {
	self.ctx.warn_work_sent_to_stopping_instance("dispatch");
	Some(ActorLifecycleError::Restarting.build())
}
LifecycleState::DestroyGrace => { /* actor.stopping, unchanged */ }
  • Handle calls do not see this error in practice. The gateway holds requests while the actor stops and sends them to the next instance.
  • A connection action sent in this window usually fails with ws.going_away (fix(rivetkit-core): close drained connections with ws.going_away #5867) instead, because the connection closes before the reply arrives. In both cases the action did not run.
  • rivetkit-rust/packages/rivetkit-core/src/error.rs: adds ActorLifecycle::Restarting (actor.restarting, "Actor is restarting. Retry the request.") with status 503. It matches the TypeScript actorRestarting() error that the client already classifies as retryable.
  • rivetkit-rust/engine/artifacts/errors/actor.restarting.json: generated error artifact.
  • docs-internal/engine/sleep-sequence.md: documents when sleep grace admits work.
  • actor-sleep-db.test.ts: broadcast works in onSleep calls getCounts through a closure. The connection proxy read getCounts.bind as an action name, so the test failed before sleep started.
  • Behavior change: an action that arrives after the sleep hooks finish now runs in the next generation instead of the current one. Keep-awake work that waits for a later action during sleep grace no longer receives it in the same generation.

This is part 4 of 6 in a stack:

@railway-app

railway-app Bot commented Oct 13, 2026

Copy link
Copy Markdown

This PR was not deployed automatically as @eersnington does not have access to the Railway project.

In order to get automatic PR deploys, please add @eersnington to your workspace on Railway.

@claude

claude Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Review

The change is small, focused and matches the PR description. The lifecycle gating reads correctly, and the error follows the existing RivetError pattern. The artifact sits alongside the other actor.* artifacts in rivetkit-rust/engine/artifacts/errors.

Things to check

  • Race at grace start. SleepGrace admits work only while core_dispatched_hook_count() > 0. Confirm begin_core_dispatched_hook is incremented in the same task step that sets lifecycle = SleepGrace, before any dispatch is processed. Otherwise an action in that window gets a spurious actor.restarting. It is benign because the client retries, but it blurs the "hooks still run" semantics.
  • Counter semantics. The counter also covers per-connection disconnect hooks. If hooks are started in sequence, the count can hit 0 and rise again, so dispatch would flip between open and closed. Please confirm all hooks are started up front.
  • Behavior change. Keep-awake work that waits for a later action during grace no longer gets it in the same generation. A client with retries disabled, or a non-TS client, will see a 503 for an action that used to succeed. It is worth a changelog or docs note.
  • Doc inconsistency. sleep-sequence.md says actor/restarting, but the code uses actor.restarting.
  • Docs generation. Per CLAUDE.md, a new RivetError requires pnpm docs:gen-api so docs/api/error-registry.json is updated. I did not see that file in the diff, so CI docs:check-api may fail.
  • Tests. wait_for_grace_hooks_done is a loop { check; sleep(10ms) } poll. It mirrors the existing wait_for_state helper, but CLAUDE.md discourages polling. Awaiting the counter via wait_zero would be deterministic. The two key cases are covered: dispatch accepted while cleanup runs, and refused after it finishes. There is no test for the connection-disconnect-hook case.
  • TS test fix. The getCounts closure fix in actor-sleep-db.test.ts is reasonable.

Overall this looks good, pending the docs-gen check and the race confirmation.

@eersnington
eersnington force-pushed the fix/rivetkit-restarting-after-on-sleep branch from 638369f to bfd367a Compare October 16, 2026 23:44

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant