Skip to content

fix(rivetkit): run onSleep after in-flight work and refuse dispatch once it starts - #5860

Open
eersnington wants to merge 2 commits into
fix/rivetkit-restarting-after-on-sleepfrom
stack/fix-rivetkit-run-onsleep-after-in-flight-work-kttnpoyx
Open

eersnington wants to merge 2 commits into
fix/rivetkit-restarting-after-on-sleepfrom
stack/fix-rivetkit-run-onsleep-after-in-flight-work-kttnpoyx

Conversation

@eersnington

@eersnington eersnington commented Oct 11, 2026 •

Copy link
Copy Markdown
Member

Run onSleep after in-flight work finishes, and refuse new work once onSleep starts. Before this change:

  • onSleep ran at the same time as in-flight actions.
  • Connections closed when sleep started, so a client lost the result of an action that still ran.
  • Every sleep of an actor with a database logged failed to sync scheduled actor alarm, because TypeScript closed SQLite before core synced the alarm. registry.shutdown() hits the same error.

Fixes #5554

Before

sequenceDiagram
	participant C as Client
	participant T as ActorTask (core)
	participant TS as TypeScript runtime
	participant A as In-flight action
	T->>TS: Stop(Sleep): onSleep and disconnects start at once
	Note over TS,A: onSleep runs while the action still runs
	T--xC: connection closed, the action result is lost
	A->>TS: action finishes
	TS->>TS: close SQLite
	T->>T: sync_alarm fails (SQLite already closed)
Loading

After

sequenceDiagram
	participant C as Client
	participant T as ActorTask (core)
	participant TS as TypeScript runtime
	participant A as In-flight action
	T->>T: Stop(Sleep): fire abort signal
	A->>C: action result
	Note over T: wait until in-flight work drains, or 3/4 of the grace period
	T->>TS: onSleep + disconnects
	C->>T: new action
	T-->>C: actor.restarting (503), not run
	T->>T: sync_alarm, then close SQLite
Loading

Changes

  • Order: ActorTask runs the sleep hooks when the shutdown counter (actions, keepAwake, waitUntil, registered tasks) and WebSocket callbacks reach zero, or at the hooks deadline. Raw onRequest and run are not waited for. Destroy runs its hooks at once, as before.

  • Reserve: SLEEP_HOOK_RESERVE_DIVISOR = 4 keeps the last quarter of the grace period for onSleep, disconnects, and the final save. With the 15 second default this is 3.75 seconds.

  • Refusal point: fix(rivetkit): refuse actions after onSleep with actor.restarting #5876 refuses dispatch with actor.restarting after the sleep hooks finish. This PR refuses it once the hooks start, because the hooks now run after in-flight work drains. Before the hooks run, dispatch is still accepted. DestroyGrace keeps actor.stopping.

    // rivetkit-core/src/actor/task.rs
    LifecycleState::SleepGrace if self.pending_hooks_deadline().is_some() => None,
  • SQLite close moves to core: On the drained path, cleanupNativeSleepRuntimeState no longer closes SQLite. Core runs sync_alarm and then closes it, so the alarm sync no longer fails. If work did not drain by the deadline, TypeScript still closes SQLite early so late work cannot write after shutdown.

  • try_finish_grace requires the hooks to have run, so an idle actor cannot finalize without onSleep.

  • docs-internal/engine/sleep-sequence.md describes the new order.

  • docs/actors/content/docs/lifecycle.mdx describes the new shutdown sequence for users: which work onSleep waits for, and what happens to actions that arrive after it starts.

Behavior change

An action that waits for something onSleep does, such as a flag that onSleep sets, now waits until the hooks deadline. Long actions should watch c.abortSignal, which still fires when sleep starts. No onSleep in the docs or examples uses this pattern (going to follow up on this)


This is part 5 of 6 in a stack:

@railway-app

railway-app Bot commented Oct 11, 2026

Copy link
Copy Markdown

This PR was not deployed automatically as @eersnington does not have access to the Railway project.

In order to get automatic PR deploys, please add @eersnington to your workspace on Railway.

@claude

claude Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Review: run onSleep after in-flight work, refuse dispatch once hooks start

The design is sound. Splitting grace into Pending { deadline } and Emitted is a clean state model, and the hook reserve is bounded and scales with the configured grace period. The try_finish_grace guard closes the "idle actor finalizes without onSleep" hole. The tests use start_paused plus an ordered log, which makes them deterministic, and include a reserve-timing assertion.

Things to check

  1. Double SQLite close on the non-drained path. cleanupNativeSleepRuntimeState still closes SQLite early when work did not drain, and core now also closes SQLite in final cleanup. Please confirm the second close is idempotent and does not log errors.
  2. Destroy and crash paths. The TS-side close was removed from the drained path. Confirm every path reaching that function has a core-side SQLite close afterwards (destroy, errored stop). Otherwise handles leak. The PR description only covers sleep.
  3. Late admission extends the wait. While Pending, dispatch is still accepted, so a steady stream of new actions can keep the shutdown counter above zero until the hooks deadline. It is bounded, but consider whether the docs should say that actions arriving after sleep begins also delay onSleep.
  4. Polling in a test. The while ctx.shutdown_task_count() != 0 { yield_now().await } loop in tests/task.rs is a busy-poll. The timeout bounds it, but CLAUDE.md prefers event-driven waits.
  5. Docs claims to verify. lifecycle.mdx says an action sent through a connection while it closes fails with ws.going_away, and that handle actions are held and run on the next instance. The core change returns actor.restarting (503) and relies on client retry. Please check the wording matches actual behavior, since ws.going_away is not covered by this diff.
  6. Behavior change. An action waiting on something onSleep sets now stalls until the hooks deadline (3.75s by default). Please link the promised follow-up.

Nits

  • The new sleep_grace_tick(hooks_deadline) select arm is duplicated in the main and drain loops. The drain-loop arm does not call try_finish_grace after emitting while the main-loop arm does. If intentional, a short comment would help.
  • Coverage: no test for DestroyGrace running hooks immediately, or for work finishing exactly at the reserve boundary.

No security concerns. Looks good to merge once items 1, 2 and 5 are confirmed.

@eersnington
eersnington force-pushed the stack/fix-rivetkit-run-onsleep-after-in-flight-work-kttnpoyx branch from bc8a66d to 7b01c80 Compare October 16, 2026 23:44

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RivetKit action can hit closed SQLite coordinator immediately after local engine restart

1 participant