Skip to content

CLI: torn run-state.json silently resumes chats with no agent context (model amnesiac, transcript intact) #1168

Description

@nordicnode

Follow-up from #1166 (the compaction wipes). While auditing that report I traced the persistence side and found a second, independent path to the same symptom — "the model lost all chat history" — with a different cause. Verified against the current code with executable repros.

The failure chain

  1. The atomic write is not durable. cli/src/utils/write-file-atomic.ts writes a temp file and renames it over run-state.json — but never fsyncs. The rename is atomic against concurrent processes, yet after a power loss / hard hang / battery pull, the filesystem can land the rename while the file's data blocks were never written. Result: a truncated or empty run-state.json exactly where the atomic rename was supposed to guarantee a complete file. (The in-code comment acknowledges torn non-atomic writes; the same tear reaches through this pattern without the fsync.)

  2. The load path has no recovery. loadMostRecentChatState (cli/src/utils/run-state-storage.ts) parses run-state.json and, on failure, falls back to a placeholder:

    runState ??= { output: { type: 'error', message: 'Previous run state could not be restored.' } } as RunState

    The placeholder has no sessionState. Meanwhile chat-messages.json is parsed independently and usually survives — so the UI transcript is fully intact.

  3. A missing sessionState means a fresh session. sdk/src/run.ts:436-467: when previousRun.sessionState is undefined, runOnce builds a brand-new initialSessionState. The next turn's model context = empty history.

Net effect: the user resumes a chat, sees their whole transcript, sends a message — and the assistant has no idea who they are or what it did. That is precisely the "lost all chat history instead of compacting" report shape, on machines that crashed or lost power mid-session. Reproduced with a temp-dir test: a truncated primary + intact transcript yields a placeholder RunState without sessionState.

Two aggravating details found on the way:

  • Recoverable state is thrown away. writeFileAtomic leaves run-state.json.<pid>.<uuid>.tmp behind when the process dies between write and rename — that temp is usually a complete generation of the state. The loader ignores it.
  • The loss is silent in the UI. Nothing tells the user the assistant's memory is gone; they just experience the model being broken.

Suggested fix (all verified in my PR)

  1. fsync the temp file before the rename in both writeFileAtomic and writeFileAtomicAsync (rename becomes durable and its data with it).
  2. Recovery order on unreadable primary: rotated run-state.json.bak (previous complete generation, rotated aside by each synchronous save) → newest complete checkpoint .tmp → give up. Self-heal the primary from whichever recovered.
  3. Surface the loss: loadMostRecentChatState reports whether agent context survived, and the resume flow prepends an error-variant notice ("agent context could not be restored; the assistant starts without memory of earlier turns; transcript is intact") instead of failing silently.

PR to follow.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions