Skip to content

fix(cli): recover agent context when run-state.json is torn - #1169

Open
nordicnode wants to merge 1 commit into
CodebuffAI:mainfrom
nordicnode:oss/runstate-recovery-1166-related
Open

fix(cli): recover agent context when run-state.json is torn#1169
nordicnode wants to merge 1 commit into
CodebuffAI:mainfrom
nordicnode:oss/runstate-recovery-1166-related

Conversation

@nordicnode

@nordicnode nordicnode commented Aug 31, 2026

Copy link
Copy Markdown

Summary

Companion to the compaction wipes in #1166 — this is the persistence-side path to the same "model lost all chat history" symptom (details and repro in #1168). A resumed chat could start amnesiac through a chain with no single point of failure:

  • writeFileAtomic/writeFileAtomicAsync now fsync the temp before renaming. Without it, a power loss could land the rename while the data blocks were never written — a torn run-state.json exactly where the atomic rename was supposed to guarantee a complete file.
  • loadMostRecentChatState recovers instead of giving up. On an unreadable primary it now tries the rotated run-state.json.bak (previous complete generation; each synchronous save rotates the current file aside before overwriting), then the newest complete checkpoint .tmp (left behind when a process dies between write and rename), self-heals the primary from whichever recovered, and only falls back to the context-less placeholder when all three are unreadable.
  • The loss is surfaced, not silent. loadMostRecentChatState returns runStateRestored, and the resume flow prepends an error-variant notice ("the assistant starts this chat without memory of earlier turns; the transcript below is intact") when it is false. Previously the transcript rendered normally and the model just "forgot" — indistinguishable from a broken assistant.
  • The async rename retries briefly on EPERM/EBUSY/EACCES. On Windows the just-closed handle can still be held by AV/indexer scans — a transient the fsync widens the window for. Bounded (4 retries, exponential backoff), non-matching errors rethrow immediately.
  • clearChatState also removes the .bak.

Validation

  • bun test cli/src/utils/__tests__/run-state-storage.test.ts cli/src/utils/__tests__/write-file-atomic.test.ts cli/src/utils/__tests__/chat-meta.test.ts cli/src/utils/__tests__/safe-json.test.ts (75 pass) — 6 new recovery tests: .bak recovery + self-heal, newest-temp recovery, healthy primary flagged restored, all-three-unreadable fallback with the loss flagged, backup rotation on save, backup cleanup
  • The placeholder-without-sessionState → amnesia chain reproduced with a temp-dir test before the fix; the recovery tests fail on the pre-fix loader
  • Full CLI suite: identical failures with and without this change (20 pre-existing env-dependent failures + 12 pre-existing import-chain errors on main; this change adds 6 passing tests, breaks nothing)
  • bunx tsc --noEmit -p cli/tsconfig.json — no new errors (10 pre-existing, all in unrelated test files: missing tar types, react-dom/server types)

Refs #1168

A resumed chat could start amnesiac — the transcript intact, the model
with no memory of earlier turns — through a chain with no single point
of failure:

- writeFileAtomic renamed the temp over the target without fsync, so a
  power loss could land the rename while the data blocks were never
  written: the 'atomic' file contained garbage. fsync the temp before
  renaming (both sync and async paths).
- loadMostRecentChatState tried only the primary and fell back to a
  RunState placeholder with no sessionState. The SDK starts a fresh
  session when previousRun.sessionState is absent, so the next turn
  silently lost every earlier turn. Now the load tries the rotated
  .bak (the previous complete generation) and then the newest complete
  checkpoint temp (a SIGKILL between write and rename leaves one
  behind), self-heals the primary from whichever recovered, and only
  gives up when all three are unreadable.
- The loss used to be invisible to the user: the transcript rendered
  normally and the model just 'forgot'. loadMostRecentChatState now
  reports whether agent context survived, and the resume flow
  prepends an error-variant notice when it did not.

Also: retry the async rename briefly on EPERM/EBUSY/EACCES — on
Windows the just-closed handle can still be held by AV/indexer scans,
which the new fsync widens the window for.

Refs CodebuffAI#1166 (persistence-side companion to the compaction wipes)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant