[yul761]

Fixes five limitations Butler (a statecore-mcp@1.0.0 consumer) documented. All five were confirmed against main 724f309.

1. Forgotten notes kept their cap slots

forget only inserted a ForgottenFact row. facts() hid the note, but it stayed in profile.notes, so a full notes facet (cap 30) stayed full and the next note evicted a live one.

• Core: new retireForgottenNotes(state, forgottenKeys, makeNow, pack) removes the forgotten notes' strings from profile.notes and marks their registry entries retiredAt / retiredReason: "forgotten". Nothing is deleted.
• Embedded forget now runs in one transaction that inserts the ForgottenFact row and retires the note in the latest snapshot. The API forgetFact does the same, with the upsert and the snapshot update in one $transaction.
• addNoteFact takes optional forgottenKeys and runs the helper before the dedup and cap checks. Stores written before this fix heal on their next note write. Both embedded remember and API addNote pass the keys.
• alreadyRegistered now requires !e.retiredAt. Before, a note that was cap-evicted and then written again stayed in profile.notes with no active registry entry behind it.

2(a). A full digest did not free forgotten slots

pruneForgottenFacts removed forgotten notes from the pipeline state. At write time carryOverConcurrentNotes saw their ids missing and appended all of them again as if they were concurrent writes, with no cap check.

• carryOverConcurrentNotes(pipeline, latest, { forgottenFactKeys, pack }) skips forgotten notes and retires any forgotten note the pipeline still holds (covers a forget that lands mid-digest). It also copies over retirements and supersessions made concurrently in the latest snapshot, and holds the notes cap: it evicts the oldest digest-produced note first, then the oldest note, retiring each as cap_evicted.
• CreateDigestWithSnapshotInput gains forgottenFactKeys and pack in both apps/mcp and apps/worker (including the worker's rebuild chain). The keys are re-read at write time.
• The forgotten-key read was improved, not just kept. Both the MCP and the worker now read every forgotten key for the prune. Only the verbatim contents fed to the stage-2 prompt stay limited to the latest 100.

3(b). Events before the first note were never pending

The first note created a placeholder Digest stamped "now". Pending means "newer than the latest Digest", so earlier events were never digested (the pipeline exited no_new_events). The placeholder Digest and its snapshot are now stamped min(ingestedAt in scope) − 1ms, or now when the scope has no events. Done in both the embedded and API paths.

4(c). A digest could evict or rewrite a caller's note — production behaviour change

• addNoteFact tags caller notes with origin: "note". Legacy caller notes are recognised by shape: facet notes, confidence 0.9, no origin. I checked that no digest path writes 0.9 (stream evidence is 0.6, documents 0.85).
• applyProfileFactsFromDigest: at the cap, a digest-produced note may evict only the oldest digest-produced note. If every slot holds a caller note, the incoming digest note is dropped and logged as cap_rejected_incoming (heldBy: "caller_notes").
• Caller notes are skipped by the ≥0.6 in-place rewrite (logged as protected_lower_authority when the text differed) and by supersession.
• Facet consolidation never sends caller notes to the model and never retires them. They keep their place in the profile.

Net effect in production: digest-extracted notes can no longer push out notes users wrote explicitly. On a scope whose notes facet is full of caller notes, new digest-extracted notes are dropped, and the selection log records the drop.

5(d). A digest reported "ran" when it had not processed everything

• Core: generateDigestStage2 adds degraded: { reason: "stage2_chunks_failed", errors, failed, total } when chunks were skipped, either because the pass threw or because every reply was unparseable. isEmptyInvalidJsonPass is exported.
• MCP: if stage 2 came back degraded on invalid_json_output with no facts, the run now throws before writing. The result is "failed", and the events stay pending (writing would have moved the watermark past events nobody read). A missing scope or no model provider is also "failed" now; both used to return "ran".
• MCP: when a digest is written but stage 2 skipped chunks, or selection left pending events out or truncated one, digestNow() / statecore-mcp digest return { ran: true, partial: true }. The field is additive. DigestRunOutcome is unchanged. A new internal runDigestAttempt carries partial, and maybeRunDigest still returns the plain outcome. Butler's parseDigestOutcome (a non-strict zod object) reads { ran: true, partial: true } as "ran".
• The worker keeps its existing behaviour of writing degraded digests anyway.

Follow-up round (review findings + a Butler regression)

• Forget, then remember a revision, keeps the chain (regression fix). Butler updates a decision by forgetting the old note and remembering the revised text. Once forget retired the note, revision matching no longer saw it, and superseded came back null. addNoteFact now also matches notes retired as "forgotten":
  • The revision supersedes the forgotten note, so superseded is returned and why shows old → new.
  • The revision takes one slot and the forgotten note takes none.
  • An embedded test mirrors Butler's scenario, and there are core tests too.
• The exact text of a forgotten note is a no-op. It stays hidden, as Butler expects, and no longer evicts a visible note at the cap. A revised text has a different key and links to the old note as described above.
• A partly successful digest is no longer thrown away.
  • A run fails only when stage 2 distilled nothing: the single pass was unparseable, or every chunk failed (isTotalStage2Failure). If only some chunks failed, the digest is written with partial: true.
  • After 3 consecutive total failures for a scope, the degraded digest is written with partial: true so the scope moves on. The count lives in a new MCP-private DigestFailure table (embedded migration 2, like DigestLock) and resets on every write.
• pruneForgottenFacts retires instead of deleting. Forgotten registry entries are marked retiredReason: "forgotten", so why keeps their history. Stage 2 never shows forgotten entries to the model.
• Scope write lock (Postgres). forgetFact, addNote (whose read-modify-write is now inside one transaction) and the worker's digest write each take pg_advisory_xact_lock(hashtext(scopeId)). They read the snapshot, and for the digest write the forgotten keys, while holding the lock. A new integration test of concurrent note writes fails without the lock.
• Legacy consolidated notes are not caller notes. A legacy notes entry at 0.9 that more than one source was merged into is not treated as a caller note, so it can still be consolidated or evicted.

Also: embedded recall factRegistry entries for notes now carry origin: "note" (an additive field). The /v1 output strips it through FactRegistryEntrySchema, so the frozen contract is untouched.

Tests

• core: fact-retire, handoff, stage2-chunking, digest-control, facet-consolidation
• worker: digest-write
• api: add-note, memory-facts.service
• mcp: embedded, digest-now, digest-trigger

No test calls a real model. MCP tests use stub chat models, and the no-provider test mocks createModelProvider and points at a dead port.

Ran locally: core 555, worker 43, statecore-mcp 113, and api 141 (integration included, against a local pgvector test DB). pnpm lint and pnpm build are clean.

Release

A changeset is included: statecore-mcp minor (→ 1.1.0 together with the pending patch changeset), @statecore/core minor, @statecore/worker and @statecore/api patch. changeset version was not run.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	🔄 Running since 2026-10-23T18:39:15.493464Z	98679dd	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.