Skip to content

fix(plugin): close dirty-closed rescan gaps: cursor starvation, cancelled-drain settlement, mid-drain re-emission - #2637

Open
syzsunshine219 wants to merge 2 commits into
mainfrom
codex/maintainer-edit-pr-2588-20261119
Open

syzsunshine219 wants to merge 2 commits into
mainfrom
codex/maintainer-edit-pr-2588-20261119

Conversation

@syzsunshine219

Copy link
Copy Markdown
Collaborator

The source branch for the change proposed in #2588 now lives in MemTensor/MemOS, so repository writers can contribute fixes through the normal upstream branch permissions.

The code and commit authors are unchanged. This PR retains target branch main and exact source head c70019cf99b938f1b66726f03b6d7dafff7ecbe4 from #2588.

Previous discussion and review remain in #2588. Review and checks must be re-evaluated for this replacement PR; no previous approval or new test result is asserted by this migration.

Previous PR description and implementation details

Restores the change proposed in #2482 for review against main after its former base branch, dev-v2.0.36, was deleted. The source commits and authors are preserved.

Source head: c70019cf99b938f1b66726f03b6d7dafff7ecbe4

This PR is open for review. Review and CI against the current base are pending; validation claims in the original description below refer to the earlier snapshot. The original discussion and evidence remain in #2482.


Original change description

Description

Fixes #2459 — two reliability gaps in the memos-local-plugin dirty-closed rescan, plus one follow-up guard found in review. All changes are in apps/memos-local-plugin/core/pipeline/memory-core.ts with new unit tests; both commits are based on dev-v2.0.36 (8511502).

Root cause

  • Gap 1 (cursor starvation): collectDirtyClosedEpisodes() paged only downward via listClosedPage({ before: cursor }) and only reset the cursor on a short page, so an episode that became dirty after the cursor paged past its position was never revisited until the walk reached the end of the closed set and the cursor reset — a full lap of ceil(closed/500) × 10 min (~70 min on the reporter's DB, hours on the 30k+-episode DBs of v2.0.2+ regression: bootstrapMemoryCoreFull() hangs with 100% CPU on databases >500MB #1787).
  • Gap 2 (interrupted drain lost the whole batch's phase-2): recoverDirtyClosedEpisodes() returned right after await handle.flush() when startupRecoveryCancelled was set, skipping settlement for every queued episode: successes kept their rewardDirty markers (re-collected and re-run later — duplicate LLM spend) and failures lost their failedAttempts backoff counters.

Fix

  • Gap 1: every scan first takes one bounded newest-first head page (before: null) merged into an id-deduplicated collected map, then continues the cursor walk (core/pipeline/memory-core.ts:1788-1831). Cursor advancement still derives from the walk page only (core/pipeline/memory-core.ts:1823-1831), so head-page rows do not disturb cursor semantics; freshly closed / freshly re-marked rows are now seen on the very next tick.
  • Gap 2: phase-2 settlement moved into try { flush } finally with one try/catch per episode (core/pipeline/memory-core.ts:1700-1775) — cancel or a flush throw mid-teardown still settles each row (clear rewardDirty on success, bump failedAttempts + lastFailureAt on failure); a per-row error logs init.dirty_closed_episodes.settle_failed and cannot drop the rest of the batch.
  • Review follow-up (second commit): recoverDirtyClosedEpisodes records each emitted episode id in a process-local in-flight set before emitting and clears it as soon as that row settles; both auto-collection sites (init classification core/pipeline/memory-core.ts:811-825 and the 10-min periodic rescan core/pipeline/memory-core.ts:1215-1227) filter in-flight ids, so a tick racing an open drain can no longer re-collect and re-emit the whole batch (duplicate LLM spend). Freshly dirtied rows are not blocked — per-episode guard, not a whole-scan mutex.
  • init.orphan_scan.failed raised from debug to warn (core/pipeline/memory-core.ts:1219-1226): a skipped startup classification used to be invisible at default log levels.

AI assistance disclosure: this change was produced with AI assistance in an automated contributor workflow; every command and test result below was executed locally as recorded.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

  • Unit Test
  • Test Script Or Test Steps (please provide)

Environment: macOS (arm64), Node + vitest 2.1.9, run from apps/memos-local-plugin.

Reproduce:

cd apps/memos-local-plugin
# targeted issue-#2459 cases (2 new in commit 1, 1 more in commit 2)
npx vitest run tests/unit/pipeline/memory-core.test.ts --reporter=dot -t "issue #2459"
# regression subset
npx vitest run tests/unit/pipeline/memory-core.test.ts tests/unit/startup-recovery.test.ts tests/unit/reward/ --reporter=dot
# full unit / integration / typecheck
npx vitest run tests/unit --reporter=dot
npx vitest run tests/integration --reporter=dot
npx tsc -p tsconfig.json --noEmit

Results (final verification at this branch head):

Note: the ticket's vitest … -q flag does not exist in vitest 2.1.9 (CACError: Unknown option -q), so --reporter=dot was used instead. CI is expected red on this scope for external PRs due to the known environment issue tracked in #2476 (autotest environment preparation fails); local green evidence is provided above.

Checklist

  • I have performed a self-review of my own code | 我已自行检查了自己的代码
  • I have commented my code in hard-to-understand areas | 我已在难以理解的地方对代码进行了注释
  • I have added tests that prove my fix is effective or that my feature works | 我已添加测试以证明我的修复有效或功能正常
  • I have created related documentation issue/PR in MemOS-Docs (if applicable) | 我已在 MemOS-Docs 中创建了相关的文档 issue/PR(如果适用)
  • I have linked the issue to this PR (if applicable) | 我已将 issue 链接到此 PR(如果适用)
  • I have mentioned the person who will review this PR | 我已提及将审查此 PR 的人

Reviewer Checklist

  • closes #xxxx (Replace xxxx with the GitHub issue number)
  • Made sure Checks passed
  • Tests have been provided

…ncelled-drain settlement

Issue #2459 reported two reliability gaps in the dirty-closed rescan:

Gap 1 — cursor-lap starvation. collectDirtyClosedEpisodes() paged only
downward (listClosedPage({ before: cursor })), so an episode that became
dirty after the cursor paged past its position was not revisited until
the walk reached the end of the closed set and the cursor reset — a full
lap of ceil(closed/500) x 10 min (~70 min on the reporter's DB, hours on
the 30k+-episode DBs of #1787). Each scan now takes one bounded
newest-first page from the head (before: null) before continuing the
cursor walk, deduplicated by episode id, so freshly closed episodes and
freshly re-marked recent rows are seen on the very next tick while the
cursor still drains the older backlog a page at a time.

Gap 2 — interrupted drains lost phase-2 accounting for the whole batch.
recoverDirtyClosedEpisodes() returned right after await handle.flush()
when startupRecoveryCancelled was set, skipping settlement for every
queued episode: successes kept their rewardDirty markers (re-collected
and re-run later — duplicate LLM spend) and failures lost their
failedAttempts backoff counters. Settlement is synchronous bookkeeping
on rows whose chains already ran, so it now runs even when cancelled and
even if the flush throws mid-teardown, best-effort with one try/catch
per row (init.dirty_closed_episodes.settle_failed).

Also raises init.orphan_scan.failed from debug to warn: a skipped
startup classification used to be invisible at default log levels (the
reporter only found it by diffing kv cursor writes).

Base: dev-v2.0.36. Fixes #2459
Signed-off-by: akushonkamen <akushonkamen@163.com>
…-drain

Review finding on #2459: the head-page pass from gap 1 re-reads the
newest page on every 10-minute tick, and episodes whose drain is still
in flight keep their rewardDirty markers until phase-2 settles — so a
tick racing an open drain re-collected and re-emitted the whole batch,
re-running the full reward chain per episode every tick (~5-6 duplicate
runs per episode for the 300+-episode / 1h+-drain scenario the issue
describes; neither the 30s scan dedup nor the capture/reward
subscribers dedupe by episodeId).

Guard: recoverDirtyClosedEpisodes records each emitted episode id in a
closure-level in-flight set before emitting and clears it as soon as
that row's settlement runs (success, backoff bump, or un-settleable);
both auto collection sites (init classification + periodic rescan)
filter in-flight ids out. Per-episode, not a whole-scan mutex, so
freshly dirtied rows are still collected on the very same tick while a
large drain is open. The set is process-local: after a hard kill the
markers on disk still drive a normal rescan on reboot.

Base: dev-v2.0.36. Refs #2459
Signed-off-by: akushonkamen <akushonkamen@163.com>
@syzsunshine219 syzsunshine219 added the area:plugin OpenClaw & Hermes label Nov 19, 2026
@Memtensor-AI Memtensor-AI added the status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 label Nov 19, 2026
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2637
Task: a29cfeafed0e33e3
Base: main
Head: codex/maintainer-edit-pr-2588-20261119
Head SHA: c70019cf99b938f1b66726f03b6d7dafff7ecbe4

🔍 OpenCodeReview found 4 issue(s) in this PR.

⚠️ Review warnings (1)

  • bounded analysis finalized with explicit task_done after reviewing all group files

1. apps/memos-local-plugin/core/pipeline/memory-core.ts (L1711)

The in-flight marker is added here, but the try { await handle.flush(); } finally { … delete … } block only starts after this loop. Any throw between the add and the try — updateMeta() (sync SQLite write), snapshotFromRecoveredEpisode() (trace DB reads), or handle.buses.session.emit() (listener exceptions) — skips the finally, leaving those ids in dirtyRescoreInFlight for the rest of the process lifetime. Both autoRescoreDirtyClosedEpisodes() and the init scan filter on this set, so the affected episodes are silently excluded from every future rescore (their rewardDirty marker stays, nothing ever retries them) until the process restarts. Suggest moving the try { above the emit loop so the settlement finally also covers loop failures — priorFailedAttempts already tracks exactly the rows that were marked, so the settlement loop stays correct. Additionally, consider re-checking dirtyRescoreInFlight right before emit() (callers pass pre-collected lists filtered at collection time, so a periodic drain that starts between collection and emit can still double-emit the same row).


2. apps/memos-local-plugin/core/pipeline/memory-core.ts (L1810-L1813)

page is reassigned to the cursor page, so the following if (page.length === 0) { … return []; } returns an empty array even though collected already holds the head page (the newest rows, including freshly dirtied episodes this fix targets). On the tick where the cursor walk reaches the end, the gap-1 fix is silently neutralized: the newest dirty rows are dropped for that tick, and the stale-marker cleanup loop below is skipped for them too. Suggest keying the early return on collected.size === 0 (or merging the head rows into the result and only resetting the cursor when the cursor page is empty), e.g. advance the cursor / delete it based on the cursor page, but still return [...collected.values()].


3. apps/memos-local-plugin/core/pipeline/memory-core.ts (L1751-L1752)

Running settlement on cancellation mis-charges never-attempted rows. startupRecoveryCancelled is set when the shutdown grace expires (and handle.shutdown is then called with flushGraceMs: 0), so the await handle.flush() in the try above can abort before the reward chains for the emitted rows actually ran. Those rows are still dirty, so they land in this stillDirty branch and get failedAttempts: prevAttempts + 1 + lastFailureAt: now despite never being attempted. Repeated crash/restart cycles can accumulate these false increments past MAX_DIRTY_REWARD_ATTEMPTS and push the rows into the 1h→24h backoff, delaying a retry that never failed. Suggest gating the increment on !startupRecoveryCancelled (still clear the marker for rows that are no longer dirty, and always clear dirtyRescoreInFlight).


4. apps/memos-local-plugin/core/pipeline/memory-core.ts (L1709-L1711)

The in-flight guard is only consulted at collection time (autoRescoreDirtyClosedEpisodes / init()), but the drain can start long after its batch was collected, and this loop never re-checks the set before emitting. Concretely: init() collects dirtyClosedForBackground synchronously, then startupRecoveryPromise first awaits recoverOpenEpisodesAsSessionEnd(stale) (one LLM chain per orphan, which can take well over 10 minutes on the 500-orphan / 30k-episode DBs from #1787). When the periodic timer fires in that window, those dirty rows are still not in-flight, so the periodic drain emits them and marks them in-flight — and afterwards the delayed background drain emits the very same rows again (this loop only skips isLightweightEpisode, not in-flight ids), reproducing the duplicate reward/L2/L3 spend this PR exists to eliminate. Add an in-flight check inside the emit loop so the batch is filtered at drain time as well:

const episodeId = ep.id as EpisodeId;
if (dirtyRescoreInFlight.has(episodeId)) continue;
💡 Suggested Change

Before:

      // Mark in-flight BEFORE emitting so a periodic tick racing this
      // drain cannot queue a duplicate run for the same row.
      dirtyRescoreInFlight.add(episodeId);

After:

      const episodeId = ep.id as EpisodeId;
      // Collection-time filtering is not enough: the init batch can sit
      // queued behind the (slow) open-episode recovery while the periodic
      // scan drains the same rows — re-check at emit time.
      if (dirtyRescoreInFlight.has(episodeId)) continue;
      // Mark in-flight BEFORE emitting so a periodic tick racing this
      // drain cannot queue a duplicate run for the same row.
      dirtyRescoreInFlight.add(episodeId);

Generated by cloud-assistant via Open Code Review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:plugin OpenClaw & Hermes status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] dirty-closed rescan misses freshly-marked episodes for a full cursor lap; interrupted drains lose phase-2 accounting

5 participants