Skip to content

Fix #2063: algorithm.lightweightMemory.enabled: true does not actually skip evolution pipel - #2074

Merged
syzsunshine219 merged 3 commits into
MemTensor:dev-v2.0.23from
Memtensor-AI:feature/autodev-2063-20260708064514092
Jul 9, 2026
Merged

syzsunshine219 merged 3 commits into
MemTensor:dev-v2.0.23from
Memtensor-AI:feature/autodev-2063-20260708064514092

Conversation

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

Description

Fixed issue #2063 — algorithm.lightweightMemory.enabled: true now actually skips the evolution pipeline in @memtensor/memos-local-plugin. Two independent root causes were addressed. First, buildPipelineSubscribers in apps/memos-local-plugin/core/pipeline/deps.ts used to attach reward / L2 / L3 / skill / feedback subscribers unconditionally, so every episode.finalized event still cascaded through the LLM-heavy chain — the fix short-circuits the subscriber wiring and returns no-op stubs whose drain/stop/detach/dispose are safe to call from flush() / shutdown(), and whose runner methods throw with a "lightweight mode" message if reached (every real caller is already gated on the flag). Second, memory-core.ts startup + periodic recovery paths (init, autoFinalizeStaleTasks, autoRescoreDirtyClosedEpisodes) used to re-emit episode.finalized for legacy episodes lacking meta.lightweightMemory === true, producing the >19k skill_generate / >4.7k world_model_generate / >2.1k policy_evolve backlog reported in the issue — those paths now close orphan / stale / dirty-closed episodes silently when the flag is on and tag their meta so subsequent restarts stay quiet. Regression coverage added at tests/unit/pipeline/lightweight-mode.test.ts: three tests exercising normal-mode listener presence, lightweight-mode listener absence on capture / reward / L2 buses, and a synthetic episode.finalized recovery emission that must produce zero reward / L2 / L3 / skill events. All 50 existing pipeline unit tests + the openclaw full-chain integration test still pass; TypeScript strict typecheck (tsc --noEmit) is clean. Three pre-existing test failures (startup-recovery, storage/migrator, v7-full-chain e2e) reproduce on the pristine base branch and are out of scope.

Related Issue (Required): Fixes #2063

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactor (does not change functionality, e.g. code style improvements, linting)
  • Documentation update

How Has This Been Tested?

Automated tests are pending.

  • Unit Test
  • Test Script Or Test Steps (please provide)
  • Pipeline Automated API Test (please provide)

Checklist

  • I have performed a self-review of my own code
  • I have commented my code in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • I have created related documentation issue/PR in MemOS-Docs (if applicable)
  • I have linked the issue to this PR (if applicable)
  • I have mentioned the person who will review this PR

@hijzy please review this PR.

Reviewer Checklist

…eightMemory is enabled (MemTensor#2063)

`buildPipelineSubscribers` used to attach reward / L2 / L3 / skill /
feedback subscribers unconditionally, so `algorithm.lightweightMemory.enabled: true`
only affected `flush()` — every `episode.finalized` still cascaded
through the LLM-heavy evolution chain.

Startup + periodic recovery compounded the bug: `recoverOpenEpisodesAsSessionEnd`
and `recoverDirtyClosedEpisodes` re-emitted `episode.finalized` for
legacy episodes that lack `meta.lightweightMemory === true`, producing
a large backlog of `skill_generate` / `world_model_generate` /
`policy_evolve` calls after every bridge restart.

- Gate subscriber attachment in `buildPipelineSubscribers` and return
  no-op stubs for the runner/handle shape callers depend on.
- Treat every orphan / stale / dirty-closed episode as lightweight in
  the recovery paths of `memory-core.init`, `autoFinalizeStaleTasks`,
  and `autoRescoreDirtyClosedEpisodes` when the flag is on.
- Regression coverage in `tests/unit/pipeline/lightweight-mode.test.ts`.
@Memtensor-AI

Copy link
Copy Markdown
Collaborator Author

🤖 Open Code Review

Target: PR #2074
Task: 00ce0051970d8558
Base: dev-v2.0.23
Head: feature/autodev-2063-20260708064514092
Head SHA: c80d351658e953ea1c5b37ff44083b15889619c6

🔍 OpenCodeReview found 5 issue(s) in this PR.

⚠️ 1 warning(s) occurred during review.


1. apps/memos-local-plugin/core/pipeline/memory-core.ts (L630-L632)

recoveredAtStartup: nowMs is written here during a periodic stale scan (called every 30 seconds via autoFinalizeStaleTasks), not during startup. This field name falsely implies the episode was handled at process startup, which will mislead any observability or debugging tooling that reads meta.recoveredAtStartup to distinguish startup recovery from periodic maintenance. Consider using a different key (e.g., closedAtMs) or omitting the field altogether from the periodic path.

💡 Suggested Change

Before:

            closeReason: "finalized",
              recoveredAtStartup: nowMs,
              recoveryReason: "lightweight_stale_topic_close",

After:

            closeReason: "lightweight_stale",
              closedAtMs: nowMs,
              recoveryReason: "lightweight_stale_topic_close",

2. apps/memos-local-plugin/core/pipeline/memory-core.ts (L626-L628)

close() and updateMeta() are not atomic. If close() succeeds but updateMeta() throws, the episode is closed without meta.lightweightMemory: true being set. Because isLightweightEpisode() checks only row.meta?.lightweightMemory === true, a future non-lightweight scan path (e.g., autoRescoreDirtyClosedEpisodes) could pick up this episode and feed it into the reward/L2/L3 pipeline — exactly the regression this change is meant to prevent. Consider reversing the order: write the meta flag first, then close; or combine the fields into a single updateMeta call before closing.

💡 Suggested Change

Before:

          try {
            handle.repos.episodes.close(ep.id as EpisodeId, nowMs, ep.rTask ?? undefined);
            handle.repos.episodes.updateMeta(ep.id as EpisodeId, {

After:

          try {
            // Write the lightweight flag first so isLightweightEpisode()
            // returns true even if close() subsequently throws.
            handle.repos.episodes.updateMeta(ep.id as EpisodeId, {
              lightweightMemory: true,
              closeReason: "lightweight_stale",
              closedAtMs: nowMs,
              recoveryReason: "lightweight_stale_topic_close",
            });
            handle.repos.episodes.close(ep.id as EpisodeId, nowMs, ep.rTask ?? undefined);

3. apps/memos-local-plugin/core/pipeline/memory-core.ts (L979-L981)

This is a nested ternary expression — the outer lightweightMode && !isLightweightEpisode(ep) is itself a compound boolean inside a ternary condition. The project checklist prohibits nested ternaries. Additionally, isLightweightEpisode(ep) is called redundantly here for every episode already included in treatAsLightweight: when lightweightMode is false, treatAsLightweight was already filtered to only lightweight episodes, so !isLightweightEpisode(ep) is always false in that branch (making "lightweight_startup_close_backfill" unreachable outside lightweight mode). Simplify with an explicit if/else.

💡 Suggested Change

Before:

            recoveryReason: lightweightMode && !isLightweightEpisode(ep)
              ? "lightweight_startup_close_backfill"
              : "lightweight_startup_close",

After:

            recoveryReason: (lightweightMode && !isLightweightEpisode(ep))
              ? "lightweight_startup_close_backfill"
              : "lightweight_startup_close",

4. apps/memos-local-plugin/core/pipeline/deps.ts (L469-L471)

Missing return before throwLightweight(...) in this async method (and in lightweightL2Handle().runOnce() below). While TypeScript accepts this because never is assignable to any type, the pattern is inconsistent with every other throwing stub in this file (all of which use return throwLightweight(...)). Without return, TypeScript's control-flow analysis does NOT treat the code after the call as unreachable, so a future developer adding a second statement after the throwLightweight call would not get a type-error warning. Prefer the consistent return throwLightweight(...) pattern throughout.

💡 Suggested Change

Before:

    async runManually() {
      throwLightweight("rewardSubscription.runManually");
    },

After:

    async runManually() {
      return throwLightweight("rewardSubscription.runManually");
    },

5. apps/memos-local-plugin/core/pipeline/deps.ts (L503-L505)

Same issue as runManually above: missing return before throwLightweight(...). All other throwing stubs in this file use return throwLightweight(...). Without return, TypeScript does not mark subsequent statements as unreachable, making it easier to accidentally add dead code after this call in the future.

💡 Suggested Change

Before:

    async runOnce() {
      throwLightweight("l2.runOnce");
    },

After:

    async runOnce() {
      return throwLightweight("l2.runOnce");
    },

🧹 Filtered 3 low-confidence OCR finding(s) before posting/fix-loop (duplicate: 3).

Generated by cloud-assistant via Open Code Review.

…ight mode

- deps.ts: return throwLightweight(...) in rewardSubscription.runManually and
  l2.runOnce so control-flow analysis marks subsequent statements unreachable,
  matching the pattern used by every other throwing stub in the file.
- memory-core.ts (autoFinalizeStaleTasks lightweight path): write the
  lightweight meta BEFORE close() so a failing close() cannot leave an
  episode without meta.lightweightMemory=true (which would let the periodic
  non-lightweight rescan pick it up). Also rename the meta fields to
  closeReason="lightweight_stale" + closedAtMs so observability tooling does
  not mistake a periodic close for startup recovery.
- memory-core.ts (init lightweight orphan close): replace the nested ternary
  for recoveryReason with an explicit if/else.
@Memtensor-AI

Copy link
Copy Markdown
Collaborator Author

🤖 Open Code Review

Target: PR #2074
Task: 00ce0051970d8558
Base: dev-v2.0.23
Head: feature/autodev-2063-20260708064514092
Head SHA: 98d3bfff9b543f7d42c1244eb73573c84d17c0f8

🔍 OpenCodeReview found 3 issue(s) in this PR.


1. apps/memos-local-plugin/core/pipeline/memory-core.ts (L985-L991)

Bug: close() must be called AFTER updateMeta(), not before.

The periodic stale-scan path (around line 632) has an explicit comment explaining that updateMeta() is intentionally called before close() so that if close() throws, isLightweightEpisode() still returns true (since it only checks row.meta?.lightweightMemory === true) and downstream rescans correctly skip the episode.

Here the order is reversed. If close() succeeds but updateMeta() throws (or vice-versa), the episode ends up with its status set to 'closed' but without lightweightMemory: true in its metadata. On the next dirty-closed rescan (autoRescoreDirtyClosedEpisodes), this episode passes !isLightweightEpisode(ep) and gets fed into the reward/L2/L3 pipeline — exactly the cascade this change is trying to prevent.

Fix: swap the order to match the periodic path:

handle.repos.episodes.updateMeta(ep.id as EpisodeId, {
  lightweightMemory: true,
  closeReason: lightweightMode && !isLightweightEpisode(ep) ? "lightweight_backfill" : "finalized",
  recoveredAtStartup: nowMs,
  recoveryReason,
});
handle.repos.episodes.close(ep.id as EpisodeId, nowMs, ep.rTask ?? undefined);

2. apps/memos-local-plugin/core/pipeline/memory-core.ts (L978-L992)

No per-episode error isolation: Unlike the periodic stale-scan loop (which wraps each episode in its own try/catch), this startup loop has no per-episode error handling. If close() or updateMeta() throws for any single episode (e.g., DB lock, constraint violation), the entire loop is aborted by the outer try/catch and logged as init.orphan_scan.failed — leaving all remaining episodes unprocessed with no per-episode error context.

This matters particularly at startup where the number of legacy orphan rows can be large. Consider adding a per-episode try/catch matching the periodic scan pattern:

for (const ep of treatAsLightweight) {
  try {
    // updateMeta first, then close (see ordering note)
    handle.repos.episodes.updateMeta(...);
    handle.repos.episodes.close(...);
  } catch (err) {
    log.debug("init.lightweight_close_error", {
      episodeId: ep.id,
      err: err instanceof Error ? err.message : String(err),
    });
  }
}

3. apps/memos-local-plugin/core/pipeline/memory-core.ts (L988-L990)

Misleading closeReason for backfill episodes. When lightweightMode=true and the episode did NOT previously have lightweightMemory: true (i.e., it is a legacy row being backfilled), closeReason is set to "finalized" even though the episode was never actually finalized through the normal evolution pipeline. The recoveryReason correctly distinguishes between "lightweight_startup_close" and "lightweight_startup_close_backfill", but closeReason: "finalized" is semantically incorrect for the backfill case and could mislead future analytics or debugging that filters on closeReason.

Consider using a distinct value such as "lightweight_backfill" when recoveryReason === "lightweight_startup_close_backfill".

Generated by cloud-assistant via Open Code Review.

…htweight close

Three issues in the init() lightweight-orphan-close loop:

1. Ordering: close() was called BEFORE updateMeta(). If close() succeeded
   but updateMeta() threw, the episode ended up closed without
   meta.lightweightMemory=true, so autoRescoreDirtyClosedEpisodes would
   later pick it up and feed it into the reward/L2/L3 pipeline. Now the
   meta write happens first — matching the periodic stale-topic path.

2. Error isolation: the loop had no per-episode try/catch, so a single
   DB lock / constraint violation would abort the whole loop and be
   swallowed by the outer scan-level try/catch. Now each episode has its
   own try/catch that emits init.lightweight_close_error at debug.

3. Backfill semantics: legacy rows being backfilled were tagged
   closeReason="finalized" even though they were never actually
   finalized. They now get closeReason="lightweight_backfill", matching
   the existing recoveryReason="lightweight_startup_close_backfill"
   distinction so downstream analytics can tell the two apart.
@Memtensor-AI

Copy link
Copy Markdown
Collaborator Author

🤖 Open Code Review

Target: PR #2074
Task: 00ce0051970d8558
Base: dev-v2.0.23
Head: feature/autodev-2063-20260708064514092
Head SHA: b9d4a305a2e132f25988016aa2210bd6ec4cea7d

🔍 OpenCodeReview found 3 issue(s) in this PR.


1. apps/memos-local-plugin/core/pipeline/memory-core.ts (L1028)

Using Array.includes() inside .filter() produces an O(n²) scan. normalOrphans and stale can each hold up to 500 entries (the repo list limit), so this can execute up to 250,000 element comparisons at startup. Use a Set keyed on episode id for O(n) performance.

💡 Suggested Change

Before:

          const recent = normalOrphans.filter((ep) => !stale.includes(ep));

After:

          const staleIds = new Set(stale.map((ep) => ep.id));
          const recent = normalOrphans.filter((ep) => !staleIds.has(ep.id));

2. apps/memos-local-plugin/core/pipeline/memory-core.ts (L980-L985)

Both branches of this if/else are exhaustive and recoveryReason is never reassigned after them. Declare it as const with a ternary to make immutability explicit and keep style consistent with the rest of the codebase.

💡 Suggested Change

Before:

          let recoveryReason: string;
          if (isBackfill) {
            recoveryReason = "lightweight_startup_close_backfill";
          } else {
            recoveryReason = "lightweight_startup_close";
          }

After:

          const recoveryReason = isBackfill
            ? "lightweight_startup_close_backfill"
            : "lightweight_startup_close";

3. apps/memos-local-plugin/core/pipeline/memory-core.ts (L1039-L1042)

The lightweightMemory.enabled flag was already captured as lightweightMode on line 974 for the orphan-close block above. Accessing handle.algorithm.lightweightMemory.enabled again here directly (rather than reusing lightweightMode) is inconsistent and means the flag is read twice from the handle instead of once. If the value could ever mutate between the two reads the branches would be inconsistent; and even when it cannot, the inconsistency reduces readability. Reuse the already-captured lightweightMode local.

💡 Suggested Change

Before:

      if (handle.algorithm.lightweightMemory.enabled) {
        // Same story for dirty-closed episodes — never rescore them
        // when the pipeline is intentionally light.
        dirtyClosedForBackground = [];

After:

      if (lightweightMode) {
        // Same story for dirty-closed episodes — never rescore them
        // when the pipeline is intentionally light.
        dirtyClosedForBackground = [];

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator Author

⚠️ Open Code Review automatic OCR fix limit reached

Open Code Review still found 3 issue(s), but the automatic OCR fix loop has reached 2/2 attempts.

I stopped auto-fixing this PR to avoid an infinite loop. Please review the latest OCR comment and fix or dismiss the remaining findings manually.

PR: #2074

@Memtensor-AI Memtensor-AI added area:plugin OpenClaw & Hermes area:memory 记忆存储、检索、更新、召回逻辑 labels Jul 8, 2026
@Memtensor-AI Memtensor-AI added the status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 label Jul 8, 2026
@Memtensor-AI

Copy link
Copy Markdown
Collaborator Author

✅ Automated Test Results: PASSED

All tests passed (36/36 executed). memos_local_plugin/unit: 36/36. Duration: 11s [advisory, non-gating] AI-generated tests on branch test/auto-gen-00ce0051970d8558-20260708232552: 76/83 passed, 7 failed — these do NOT affect the PR verdict; review the branch manually.

Branch: feature/autodev-2063-20260708064514092

@Memtensor-AI Memtensor-AI removed the area:memory 记忆存储、检索、更新、召回逻辑 label Jul 9, 2026
@syzsunshine219
syzsunshine219 merged commit de03ab2 into MemTensor:dev-v2.0.23 Jul 9, 2026
16 checks passed
steven-stack-s pushed a commit to steven-stack-s/dsh-memos-local that referenced this pull request Sep 18, 2026
…t actually skip evolution pipel (MemTensor#2074)

* fix(memos-local-plugin): actually skip evolution pipeline when lightweightMemory is enabled (MemTensor#2063)

`buildPipelineSubscribers` used to attach reward / L2 / L3 / skill /
feedback subscribers unconditionally, so `algorithm.lightweightMemory.enabled: true`
only affected `flush()` — every `episode.finalized` still cascaded
through the LLM-heavy evolution chain.

Startup + periodic recovery compounded the bug: `recoverOpenEpisodesAsSessionEnd`
and `recoverDirtyClosedEpisodes` re-emitted `episode.finalized` for
legacy episodes that lack `meta.lightweightMemory === true`, producing
a large backlog of `skill_generate` / `world_model_generate` /
`policy_evolve` calls after every bridge restart.

- Gate subscriber attachment in `buildPipelineSubscribers` and return
  no-op stubs for the runner/handle shape callers depend on.
- Treat every orphan / stale / dirty-closed episode as lightweight in
  the recovery paths of `memory-core.init`, `autoFinalizeStaleTasks`,
  and `autoRescoreDirtyClosedEpisodes` when the flag is on.
- Regression coverage in `tests/unit/pipeline/lightweight-mode.test.ts`.

* fix(memos-local-plugin): address open code review findings on lightweight mode

- deps.ts: return throwLightweight(...) in rewardSubscription.runManually and
  l2.runOnce so control-flow analysis marks subsequent statements unreachable,
  matching the pattern used by every other throwing stub in the file.
- memory-core.ts (autoFinalizeStaleTasks lightweight path): write the
  lightweight meta BEFORE close() so a failing close() cannot leave an
  episode without meta.lightweightMemory=true (which would let the periodic
  non-lightweight rescan pick it up). Also rename the meta fields to
  closeReason="lightweight_stale" + closedAtMs so observability tooling does
  not mistake a periodic close for startup recovery.
- memory-core.ts (init lightweight orphan close): replace the nested ternary
  for recoveryReason with an explicit if/else.

* fix(memos-local-plugin): address round-2 open code review on init lightweight close

Three issues in the init() lightweight-orphan-close loop:

1. Ordering: close() was called BEFORE updateMeta(). If close() succeeded
   but updateMeta() threw, the episode ended up closed without
   meta.lightweightMemory=true, so autoRescoreDirtyClosedEpisodes would
   later pick it up and feed it into the reward/L2/L3 pipeline. Now the
   meta write happens first — matching the periodic stale-topic path.

2. Error isolation: the loop had no per-episode try/catch, so a single
   DB lock / constraint violation would abort the whole loop and be
   swallowed by the outer scan-level try/catch. Now each episode has its
   own try/catch that emits init.lightweight_close_error at debug.

3. Backfill semantics: legacy rows being backfilled were tagged
   closeReason="finalized" even though they were never actually
   finalized. They now get closeReason="lightweight_backfill", matching
   the existing recoveryReason="lightweight_startup_close_backfill"
   distinction so downstream analytics can tell the two apart.

---------

Co-authored-by: autodev <autodev@memtensor.local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:generated Generated or modified by AI | 由 AI 生成或修改 area:plugin OpenClaw & Hermes status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants