Skip to content

fix(plugin): honor explicit storage list limits instead of clamping to 500 - #2535

Open
linjicong wants to merge 3 commits into
MemTensor:mainfrom
linjicong:fix/plugin-honor-explicit-list-limits
Open

linjicong wants to merge 3 commits into
MemTensor:mainfrom
linjicong:fix/plugin-honor-explicit-list-limits

Conversation

@linjicong

Copy link
Copy Markdown

Description

clampLimit() in apps/memos-local-plugin/core/storage/repos/_helpers.ts rewrote every explicitly requested page size to the 500-row default page:

export function clampLimit(n: number): number {
  if (!Number.isFinite(n) || n <= 0) return 500;
  return Math.min(Math.trunc(n), 500);   // ← explicit 10_000 became 500
}

Bulk callers therefore silently lost rows:

  • metrics() (memory-core.ts) reads traces.list({ limit: 10_000 }) to bucket the viewer's "memory writes per day" chart — it only ever saw the newest 500 traces. On a real production DB (9,362 traces over 52 days, ~250/day) the chart showed data for exactly the last two days (253 + 247 = 500) and zero-filled the rest, while the sessions card reported 14 instead of 163 and embeddings was similarly undercounted.
  • World-model maintenance (worldModel.list({ limit: 100_000 })) received 500 rows.

Fix: explicit positive limits are now honored up to a MAX_QUERY_LIMIT ceiling of 100,000 (the largest limit any call site asks for today). The default page for callers that omit limit stays at 500 — no existing call site changes behavior, and unbounded scans remain impossible. No new dependencies.

Relation to #2464 (cc @akushonkamen): same root cause, different contract decision. #2464 additionally changes omitted-limit to mean "all rows" (no LIMIT clause), which fixes more of #2401 but re-opens unbounded-scan risk on very large databases (the #2076 4.2 GB / 518k-row report). This PR is intentionally the minimal, zero-behavior-change subset: it can land independently (or serve as the safe core of that effort) while the omitted-limit semantics are decided separately.

Related Issue (Required): Addresses the explicit-limit half of #2401 (no Fixes — the omitted-limit call sites it also reports are intentionally out of scope here).

Type of change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

cd apps/memos-local-plugin && npm ci && npx vitest run tests/unit

  • Unit Test
    • New tests/unit/storage/explicit-limit.test.ts: clampLimit honors explicit limits up to MAX_QUERY_LIMIT, invalid input keeps the 500 default, and repo.list({ limit: 10_000 }) over 600 seeded rows (3 distinct days) returns all 600 — including the oldest rows a day-bucketing caller needs — while repo.list() keeps the 500-row default page.
    • New regression in tests/unit/pipeline/memory-core.test.ts ("MemoryCore metrics sees beyond the 500-row page"): 520 traces "now" + 100 traces 3 days earlier; metrics({ days: 7 }) must report the older day bucket as 100 and sessions as 2. Fails on the old code with exactly the shipped symptom (expected +0 to be 100) and passes with this change.
    • Updated the stale assertion in traces-count.test.ts that enshrined the old clamp (list({ limit: 10000 }) → 500) to the new contract (→ 600).
  • Test Script Or Test Steps (please provide)
    • npm run lint (tsc --noEmit): clean.
    • Full vitest run tests/unit on this branch: 9 pre-existing failing files, byte-identical to the same run on pristine main (config/paths, install scripts, server http/sse — Windows-environment failures; see also the CI "ENV ISSUE" already reported on fix(plugin): stop clamping storage list queries to 500 rows #2464). This change adds zero new failures and 3 new passing tests.

Checklist

…o 500

clampLimit rewrote every explicitly requested page size to the 500-row
default, so bulk callers silently lost rows: metrics() asked for 10_000
traces to bucket the viewer's "memory writes per day" chart but only
ever saw the newest 500 (~2 days at typical volume), sessions/embeddings
undercounted, and world-model maintenance asking for 100_000 rows got 500.

Explicit positive limits are now honored up to a MAX_QUERY_LIMIT ceiling
of 100_000. The default page for callers that omit limit stays at 500,
so no existing call site changes behavior.

Addresses part of MemTensor#2401. Regression tests in
tests/unit/storage/explicit-limit.test.ts and memory-core.test.ts fail on
the old code and pass with this change.
@Memtensor-AI Memtensor-AI added area:plugin OpenClaw & Hermes status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Oct 22, 2026
@Memtensor-AI

Memtensor-AI commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2535
Task: 6424883cdfd0c4fc
Base: main
Head: fix/plugin-honor-explicit-list-limits
Head SHA: 5aef4001d0d6aed60b7d9f4f1a33618f69d91973

🔍 OpenCodeReview found 4 issue(s) in this PR.

⚠️ Review warnings (3)

  • bounded analysis finalized with explicit task_done after reviewing all group files
  • bounded analysis finalized with explicit task_done after reviewing all group files
  • Filtered 1 low-confidence OCR finding(s) before posting/fix-loop.

1. apps/memos-local-plugin/server/routes/params.ts (L21-L23)

Fractional values between 0 and 1 slip through: parsed <= 0 is checked before truncation, so ?limit=0.5 passes the guard, gets truncated to 0, and the helper returns 0 — violating its contract of returning a safe page size. Downstream this is observable: policies.ts/policies.ts (world-models) compute nextOffset via policies.length === limit, which is never true for limit === 0 (pagination silently stops); session.ts's filtered path does rows.slice(offset, offset + 0) (empty page); and storage's clampLimit(0) falls back to 500 rows while the response reports limit: 0. Truncate before validating, and also clamp a caller-supplied fallback so it can never exceed MAX_HTTP_PAGE_LIMIT.

💡 Suggested Change

Before:

  const parsed = Number(raw);
  if (!Number.isFinite(parsed) || parsed <= 0) return fallback;
  return Math.min(Math.trunc(parsed), MAX_HTTP_PAGE_LIMIT);

After:

  const parsed = Math.trunc(Number(raw));
  if (!Number.isFinite(parsed) || parsed <= 0) {
    return Math.min(fallback, MAX_HTTP_PAGE_LIMIT);
  }
  return Math.min(parsed, MAX_HTTP_PAGE_LIMIT);

2. apps/memos-local-plugin/core/storage/repos/_helpers.ts (L76)

Raising the single repo-level ceiling from 500 to 100_000 removes the safety net for every caller, not just the five routes updated in this PR. Some remaining paths compose the scan window from caller-supplied offset:

  • server/routes/skill.ts (GET /api/v1/skills): limit: pageSize + offset + 1 with offset read straight from the query string and never capped → ?limit=500&offset=99500 now makes the repo load ~100k skill rows, including the vec BLOB and invocation_guide columns (previously ≤500 rows).
  • core/pipeline/memory-core.ts listPolicies / listWorldModels: limit: limit + offset + ... with offset taken from bridge/methods.ts RPC params.

Since clampLimit is the only shared guard, either cap offset/limit + offset before it reaches the repo (as parseLimitParam does for HTTP), or keep a separate bounded variant for paged/offset-based reads so the trusted bulk-read window can't be driven by untrusted input.


3. apps/memos-local-plugin/core/storage/repos/_helpers.ts (L62)

The docstring promises bulk callers "must actually receive what they ask for", but clampLimit still silently rewrites anything above 100_000, and there is no signal (log/throw/truncated flag) that it happened. That reintroduces the same failure mode as #2401 one order of magnitude higher: e.g. exportBundle() (memory-core.ts:4352) asks for traces.list({ limit: 100_000 }) and would silently drop data for stores with more than 100k traces (the referenced incident had 518k rows). At minimum, log/emit a warning when a request exceeds MAX_QUERY_LIMIT so truncation is observable, and/or have the known bulk call sites page through a cursor instead of relying on one oversized LIMIT.


4. apps/memos-local-plugin/server/routes/params.ts (L17-L19)

The fallback branch bypasses the cap. The docstring promises a "safe page size", but a caller that passes a fallback above MAX_HTTP_PAGE_LIMIT (e.g. parseLimitParam(raw, FILTER_SCAN_LIMIT) wiring a 5_000 scan window) gets 5_000 returned verbatim, which defeats the very bound this helper exists to enforce. Clamp the fallback as well (and normalise it to a positive integer) so every return path is guaranteed to be within 1..MAX_HTTP_PAGE_LIMIT.

💡 Suggested Change

Before:

export function parseLimitParam(
  raw: string | null,
  fallback = 50,

After:

export function parseLimitParam(
  raw: string | null,
  fallback = 50,

🧹 Filtered 1 low-confidence OCR finding(s) before posting/fix-loop (duplicate: 1).

Generated by cloud-assistant via Open Code Review.

Review follow-up:

- Extract DEFAULT_PAGE_SIZE so the 500 default lives in one place
  (buildPageClauses and clampLimit both referenced a bare literal).
- Bound caller-supplied limit params at the routes. Storage-level
  clampLimit now honors explicit limits up to MAX_QUERY_LIMIT for trusted
  in-process bulk reads, which relaxed the untrusted HTTP surface:
  GET /api/v1/traces (downstream listTraces clamps at 10_000),
  /api/v1/episodes (listEpisodes does not clamp at all),
  /api/v1/policies and /api/v1/world-models could each be asked for up
  to MAX_QUERY_LIMIT rows per request. New server/routes/params.ts
  parseLimitParam() restores the 500-row ceiling these endpoints had
  before this PR, and GET /api/v1/traces' documented "max 500" is now
  actually enforced at the route.
- MAX_QUERY_LIMIT doc now states it is for trusted in-process bulk reads
  only; untrusted input is bounded by MAX_HTTP_PAGE_LIMIT.
@linjicong

Copy link
Copy Markdown
Author

Thanks for the review — both points addressed in 7a0f9b8.

Point 2 (duplicated 500 magic number) — accepted verbatim: extracted DEFAULT_PAGE_SIZE = 500, now referenced by both buildPageClauses and clampLimit.

Point 1 (global 100k ceiling too coarse-grained) — accepted, with one amendment to the premise: I audited every untrusted entry before choosing where to fix it, and the actual exposure was not "any caller can fetch 100k rows" but a specific, smaller set:

untrusted entry downstream effective max after the first commit
GET /api/v1/traces?limit= core.listTraces clamps at 10_000 10,000 (doc says "max 500")
GET /api/v1/episodes?limit= core.listEpisodes — no clamp 100,000
GET /api/v1/policies / world-models?limit= listPolicies clamps at 500 but its internal over-fetch (limit + offset + 200) was previously hard-capped at 500, now bounded by MAX_QUERY_LIMIT amplified internal reads
GET /api/v1/api-logs?limit= core.listApiLogs clamps at 500 500 (unaffected)
GET /api/v1/skills?limit= route-local Math.min(n, 500) 500 (unaffected)

Rather than threading per-caller caps through clampLimit's signature (which would churn every bulk call site — the exact invasiveness this PR tries to avoid), the boundary is now enforced at the boundary: new server/routes/params.ts exposes parseLimitParam(raw, fallback) + MAX_HTTP_PAGE_LIMIT = 500, and all four exposed routes use it. Every untrusted page is back to its pre-PR 500 ceiling (and GET /api/v1/traces's documented "max 500" is enforced for the first time), while trusted in-process bulk readers (metrics(), world-model maintenance) keep the MAX_QUERY_LIMIT ceiling. MAX_QUERY_LIMIT's doc comment now states that split explicitly.

Side note from the audit: the policies over-fetch (limit + offset + 200) used to silently under-fill pages once offset exceeded ~300 because the repo fetch was clamped at 500 — honoring explicit limits fixes that latent paging bug as a side effect.

Verification — npm run lint clean; vitest run tests/unit: new tests/unit/server/params.test.ts (8 assertions incl. clamp-at-500 and 1e9 input) plus all prior tests pass; full-suite failure set is still byte-identical to pristine main (the 9 pre-existing Windows-environment failures), zero new failures.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: ENV ISSUE

The test environment encountered an issue that requires manual attention.

Details: Executor error: Node dependency preparation failed for memos_local_plugin: Command failed with exit code 1: npm ci --include=dev --no-audit --no-fund
stdout:
stderr: npm warn deprecated boolean@3.2.0: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.
npm error code 1
npm error path /data/test-workspaces/6424883cdfd0c4fc/repo/apps/memos-local-plugin/node_modules/onnxruntime-node
npm error command failed
npm error command sh -c node ./script/install
npm error /data/test-workspaces/6424883cdfd0c4fc/repo/apps/memos-local-plugin/node_modules/onnxruntime-node/script/install-utils.js:57
npm error reject(new Error(Failed to download build list. HTTP status code = ${statusCode}));
npm error ^
npm error
npm error Error: Failed to download build list. HTTP status code = 302
npm error at ClientRequest. (/data/test-workspaces/6424883cdfd0c4fc/repo/apps/memos-local-plugin/node_modules/onnxruntime-node/script/install-utils.js:57:18)
npm error at Object.onceWrapper (node:events:634:26)
npm error at ClientRequest.emit (node:events:519:28)
npm error at ClientRequest.emit (node:domain:489:12)
npm error at HTTPParser.parserOnIncomingClient (node:_http_client:780:27)
npm error at HTTPParser.parserOnHeadersComplete (node:_http_common:125:17)
npm error at TLSSocket.socketOnData (node:_http_client:615:22)
npm error at TLSSocket.emit (node:events:519:28)
npm error at TLSSocket.emit (node:domain:489:12)
npm error at addChunk (node:internal/streams/readable:561:12)
npm error
npm error Node.js v22.22.2
npm error A complete log of this run can be found in: /data/test-workspaces/_shared/npm-downloads/_logs/2026-11-11T03_02_05_948Z-debug-0.log
Branch: fix/plugin-honor-explicit-list-limits

Tested source: 5aef4001d0d6aed60b7d9f4f1a33618f69d91973
Tested base: 3a16e5fe3048a209f722806f4123fa2e533588fd
Tested commit: 5aef4001d0d6aed60b7d9f4f1a33618f69d91973
These results apply only to this immutable source/base snapshot.

@Memtensor-AI Memtensor-AI added status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发 and removed status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:plugin OpenClaw & Hermes status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants