Repository navigation
Description
Summary
I observed a high-volume paid LLM request spike that appears to correlate with MemOS Local Plugin running under Hermes. The strongest signal is that local api_logs contain a very large number of system_model_status records, close to the paid provider's request count for the same billing window.
This looks like a background model-status / health-check / retry loop may be sending real paid LLM requests instead of a non-billable local/provider availability check.
No API keys, account identifiers, full prompts, or private logs are included here.
Environment
- Project:
memos-local-plugin - Host: Hermes
- Runtime data path:
~/.hermes/memos-plugin/ - Provider: DeepSeek
- Model:
deepseek-v4-flash - Date observed: 2026-06-08
Evidence
Provider billing export for the dedicated MemOS key:
2026-06-08 UTC
model: deepseek-v4-flash
request_count: 11,344
cost: approx 19.89 CNY
Local MemOS database/API log summary:
api_logs total: 14,632
system_model_status: 12,900
system_error: 811
skill_generate: 80
world_model_generate: 40
policy_evolve: 85
memory_add: 231
local log coverage: roughly 2026-06-07 12:00 to 2026-06-08 11:41
The suspicious part is system_model_status = 12,900, which is very close to the provider-side request_count = 11,344 for deepseek-v4-flash.
During the abnormal window, system_error also increased rapidly, with LLM-role errors recurring every few seconds. This suggests the plugin may continue retrying after provider errors such as insufficient balance, rate limit, or transient provider failures.
Why this seems problematic
A system/model status check should not repeatedly call a paid chat/completions endpoint by default. If system_model_status is implemented as a real LLM generation request, it can silently consume paid API quota in the background.
This issue may be related to #1620, where LLM/embedding fetchers retry 429 responses without respecting Retry-After, potentially increasing failed requests and wasting paid API calls. In this case, the visible symptom is even more severe because the high-volume tool is system_model_status.
Expected behavior
system_model_statusshould not perform paid LLM generation by default.- Health checks should use non-generative checks where possible, such as config validation, lightweight endpoint reachability, or model-list/balance-style provider APIs if available.
- Paid LLM calls should have per-tool rate limits.
- Background jobs should have daily/hourly budget caps.
- 402 / insufficient balance / unauthorized / invalid key should immediately open a circuit breaker instead of retrying.
- 429 should respect
Retry-After. - Repeated provider failures should exponentially back off and eventually disable the provider until user action.
- The viewer/logs should clearly distinguish non-billable status checks from billable LLM calls.
Suggested safeguards
- Add a circuit breaker for provider errors:
402 / insufficient balance: disable provider immediately
401 / invalid key: disable provider immediately
429: respect Retry-After, then back off
5xx/network errors: bounded retries only
- Add a hard throttle for
system_model_status, for example:
max 1 status check per 10-30 minutes
never run status checks in a tight loop
never retry status checks every few seconds
- Add per-tool billing guardrails:
system_model_status: disabled or non-billable by default
memory_add / policy_evolve / world_model_generate / skill_generate: bounded per hour/day
- Make paid background calls visible in the UI/logs:
tool_name, provider, model, billable=true/false, retry_count, backoff_ms, circuit_breaker_state
Current workaround
I disabled the MemOS plugin / removed the dedicated DeepSeek key to stop further spend. I can provide redacted query outputs or sampled system_model_status rows if needed, but I do not want to post complete private prompts or credentials publicly.
Activity
Memtensor-AI commentedon Jun 8, 2026
🤖 AutoDev has picked up this issue and started working on it.
Task ID:
a995d3f2c525e73dWorking branch:
bugfix/autodev-1897Target branch:
mainWorkflow: opsp (analysis → coding → testing → PR)
I will post the PR link here once done. If I need more information, I will ask in the comments.
Fix #1897: fix(memos-local-plugin): add LLM circuit breaker for termi…
Memtensor-AI commentedon Jun 8, 2026
✅ AutoDev task
a995d3f2c525e73dcompleted.Summary: Fixes #1897: paid LLM request storm on Hermes when the configured provider returns terminal errors (402 insufficient balance / 401 invalid key / 403). Issue reporter saw ~12,900 paid DeepSeek requests in 24 h tracking exactly the local
system_model_statusaudit-row count.Root cause: the
system_model_statusrow name is misleading. It is not a health probe; it is the audit row written once per LLM call insideapps/memos-local-plugin/core/llm/client.ts::callWithFallback. With no circuit breaker, every pipeline subscriber (capture / session-relation / reward / L2 / L3 / skill / retrieval LLM filter / world-model) kept firing on every turn, induction, and closed episode, generating one paid request each even though the provider was permanently broken until the operator fixed billing.Fix: per-
LlmClientcircuit breaker that trips on terminal provider errors (HTTP 401/402/403 or messages containing "insufficient balance" / "invalid api key" / "unauthorized" / "account suspended" / "billing"). When open, calls short-circuit inside the facade — zero paid requests, no furthersystem_model_status="error"rows. Half-open after a 5-minute cool-down (configurable, min 30 s); the next call probes the provider and either closes the breaker on success or re-opens it on terminal failure. Host fallback rescues a call without tripping the breaker. A coalescedsystem_model_status="circuit_open"row is emitted at most once per ~25 s so the Logs viewer still surfaces the suppression event without replacing paid spam with audit-row spam. The breaker is enabled by default; legacy behaviour is available viacircuitBreaker.enabled = false.Tests: 9 new vitest cases under
apps/memos-local-plugin/tests/unit/llm/client.test.tscovering trip on 402, trip on "insufficient balance" message, no trip on generic transient, coalescing under load, half-open close on success, host-fallback rescues without trip, legacy disabled mode, stats exposure, and re-open on terminal probe failure. All 59 LLM and 28 pipeline tests pass;tsc --noEmitclean. Three full-suite failures (storage migrator / traces-count / e2e episode merge) are pre-existing on the base branch dev-20260604-v2.0.19 and unrelated to this change.Out of scope (tracked separately): 429
Retry-Afterhandling (issue #1620), per-tool rate limits, daily budget caps.Base branch:
mainBranch:
bugfix/autodev-1897Commit:
d538031PR: #1899
Assigned to: @MatthewZhuang, @CarltonXiang, @syzsunshine219
Reviewers: @MatthewZhuang, @CarltonXiang, @syzsunshine219
TianchunGu commentedon Jun 17, 2026
Additional reporter-side finding, shared in a sanitized form: #1899 / #1938 address the terminal-provider-error retry storm, but there appears to be a second request-amplification path that can still produce many paid LLM calls even when the provider is healthy.
The second path is startup recovery of dirty closed episodes.
High-level shape observed from a local aggregate DB audit, without sharing raw prompts, tool payloads, keys, timestamps, or local episode IDs:
system_model_statusaudit rows, reaching the>10kscale by itself.startedAtandendedAtintool_calls_json.reflectionand the episode has nor_task, startup recovery treats it as dirty and replays reflect/reward work.Observed code path:
core/pipeline/memory-core.ts::recoverDirtyClosedEpisodes()emitsepisode.finalizedfor dirty closed episodes.core/capture/subscriber.tscallsrunner.runReflect(...).core/capture/capture.ts::runReflect()reconstructs normalized steps and matches them against existing traces bystepIdentitySignature()/traceIdentitySignature().ts. During recovered snapshot replay,extractSteps()may assign shifted unique timestamps, so semantically identical tool calls no longer match their existing trace rows.reflect.orphan_stepsand then passed throughrunSummarize(..., { phase: "reflect" }), which invokes the configured LLM for each orphan before inserting more rows.runEpisodeBatchScoring()runs windowedcapture.reflection.batch.v10calls across the recovered episode, with retry/degrade behavior that can further amplify requests.A local read-only replay of the current matching logic showed that the replay orphans were overwhelmingly tool-call steps and were matchable if timestamp/timing was ignored. So this looks like a recovery/signature stability issue rather than genuinely missing trace content.
Suggested follow-up fixes:
reflect.orphan_steps; prefer existing summaries or heuristic summaries.startedAt/endedAt, especially in recovered snapshots.alphaScoring/synthReflectionsare no-ops under the windowed binary pipeline.I can split this into a separate issue if maintainers prefer, but it seems closely related to #1897 because it explains how local recovery/replay can create
system_model_statusrows at the same scale as the reported bill spike, independent of external key theft.4 remaining items
Dev 20260604 v2.0.19 (#2033)
Fix #1523: [Bug] timed_with_status decorator silently swallows except…
feat: chunk batch reflection scoring (#1957)
fix(adapter): fire session.close in daemon thread to unblock event lo…
Fix #1897 recovery replay request storm (#1939)
fix: remove 500-row cap that truncated viewer count displays (#1954)
feat: chunk batch reflection scoring (MemTensor#1957)
feat: chunk batch reflection scoring (MemTensor#1957)
feat: chunk batch reflection scoring (MemTensor#1957)
release: merge dev-v2.0.25 into main (#2156)
Dev 20260604 v2.0.19 (MemTensor#2033)
feat: chunk batch reflection scoring (MemTensor#1957)
release: merge dev-v2.0.25 into main (MemTensor#2156)