Skip to content

bug(memos-local-plugin): Hermes background model status checks can trigger paid LLM request storm #1897

Closed

Description

@TianchunGu

Summary

I observed a high-volume paid LLM request spike that appears to correlate with MemOS Local Plugin running under Hermes. The strongest signal is that local api_logs contain a very large number of system_model_status records, close to the paid provider's request count for the same billing window.

This looks like a background model-status / health-check / retry loop may be sending real paid LLM requests instead of a non-billable local/provider availability check.

No API keys, account identifiers, full prompts, or private logs are included here.

Environment

  • Project: memos-local-plugin
  • Host: Hermes
  • Runtime data path: ~/.hermes/memos-plugin/
  • Provider: DeepSeek
  • Model: deepseek-v4-flash
  • Date observed: 2026-06-08

Evidence

Provider billing export for the dedicated MemOS key:

2026-06-08 UTC
model: deepseek-v4-flash
request_count: 11,344
cost: approx 19.89 CNY

Local MemOS database/API log summary:

api_logs total: 14,632
system_model_status: 12,900
system_error: 811
skill_generate: 80
world_model_generate: 40
policy_evolve: 85
memory_add: 231
local log coverage: roughly 2026-06-07 12:00 to 2026-06-08 11:41

The suspicious part is system_model_status = 12,900, which is very close to the provider-side request_count = 11,344 for deepseek-v4-flash.

During the abnormal window, system_error also increased rapidly, with LLM-role errors recurring every few seconds. This suggests the plugin may continue retrying after provider errors such as insufficient balance, rate limit, or transient provider failures.

Why this seems problematic

A system/model status check should not repeatedly call a paid chat/completions endpoint by default. If system_model_status is implemented as a real LLM generation request, it can silently consume paid API quota in the background.

This issue may be related to #1620, where LLM/embedding fetchers retry 429 responses without respecting Retry-After, potentially increasing failed requests and wasting paid API calls. In this case, the visible symptom is even more severe because the high-volume tool is system_model_status.

Expected behavior

  • system_model_status should not perform paid LLM generation by default.
  • Health checks should use non-generative checks where possible, such as config validation, lightweight endpoint reachability, or model-list/balance-style provider APIs if available.
  • Paid LLM calls should have per-tool rate limits.
  • Background jobs should have daily/hourly budget caps.
  • 402 / insufficient balance / unauthorized / invalid key should immediately open a circuit breaker instead of retrying.
  • 429 should respect Retry-After.
  • Repeated provider failures should exponentially back off and eventually disable the provider until user action.
  • The viewer/logs should clearly distinguish non-billable status checks from billable LLM calls.

Suggested safeguards

  1. Add a circuit breaker for provider errors:
402 / insufficient balance: disable provider immediately
401 / invalid key: disable provider immediately
429: respect Retry-After, then back off
5xx/network errors: bounded retries only
  1. Add a hard throttle for system_model_status, for example:
max 1 status check per 10-30 minutes
never run status checks in a tight loop
never retry status checks every few seconds
  1. Add per-tool billing guardrails:
system_model_status: disabled or non-billable by default
memory_add / policy_evolve / world_model_generate / skill_generate: bounded per hour/day
  1. Make paid background calls visible in the UI/logs:
tool_name, provider, model, billable=true/false, retry_count, backoff_ms, circuit_breaker_state

Current workaround

I disabled the MemOS plugin / removed the dedicated DeepSeek key to stop further spend. I can provide redacted query outputs or sampled system_model_status rows if needed, but I do not want to post complete private prompts or credentials publicly.

Activity

  1. added
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    types:bugSomething isn't working | 功能异常
    contrib:help wantedExtra attention is needed | 需要社区帮助
    ai:codingAI agent is coding | AI 正在编码
    on Jun 8, 2026
  2. self-assigned this
    on Jun 8, 2026
  3. Memtensor-AI commented on Jun 8, 2026

    @Memtensor-AI
    Collaborator

    🤖 AutoDev has picked up this issue and started working on it.

    Task ID: a995d3f2c525e73d
    Working branch: bugfix/autodev-1897
    Target branch: main
    Workflow: opsp (analysis → coding → testing → PR)

    I will post the PR link here once done. If I need more information, I will ask in the comments.

  4. added a commit that references this issue on Jun 8, 2026
    d538031
  5. Memtensor-AI commented on Jun 8, 2026

    @Memtensor-AI
    Collaborator

    ✅ AutoDev task a995d3f2c525e73d completed.

    Summary: Fixes #1897: paid LLM request storm on Hermes when the configured provider returns terminal errors (402 insufficient balance / 401 invalid key / 403). Issue reporter saw ~12,900 paid DeepSeek requests in 24 h tracking exactly the local system_model_status audit-row count.

    Root cause: the system_model_status row name is misleading. It is not a health probe; it is the audit row written once per LLM call inside apps/memos-local-plugin/core/llm/client.ts::callWithFallback. With no circuit breaker, every pipeline subscriber (capture / session-relation / reward / L2 / L3 / skill / retrieval LLM filter / world-model) kept firing on every turn, induction, and closed episode, generating one paid request each even though the provider was permanently broken until the operator fixed billing.

    Fix: per-LlmClient circuit breaker that trips on terminal provider errors (HTTP 401/402/403 or messages containing "insufficient balance" / "invalid api key" / "unauthorized" / "account suspended" / "billing"). When open, calls short-circuit inside the facade — zero paid requests, no further system_model_status="error" rows. Half-open after a 5-minute cool-down (configurable, min 30 s); the next call probes the provider and either closes the breaker on success or re-opens it on terminal failure. Host fallback rescues a call without tripping the breaker. A coalesced system_model_status="circuit_open" row is emitted at most once per ~25 s so the Logs viewer still surfaces the suppression event without replacing paid spam with audit-row spam. The breaker is enabled by default; legacy behaviour is available via circuitBreaker.enabled = false.

    Tests: 9 new vitest cases under apps/memos-local-plugin/tests/unit/llm/client.test.ts covering trip on 402, trip on "insufficient balance" message, no trip on generic transient, coalescing under load, half-open close on success, host-fallback rescues without trip, legacy disabled mode, stats exposure, and re-open on terminal probe failure. All 59 LLM and 28 pipeline tests pass; tsc --noEmit clean. Three full-suite failures (storage migrator / traces-count / e2e episode merge) are pre-existing on the base branch dev-20260604-v2.0.19 and unrelated to this change.

    Out of scope (tracked separately): 429 Retry-After handling (issue #1620), per-tool rate limits, daily budget caps.

    Base branch: main
    Branch: bugfix/autodev-1897
    Commit: d538031
    PR: #1899
    Assigned to: @MatthewZhuang, @CarltonXiang, @syzsunshine219
    Reviewers: @MatthewZhuang, @CarltonXiang, @syzsunshine219

  6. added
    ai:testingAI agent is running tests | AI 正在运行测试
    ai:pr-readyAI-created PR is ready for review | AI 生成的 PR 已等待评审
    and removed
    ai:codingAI agent is coding | AI 正在编码
    ai:testingAI agent is running tests | AI 正在运行测试
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    on Jun 8, 2026
  7. TianchunGu commented on Jun 17, 2026

    @TianchunGu
    ContributorAuthor

    Additional reporter-side finding, shared in a sanitized form: #1899 / #1938 address the terminal-provider-error retry storm, but there appears to be a second request-amplification path that can still produce many paid LLM calls even when the provider is healthy.

    The second path is startup recovery of dirty closed episodes.

    High-level shape observed from a local aggregate DB audit, without sharing raw prompts, tool payloads, keys, timestamps, or local episode IDs:

    • A single closed episode can dominate the system_model_status audit rows, reaching the >10k scale by itself.
    • That episode can contain thousands of trace rows, mostly tool-call traces.
    • Many tool-call traces may lack both startedAt and endedAt in tool_calls_json.
    • If those traces are still missing reflection and the episode has no r_task, startup recovery treats it as dirty and replays reflect/reward work.

    Observed code path:

    1. core/pipeline/memory-core.ts::recoverDirtyClosedEpisodes() emits episode.finalized for dirty closed episodes.
    2. core/capture/subscriber.ts calls runner.runReflect(...).
    3. core/capture/capture.ts::runReflect() reconstructs normalized steps and matches them against existing traces by stepIdentitySignature() / traceIdentitySignature().
    4. For tool-call traces without real timing, the signature falls back to ts. During recovered snapshot replay, extractSteps() may assign shifted unique timestamps, so semantically identical tool calls no longer match their existing trace rows.
    5. Those steps are classified as reflect.orphan_steps and then passed through runSummarize(..., { phase: "reflect" }), which invokes the configured LLM for each orphan before inserting more rows.
    6. After that, runEpisodeBatchScoring() runs windowed capture.reflection.batch.v10 calls across the recovered episode, with retry/degrade behavior that can further amplify requests.

    A local read-only replay of the current matching logic showed that the replay orphans were overwhelmingly tool-call steps and were matchable if timestamp/timing was ignored. So this looks like a recovery/signature stability issue rather than genuinely missing trace content.

    Suggested follow-up fixes:

    • Add a hard per-episode LLM budget for startup recovery / reflect replay.
    • In recovery mode, avoid LLM summarization for reflect.orphan_steps; prefer existing summaries or heuristic summaries.
    • Add a timing-insensitive fallback signature for tool calls that lack startedAt/endedAt, especially in recovered snapshots.
    • Stop the remaining window pass early after terminal provider errors instead of continuing through every window.
    • Add an actual config gate to disable capture/reflection LLM calls; current comments say alphaScoring / synthReflections are no-ops under the windowed binary pipeline.

    I can split this into a separate issue if maintainers prefer, but it seems closely related to #1897 because it explains how local recovery/replay can create system_model_status rows at the same scale as the reported bill spike, independent of external key theft.

  8. 4 remaining items

  9. removed
    ai:pr-readyAI-created PR is ready for review | AI 生成的 PR 已等待评审
    on Jun 30, 2026
  10. added a commit that references this issue on Jul 2, 2026
    4e9bbce
  11. added a commit that references this issue on Jul 2, 2026
    ca75756
  12. added 4 commits that reference this issue on Jul 23, 2026
    21a27ed
    db45f40
    ded4c3f
    63ddb97
  13. added 3 commits that reference this issue on Jul 23, 2026
    d9130a7
    87b0738
    f8a887d
  14. added a commit that references this issue on Jul 24, 2026
    e820406
  15. added 3 commits that reference this issue on Sep 18, 2026
    8001941
    c9a80d5
    a447579
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:pluginOpenClaw & Hermescontrib:help wantedExtra attention is needed | 需要社区帮助types:bugSomething isn't working | 功能异常

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions