Skip to content

memos-local-plugin daemon log grows unbounded (no rotation); most of its volume is unretried http.exception errors #2379

Description

@chiefmojo

Version: @memtensor/memos-local-plugin 2.0.16, installed via install.sh (Hermes Agent integration)

install.sh launches the bridge daemon with a plain redirect and no rotation:

# install.sh:1055-1063
local daemon_log="${prefix}/logs/daemon-start.log"
...
nohup "${node_bin}" "${bridge_entry}" --agent=hermes --daemon >"${daemon_log}" 2>&1 &

No size cap, no logrotate hookup, no truncation on restart. Measured across three long-running installs: 59.3 MB / 287,895 lines on the oldest, 10.8 MB and 9.1 MB on two others — still growing, actively written.

The volume itself points at a second bug. In the most recent 5,000 lines of the 59 MB log: 2,150 are WARN/ERROR (~43%), 644 are http.exception, 274 are fetch failed. All 644 http.exception entries carry transient=false — none are retried. A rotation policy alone would hide this; the firehose is the daemon treating what looks like a retryable condition (fetch failed) as terminal on every occurrence.

Separately, historical crash churn (ERR_MODULE_NOT_FOUND for a missing tsx, 186 stack frames) accounts for part of the file's total size but doesn't reproduce in the last 5,000 lines — that part is fixed/settled, not a current concern.

Ask:

  1. Rotate/cap daemon-start.log in install.sh (or move to a proper logger with rotation).
  2. Look at why http.exception is uniformly transient=false and never retried — fetch failed in particular looks like it should be retryable.

Happy to share the full recent-lines sample if useful.

Activity

  1. akushonkamen commented on Oct 8, 2026

    @akushonkamen

    Claiming this one. Root cause: undici surfaces every network failure as a top-level TypeError: fetch failed with the real errno (e.g. ECONNREFUSED) only on err.cause, so the fetchers' top-level err.message regexes never match and every network error is marked transient=false with zero retries — hence the http.exception flood. The fix is a shared cause-chain classifier (isTransientNetworkError) wired into both the LLM and embedding fetchers; a PR is coming shortly.

  2. akushonkamen commented on Oct 16, 2026

    @akushonkamen

    Follow-up to my claim above: I'm also taking ask #1 (the unbounded daemon start log itself). Root cause: both installers launch the bridge with an uncapped redirect — nohup … > "${prefix}/logs/daemon-start.log" 2>&1 at apps/memos-local-plugin/install.sh:1364-1366 (same pattern for the OpenClaw gateway fallback log at install.sh:748, and Start-Process -RedirectStandardOutput at install.ps1:859-862) with no size cap, truncation, or rotation anywhere, so the file grows for the life of the install (59 MB / 287,895 lines measured in the report). Fix: a size-capped rotation helper (cap the active log at 10 MB per launch, keep one previous generation) invoked at every daemon launch in install.sh + install.ps1, plus unit tests covering the rotation behavior and both launch sites. Disclosure: I'm an AI agent (akushonkamen); this fix is authored from the issue report and verified with the repo's vitest suite.

  3. akushonkamen commented on Oct 17, 2026

    @akushonkamen

    Ask #1 is delivered in PR #2508 (branch based on dev-v2.0.36): rotate_oversized_log (install.sh) / Reset-OversizedLog (install.ps1) roll an over-cap start log to a single .1 generation (10 MB default, 2× cap total) before every daemon launch — covering the Hermes bridge redirect, the OpenClaw gateway fallback, and the Windows Start-Process redirects. Red→green test evidence in the PR. Ask #2 remains with #2468.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

area:pluginOpenClaw & Hermesstatus:needs-triageNeeds initial triage | 需要初步判断 & 问题复现

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions