Skip to content

l3.abstraction: misleading "title must be a non-empty string" error — output truncation at maxTokens, parseLlmJson falls back to a nested block #2356

Closed as not planned

Description

@zenhtml

Summary

With @memtensor/memos-local-plugin v2.0.18 (Hermes local adapter, full self-evolution mode), L3 world-model generation fails almost always with a misleading validation error:

[core.memory.l3.abstract] abstract.llm_failed err="l3.abstraction: 'title' must be a non-empty string"
[llm.json] malformed op="l3.abstraction.v2"

The model's answer is actually valid JSON with a non-empty title at the top level. The real failure chain is:

  1. The L3 answer is truncated at maxTokens (long answer + a reasoning model burning 5–12k tokens on thoughts).
  2. JSON.parse on the full text fails (Unterminated string).
  3. parseLlmJson falls back to extractFirstJsonBlock, which returns the first balanced {...} / [...] — a nested block (in our case the domain_tags array).
  4. validate then runs against that nested value and reports 'title' must be a non-empty string — pointing the debugging effort at the model, not at the truncation.

Related: #1755 lists these schema failures as a secondary symptom; the root cause was not analyzed there.

Environment

  • Plugin: @memtensor/memos-local-plugin v2.0.18 (Hermes local adapter, Windows)
  • Memory mode: full self-evolution (algorithm.lightweightMemory.enabled: false)
  • LLM: OpenAI-compatible endpoint, deepseek-v4-flash (reasoning model), temperature: 0, response_format: {"type":"json_object"}
  • Config before the fix: llm.maxTokens: 8192, llm.timeoutMs: 120000
  • Volume: world_model_generate — 6 successes out of 456 calls (1/22 today).

Evidence (raw answers captured from completeJson)

attempt 1: len 13082, JSON.parse error: Unterminated string starting at: line 1 column 11856 (char 11855)
           raw starts with: {"title":"...","domain_tags":["network","http","local-services","cron"],...
attempt 2: len 0  -> "LLM output not valid JSON: empty response"
parsedShape (what parseLlmJson actually returned): array[4]   <- the `domain_tags` array

Model-side limit probe against the same endpoint:

max_tokens finish_reason completion_tokens reasoning_tokens
8192 length 8192 5004
16384 length 16384 5081
32768 stop 17374 11827

The prompt asks for a body field (full markdown rendering that duplicates environment/inference/constraints), so an L3 answer is ~13k chars — reasoning + answer do not fit in the old 8192 budget.

Fix that works locally

llm:
  maxTokens: 32768
  timeoutMs: 300000

After restarting the bridge/daemon, a manual L3 run over the live DB:

run.done: clusters=6 created=0 merged=6 skipped=0

5 world models updated, confidence ~1.0, no further malformed warnings.

Suggested improvements

  1. Don't let the parse fallback masquerade as a schema error. When the top-level text fails to parse, either restrict the extracted-block fallback to a value that satisfies the caller's shape, or surface the original parse error (with rawPreview) instead of validating a nested value.
  2. Log the raw answer on malformed output. logs/llm.jsonl stays empty and memos.log only carries "msg":"malformed" without the payload — which makes this whole bug class invisible.
  3. Handle finishReason === "length" explicitly: retry with a "shorter answer" instruction (or raise the budget) instead of a blind retry; the current retry sometimes returns an empty response.
  4. Trim the L3 schema: body duplicates the three structured sections; a short body (or a derived one) would roughly halve the answer size.

Happy to attach the small local diagnostic patch (adds rawPreview / rawLen / parsedShape to the malformed warning) if it helps.

Activity

  1. added
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理
    types:bugSomething isn't working | 功能异常
    on Sep 10, 2026
  2. self-assigned this
    on Sep 10, 2026
  3. Memtensor-AI commented on Sep 10, 2026

    @Memtensor-AI
    Collaborator

    🤖 AutoDev has picked up this issue and started working on it.

    Task ID: 035ad74cb9dcd82a
    Working branch: bugfix/autodev-2356-20260910132328679
    Target branch: latest dev* branch
    Workflow: opsp (analysis → coding → testing → PR)

    I will post the PR link here once done. If I need more information, I will ask in the comments.

  4. added a commit that references this issue on Sep 10, 2026
    4ce26cc
  5. Memtensor-AI commented on Sep 10, 2026

    @Memtensor-AI
    Collaborator

    ✅ AutoDev task 035ad74cb9dcd82a completed.

    Summary: Fixed the misleading "title must be a non-empty string" error in L3 world-model generation caused by two cooperating bugs.

    Bug 1 — extractFirstJsonBlock (json-mode.ts): The old loop continued scanning for a nested balanced sub-block when the root opener returned null from walkToClose. A truncated root object like {"title":"...","domain_tags":["network","http"],... left the root { unbalanced, causing the loop to continue and find the inner domain_tags array, which was fully balanced. JSON.parse then succeeded on that array, and the caller's validate() received ["network","http","local-services","cron"] instead of the full object — correctly reporting 'title' must be a non-empty string but pointing blame at the model rather than at token truncation. Fix: stop at the first opener; if walkToClose returns null, return null immediately.

    Bug 2 — completeJson (client.ts): finishReason === "length" was already present on LlmCompletion but completeJson passed the truncated text straight to parseLlmJson without inspecting it. Fix: check finishReason === "length" before parsing; throw LLM_OUTPUT_MALFORMED immediately with finishReason: "length" and rawLen in the error details. Also added rawLen to all existing malformed warn log rows for easier token-budget diagnosis.

    Tests: 5 new test cases added (2 in json-mode.test.ts, 3 in client.test.ts). All 43 LLM unit tests pass. Pre-existing failures in subscriber.test.ts are unrelated (SQLite native binding absent in this environment).

    Base branch: main
    Branch: bugfix/autodev-2356-20260910132328679
    Commit: 4ce26cc
    PR: #2357
    Assigned to: @wustzdy
    Reviewers: @whipser030, @hijzy

  6. added
    ai:testingAI agent is running tests | AI 正在运行测试
    and removed
    ai:codingAI agent is coding | AI 正在编码
    on Sep 10, 2026
  7. github-actions commented on Oct 11, 2026

    @github-actions
    Contributor

    This issue has been automatically marked as stale due to inactivity.

  8. added
    status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.
    on Oct 11, 2026
  9. github-actions commented on Oct 19, 2026

    @github-actions
    Contributor

    This issue has been automatically closed due to inactivity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务ai:testingAI agent is running tests | AI 正在运行测试area:pluginOpenClaw & Hermesstatus:in-progressSomeone or AI is working on it | 人工或 AI 正在处理status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.types:bugSomething isn't working | 功能异常

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions