Skip to content

fix: skill crystallization retries the same failing policies forever #2319

Closed as not planned

Description

@smoryan

Pre-submission checklist

  • I have searched existing issues and this hasn't been mentioned before
  • I have read the project documentation and confirmed this issue doesn't already exist
  • This issue is specific to MemOS and not a general software issue

Bug Description

Skill crystallization has no failure back-off. When crystallize() or the verifier fails for a policy, nothing changes in that policy's eligibility — so every skill tick re-attempts the same failing policy forever.

In a 25-day audit of our local deployment, this accumulated 2,640 repeated crystallization failures across a small set of permanently failing policies (LLM refusals, verifier mismatches, permanently unqualifying evidence). Each retry burns LLM budget and log volume for a policy that can never succeed.

How to Reproduce

  1. Seed one graduated policy whose evidence can never pass the crystallizer (e.g. its traces contain no anchorable tool results)
  2. Let skill ticks run repeatedly
  3. Observe the same policy selected for crystallization every tick, failing every time, with no upper bound

Environment

  • memos-local-plugin: 2.0.18 (apps/memos-local-plugin)
  • Node.js: 26
  • OS: macOS

Willingness to Implement

  • I'm willing to implement this myself

Activity

  1. added
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    status:in-progressSomeone or AI is working on it | 人工或 AI 正在处理
    types:bugSomething isn't working | 功能异常
    on Sep 2, 2026
  2. changed the title [-]### Pre-submission checklist[/-] [+]fix: skill crystallization retries the same failing policies forever[/+] on Sep 2, 2026
  3. Memtensor-AI commented on Sep 2, 2026

    @Memtensor-AI
    Collaborator

    🤖 AutoDev has picked up this issue and started working on it.

    Task ID: 6694bfe346157994
    Working branch: bugfix/autodev-2319-20260902004643085
    Target branch: latest dev* branch
    Workflow: opsp (analysis → coding → testing → PR)

    I will post the PR link here once done. If I need more information, I will ask in the comments.

  4. self-assigned this
    on Sep 2, 2026
  5. Memtensor-AI commented on Sep 2, 2026

    @Memtensor-AI
    Collaborator

    ✅ AutoDev task 6694bfe346157994 completed.

    Summary: Fix for issue #2319 (skill crystallization retry storm) is implemented, tested, and pushed to the working branch.

    The root cause was that runSkill in apps/memos-local-plugin/core/skill/skill.ts recorded no per-policy state on failure, and the documented skill.cooldownMs config was dead code — so every subscriber trigger (l2.policy.induced, l2.policy.updated, reward.updated) re-attempted the same permanently-failing policies forever, producing the observed 2,640 repeated failures over 25 days.

    The fix adds a persistent per-policy back-off state to the policies table via additive migration 019 (four columns: attempts, backoff_until, last_attempt_at, last_failure_reason). Three new SkillConfig fields (crystallizationBackoffBaseMs, crystallizationBackoffMaxMs, crystallizationMaxAttempts, defaults 5 min / 24 h / 8 attempts) drive an exponential back-off. evaluateEligibility gains a pre-gate that skips policies inside their back-off window or in quarantine, auto-invalidating whenever policy.updatedAt > lastAttemptAt so that new evidence always retries immediately. runSkill records failures on all three branches (no-evidence, crystallize, verify) and resets state on successful upsert. Docs (README.md, ALGORITHMS.md §9) updated; the legacy cooldownMs field is preserved as a documented no-op for YAML compatibility.

    Verification: npm run lint clean; npm test reports 1,564 passed / 2 skipped across 183 files with zero regressions. Four new eligibility tests and three new integration tests explicitly cover the back-off window, quarantine, auto-invalidation, and success reset — including a regression test that asserts the LLM is NOT re-invoked while back-off is active.

    Under the observed workload this turns 2,640 retries into ~8 then zero, a >300× budget reduction on the worst-case permanently-failing policy.

    Base branch: main
    Branch: bugfix/autodev-2319-20260902004643085
    Commit: 68aa948
    PR: #2325
    Assigned to: @whipser030
    Reviewers: @whipser030, @hijzy

  6. added
    ai:testingAI agent is running tests | AI 正在运行测试
    and removed
    ai:codingAI agent is coding | AI 正在编码
    on Sep 2, 2026
  7. added a commit that references this issue on Sep 2, 2026
  8. added
    ai:pr-readyAI-created PR is ready for review | AI 生成的 PR 已等待评审
    and removed
    ai:testingAI agent is running tests | AI 正在运行测试
    ai:taskDispatched to AI coding agent | 已派发给 AI 编码任务
    on Sep 2, 2026
  9. github-actions commented on Oct 5, 2026

    @github-actions
    Contributor

    This issue has been automatically marked as stale due to inactivity.

  10. added
    status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.
    on Oct 5, 2026
  11. github-actions commented on Oct 12, 2026

    @github-actions
    Contributor

    This issue has been automatically closed due to inactivity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

ai:pr-readyAI-created PR is ready for review | AI 生成的 PR 已等待评审area:pluginOpenClaw & Hermesstatus:in-progressSomeone or AI is working on it | 人工或 AI 正在处理status:staleAuto-marked stale after 30d inactivity; closes 7d later unless exempt.types:bugSomething isn't working | 功能异常

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions