You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix: skill crystallization retries the same failing policies forever #2319
I have searched existing issues and this hasn't been mentioned before
I have read the project documentation and confirmed this issue doesn't already exist
This issue is specific to MemOS and not a general software issue
Bug Description
Skill crystallization has no failure back-off. When crystallize() or the verifier fails for a policy, nothing changes in that policy's eligibility — so every skill tick re-attempts the same failing policy forever.
In a 25-day audit of our local deployment, this accumulated 2,640 repeated crystallization failures across a small set of permanently failing policies (LLM refusals, verifier mismatches, permanently unqualifying evidence). Each retry burns LLM budget and log volume for a policy that can never succeed.
How to Reproduce
Seed one graduated policy whose evidence can never pass the crystallizer (e.g. its traces contain no anchorable tool results)
Let skill ticks run repeatedly
Observe the same policy selected for crystallization every tick, failing every time, with no upper bound
Summary: Fix for issue #2319 (skill crystallization retry storm) is implemented, tested, and pushed to the working branch.
The root cause was that runSkill in apps/memos-local-plugin/core/skill/skill.ts recorded no per-policy state on failure, and the documented skill.cooldownMs config was dead code — so every subscriber trigger (l2.policy.induced, l2.policy.updated, reward.updated) re-attempted the same permanently-failing policies forever, producing the observed 2,640 repeated failures over 25 days.
The fix adds a persistent per-policy back-off state to the policies table via additive migration 019 (four columns: attempts, backoff_until, last_attempt_at, last_failure_reason). Three new SkillConfig fields (crystallizationBackoffBaseMs, crystallizationBackoffMaxMs, crystallizationMaxAttempts, defaults 5 min / 24 h / 8 attempts) drive an exponential back-off. evaluateEligibility gains a pre-gate that skips policies inside their back-off window or in quarantine, auto-invalidating whenever policy.updatedAt > lastAttemptAt so that new evidence always retries immediately. runSkill records failures on all three branches (no-evidence, crystallize, verify) and resets state on successful upsert. Docs (README.md, ALGORITHMS.md §9) updated; the legacy cooldownMs field is preserved as a documented no-op for YAML compatibility.
Verification: npm run lint clean; npm test reports 1,564 passed / 2 skipped across 183 files with zero regressions. Four new eligibility tests and three new integration tests explicitly cover the back-off window, quarantine, auto-invalidation, and success reset — including a regression test that asserts the LLM is NOT re-invoked while back-off is active.
Under the observed workload this turns 2,640 retries into ~8 then zero, a >300× budget reduction on the worst-case permanently-failing policy.
Base branch:main Branch:bugfix/autodev-2319-20260902004643085 Commit:68aa948 PR:#2325 Assigned to:@whipser030 Reviewers:@whipser030, @hijzy
Activity
[-]### Pre-submission checklist[/-][+]fix: skill crystallization retries the same failing policies forever[/+]Memtensor-AI commentedon Sep 2, 2026
🤖 AutoDev has picked up this issue and started working on it.
Task ID:
6694bfe346157994Working branch:
bugfix/autodev-2319-20260902004643085Target branch:
latest dev* branchWorkflow: opsp (analysis → coding → testing → PR)
I will post the PR link here once done. If I need more information, I will ask in the comments.
fix(memos-local-plugin): add persistent back-off for skill crystalliz…
Memtensor-AI commentedon Sep 2, 2026
✅ AutoDev task
6694bfe346157994completed.Summary: Fix for issue #2319 (skill crystallization retry storm) is implemented, tested, and pushed to the working branch.
The root cause was that
runSkillinapps/memos-local-plugin/core/skill/skill.tsrecorded no per-policy state on failure, and the documentedskill.cooldownMsconfig was dead code — so every subscriber trigger (l2.policy.induced,l2.policy.updated,reward.updated) re-attempted the same permanently-failing policies forever, producing the observed 2,640 repeated failures over 25 days.The fix adds a persistent per-policy back-off state to the
policiestable via additive migration 019 (four columns: attempts, backoff_until, last_attempt_at, last_failure_reason). Three newSkillConfigfields (crystallizationBackoffBaseMs,crystallizationBackoffMaxMs,crystallizationMaxAttempts, defaults 5 min / 24 h / 8 attempts) drive an exponential back-off.evaluateEligibilitygains a pre-gate that skips policies inside their back-off window or in quarantine, auto-invalidating wheneverpolicy.updatedAt > lastAttemptAtso that new evidence always retries immediately.runSkillrecords failures on all three branches (no-evidence, crystallize, verify) and resets state on successful upsert. Docs (README.md, ALGORITHMS.md §9) updated; the legacycooldownMsfield is preserved as a documented no-op for YAML compatibility.Verification:
npm run lintclean;npm testreports 1,564 passed / 2 skipped across 183 files with zero regressions. Four new eligibility tests and three new integration tests explicitly cover the back-off window, quarantine, auto-invalidation, and success reset — including a regression test that asserts the LLM is NOT re-invoked while back-off is active.Under the observed workload this turns 2,640 retries into ~8 then zero, a >300× budget reduction on the worst-case permanently-failing policy.
Base branch:
mainBranch:
bugfix/autodev-2319-20260902004643085Commit:
68aa948PR: #2325
Assigned to: @whipser030
Reviewers: @whipser030, @hijzy
fix(memos-local-plugin): address PR MemTensor#2325 open code review f…
github-actions commentedon Oct 5, 2026
This issue has been automatically marked as stale due to inactivity.
github-actions commentedon Oct 12, 2026
This issue has been automatically closed due to inactivity.