Skip to content

feat(reward): sentinel-based exchange-count bypass for cron episodes - #1848

Closed
chiefmojo wants to merge 6 commits into
MemTensor:mainfrom
chiefmojo:pr/cron-episode-scoring-bypass
Closed

chiefmojo wants to merge 6 commits into
MemTensor:mainfrom
chiefmojo:pr/cron-episode-scoring-bypass

Conversation

@chiefmojo

@chiefmojo chiefmojo commented Jun 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds a config-driven bypass so cron-initiated episodes can be scored even when they fall below the minExchangesForCompletion threshold.

Problem

Scheduled agent jobs usually produce a single exchange: one cron prompt and one agent response. Deployments with a stricter exchange floor can therefore discard substantive scheduled work before reward scoring.

Solution

Adds RewardConfig.cronSentinels, an array of user-turn prefixes. When the first user turn starts with a configured prefix, only the exchange-count gate is bypassed. Content length, triviality, tool-heavy, and repetition gates remain active.

The default is an empty array ([]), so existing deployments keep their current behavior. Operators can opt in through config.yaml:

algorithm:
  reward:
    cronSentinels: ["[IMPORTANT: You are running as a scheduled cron job"]

Design notes

  • Matching is prefix-based against the first user turn.
  • A blank first materialized user turn may fall back to string-valued meta.initialUserText; later user turns are never used for sentinel matching.
  • Empty, whitespace-only, and leading-whitespace prefixes are rejected by config validation; blank values are also ignored defensively at runtime.
  • No other reward quality gate is bypassed.
  • cronSentinels remains optional in the exported RewardConfig interface for backward compatibility with programmatic callers; resolved config always supplies [].

Test plan

  • One exchange with a matching sentinel reaches reward scoring.
  • A non-matching sentinel still fails the exchange-count gate.
  • An empty sentinel list preserves existing behavior.
  • Empty, whitespace-only, and leading-whitespace sentinel values are rejected.
  • A blank first materialized user turn can use meta.initialUserText.
  • A matching but short cron episode still fails the content gate.
  • Reward/config unit suites pass (103/103).
  • TypeScript lint and package build pass.

Generated with Claude Code

Cron jobs always produce exactly 1 user↔agent exchange — the task prompt
plus one reply — so minExchangesForCompletion: 2 zero-scores every cron
episode before content is even evaluated. This starves L2 induction of
signal after the bridge stabilises.

Adds `cronSentinels` to RewardConfig (schema, defaults, types). When
the first user turn starts with a sentinel prefix, check 1 (exchange
count) is skipped; content/triviality checks still apply. Default
sentinel covers the Hermes cron prompt.

The `snapshot.meta?.initialUserText` fallback handles episodes scored
during recovery when turns aren't materialised. If the field is absent
the episode falls back to the old skip behaviour — no false positives.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Memtensor-AI
Memtensor-AI changed the base branch from main to dev-20260604-v2.0.19 June 10, 2026 15:41
@Memtensor-AI
Memtensor-AI changed the base branch from dev-20260604-v2.0.19 to dev-v2.0.22 July 1, 2026 13:16
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

Automated Test Results: PASSED

Cloud test-engine rerun against dev-v2.0.22 completed successfully.

  • Run: tr-dfe46f67-699 on cloud test-engine 10012
  • memos_local_plugin/unit: 6 passed, 0 failed, 0 skipped

Manual code review is still required before merge.

@CarltonXiang
CarltonXiang deleted the branch MemTensor:main July 3, 2026 07:25
@syzsunshine219 syzsunshine219 reopened this Jul 3, 2026
@syzsunshine219
syzsunshine219 changed the base branch from dev-v2.0.22 to main July 3, 2026 08:22
@Memtensor-AI Memtensor-AI added area:plugin OpenClaw & Hermes area:memory 记忆存储、检索、更新、召回逻辑 labels Jul 8, 2026
@Memtensor-AI
Memtensor-AI requested a review from bittergreen July 8, 2026 11:44
@Memtensor-AI Memtensor-AI added area:memory 记忆存储、检索、更新、召回逻辑 area:model llm + embedder + reranker and removed area:memory 记忆存储、检索、更新、召回逻辑 labels Jul 9, 2026
@Memtensor-AI Memtensor-AI removed area:memory 记忆存储、检索、更新、召回逻辑 area:model llm + embedder + reranker needs-audit labels Jul 9, 2026
@Memtensor-AI Memtensor-AI added the status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 label Aug 6, 2026
@Memtensor-AI

Memtensor-AI commented Aug 6, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #1848
Task: 289fee9a4f552702
Base: main
Head: pr/cron-episode-scoring-bypass

🔍 OpenCodeReview found 5 issue(s) in this PR.


1. apps/memos-local-plugin/core/config/defaults.ts (L172)

The cronSentinels field is declared as cronSentinels?: string[] (optional) in RewardConfig (types.ts), but it is provided here as a required concrete value [] in DEFAULT_CONFIG. This optional typing in the interface means every call site in reward.ts must guard with ?? [] defensively.

For consistency and to eliminate the runtime defensive fallback, consider making the field required in RewardConfig:

cronSentinels: string[];

This aligns the type with the schema (which always provides a default []) and with this default config entry, making the ?? [] in reward.ts unnecessary.

💡 Suggested Change

Before:

      cronSentinels: [],

After:

      cronSentinels: [],

2. apps/memos-local-plugin/core/config/schema.ts (L279-L281)

The pattern: "^\\S" constraint only validates that each sentinel does not start with whitespace — it does not guarantee the string is non-empty (that is handled separately by minLength: 1). However, a single non-whitespace character followed by only whitespace (e.g. "A ") would pass both constraints and still produce a match for any user content beginning with "A", including content whose effective trimmed start is very different.

More critically, a sentinel like " " (a single space) would fail ^\\S but "\t" (a tab) would also fail — those are caught. However, a sentinel of "_" (one underscore) passes both validations and would match enormous swaths of user content that happen to start with _. Consider tightening the pattern to something like "^\\S+" (requires at least one trailing non-whitespace, but that is still just one char), or alternatively document clearly that sentinels should be long, distinctive prefixes and add a minLength of, say, 3–5 to reduce accidental over-matching.

Also note that the runtime code in reward.ts uses firstUserContent.startsWith(sentinel) without trimming sentinel, so a trailing-whitespace sentinel (e.g. "CRON: ") would only match if the user text contains the exact trailing space — this is subtle and inconsistent with the trimming applied in the validation guard (sentinel.trim().length > 0).

💡 Suggested Change

Before:

    cronSentinels: Type.Array(Type.String({ minLength: 1, pattern: "^\\S" }), {
      default: [],
    }),

After:

    cronSentinels: Type.Array(Type.String({ minLength: 3, pattern: "^\\S+" }), {
      default: [],
    }),

3. apps/memos-local-plugin/core/reward/reward.ts (L469-L474)

Nested ternary expression: the firstUserContent assignment uses a nested ternary (condition ? A : condition2 ? B : C), which is explicitly prohibited by the code quality checklist. Refactor into an if/else block or a helper variable for clarity and maintainability.

💡 Suggested Change

Before:

    const firstUserContent =
      userContents[0]?.trim().length
        ? userContents[0]
        : typeof metaInitialUserText === "string"
          ? metaInitialUserText
          : "";

After:

    let firstUserContent = "";
    if (userContents[0]?.trim().length) {
      firstUserContent = userContents[0];
    } else if (typeof metaInitialUserText === "string") {
      firstUserContent = metaInitialUserText;
    }

4. apps/memos-local-plugin/core/reward/reward.ts (L475-L480)

cronSentinels is already typed as string[], and the schema enforces minLength: 1 and a non-whitespace leading character (^\S). The runtime guards typeof sentinel === "string" and sentinel.trim().length > 0 are therefore redundant and add noise. Consider removing them to keep the predicate concise.

💡 Suggested Change

Before:

    const isCronEpisode = (cfg.cronSentinels ?? []).some(
      (sentinel) =>
        typeof sentinel === "string" &&
        sentinel.trim().length > 0 &&
        firstUserContent.startsWith(sentinel),
    );

After:

    const isCronEpisode = (cfg.cronSentinels ?? []).some(
      (sentinel) => firstUserContent.startsWith(sentinel),
    );

5. apps/memos-local-plugin/core/reward/types.ts (L75)

The cronSentinels field is declared as optional (string[] | undefined) in the interface, but in reward.ts the .startsWith(sentinel) check is performed on firstUserContent — which is also a plain string. If firstUserContent is an empty string ("") and a sentinel is also "", "".startsWith("") returns true, marking every under-exchanged episode as a cron episode and bypassing the exchange-count gate entirely.

The schema in schema.ts guards against this with minLength: 1 and pattern: "^\\S", but those constraints apply only to config loaded through the schema validator. A cronSentinels value constructed/injected programmatically (e.g. in tests or via a future code path) can still contain empty strings. The runtime guard in reward.ts (sentinel.trim().length > 0) correctly filters blank sentinels, so that code path is safe — but the type definition gives no indication of this constraint, making the contract easy to misuse.

Consider either:

  1. Tightening the type to [string, ...string[]] (non-empty tuple) or using a branded type/nominal type to signal the non-blank invariant, or
  2. Adding a brief inline note in the JSDoc that sentinels must be non-blank (as the schema enforces), so future callers don't accidentally pass [""].
💡 Suggested Change

Before:

  cronSentinels?: string[];

After:

  /**
   * ...
   * Each sentinel must be a non-blank string; blank sentinels are silently
   * ignored at runtime but may cause unexpected behaviour if introduced
   * programmatically without going through the schema validator.
   */
  cronSentinels?: [string, ...string[]] | string[];

Generated by cloud-assistant via Open Code Review.

@syzsunshine219

Copy link
Copy Markdown
Collaborator

OCR follow-up in 0e98ecda:

  • Addressed finding 1 by rejecting sentinel prefixes with leading whitespace.
  • Addressed finding 2 by falling back to string-valued meta.initialUserText only when the first materialized user turn is blank. The implementation deliberately does not scan later user turns, so the bypass remains limited to the initial prompt.
  • Kept cronSentinels optional in exported RewardConfig for backward compatibility with existing programmatic callers. The resolved config still always supplies [], and runtime guards remain intentional defense in depth.

Local verification after the follow-up: reward/config tests 103/103, TypeScript lint passed, package build passed.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

✅ Automated Test Results: PASSED

All tests passed (53/53 executed). memos_local_plugin/unit: 53/53. Duration: 5s

Branch: pr/cron-episode-scoring-bypass

@Memtensor-AI Memtensor-AI added status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发 and removed status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Aug 6, 2026
@Hun-ger

Hun-ger commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the contribution. The default minExchangesForCompletion on main is now 1, so single-exchange scheduled tasks are no longer skipped in the standard configuration. The remaining use case only applies to deployments that explicitly configure a stricter threshold, and we do not want to identify scheduled runs using user-text prefixes. If we support this distinction later, it should use structured adapter metadata. Closing this PR as no longer needed in its current form.

@Hun-ger Hun-ger closed this Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:plugin OpenClaw & Hermes status:ready Ready for implementation; waiting for assignee or AI dispatch | 可进入实现,等待认领或派发

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants