feat: add Hermes Agent as a first-class backend and transcript source

[fgscaglioni]

This adds full support for Nous Research's Hermes Agent across all three SkillOpt layers — core model backend, sleep-cycle backend, and transcript harvesting.

Core model layer (skillopt/model/):

• Register the Hermes CLI as a chat backend (hermes_chat) alongside the existing Claude, Codex, Qwen, and MiniMax backends.
• All chat entry points (optimizer, target, messages, deployment) route through the Hermes adapter.

Sleep-cycle backend (skillopt_sleep/backend.py):

• Add HermesBackend extending CliBackend, driving hermes --profile chat -Q -q for attempt / judge / reflect calls.
• CLI output is filtered to strip notices, warnings, and traceback debris so the pipeline sees only the model response.
• Registered under get_backend("hermes") with aliases hermes_chat and hermes_cli.

Transcript harvesting (skillopt_sleep/harvest_hermes.py):

• New harvest source reads Hermes session data from ~/.hermes/state.db (SQLite), building SessionDigest objects from sessions + messages.
• Skips engine-internal sessions (temp dirs matching skillopt_sleep_*) so the mine step only sees real user sessions.
• Supports the standard harvest contract: scope, since, limit, project.

Config additions (skillopt_sleep/config.py):

• memory_filename — controls the project memory file name. Defaults to "CLAUDE.md" for backward compatibility. Set to "AGENTS.md" for Codex and Hermes.
• hermes_home — path to Hermes state directory, defaults to ~/.hermes, overridable via $HERMES_HOME.

Tests:

• 7 new tests in TestHermesBackendCli covering backend registration, command construction, error capture, output filtering, and env vars.

Backward compatible — defaults unchanged, all existing backends intact.

[fgscaglioni]

@microsoft-github-policy-service agree

[Yif-Yang]

Thanks for the substantial Hermes integration. Supporting another agent/backend can improve SkillOpt, and we would like to preserve the work under your authorship, but the current PR is too broad and has several correctness issues that block merge.

Confirmed issues:

• The branch conflicts with current backend registry files. Please rebase onto the latest main.
• set_*_backend accepts hermes_chat, but is_optimizer_chat_backend() and is_target_chat_backend() still return false.
• The public deployment setters do not dispatch to Hermes, and the Hermes setter changes a deployment variable while calls use profile constants.
• Hermes and Claude share the same global token tracker, then both summaries are aggregated, so usage is double-counted.
• Message APIs discard tools, tool_choice, retries, and return_message; return_message=True still returns a string. This violates the backend contract for tool-using environments.
• Sleep CLI launch exceptions are swallowed without last_call_error, recreating the silent-zero failure fixed for other backends.
• Output beginning with Exception can be a legitimate answer but is filtered to empty.
• Harvesting filters legitimate /tmp projects, caps limit=0 at 200, drops tool-role messages, and never produces feedback signals.
• New tests cover only a mocked Sleep CLI subset, not core routing, setters, token accounting, message/tool contracts, database schema compatibility, or secret handling.

We recommend splitting this into three reviewable PRs: core model backend, Sleep CLI backend, and transcript harvester. Each should include contract tests and one opt-in real Hermes smoke test. Please ensure backend invocations disable unintended tools/plugins/MCP when processing harvested untrusted text.

We are not rejecting the feature direction; we are asking for a rebase and a narrower, test-backed design so it can be merged safely while keeping your contribution attribution.

[Yif-Yang]

Requesting changes based on the backend-contract, routing, token-accounting, error-surfacing, harvesting, and test gaps detailed in the maintainer comment above. Please rebase and narrow or split the implementation before re-review.

[fgscaglioni]

Thanks for the thorough review, Yif-Yang. I've split the implementation into three branches as suggested and will open separate PRs shortly.

[Yif-Yang]

Hi @fgscaglioni — following up. The CLA here is passed, but the branch is currently conflicting, so the original PR remains blocked as-is. You mentioned splitting the work into three separate branches; could you share their status and open the replacement PRs when they are ready? We do not see them yet, and we will be happy to review each focused PR as it arrives.

[Copilot]

Pull request overview

Adds first-class support for Nous Research’s Hermes Agent across SkillOpt’s model backend layer and the SkillOpt-Sleep pipeline, including CLI execution and transcript harvesting.

Changes:

• Introduces a new hermes_chat model backend (skillopt/model/hermes_backend.py) and routes chat entry points through it when selected.
• Adds a Hermes CLI backend for SkillOpt-Sleep plus a new harvest source that reads Hermes transcripts from ~/.hermes/state.db.
• Extends configuration/CLI flags (e.g., memory_filename, Hermes home/source options) and adds tests for Hermes behavior.

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/test_sleep_engine.py	Adds unittest coverage for SkillOpt-Sleep HermesBackend CLI behavior (registration, command flags, filtering, env).
tests/test_hermes_backend.py	Adds contract tests for Hermes model backend routing, message APIs, and token tracking behavior.
skillopt/model/hermes_backend.py	New Hermes CLI-backed chat implementation for core SkillOpt model APIs.
skillopt/model/common.py	Registers hermes_chat default model and normalization aliases.
skillopt/model/backend_config.py	Allows hermes_chat as a valid optimizer/target backend and updates recognition helpers.
skillopt/model/init.py	Routes chat functions and token summary aggregation through Hermes backend when selected.
skillopt_sleep/harvest_sources.py	Adds Hermes as a transcript source and wires it into harvest selection.
skillopt_sleep/harvest_hermes.py	New SQLite-based harvester that builds SessionDigest from Hermes sessions/messages.
skillopt_sleep/cycle.py	Makes the “live memory” filename configurable via memory_filename.
skillopt_sleep/config.py	Adds hermes_home, transcript_source=hermes, and configurable memory_filename.
skillopt_sleep/backend.py	Adds HermesBackend (CLI-driven) and registers aliases in get_backend().
skillopt_sleep/main.py	Exposes Hermes via CLI --backend and --source choices/help.
.gitignore	Ignores .codegraph/ artifacts.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (8)

skillopt_sleep/harvest_hermes.py:195

• harvest_hermes() currently (1) filters out sessions with empty/NULL cwd in SQL, even though _filter_engine_sessions() explicitly keeps no-cwd sessions, and (2) treats limit=0 as "no cap" in the docstring but still caps the SQL query to 200 rows. This can silently drop valid sessions and contradicts the limit contract.

    # Build query with optional since filter
    where = "WHERE cwd IS NOT NULL AND cwd != '' AND ended_at IS NOT NULL"
    params: List[Any] = []

skillopt_sleep/harvest_hermes.py:229

• SessionDigest.raw_path is currently built from the module-level STATE_DB (derived from $HERMES_HOME at import time), but harvest_for_config() can pass an overridden db_path. That makes raw_path misleading when cfg["hermes_home"] differs from the environment/default.

        if digest is None:
            continue
        digests.append(digest)
        if limit and len(digests) >= limit:

skillopt/model/hermes_backend.py:42

• _call_hermes()'s docstring says it sets last_call_error on the module, but there is no such variable in this module. This is misleading for callers/debugging.

    """Call hermes CLI and return (response_text, token_info).

    Retries on non-zero exit with exponential backoff.
    Sets ``last_call_error`` on the module on persistent failure.
    """

tests/test_sleep_engine.py:2388

• This test name refers to "hermes_home", but it checks the default profile/bin fallback when env overrides are absent. Renaming helps align the test output with what is being asserted.

    def test_hermes_home_falls_back_to_default(self):
        """Without env overrides, hermes_profile defaults to HERMES_TARGET_PROFILE or 'default'."""

skillopt_sleep/backend.py:1898

• HermesBackend._call() increments self._tokens, but CliBackend._cached_call() and CliBackend.reflect() already account for tokens around _call(). This will double-count token usage for the Hermes backend.

        result = "\n".join(body).strip()
        self._tokens += len(prompt) // 4 + len(result) // 4
        return result

skillopt/model/hermes_backend.py:7

• The module docstring is partly in Portuguese, while the other backends use English docstrings (e.g., skillopt/model/claude_backend.py:1, skillopt/model/qwen_backend.py:1). For consistency and to keep the public backend docs accessible, please translate this docstring to English.

"""Hermes CLI chat backend for SkillOpt.

Chama `hermes --profile <name> chat -q "<prompt>"` como target/optimizer.
Possui token tracker próprio (separado do Claude/OpenAI) para evitar
double-count. Profiles são mutáveis via set_target_deployment /
set_optimizer_deployment.
"""

skillopt/model/hermes_backend.py:60

• The local elapsed variable is computed but never used.

        elapsed = time.time() - t0

tests/test_sleep_engine.py:2375

• This test name refers to "hermes_home", but it actually verifies env overrides for HERMES_BIN and SKILLOPT_SLEEP_HERMES_PROFILE. Renaming it would make intent clearer when scanning test output.

This issue also appears on line 2387 of the same file.

    def test_hermes_home_overridable(self):
        """SKILLOPT_SLEEP_HERMES_PROFILE and HERMES_BIN are read from env."""

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (11)

skillopt/model/init.py:497

• Same issue as chat_messages_with_deployment: chat_with_deployment routes to Hermes when only one role is hermes_chat, which can silently ignore a non-Hermes backend’s deployment semantics in mixed configurations. Prefer routing only when both backends are Hermes, otherwise raise to avoid ambiguous behavior.

    if get_optimizer_backend() == "hermes_chat" or get_target_backend() == "hermes_chat":
        # TODO: Same limitation as chat_messages_with_deployment — dispatches to
        # Hermes if EITHER role is hermes_chat. A proper fix needs deployment
        # role awareness.
        return _hermes.chat_with_deployment(

skillopt/model/hermes_backend.py:292

• set_optimizer_deployment has the same fallback issue as set_target_deployment: it uses default_model_for_backend("hermes") ("hermes") as a profile name when deployment is falsy, instead of reverting to the default profile ("default").

    global _optimizer_profile
    _optimizer_profile = deployment or default_model_for_backend("hermes")
    os.environ["HERMES_OPTIMIZER_PROFILE"] = _optimizer_profile

skillopt_sleep/cycle.py:289

• The Hermes-specific default memory filename logic doesn’t do what the comment says when using the default config. memory_filename is always present in the default config ("CLAUDE.md"), so cfg.get("memory_filename", default_memory) returns "CLAUDE.md" even when transcript_source == "hermes", and the code never switches to "AGENTS.md".

    # When transcript_source is "hermes", default to AGENTS.md instead of
    # CLAUDE.md unless the user explicitly set memory_filename.
    default_memory = "CLAUDE.md"
    if cfg.get("transcript_source") == "hermes":
        default_memory = "AGENTS.md"
    memory_filename = cfg.get("memory_filename", default_memory)
    if memory_filename == default_memory and cfg.get("transcript_source") == "hermes":
        # User did not explicitly override; use Hermes convention
        memory_filename = "AGENTS.md"
    live_memory_path = os.path.join(project, memory_filename)

skillopt_sleep/harvest_hermes.py:220

• limit is applied in SQL before filtering out engine sessions and before scope/project filtering, so harvest_hermes(..., limit=N) can return fewer than N digests even when more qualifying sessions exist. Other harvesters apply limit after filtering.

    limit_clause = ""
    cap = int(limit or 0)
    if cap > 0:
        limit_clause = " LIMIT ?"
        params.append(cap)

    cursor.execute(
        f"""SELECT id, cwd, title, started_at, ended_at, model
            FROM sessions
            {where}
            ORDER BY ended_at DESC{limit_clause}""",
        params,
    )

skillopt_sleep/harvest_hermes.py:37

• The /tmp/ heuristic in _filter_engine_sessions will drop legitimate user sessions whose cwd is under /tmp (common for scratch repos/containers). This can silently exclude real transcripts.

        elif cwd.startswith("/tmp/") and len(cwd.split("/", 3)) <= 4:
            # Very short-lived temp sessions; likely programmatic
            continue

skillopt/model/init.py:468

• chat_messages_with_deployment dispatches to Hermes when either optimizer or target backend is hermes_chat. In mixed-backend configurations this can route a deployment-scoped call to the wrong backend silently (the TODO notes this). It’s safer to only route to Hermes when both roles are Hermes, and otherwise fail fast instead of making an ambiguous choice.

This issue also appears on line 493 of the same file.

    if get_optimizer_backend() == "hermes_chat" or get_target_backend() == "hermes_chat":
        # TODO: This dispatches to Hermes if EITHER optimizer OR target is
        # hermes_chat, which breaks in a dual-backend scenario (e.g.
        # optimizer=openai_chat, target=hermes_chat). The function receives
        # ``deployment`` but doesn't know which role it applies to. A proper
        # fix would route based on the deployment's role, or add a ``role``
        # parameter. For now the ``or`` condition is conservative (Hermes
        # handles both) but may need revisiting for dual-backend setups.
        return _hermes.chat_messages_with_deployment(

skillopt/model/hermes_backend.py:282

• set_target_deployment falls back to default_model_for_backend("hermes") ("hermes") when deployment is falsy. This function is setting a profile name (default elsewhere is "default" via HERMES_TARGET_PROFILE), so resetting with ""/None can unexpectedly switch the CLI profile to "hermes".

This issue also appears on line 290 of the same file.

    global _target_profile
    _target_profile = deployment or default_model_for_backend("hermes")
    os.environ["HERMES_TARGET_PROFILE"] = _target_profile

skillopt/model/hermes_backend.py:27

• New comments are in Portuguese while the surrounding codebase appears to use English comments/docstrings (e.g. skillopt/model/claude_backend.py:1-25). This makes the module harder to maintain for most contributors.

# Profiles mutáveis — setters alteram estas variáveis
_target_profile: str = os.environ.get("HERMES_TARGET_PROFILE", "default")
_optimizer_profile: str = os.environ.get("HERMES_OPTIMIZER_PROFILE", "default")

# Token tracker próprio — não usa o global de common.py
_hermes_tracker = TokenTracker()

tests/test_hermes_backend.py:13

• Typo in docstring: "ptest" → "pytest".

Follows the same ptest + monkeypatch pattern as test_qwen_backend.py.

skillopt_sleep/backend.py:1822

• The HermesBackend docstring documents chat -q but the implementation uses chat -Q -q. Keeping the docstring in sync helps avoid confusion when debugging CLI invocation.

class HermesBackend(CliBackend):
    """Drives Hermes Agent CLI: `hermes --profile <name> chat -q "<prompt>"`."""

skillopt/model/hermes_backend.py:49

• t0 = time.time() is assigned but never used in _call_hermes, which is dead code and can trip linters.

    for attempt in range(retries):
        t0 = time.time()
        try:

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (5)

skillopt/model/init.py:494

• Same issue as chat_messages_with_deployment above: this routes to Hermes only when both backends are hermes_chat and otherwise silently falls back to OpenAI. If deployment is interpreted as a Hermes profile, mixed-backend calls should fail fast instead of using OpenAI with a likely-invalid deployment name.

    if get_optimizer_backend() == "hermes_chat" and get_target_backend() == "hermes_chat":
        # Route to Hermes only when BOTH backends are hermes_chat.  Same
        # rationale as chat_messages_with_deployment above.
        return _hermes.chat_with_deployment(

skillopt_sleep/cycle.py:286

• The new memory_filename setting is never read here: memory_filename is always set to default_memory, so users cannot override the memory file name via config despite the comment saying they can. This also only defaults to AGENTS.md for Hermes, not Codex as described in the config comment.

    default_memory = "CLAUDE.md"
    if cfg.get("transcript_source") == "hermes":
        default_memory = "AGENTS.md"
    memory_filename = default_memory
    live_memory_path = os.path.join(project, memory_filename)

skillopt/model/hermes_backend.py:18

• default_model_for_backend is imported but never used in this module, which will trigger unused-import tooling (and adds noise for readers).

from skillopt.model.common import (
    CompatAssistantMessage,
    TokenTracker,
    default_model_for_backend,
)

skillopt/model/init.py:466

• This deployment helper only routes to Hermes when both optimizer and target backends are hermes_chat; in a mixed-backend configuration it silently routes to OpenAI. That contradicts the PR description (“all chat entry points … route through the Hermes adapter”) and is likely to surprise callers passing a Hermes profile as deployment. Consider failing fast when either backend is Hermes but both are not, so misconfiguration is surfaced immediately.

This issue also appears on line 491 of the same file.

    if get_optimizer_backend() == "hermes_chat" and get_target_backend() == "hermes_chat":
        # Route to Hermes only when BOTH backends are hermes_chat.  When only
        # one side is hermes_chat (dual-backend scenario) the function routes
        # to OpenAI, which handles both sides via the generic OpenAI backend.
        # A deployment-level ``role`` parameter would be cleaner but requires

skillopt_sleep/harvest_hermes.py:159

• scope is documented as allowing a list of project paths, and the rest of SkillOpt-Sleep supports that (see skillopt_sleep/harvest.py:_project_matches). This Hermes implementation ignores list/tuple scopes and will effectively fall back to invoked-project matching, so config like projects: ["/a", "/b"] won’t filter Hermes sessions correctly.

def _project_matches(project: str, scope: str, invoked: str) -> bool:
    """Check whether ``project`` matches the scope."""
    if not invoked or scope == "all":
        return True
    if not project:
        return True  # no cwd → can't filter, accept
    a = os.path.abspath(project)
    b = os.path.abspath(invoked)
    return a == b or a.startswith(b + os.sep) or b.startswith(a + os.sep)