test(cli): Add route honesty eval fixtures and finalize change

openspec/changes/local-rule-routing/tasks.md → openspec/changes/archive/2026-06-12-local-rule-routing/tasks.md

@@ -33,7 +33,7 @@
 
 ## 5. Validation + quality gate
 
-- [ ] 5.1 Run `pnpm openspec validate local-rule-routing` and resolve any issues
-- [ ] 5.2 Add/curate the honesty eval fixtures (labeled request → expected route) and assert the `route` heuristic against both failure directions: under-confident (escalating a locally-solvable request to login) and over-confident (claiming local for a request that needs the service). Use the fixtures to calibrate the "confident enough for local" threshold
-- [ ] 5.3 Run `pnpm typecheck` and `pnpm lint`; fix all failures
-- [ ] 5.4 Manual smoke: `taskless detect --json`, then `taskless help route`/`existing`/`static`/`remote` resolve and read coherently end-to-end
+- [x] 5.1 Run `pnpm openspec validate local-rule-routing` and resolve any issues
+- [x] 5.2 Add/curate the honesty eval fixtures (labeled request → expected route) and assert the `route` heuristic against both failure directions: under-confident (escalating a locally-solvable request to login) and over-confident (claiming local for a request that needs the service). Use the fixtures to calibrate the "confident enough for local" threshold — dataset at `packages/cli/test/fixtures/route-eval.json` with a coverage test (`packages/cli/test/route-eval.test.ts`); route decision is agent-made (recipe-followed), so the test guards dataset balance across routes + every declared trap rather than running a code classifier
+- [x] 5.3 Run `pnpm typecheck` and `pnpm lint`; fix all failures
+- [x] 5.4 Manual smoke: `taskless detect --json`, then `taskless help route`/`existing`/`static`/`remote` resolve and read coherently end-to-end

openspec/specs/cli-detect/spec.md

@@ -0,0 +1,100 @@
+# cli-detect Specification
+
+## Purpose
+
+TBD - created by archiving change local-rule-routing. Update Purpose after archive.
+
+## Requirements
+
+### Requirement: Detect subcommand exists
+
+The CLI SHALL provide a `taskless detect` subcommand registered in the top-level
+command list, with a `--json` flag and the standard `--dir`/`-d` working-directory
+flag.
+
+#### Scenario: Detect is registered
+
+- **WHEN** `taskless detect --help` is run
+- **THEN** the command SHALL be recognized and print its usage
+- **AND** the command SHALL accept `--json` and `--dir`/`-d`
+
+### Requirement: Detect scans deterministic repo signals only
+
+The `detect` command SHALL emit only deterministic signals derived from files on
+disk: configured linters, detected languages, and the styles of the repo's own
+existing rules. It SHALL NOT perform any LLM inference and SHALL NOT match the
+request against any catalog of known packaged linter rules.
+
+Detection follows a languages → linters flow: languages are inferred first, and
+a linter's dependency evidence is then read from the manifest of that linter's
+own language (a node dependency from `package.json`, a Python dependency from
+`pyproject.toml`/`requirements.txt`) rather than conflating ecosystems. A
+recognized linter config file on disk is honored regardless of the languages
+inferred.
+
+#### Scenario: Linter configs are detected from disk
+
+- **WHEN** the working directory contains a recognized linter config (for
+  example `.eslintrc*`, `eslint.config.js`, `ruff.toml`, a `[tool.ruff]` block in
+  `pyproject.toml`, `.rubocop.yml`, `biome.json`, or `stylelint` config)
+- **THEN** `detect --json` SHALL report each configured linter it found
+
+#### Scenario: Languages are reported
+
+- **WHEN** `detect --json` runs in a repository
+- **THEN** the output SHALL include the languages inferred from manifest and
+  marker files present on disk and from the linters detected
+
+#### Scenario: A linter dependency is sourced from its own language's manifest
+
+- **WHEN** a dependency-evidenced linter (for example `ruff`) is named only in a
+  manifest belonging to a different language (for example `package.json`)
+- **THEN** `detect --json` SHALL NOT report that linter from the mismatched
+  manifest
+
+#### Scenario: Configs in monorepo sub-packages are detected
+
+- **WHEN** a linter config or language manifest lives in a sub-package rather
+  than the repository root (for example `packages/api/.eslintrc.json`)
+- **THEN** `detect --json` SHALL detect it and SHALL carry the path it was found
+  at in the linter's evidence
+- **AND** the scan SHALL prune a curated set of ignored directories (for example
+  `node_modules`, `.git`, build output) and SHALL bound traversal depth
+
+#### Scenario: The repo's own rule styles are surfaced
+
+- **WHEN** the working directory contains existing rule definitions (for example
+  custom linter rules or `.taskless/rules/`)
+- **THEN** `detect --json` SHALL surface a description of those existing rule
+  styles for downstream authoring
+
+#### Scenario: No packaged-rule catalog matching
+
+- **WHEN** `detect --json` runs
+- **THEN** the output SHALL NOT claim a request maps to a specific named packaged
+  rule (such matching is left to the authoring recipe, not the command)
+
+### Requirement: Detect runs offline with no network or auth
+
+The `detect` command SHALL complete without network access and without
+authentication.
+
+#### Scenario: Detect works without login or network
+
+- **WHEN** `detect --json` runs while logged out and offline
+- **THEN** it SHALL produce its signal output successfully
+- **AND** it SHALL NOT require or prompt for authentication
+
+### Requirement: Detect emits a stable JSON shape
+
+When `--json` is set, `detect` SHALL emit a single structured JSON object whose
+shape is validated internally against a stable Zod output schema before being
+printed, consistent with how other `--json` commands in the CLI (e.g. `info`,
+`check`) validate their output. The schema is an internal contract, not a
+published artifact, and `detect` does not expose a `--schema` mode.
+
+#### Scenario: JSON output validates against the internal schema
+
+- **WHEN** `detect --json` succeeds
+- **THEN** stdout SHALL be a single JSON object that the command has validated
+  against its internal output schema (linters, languages, existing rule styles)

openspec/specs/cli-help/spec.md

@@ -155,6 +155,37 @@ The help command's existing intent-telemetry requirement SHALL extend naturally
 - **WHEN** an agent runs `taskless help onboard`
 - **THEN** PostHog SHALL receive a `help_onboard` event
 
+### Requirement: Routing topics are registered in the help system
+
+The help system SHALL register the routing recipes `route`, `existing`, `static`,
+and `remote` as embedded help topics, retrievable via `taskless help <topic>` and
+listed in the help index, consistent with the existing topic embedding and format
+requirements.
+
+#### Scenario: Routing topics resolve
+
+- **WHEN** `taskless help route`, `taskless help existing`,
+  `taskless help static`, or `taskless help remote` is run
+- **THEN** the corresponding recipe text SHALL be returned
+- **AND** an unknown-topic error SHALL NOT be raised for any of the four
+
+#### Scenario: Routing topics appear in the index
+
+- **WHEN** `taskless help` (no arguments) is run
+- **THEN** the topic index SHALL include the routing topics so an agent can
+  discover the authoring front door
+
+### Requirement: Routing topics emit intent telemetry
+
+Fetching a routing recipe SHALL emit a per-topic intent telemetry event,
+consistent with the existing `help_<topic>` telemetry convention.
+
+#### Scenario: Help topic intent is captured for routing recipes
+
+- **WHEN** the agent fetches `route`, `existing`, `static`, or `remote`
+- **THEN** the help command SHALL capture the corresponding `help_<topic>` intent
+  event with the topic name
+
 ## Goal
 
 <one paragraph stating what this recipe accomplishes>

openspec/specs/cli-rule-routing/spec.md

@@ -0,0 +1,209 @@
+# cli-rule-routing Specification
+
+## Purpose
+
+TBD - created by archiving change local-rule-routing. Update Purpose after archive.
+
+## Requirements
+
+### Requirement: Route is the local authoring classifier
+
+The CLI SHALL provide a `route` help recipe that instructs the agent to classify
+a rule-authoring request into one of three destinations — `existing`, `static`,
+or `remote` — using `taskless detect --json` signals plus the user's intent. The
+`route` recipe SHALL be biased to stay local: it SHALL prefer `existing` or
+`static` and SHALL treat `remote` as the escalation of last resort.
+
+#### Scenario: Route fetches detection before classifying
+
+- **WHEN** the agent fetches the `route` recipe to author a rule
+- **THEN** the recipe SHALL direct the agent to run `taskless detect --json` and
+  use its signals as input to the classification
+
+#### Scenario: Route classifies into one of three destinations
+
+- **WHEN** the agent follows `route`
+- **THEN** it SHALL select exactly one of `existing`, `static`, or `remote`
+- **AND** it SHALL fetch the corresponding recipe to perform the authoring
+
+### Requirement: Route states reasoning before naming a destination
+
+The `route` recipe SHALL require the agent to write an explicit rationale before
+naming a destination. The rationale SHALL cover what the `detect` signals show,
+whether an existing linter plausibly already covers the request, whether the
+pattern is expressible as a simple static ast-grep rule, and the resulting
+confidence that the request is locally solvable. The destination SHALL be emitted
+only after this rationale, and SHALL follow from it.
+
+#### Scenario: Rationale precedes the route decision
+
+- **WHEN** the agent follows `route` to classify a request
+- **THEN** it SHALL produce a written rationale covering the detection signals,
+  existing-linter coverage, ast-grep expressibility, and local-solvability
+  confidence
+- **AND** it SHALL name the destination (`existing`, `static`, or `remote`) only
+  after that rationale
+
+#### Scenario: Route is not named before reasoning
+
+- **WHEN** the agent has not yet articulated its reasoning
+- **THEN** the recipe SHALL NOT permit committing to a destination
+- **AND** the destination SHALL be a conclusion of the rationale, not asserted
+  ahead of it
+
+### Requirement: Route commits to the believed-correct path on reasonable confidence
+
+The `route` recipe SHALL determine the destination upfront from `detect` signals
+and the user's intent, committing to the path it believes is correct. The bar to
+commit to a local path SHALL be **reasonable confidence**, not certainty. Routing
+distinguishes three states: reasonable confidence the request IS locally solvable
+selects a local path; reasonable belief the request is NOT locally solvable selects
+`remote` directly, without first attempting a local rule; genuine inability to
+judge either way is uncertainty, which SHALL be resolved by asking the user (see
+the clarifying-question scenario) and SHALL NOT by itself select `remote`. The
+recipe SHALL NOT use a deliberate local attempt-and-fail with no genuine belief of
+success as the mechanism for choosing `remote`.
+
+#### Scenario: Reasonably-confident-local commits locally without a justification probe
+
+- **WHEN** `route` is reasonably confident the request fits an existing linter or
+  a simple static ast-grep pattern
+- **THEN** it SHALL select `existing` or `static` and proceed locally
+- **AND** it SHALL NOT run a throwaway local attempt whose only purpose is to
+  justify the choice
+
+#### Scenario: Believed-not-local routes remote upfront
+
+- **WHEN** `route` reasonably believes the request cannot be solved locally — a
+  positive judgment, not mere inability to tell
+- **THEN** it SHALL select `remote` directly
+- **AND** it SHALL NOT manufacture a deliberate local failure to reach that
+  decision
+
+#### Scenario: Uncertainty biases toward asking, not toward login
+
+- **WHEN** `route` cannot reasonably place a request as local or remote
+- **THEN** it SHALL prefer clarifying with the user over defaulting to `remote`
+- **AND** uncertainty alone SHALL NOT be treated as a reason to consume a
+  generation via `remote`
+
+### Requirement: A believed-local path that fails escalates only after confirmation
+
+The `route` recipe SHALL treat try-verify-escalate as a legitimate failure
+fallback: when it committed to a local path on reasonable confidence and the
+authored rule then fails verification against the user's success/failure cases, it
+SHALL surface the failure and SHALL obtain explicit user confirmation before
+calling the Taskless service. The recipe SHALL NOT silently fall through from a
+failed local attempt to a service call.
+
+#### Scenario: Failed local attempt prompts before spending a generation
+
+- **WHEN** a `static` rule the agent committed to fails verification
+- **THEN** the recipe SHALL inform the user the local rule could not capture the
+  cases
+- **AND** SHALL state that generating via the Taskless service uses a generation
+  and requires login
+- **AND** SHALL call the service only after the user confirms
+
+#### Scenario: No silent fall-through to the service
+
+- **WHEN** a believed-local attempt fails
+- **THEN** the recipe SHALL NOT invoke `remote` / the service without an explicit
+  user confirmation step
+
+### Requirement: Route asks the user when multiple paths fit
+
+The `route` recipe SHALL present the viable options to the user with their
+trade-offs, rather than silently selecting one, whenever more than one destination
+genuinely fits a request (most commonly both `existing` and `static`). The
+trade-off framing SHALL note that `remote` consumes a generation and requires
+login, so it is appropriate when a request cannot be solved locally rather than as
+a default.
+
+#### Scenario: Both local paths viable surfaces a choice
+
+- **WHEN** the repository has a detected linter that fits AND the pattern is a
+  clean local static ast-grep rule
+- **THEN** `route` SHALL present both `existing` and `static` with their
+  trade-offs and let the user choose
+
+#### Scenario: Trade-off framing names the generation cost of remote
+
+- **WHEN** `route` presents options that include `remote`
+- **THEN** it SHALL state that `remote` consumes a generation and requires login
+- **AND** SHALL frame `remote` as the path for what cannot be solved locally
+
+### Requirement: Existing recipe authors in the detected linter's dialect
+
+The CLI SHALL provide an `existing` help recipe that instructs the agent to
+author a rule in a linter already detected in the repository, expressed in that
+tool's own dialect. The recipe SHALL direct the agent to source authoring
+knowledge first from the repository's own existing rules and only then from the
+agent's own web research. The recipe SHALL NOT embed or rely on a Taskless-
+maintained catalog of linter rules.
+
+#### Scenario: Repo-first knowledge sourcing
+
+- **WHEN** the agent follows `existing` for a detected linter
+- **THEN** it SHALL first mine the repository's existing rules of that kind for
+  house style
+- **AND** SHALL fall back to web research (WebFetch/WebSearch) only when the
+  repository signal is insufficient
+
+#### Scenario: Existing path is author-only
+
+- **WHEN** the agent authors a rule via `existing`
+- **THEN** the recipe SHALL make clear the user's own toolchain runs the rule and
+  that `taskless check` does not execute the external linter
+
+### Requirement: Static recipe authors a verified local ast-grep rule
+
+The CLI SHALL provide a `static` help recipe that instructs the agent to author a
+local ast-grep rule on-device, without calling the Taskless service, and to
+verify it against the user's success and failure cases before reporting success.
+The recipe SHALL produce the canonical on-disk rule shape and paths used by remote
+generation so that `check`, `improve`, and `verify` see a single dialect.
+
+#### Scenario: Local authoring without the service
+
+- **WHEN** the agent follows `static`
+- **THEN** it SHALL write the rule on-device without requiring login or the
+  Taskless API
+
+#### Scenario: Verification gates success
+
+- **WHEN** the agent authors a static rule
+- **THEN** it SHALL verify the rule against the provided success/failure cases
+  before reporting the rule as complete
+
+#### Scenario: Canonical output shape
+
+- **WHEN** the agent writes a static rule to disk
+- **THEN** the files, paths, and shape SHALL match those produced by remote
+  generation
+
+### Requirement: Remote recipe collects inputs and delegates to the service
+
+The CLI SHALL provide a `remote` help recipe that instructs the agent to gather
+the inputs required to call the Taskless service and to invoke the existing rule
+generation backend, which runs the service-side classifier and returns either a
+static or a runtime rule. The `remote` recipe SHALL require authentication and
+SHALL NOT itself decide static versus runtime.
+
+#### Scenario: Remote requires authentication
+
+- **WHEN** the agent follows `remote` while logged out
+- **THEN** the recipe SHALL direct the agent to the authentication flow before
+  submitting the request
+
+#### Scenario: Static-versus-runtime is decided by the service
+
+- **WHEN** the agent submits an authored request via `remote`
+- **THEN** the recipe SHALL rely on the service to classify static versus runtime
+- **AND** SHALL NOT make that determination locally
+
+#### Scenario: Remote output matches local on-disk shape
+
+- **WHEN** the service returns a generated rule via `remote`
+- **THEN** the written files and paths SHALL match the shape produced by the
+  local `static` path