Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
7a29587
feat(skill): Engage routing when a linter is named
thecodedrift Jun 11, 2026
1ecf2a5
test(cli): Add route honesty eval fixtures and finalize change
thecodedrift Jun 11, 2026
aaacd87
Merge branch 'feat/local-rule-help' into feat/local-rule-skill
thecodedrift Jun 12, 2026
f1ca12a
Merge branch 'feat/local-rule-skill' into feat/local-rule-eval
thecodedrift Jun 12, 2026
546f988
fix(skill): Target @taskless/cli in the changeset; align tool lists
thecodedrift Jun 12, 2026
6357302
Merge branch 'feat/local-rule-help' into feat/local-rule-skill
thecodedrift Jun 12, 2026
82c9177
Merge branch 'feat/local-rule-skill' into feat/local-rule-eval
thecodedrift Jun 12, 2026
7d9f4ce
test(cli): Cover every eval trap and fix the fixtures path
thecodedrift Jun 12, 2026
1b9ef65
chore(openspec): Archive local-rule-routing and sync specs
thecodedrift Jun 12, 2026
0da9de5
Merge branch 'feat/local-rule-help' into feat/local-rule-skill
thecodedrift Jun 12, 2026
21d468f
Merge branch 'feat/local-rule-skill' into feat/local-rule-eval
thecodedrift Jun 12, 2026
cb0e2d1
Merge branch 'feat/local-rule-help' into feat/local-rule-skill
thecodedrift Jun 14, 2026
f66fd50
Merge branch 'feat/local-rule-skill' into feat/local-rule-eval
thecodedrift Jun 14, 2026
3d838a9
docs(cli): Sync cli-detect spec to new shape; document test layers
thecodedrift Jun 14, 2026
7e0bedd
Merge branch 'feat/local-rule-help' into feat/local-rule-skill
thecodedrift Jun 14, 2026
4c196aa
Merge branch 'feat/local-rule-skill' into feat/local-rule-eval
thecodedrift Jun 14, 2026
d386942
Merge branch 'feat/local-rule-help' into feat/local-rule-skill
thecodedrift Jun 14, 2026
116190a
Merge branch 'feat/local-rule-skill' into feat/local-rule-eval
thecodedrift Jun 14, 2026
d30341c
chore(cli): Anchor the telemetry-taxonomy reconciliation in code
thecodedrift Jun 14, 2026
26c7ced
chore(cli): Point the telemetry-taxonomy TODOs at issue #39
thecodedrift Jun 14, 2026
924bfa9
Merge branch 'feat/local-rule-help' into feat/local-rule-skill
thecodedrift Jun 14, 2026
9a832c5
Merge branch 'feat/local-rule-skill' into feat/local-rule-eval
thecodedrift Jun 14, 2026
d60e74f
chore(cli): Resolve #39 — conform detect/help to the cli_ taxonomy
thecodedrift Jun 14, 2026
6792837
Merge branch 'feat/local-rule-help' into feat/local-rule-skill
thecodedrift Jun 14, 2026
cbb8259
Merge branch 'feat/local-rule-skill' into feat/local-rule-eval
thecodedrift Jun 14, 2026
e4f7868
Merge pull request #28 from taskless/feat/local-rule-eval
thecodedrift Jun 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .changeset/local-rule-routing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
"@taskless/cli": minor
---
Comment thread
thecodedrift marked this conversation as resolved.

Add a local-first rule-routing layer. A new deterministic `taskless detect`
command plus `route`/`existing`/`static`/`remote` recipes let the agent author
rules in an existing linter or as a local ast-grep rule on-device, only
escalating to the login-gated service (with confirmation) when a rule cannot be
built locally. The skill now engages this routing flow when a user names a
linter instead of suppressing itself.
Original file line number Diff line number Diff line change
Expand Up @@ -25,15 +25,15 @@

## 4. Skill routing posture (skill-taskless)

- [ ] 4.1 Update `skills/taskless/SKILL.md` `description`: replace the named-tool suppression clause so naming a linter engages routing via `taskless help route`; tighten (reword shorter) rather than append trigger text
- [ ] 4.1a Measure the resulting `description` length and assert it is ≤ 1024 chars (Agent Skills ceiling); treat overflow as a blocking failure and trim trigger wording until it fits
- [ ] 4.2 Update the skill body to route authoring requests through `taskless help route` (not `rule create` directly); remove the "quiet suggestion" suppression path; keep the skill a thin router with no linter knowledge
- [ ] 4.3 Bump the skill `metadata.version` per the file conventions
- [ ] 4.4 Verify the skill change against the updated `skill-taskless` scenarios (routing on named tool, no suppression wording, local-first before login)
- [x] 4.1 Update `skills/taskless/SKILL.md` `description`: replace the named-tool suppression clause so naming a linter engages routing via `taskless help route`; tighten (reword shorter) rather than append trigger text
- [x] 4.1a Measure the resulting `description` length and assert it is ≤ 1024 chars (Agent Skills ceiling); treat overflow as a blocking failure and trim trigger wording until it fits — measured 835 chars
- [x] 4.2 Update the skill body to route authoring requests through `taskless help route` (not `rule create` directly); remove the "quiet suggestion" suppression path; keep the skill a thin router with no linter knowledge
- [x] 4.3 Bump the skill `metadata.version` per the file conventions — skill version is build-locked to the `@taskless/cli` package version (assertSkillVersions), which `scripts/sync-skill-versions.ts` treats as the source of truth; added a `minor` changeset targeting `@taskless/cli` (the root `@taskless/skills` package is private/unpublished) so the release path bumps the CLI version and syncs the skill version in lockstep, rather than hand-editing a mismatch
- [x] 4.4 Verify the skill change against the updated `skill-taskless` scenarios (routing on named tool, no suppression wording, local-first before login)

## 5. Validation + quality gate

- [ ] 5.1 Run `pnpm openspec validate local-rule-routing` and resolve any issues
- [ ] 5.2 Add/curate the honesty eval fixtures (labeled request → expected route) and assert the `route` heuristic against both failure directions: under-confident (escalating a locally-solvable request to login) and over-confident (claiming local for a request that needs the service). Use the fixtures to calibrate the "confident enough for local" threshold
- [ ] 5.3 Run `pnpm typecheck` and `pnpm lint`; fix all failures
- [ ] 5.4 Manual smoke: `taskless detect --json`, then `taskless help route`/`existing`/`static`/`remote` resolve and read coherently end-to-end
- [x] 5.1 Run `pnpm openspec validate local-rule-routing` and resolve any issues
- [x] 5.2 Add/curate the honesty eval fixtures (labeled request → expected route) and assert the `route` heuristic against both failure directions: under-confident (escalating a locally-solvable request to login) and over-confident (claiming local for a request that needs the service). Use the fixtures to calibrate the "confident enough for local" threshold — dataset at `packages/cli/test/fixtures/route-eval.json` with a coverage test (`packages/cli/test/route-eval.test.ts`); route decision is agent-made (recipe-followed), so the test guards dataset balance across routes + every declared trap rather than running a code classifier
- [x] 5.3 Run `pnpm typecheck` and `pnpm lint`; fix all failures
- [x] 5.4 Manual smoke: `taskless detect --json`, then `taskless help route`/`existing`/`static`/`remote` resolve and read coherently end-to-end
100 changes: 100 additions & 0 deletions openspec/specs/cli-detect/spec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# cli-detect Specification

## Purpose

TBD - created by archiving change local-rule-routing. Update Purpose after archive.

## Requirements

### Requirement: Detect subcommand exists

The CLI SHALL provide a `taskless detect` subcommand registered in the top-level
command list, with a `--json` flag and the standard `--dir`/`-d` working-directory
flag.

#### Scenario: Detect is registered

- **WHEN** `taskless detect --help` is run
- **THEN** the command SHALL be recognized and print its usage
- **AND** the command SHALL accept `--json` and `--dir`/`-d`

### Requirement: Detect scans deterministic repo signals only

The `detect` command SHALL emit only deterministic signals derived from files on
disk: configured linters, detected languages, and the styles of the repo's own
existing rules. It SHALL NOT perform any LLM inference and SHALL NOT match the
request against any catalog of known packaged linter rules.

Detection follows a languages → linters flow: languages are inferred first, and
a linter's dependency evidence is then read from the manifest of that linter's
own language (a node dependency from `package.json`, a Python dependency from
`pyproject.toml`/`requirements.txt`) rather than conflating ecosystems. A
recognized linter config file on disk is honored regardless of the languages
inferred.

#### Scenario: Linter configs are detected from disk

- **WHEN** the working directory contains a recognized linter config (for
example `.eslintrc*`, `eslint.config.js`, `ruff.toml`, a `[tool.ruff]` block in
`pyproject.toml`, `.rubocop.yml`, `biome.json`, or `stylelint` config)
- **THEN** `detect --json` SHALL report each configured linter it found

#### Scenario: Languages are reported

- **WHEN** `detect --json` runs in a repository
- **THEN** the output SHALL include the languages inferred from manifest and
marker files present on disk and from the linters detected

#### Scenario: A linter dependency is sourced from its own language's manifest

- **WHEN** a dependency-evidenced linter (for example `ruff`) is named only in a
manifest belonging to a different language (for example `package.json`)
- **THEN** `detect --json` SHALL NOT report that linter from the mismatched
manifest

#### Scenario: Configs in monorepo sub-packages are detected

- **WHEN** a linter config or language manifest lives in a sub-package rather
than the repository root (for example `packages/api/.eslintrc.json`)
- **THEN** `detect --json` SHALL detect it and SHALL carry the path it was found
at in the linter's evidence
- **AND** the scan SHALL prune a curated set of ignored directories (for example
`node_modules`, `.git`, build output) and SHALL bound traversal depth

#### Scenario: The repo's own rule styles are surfaced

- **WHEN** the working directory contains existing rule definitions (for example
custom linter rules or `.taskless/rules/`)
- **THEN** `detect --json` SHALL surface a description of those existing rule
styles for downstream authoring

#### Scenario: No packaged-rule catalog matching

- **WHEN** `detect --json` runs
- **THEN** the output SHALL NOT claim a request maps to a specific named packaged
rule (such matching is left to the authoring recipe, not the command)

### Requirement: Detect runs offline with no network or auth

The `detect` command SHALL complete without network access and without
authentication.

#### Scenario: Detect works without login or network

- **WHEN** `detect --json` runs while logged out and offline
- **THEN** it SHALL produce its signal output successfully
- **AND** it SHALL NOT require or prompt for authentication

### Requirement: Detect emits a stable JSON shape

When `--json` is set, `detect` SHALL emit a single structured JSON object whose
shape is validated internally against a stable Zod output schema before being
printed, consistent with how other `--json` commands in the CLI (e.g. `info`,
`check`) validate their output. The schema is an internal contract, not a
published artifact, and `detect` does not expose a `--schema` mode.

#### Scenario: JSON output validates against the internal schema

- **WHEN** `detect --json` succeeds
- **THEN** stdout SHALL be a single JSON object that the command has validated
against its internal output schema (linters, languages, existing rule styles)
31 changes: 31 additions & 0 deletions openspec/specs/cli-help/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,37 @@ The help command's existing intent-telemetry requirement SHALL extend naturally
- **WHEN** an agent runs `taskless help onboard`
- **THEN** PostHog SHALL receive a `help_onboard` event

### Requirement: Routing topics are registered in the help system

The help system SHALL register the routing recipes `route`, `existing`, `static`,
and `remote` as embedded help topics, retrievable via `taskless help <topic>` and
listed in the help index, consistent with the existing topic embedding and format
requirements.

#### Scenario: Routing topics resolve

- **WHEN** `taskless help route`, `taskless help existing`,
`taskless help static`, or `taskless help remote` is run
- **THEN** the corresponding recipe text SHALL be returned
- **AND** an unknown-topic error SHALL NOT be raised for any of the four

#### Scenario: Routing topics appear in the index

- **WHEN** `taskless help` (no arguments) is run
- **THEN** the topic index SHALL include the routing topics so an agent can
discover the authoring front door

### Requirement: Routing topics emit intent telemetry

Fetching a routing recipe SHALL emit a per-topic intent telemetry event,
consistent with the existing `help_<topic>` telemetry convention.

#### Scenario: Help topic intent is captured for routing recipes

- **WHEN** the agent fetches `route`, `existing`, `static`, or `remote`
- **THEN** the help command SHALL capture the corresponding `help_<topic>` intent
event with the topic name

## Goal

<one paragraph stating what this recipe accomplishes>
Expand Down
209 changes: 209 additions & 0 deletions openspec/specs/cli-rule-routing/spec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,209 @@
# cli-rule-routing Specification

## Purpose

TBD - created by archiving change local-rule-routing. Update Purpose after archive.

## Requirements

### Requirement: Route is the local authoring classifier

The CLI SHALL provide a `route` help recipe that instructs the agent to classify
a rule-authoring request into one of three destinations — `existing`, `static`,
or `remote` — using `taskless detect --json` signals plus the user's intent. The
`route` recipe SHALL be biased to stay local: it SHALL prefer `existing` or
`static` and SHALL treat `remote` as the escalation of last resort.

#### Scenario: Route fetches detection before classifying

- **WHEN** the agent fetches the `route` recipe to author a rule
- **THEN** the recipe SHALL direct the agent to run `taskless detect --json` and
use its signals as input to the classification

#### Scenario: Route classifies into one of three destinations

- **WHEN** the agent follows `route`
- **THEN** it SHALL select exactly one of `existing`, `static`, or `remote`
- **AND** it SHALL fetch the corresponding recipe to perform the authoring

### Requirement: Route states reasoning before naming a destination

The `route` recipe SHALL require the agent to write an explicit rationale before
naming a destination. The rationale SHALL cover what the `detect` signals show,
whether an existing linter plausibly already covers the request, whether the
pattern is expressible as a simple static ast-grep rule, and the resulting
confidence that the request is locally solvable. The destination SHALL be emitted
only after this rationale, and SHALL follow from it.

#### Scenario: Rationale precedes the route decision

- **WHEN** the agent follows `route` to classify a request
- **THEN** it SHALL produce a written rationale covering the detection signals,
existing-linter coverage, ast-grep expressibility, and local-solvability
confidence
- **AND** it SHALL name the destination (`existing`, `static`, or `remote`) only
after that rationale

#### Scenario: Route is not named before reasoning

- **WHEN** the agent has not yet articulated its reasoning
- **THEN** the recipe SHALL NOT permit committing to a destination
- **AND** the destination SHALL be a conclusion of the rationale, not asserted
ahead of it

### Requirement: Route commits to the believed-correct path on reasonable confidence

The `route` recipe SHALL determine the destination upfront from `detect` signals
and the user's intent, committing to the path it believes is correct. The bar to
commit to a local path SHALL be **reasonable confidence**, not certainty. Routing
distinguishes three states: reasonable confidence the request IS locally solvable
selects a local path; reasonable belief the request is NOT locally solvable selects
`remote` directly, without first attempting a local rule; genuine inability to
judge either way is uncertainty, which SHALL be resolved by asking the user (see
the clarifying-question scenario) and SHALL NOT by itself select `remote`. The
recipe SHALL NOT use a deliberate local attempt-and-fail with no genuine belief of
success as the mechanism for choosing `remote`.

#### Scenario: Reasonably-confident-local commits locally without a justification probe

- **WHEN** `route` is reasonably confident the request fits an existing linter or
a simple static ast-grep pattern
- **THEN** it SHALL select `existing` or `static` and proceed locally
- **AND** it SHALL NOT run a throwaway local attempt whose only purpose is to
justify the choice

#### Scenario: Believed-not-local routes remote upfront

- **WHEN** `route` reasonably believes the request cannot be solved locally — a
positive judgment, not mere inability to tell
- **THEN** it SHALL select `remote` directly
- **AND** it SHALL NOT manufacture a deliberate local failure to reach that
decision

#### Scenario: Uncertainty biases toward asking, not toward login

- **WHEN** `route` cannot reasonably place a request as local or remote
- **THEN** it SHALL prefer clarifying with the user over defaulting to `remote`
- **AND** uncertainty alone SHALL NOT be treated as a reason to consume a
generation via `remote`

### Requirement: A believed-local path that fails escalates only after confirmation

The `route` recipe SHALL treat try-verify-escalate as a legitimate failure
fallback: when it committed to a local path on reasonable confidence and the
authored rule then fails verification against the user's success/failure cases, it
SHALL surface the failure and SHALL obtain explicit user confirmation before
calling the Taskless service. The recipe SHALL NOT silently fall through from a
failed local attempt to a service call.

#### Scenario: Failed local attempt prompts before spending a generation

- **WHEN** a `static` rule the agent committed to fails verification
- **THEN** the recipe SHALL inform the user the local rule could not capture the
cases
- **AND** SHALL state that generating via the Taskless service uses a generation
and requires login
- **AND** SHALL call the service only after the user confirms

#### Scenario: No silent fall-through to the service

- **WHEN** a believed-local attempt fails
- **THEN** the recipe SHALL NOT invoke `remote` / the service without an explicit
user confirmation step

### Requirement: Route asks the user when multiple paths fit

The `route` recipe SHALL present the viable options to the user with their
trade-offs, rather than silently selecting one, whenever more than one destination
genuinely fits a request (most commonly both `existing` and `static`). The
trade-off framing SHALL note that `remote` consumes a generation and requires
login, so it is appropriate when a request cannot be solved locally rather than as
a default.

#### Scenario: Both local paths viable surfaces a choice

- **WHEN** the repository has a detected linter that fits AND the pattern is a
clean local static ast-grep rule
- **THEN** `route` SHALL present both `existing` and `static` with their
trade-offs and let the user choose

#### Scenario: Trade-off framing names the generation cost of remote

- **WHEN** `route` presents options that include `remote`
- **THEN** it SHALL state that `remote` consumes a generation and requires login
- **AND** SHALL frame `remote` as the path for what cannot be solved locally

### Requirement: Existing recipe authors in the detected linter's dialect

The CLI SHALL provide an `existing` help recipe that instructs the agent to
author a rule in a linter already detected in the repository, expressed in that
tool's own dialect. The recipe SHALL direct the agent to source authoring
knowledge first from the repository's own existing rules and only then from the
agent's own web research. The recipe SHALL NOT embed or rely on a Taskless-
maintained catalog of linter rules.

#### Scenario: Repo-first knowledge sourcing

- **WHEN** the agent follows `existing` for a detected linter
- **THEN** it SHALL first mine the repository's existing rules of that kind for
house style
- **AND** SHALL fall back to web research (WebFetch/WebSearch) only when the
repository signal is insufficient

#### Scenario: Existing path is author-only

- **WHEN** the agent authors a rule via `existing`
- **THEN** the recipe SHALL make clear the user's own toolchain runs the rule and
that `taskless check` does not execute the external linter

### Requirement: Static recipe authors a verified local ast-grep rule

The CLI SHALL provide a `static` help recipe that instructs the agent to author a
local ast-grep rule on-device, without calling the Taskless service, and to
verify it against the user's success and failure cases before reporting success.
The recipe SHALL produce the canonical on-disk rule shape and paths used by remote
generation so that `check`, `improve`, and `verify` see a single dialect.

#### Scenario: Local authoring without the service

- **WHEN** the agent follows `static`
- **THEN** it SHALL write the rule on-device without requiring login or the
Taskless API

#### Scenario: Verification gates success

- **WHEN** the agent authors a static rule
- **THEN** it SHALL verify the rule against the provided success/failure cases
before reporting the rule as complete

#### Scenario: Canonical output shape

- **WHEN** the agent writes a static rule to disk
- **THEN** the files, paths, and shape SHALL match those produced by remote
generation

### Requirement: Remote recipe collects inputs and delegates to the service

The CLI SHALL provide a `remote` help recipe that instructs the agent to gather
the inputs required to call the Taskless service and to invoke the existing rule
generation backend, which runs the service-side classifier and returns either a
static or a runtime rule. The `remote` recipe SHALL require authentication and
SHALL NOT itself decide static versus runtime.

#### Scenario: Remote requires authentication

- **WHEN** the agent follows `remote` while logged out
- **THEN** the recipe SHALL direct the agent to the authentication flow before
submitting the request

#### Scenario: Static-versus-runtime is decided by the service

- **WHEN** the agent submits an authored request via `remote`
- **THEN** the recipe SHALL rely on the service to classify static versus runtime
- **AND** SHALL NOT make that determination locally

#### Scenario: Remote output matches local on-disk shape

- **WHEN** the service returns a generated rule via `remote`
- **THEN** the written files and paths SHALL match the shape produced by the
local `static` path
Loading