-
Notifications
You must be signed in to change notification settings - Fork 0
feat(skill): Engage routing when a linter is named #27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
26 commits
Select commit
Hold shift + click to select a range
7a29587
feat(skill): Engage routing when a linter is named
thecodedrift 1ecf2a5
test(cli): Add route honesty eval fixtures and finalize change
thecodedrift aaacd87
Merge branch 'feat/local-rule-help' into feat/local-rule-skill
thecodedrift f1ca12a
Merge branch 'feat/local-rule-skill' into feat/local-rule-eval
thecodedrift 546f988
fix(skill): Target @taskless/cli in the changeset; align tool lists
thecodedrift 6357302
Merge branch 'feat/local-rule-help' into feat/local-rule-skill
thecodedrift 82c9177
Merge branch 'feat/local-rule-skill' into feat/local-rule-eval
thecodedrift 7d9f4ce
test(cli): Cover every eval trap and fix the fixtures path
thecodedrift 1b9ef65
chore(openspec): Archive local-rule-routing and sync specs
thecodedrift 0da9de5
Merge branch 'feat/local-rule-help' into feat/local-rule-skill
thecodedrift 21d468f
Merge branch 'feat/local-rule-skill' into feat/local-rule-eval
thecodedrift cb0e2d1
Merge branch 'feat/local-rule-help' into feat/local-rule-skill
thecodedrift f66fd50
Merge branch 'feat/local-rule-skill' into feat/local-rule-eval
thecodedrift 3d838a9
docs(cli): Sync cli-detect spec to new shape; document test layers
thecodedrift 7e0bedd
Merge branch 'feat/local-rule-help' into feat/local-rule-skill
thecodedrift 4c196aa
Merge branch 'feat/local-rule-skill' into feat/local-rule-eval
thecodedrift d386942
Merge branch 'feat/local-rule-help' into feat/local-rule-skill
thecodedrift 116190a
Merge branch 'feat/local-rule-skill' into feat/local-rule-eval
thecodedrift d30341c
chore(cli): Anchor the telemetry-taxonomy reconciliation in code
thecodedrift 26c7ced
chore(cli): Point the telemetry-taxonomy TODOs at issue #39
thecodedrift 924bfa9
Merge branch 'feat/local-rule-help' into feat/local-rule-skill
thecodedrift 9a832c5
Merge branch 'feat/local-rule-skill' into feat/local-rule-eval
thecodedrift d60e74f
chore(cli): Resolve #39 — conform detect/help to the cli_ taxonomy
thecodedrift 6792837
Merge branch 'feat/local-rule-help' into feat/local-rule-skill
thecodedrift cbb8259
Merge branch 'feat/local-rule-skill' into feat/local-rule-eval
thecodedrift e4f7868
Merge pull request #28 from taskless/feat/local-rule-eval
thecodedrift File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| --- | ||
| "@taskless/cli": minor | ||
| --- | ||
|
|
||
| Add a local-first rule-routing layer. A new deterministic `taskless detect` | ||
| command plus `route`/`existing`/`static`/`remote` recipes let the agent author | ||
| rules in an existing linter or as a local ast-grep rule on-device, only | ||
| escalating to the login-gated service (with confirmation) when a rule cannot be | ||
| built locally. The skill now engages this routing flow when a user names a | ||
| linter instead of suppressing itself. | ||
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| # cli-detect Specification | ||
|
|
||
| ## Purpose | ||
|
|
||
| TBD - created by archiving change local-rule-routing. Update Purpose after archive. | ||
|
|
||
| ## Requirements | ||
|
|
||
| ### Requirement: Detect subcommand exists | ||
|
|
||
| The CLI SHALL provide a `taskless detect` subcommand registered in the top-level | ||
| command list, with a `--json` flag and the standard `--dir`/`-d` working-directory | ||
| flag. | ||
|
|
||
| #### Scenario: Detect is registered | ||
|
|
||
| - **WHEN** `taskless detect --help` is run | ||
| - **THEN** the command SHALL be recognized and print its usage | ||
| - **AND** the command SHALL accept `--json` and `--dir`/`-d` | ||
|
|
||
| ### Requirement: Detect scans deterministic repo signals only | ||
|
|
||
| The `detect` command SHALL emit only deterministic signals derived from files on | ||
| disk: configured linters, detected languages, and the styles of the repo's own | ||
| existing rules. It SHALL NOT perform any LLM inference and SHALL NOT match the | ||
| request against any catalog of known packaged linter rules. | ||
|
|
||
| Detection follows a languages → linters flow: languages are inferred first, and | ||
| a linter's dependency evidence is then read from the manifest of that linter's | ||
| own language (a node dependency from `package.json`, a Python dependency from | ||
| `pyproject.toml`/`requirements.txt`) rather than conflating ecosystems. A | ||
| recognized linter config file on disk is honored regardless of the languages | ||
| inferred. | ||
|
|
||
| #### Scenario: Linter configs are detected from disk | ||
|
|
||
| - **WHEN** the working directory contains a recognized linter config (for | ||
| example `.eslintrc*`, `eslint.config.js`, `ruff.toml`, a `[tool.ruff]` block in | ||
| `pyproject.toml`, `.rubocop.yml`, `biome.json`, or `stylelint` config) | ||
| - **THEN** `detect --json` SHALL report each configured linter it found | ||
|
|
||
| #### Scenario: Languages are reported | ||
|
|
||
| - **WHEN** `detect --json` runs in a repository | ||
| - **THEN** the output SHALL include the languages inferred from manifest and | ||
| marker files present on disk and from the linters detected | ||
|
|
||
| #### Scenario: A linter dependency is sourced from its own language's manifest | ||
|
|
||
| - **WHEN** a dependency-evidenced linter (for example `ruff`) is named only in a | ||
| manifest belonging to a different language (for example `package.json`) | ||
| - **THEN** `detect --json` SHALL NOT report that linter from the mismatched | ||
| manifest | ||
|
|
||
| #### Scenario: Configs in monorepo sub-packages are detected | ||
|
|
||
| - **WHEN** a linter config or language manifest lives in a sub-package rather | ||
| than the repository root (for example `packages/api/.eslintrc.json`) | ||
| - **THEN** `detect --json` SHALL detect it and SHALL carry the path it was found | ||
| at in the linter's evidence | ||
| - **AND** the scan SHALL prune a curated set of ignored directories (for example | ||
| `node_modules`, `.git`, build output) and SHALL bound traversal depth | ||
|
|
||
| #### Scenario: The repo's own rule styles are surfaced | ||
|
|
||
| - **WHEN** the working directory contains existing rule definitions (for example | ||
| custom linter rules or `.taskless/rules/`) | ||
| - **THEN** `detect --json` SHALL surface a description of those existing rule | ||
| styles for downstream authoring | ||
|
|
||
| #### Scenario: No packaged-rule catalog matching | ||
|
|
||
| - **WHEN** `detect --json` runs | ||
| - **THEN** the output SHALL NOT claim a request maps to a specific named packaged | ||
| rule (such matching is left to the authoring recipe, not the command) | ||
|
|
||
| ### Requirement: Detect runs offline with no network or auth | ||
|
|
||
| The `detect` command SHALL complete without network access and without | ||
| authentication. | ||
|
|
||
| #### Scenario: Detect works without login or network | ||
|
|
||
| - **WHEN** `detect --json` runs while logged out and offline | ||
| - **THEN** it SHALL produce its signal output successfully | ||
| - **AND** it SHALL NOT require or prompt for authentication | ||
|
|
||
| ### Requirement: Detect emits a stable JSON shape | ||
|
|
||
| When `--json` is set, `detect` SHALL emit a single structured JSON object whose | ||
| shape is validated internally against a stable Zod output schema before being | ||
| printed, consistent with how other `--json` commands in the CLI (e.g. `info`, | ||
| `check`) validate their output. The schema is an internal contract, not a | ||
| published artifact, and `detect` does not expose a `--schema` mode. | ||
|
|
||
| #### Scenario: JSON output validates against the internal schema | ||
|
|
||
| - **WHEN** `detect --json` succeeds | ||
| - **THEN** stdout SHALL be a single JSON object that the command has validated | ||
| against its internal output schema (linters, languages, existing rule styles) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,209 @@ | ||
| # cli-rule-routing Specification | ||
|
|
||
| ## Purpose | ||
|
|
||
| TBD - created by archiving change local-rule-routing. Update Purpose after archive. | ||
|
|
||
| ## Requirements | ||
|
|
||
| ### Requirement: Route is the local authoring classifier | ||
|
|
||
| The CLI SHALL provide a `route` help recipe that instructs the agent to classify | ||
| a rule-authoring request into one of three destinations — `existing`, `static`, | ||
| or `remote` — using `taskless detect --json` signals plus the user's intent. The | ||
| `route` recipe SHALL be biased to stay local: it SHALL prefer `existing` or | ||
| `static` and SHALL treat `remote` as the escalation of last resort. | ||
|
|
||
| #### Scenario: Route fetches detection before classifying | ||
|
|
||
| - **WHEN** the agent fetches the `route` recipe to author a rule | ||
| - **THEN** the recipe SHALL direct the agent to run `taskless detect --json` and | ||
| use its signals as input to the classification | ||
|
|
||
| #### Scenario: Route classifies into one of three destinations | ||
|
|
||
| - **WHEN** the agent follows `route` | ||
| - **THEN** it SHALL select exactly one of `existing`, `static`, or `remote` | ||
| - **AND** it SHALL fetch the corresponding recipe to perform the authoring | ||
|
|
||
| ### Requirement: Route states reasoning before naming a destination | ||
|
|
||
| The `route` recipe SHALL require the agent to write an explicit rationale before | ||
| naming a destination. The rationale SHALL cover what the `detect` signals show, | ||
| whether an existing linter plausibly already covers the request, whether the | ||
| pattern is expressible as a simple static ast-grep rule, and the resulting | ||
| confidence that the request is locally solvable. The destination SHALL be emitted | ||
| only after this rationale, and SHALL follow from it. | ||
|
|
||
| #### Scenario: Rationale precedes the route decision | ||
|
|
||
| - **WHEN** the agent follows `route` to classify a request | ||
| - **THEN** it SHALL produce a written rationale covering the detection signals, | ||
| existing-linter coverage, ast-grep expressibility, and local-solvability | ||
| confidence | ||
| - **AND** it SHALL name the destination (`existing`, `static`, or `remote`) only | ||
| after that rationale | ||
|
|
||
| #### Scenario: Route is not named before reasoning | ||
|
|
||
| - **WHEN** the agent has not yet articulated its reasoning | ||
| - **THEN** the recipe SHALL NOT permit committing to a destination | ||
| - **AND** the destination SHALL be a conclusion of the rationale, not asserted | ||
| ahead of it | ||
|
|
||
| ### Requirement: Route commits to the believed-correct path on reasonable confidence | ||
|
|
||
| The `route` recipe SHALL determine the destination upfront from `detect` signals | ||
| and the user's intent, committing to the path it believes is correct. The bar to | ||
| commit to a local path SHALL be **reasonable confidence**, not certainty. Routing | ||
| distinguishes three states: reasonable confidence the request IS locally solvable | ||
| selects a local path; reasonable belief the request is NOT locally solvable selects | ||
| `remote` directly, without first attempting a local rule; genuine inability to | ||
| judge either way is uncertainty, which SHALL be resolved by asking the user (see | ||
| the clarifying-question scenario) and SHALL NOT by itself select `remote`. The | ||
| recipe SHALL NOT use a deliberate local attempt-and-fail with no genuine belief of | ||
| success as the mechanism for choosing `remote`. | ||
|
|
||
| #### Scenario: Reasonably-confident-local commits locally without a justification probe | ||
|
|
||
| - **WHEN** `route` is reasonably confident the request fits an existing linter or | ||
| a simple static ast-grep pattern | ||
| - **THEN** it SHALL select `existing` or `static` and proceed locally | ||
| - **AND** it SHALL NOT run a throwaway local attempt whose only purpose is to | ||
| justify the choice | ||
|
|
||
| #### Scenario: Believed-not-local routes remote upfront | ||
|
|
||
| - **WHEN** `route` reasonably believes the request cannot be solved locally — a | ||
| positive judgment, not mere inability to tell | ||
| - **THEN** it SHALL select `remote` directly | ||
| - **AND** it SHALL NOT manufacture a deliberate local failure to reach that | ||
| decision | ||
|
|
||
| #### Scenario: Uncertainty biases toward asking, not toward login | ||
|
|
||
| - **WHEN** `route` cannot reasonably place a request as local or remote | ||
| - **THEN** it SHALL prefer clarifying with the user over defaulting to `remote` | ||
| - **AND** uncertainty alone SHALL NOT be treated as a reason to consume a | ||
| generation via `remote` | ||
|
|
||
| ### Requirement: A believed-local path that fails escalates only after confirmation | ||
|
|
||
| The `route` recipe SHALL treat try-verify-escalate as a legitimate failure | ||
| fallback: when it committed to a local path on reasonable confidence and the | ||
| authored rule then fails verification against the user's success/failure cases, it | ||
| SHALL surface the failure and SHALL obtain explicit user confirmation before | ||
| calling the Taskless service. The recipe SHALL NOT silently fall through from a | ||
| failed local attempt to a service call. | ||
|
|
||
| #### Scenario: Failed local attempt prompts before spending a generation | ||
|
|
||
| - **WHEN** a `static` rule the agent committed to fails verification | ||
| - **THEN** the recipe SHALL inform the user the local rule could not capture the | ||
| cases | ||
| - **AND** SHALL state that generating via the Taskless service uses a generation | ||
| and requires login | ||
| - **AND** SHALL call the service only after the user confirms | ||
|
|
||
| #### Scenario: No silent fall-through to the service | ||
|
|
||
| - **WHEN** a believed-local attempt fails | ||
| - **THEN** the recipe SHALL NOT invoke `remote` / the service without an explicit | ||
| user confirmation step | ||
|
|
||
| ### Requirement: Route asks the user when multiple paths fit | ||
|
|
||
| The `route` recipe SHALL present the viable options to the user with their | ||
| trade-offs, rather than silently selecting one, whenever more than one destination | ||
| genuinely fits a request (most commonly both `existing` and `static`). The | ||
| trade-off framing SHALL note that `remote` consumes a generation and requires | ||
| login, so it is appropriate when a request cannot be solved locally rather than as | ||
| a default. | ||
|
|
||
| #### Scenario: Both local paths viable surfaces a choice | ||
|
|
||
| - **WHEN** the repository has a detected linter that fits AND the pattern is a | ||
| clean local static ast-grep rule | ||
| - **THEN** `route` SHALL present both `existing` and `static` with their | ||
| trade-offs and let the user choose | ||
|
|
||
| #### Scenario: Trade-off framing names the generation cost of remote | ||
|
|
||
| - **WHEN** `route` presents options that include `remote` | ||
| - **THEN** it SHALL state that `remote` consumes a generation and requires login | ||
| - **AND** SHALL frame `remote` as the path for what cannot be solved locally | ||
|
|
||
| ### Requirement: Existing recipe authors in the detected linter's dialect | ||
|
|
||
| The CLI SHALL provide an `existing` help recipe that instructs the agent to | ||
| author a rule in a linter already detected in the repository, expressed in that | ||
| tool's own dialect. The recipe SHALL direct the agent to source authoring | ||
| knowledge first from the repository's own existing rules and only then from the | ||
| agent's own web research. The recipe SHALL NOT embed or rely on a Taskless- | ||
| maintained catalog of linter rules. | ||
|
|
||
| #### Scenario: Repo-first knowledge sourcing | ||
|
|
||
| - **WHEN** the agent follows `existing` for a detected linter | ||
| - **THEN** it SHALL first mine the repository's existing rules of that kind for | ||
| house style | ||
| - **AND** SHALL fall back to web research (WebFetch/WebSearch) only when the | ||
| repository signal is insufficient | ||
|
|
||
| #### Scenario: Existing path is author-only | ||
|
|
||
| - **WHEN** the agent authors a rule via `existing` | ||
| - **THEN** the recipe SHALL make clear the user's own toolchain runs the rule and | ||
| that `taskless check` does not execute the external linter | ||
|
|
||
| ### Requirement: Static recipe authors a verified local ast-grep rule | ||
|
|
||
| The CLI SHALL provide a `static` help recipe that instructs the agent to author a | ||
| local ast-grep rule on-device, without calling the Taskless service, and to | ||
| verify it against the user's success and failure cases before reporting success. | ||
| The recipe SHALL produce the canonical on-disk rule shape and paths used by remote | ||
| generation so that `check`, `improve`, and `verify` see a single dialect. | ||
|
|
||
| #### Scenario: Local authoring without the service | ||
|
|
||
| - **WHEN** the agent follows `static` | ||
| - **THEN** it SHALL write the rule on-device without requiring login or the | ||
| Taskless API | ||
|
|
||
| #### Scenario: Verification gates success | ||
|
|
||
| - **WHEN** the agent authors a static rule | ||
| - **THEN** it SHALL verify the rule against the provided success/failure cases | ||
| before reporting the rule as complete | ||
|
|
||
| #### Scenario: Canonical output shape | ||
|
|
||
| - **WHEN** the agent writes a static rule to disk | ||
| - **THEN** the files, paths, and shape SHALL match those produced by remote | ||
| generation | ||
|
|
||
| ### Requirement: Remote recipe collects inputs and delegates to the service | ||
|
|
||
| The CLI SHALL provide a `remote` help recipe that instructs the agent to gather | ||
| the inputs required to call the Taskless service and to invoke the existing rule | ||
| generation backend, which runs the service-side classifier and returns either a | ||
| static or a runtime rule. The `remote` recipe SHALL require authentication and | ||
| SHALL NOT itself decide static versus runtime. | ||
|
|
||
| #### Scenario: Remote requires authentication | ||
|
|
||
| - **WHEN** the agent follows `remote` while logged out | ||
| - **THEN** the recipe SHALL direct the agent to the authentication flow before | ||
| submitting the request | ||
|
|
||
| #### Scenario: Static-versus-runtime is decided by the service | ||
|
|
||
| - **WHEN** the agent submits an authored request via `remote` | ||
| - **THEN** the recipe SHALL rely on the service to classify static versus runtime | ||
| - **AND** SHALL NOT make that determination locally | ||
|
|
||
| #### Scenario: Remote output matches local on-disk shape | ||
|
|
||
| - **WHEN** the service returns a generated rule via `remote` | ||
| - **THEN** the written files and paths SHALL match the shape produced by the | ||
| local `static` path |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.