diff --git a/.changeset/detect-monorepo-node22.md b/.changeset/detect-monorepo-node22.md new file mode 100644 index 00000000..0a780825 --- /dev/null +++ b/.changeset/detect-monorepo-node22.md @@ -0,0 +1,23 @@ +--- +"@taskless/cli": minor +--- + +Require Node.js 22+ and make `taskless detect` monorepo-aware. + +- **Node floor raised to 22+.** Node 20 reached end-of-life, and detect now uses + the built-in `fs.glob` walker (Node 22+). This is a breaking engine change, + which pre-1.0 is a minor bump. +- **`detect` is monorepo-aware.** A single bounded tree walk (curated ignore + list + depth cap) finds linter configs and language manifests anywhere in the + repo, not just the root, so a linter configured in a sub-package is detected + with its path as evidence. +- **languages → linters flow.** A linter's dependency evidence is read only from + its own language's manifest (`package.json` for node, `pyproject.toml` / + `requirements.txt` for Python), parsed with real parsers (`smol-toml`, + `yaml`), instead of conflating ecosystems. A malformed manifest drops only its + own signal. +- **Dropped the `frameworks` field** from `detect` output. The routing recipe + never consumed it; the contract now matches its sole consumer. +- **Filled obvious linter gaps** for languages detect already recognizes: + golangci-lint (Go), Clippy (Rust), and PHPStan / PHP_CodeSniffer / Psalm + (PHP). diff --git a/.changeset/local-rule-routing.md b/.changeset/local-rule-routing.md new file mode 100644 index 00000000..ac842d84 --- /dev/null +++ b/.changeset/local-rule-routing.md @@ -0,0 +1,10 @@ +--- +"@taskless/cli": minor +--- + +Add a local-first rule-routing layer. A new deterministic `taskless detect` +command plus `route`/`existing`/`static`/`remote` recipes let the agent author +rules in an existing linter or as a local ast-grep rule on-device, only +escalating to the login-gated service (with confirmation) when a rule cannot be +built locally. The skill now engages this routing flow when a user names a +linter instead of suppressing itself. diff --git a/openspec/changes/archive/2026-06-12-local-rule-routing/.openspec.yaml b/openspec/changes/archive/2026-06-12-local-rule-routing/.openspec.yaml new file mode 100644 index 00000000..e0c0898f --- /dev/null +++ b/openspec/changes/archive/2026-06-12-local-rule-routing/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-06-11 diff --git a/openspec/changes/archive/2026-06-12-local-rule-routing/design.md b/openspec/changes/archive/2026-06-12-local-rule-routing/design.md new file mode 100644 index 00000000..d74d09d3 --- /dev/null +++ b/openspec/changes/archive/2026-06-12-local-rule-routing/design.md @@ -0,0 +1,282 @@ +## Context + +The Taskless skill is a thin router: it holds no recipes and delegates to +`npx @taskless/cli help `, which returns a bundled `.txt` recipe embedded +at build time. Today the only authoring entry is `rule create`, which is +rule-type-agnostic, login-gated, and runs generation off the developer's machine +via the Taskless service. The service-side classifier that decides static vs +runtime (`classifyStep`, Runtime Rules project) is **not reachable locally** — +it is login-only by design, because it carries inference off-machine. + +Three constraints shape this design: + +1. **No local classifyStep.** The local path cannot call the service classifier. + It must make a coarser, cheaper decision on-device. +2. **Login is a wall.** Reaching the service requires auth. Sending a user there + unnecessarily is the failure mode we are designing against. +3. **The skill adds no rule knowledge.** Per the existing router philosophy, all + judgment lives in fetched recipes; the CLI ships deterministic signals and + recipe text, never a maintained catalog of linter rules. + +The check pipeline already models multi-scanner aggregation (`source` field), and +remote rule generation already writes the same on-disk shape and paths as local +generation (canonical shape pinned by the Runtime Rules project). So downstream +tools (`check`, `improve`, `verify`) see a single dialect regardless of origin. + +## Goals / Non-Goals + +**Goals:** + +- A deterministic, offline `taskless detect --json` that reports repo signals + (linters configured, languages, the repo's own rule styles). +- A local routing recipe (`route`) that reasons first, then commits to the + believed-correct destination (`existing | static | remote`) on reasonable + confidence, biased to stay local. +- Try-verify-escalate as the _failure fallback_: a path committed to on reasonable + belief that fails verification escalates — and escalation to the service prompts + and confirms with the user before spending a generation. +- Asking the user when multiple paths fit, with trade-offs (including the + generation cost of `remote`). +- Authoring recipes for each destination (`existing`, `static`, `remote`). +- Reverse the skill's named-tool suppression so naming a linter engages routing. + +**Non-Goals:** + +- Defining or computing static-vs-runtime classification locally. That cut is + owned by the Runtime Rules project and lives behind `remote`. +- Maintaining any catalog/knowledge-graph of linter rules or a `howto` API + (explicitly rejected — see Decisions). Linter knowledge is sourced at author + time from the repo and the agent's WebSearch/WebFetch. +- Changing the `rule create` remote backend, the canonical on-disk rule shape, or + `check`/`improve`/`verify` behavior. +- Running external linters inside `taskless check` (the `existing` path is + author-only; the user's own toolchain runs the rule). + +## Decisions + +### D1 — `detect` is a real command; routing is recipes + +`taskless detect --json` is an executable, deterministic command. `route`, +`existing`, `static`, `remote` are `help` recipes the agent follows. + +_Why:_ A CLI cannot classify natural-language intent without an LLM, so the +"what kind of rule is this" judgment must live in a recipe the agent executes. +But "what linters/languages are present" is pure signal and benefits from +determinism — making it a command gives the recipe a stable, testable input and +keeps the agent from hallucinating the repo's tooling. + +_Alternative considered:_ a single `taskless route ""` command that +classifies directly. Rejected — it would need an on-device LLM call or a brittle +heuristic, and it couples a deterministic signal scan to a judgment that belongs +in the agent. + +### D2 — `route` commits to the believed-correct path; escalation is the failure fallback + +`route` determines the destination **upfront** from `detect` signals plus intent, +choosing the path it _believes_ is correct. The bar to commit to a local path is +**reasonable confidence**, not certainty — because the safety net is an honest +fallback, not a perfect prediction. The flow: + +``` +route (reason FIRST, then decide, then act): + 0. write the rationale BEFORE naming a destination: + - what the `detect` signals show (linters, languages, repo rule styles) + - whether an existing linter plausibly already covers this + - whether the pattern is expressible as a simple static ast-grep rule + - the resulting confidence that it is locally solvable + 1. existing AND static both fit? -> ASK the user, explain trade-offs (D8) + 2. existing linter clearly fits? -> existing (no login) + 3. reasonably confident simple static? -> static (author + verify) (no login) + 4. not reasonably confident it's local? -> remote (login) + + FALLBACK (failure state): a path we believed in (static) fails verification + -> PROMPT the user and CONFIRM before calling the service + ("local rule couldn't capture the cases; generate via Taskless? + this uses a generation and needs login") -> on yes -> remote +``` + +The destination is emitted **after** the rationale and must follow from it. The +recipe forbids naming a route before the reasoning is written. + +_Why (reason-before-route):_ Confidence assessments are unreliable when the model +names a destination first and justifies it after — it rationalizes a snap call. +Requiring the rationale to be written _before_ the route is named conditions the +decision on the contextual work (what `detect` shows, whether a packaged rule +exists, whether ast-grep can express the pattern), so the confidence judgment is +earned rather than asserted. The route must be a conclusion of the reasoning. + +_Why (commit on reasonable confidence, not certainty):_ Demanding high confidence +to act locally would push borderline-but-solvable requests to the service +unnecessarily — and the service costs generations (D8). Reasonable confidence is +enough to _commit_ to local because try-verify-escalate backstops a wrong bet: +the `static` recipe authors the rule and verifies it against the user's cases, and +if that believed-correct attempt fails, the recipe escalates. This is the +legitimate role of try-verify-escalate — a **failure handler for a path we +believed in**, not a route-selection probe. + +_Why escalation prompts and confirms:_ The service consumes a generation and +requires login. A local attempt that fails must NOT silently fall through to the +service — the recipe surfaces the failure and asks the user to confirm the +service call. This keeps generations from being spent without consent and makes +the (now justified) login ask explicit. + +_The distinction that matters:_ what is rejected is **fail-first as the selection +mechanism** — deliberately attempting local with no real belief it will work, just +to manufacture a justification for `remote`. What is kept is **try-verify-escalate +as a fallback** — committing to local on reasonable belief, and escalating (with +confirmation) only when that genuine attempt fails. `route` never names "runtime"; +it only judges "am I reasonably confident this is a simple local rule?" and, when +not, routes remote upfront. + +_Alternative considered (rejected):_ escalate to `remote` on _uncertainty alone_. +Rejected — uncertainty must not by itself send users to a generation-consuming, +login-gated path; only a reasoned judgment (confident-local, or a verified local +failure) drives the choice. + +### D3 — `detect` emits pure repo signals only; no rule-pattern detection + +`detect` reports linters present, languages, and the repo's own rule-authoring +styles. It does **not** try to match a request against known packaged rules +(e.g. "this is `no-console`"). + +_Why:_ The space of packaged-rule detections is effectively infinite and changes +constantly across ecosystems — a poor determinism target that would drag a +maintained catalog back into the CLI. That judgment is cheap for the LLM in the +`existing` recipe (repo + WebFetch) and expensive to keep correct in code. + +_Why no `frameworks` signal:_ An earlier cut also reported detected frameworks +(React, Django, …). It was dropped: the sole consumer is `route`, whose rationale +(D2 step 0) reads linters, languages, and repo rule styles — never frameworks. +Framework presence does not change which authoring destination fits, so emitting +it was unused surface. Dropping it aligns the contract with its consumer. + +_Detection shape (languages → linters):_ languages are inferred first (manifest +and marker files), then each linter is probed. A linter is tagged with the +language(s) it serves, so its dependency evidence is read from that language's +own manifest — a node dependency from `package.json`, a Python dependency from +`pyproject.toml`/`requirements.txt` — instead of conflating ecosystems. Config +files are parsed with real parsers (`smol-toml` for `pyproject.toml`, alongside +the existing `yaml`); a malformed manifest drops only its own derived signal and +never fails the scan, because other files in the repo are independent tells. A +config file present on disk is honored regardless of inferred language, per the +"Linter configs are detected from disk" requirement. + +### D4 — Knowledge is sourced at author time, not maintained by Taskless + +The `existing` recipe instructs the agent to mine the repo's own rules of the +relevant kind first, then WebFetch the linter's current docs only if that is +thin. Taskless ships the recipe (where to look), not the knowledge. + +_Why:_ A Taskless-hosted knowledge graph / `howto` endpoint is WebSearch/WebFetch +in disguise with a perpetual scraping-and-staleness maintenance bill. The agent +already has fresh web access; the repo is the highest-signal source for house +style. (There is precedent for `/cli/api/*` endpoints if server-fresh recipes are +ever needed, but bundled-first / API-as-enrichment remains the posture.) + +### D5 — `route` replaces `rule create` as the authoring front door + +The skill and topic index point "author a rule" requests at `route`, not directly +at `rule create`. `rule create` remains the executable backend that the `remote` +recipe invokes; it is unchanged. + +_Why:_ It inserts the local-first decision before any login-gated path, without +disturbing the working remote backend. + +### D6 — Local `static` output matches the canonical shape + +The `static` recipe writes the same on-disk rule shape and paths as remote +generation (the canonical shape pinned upstream). + +_Why:_ `check`, `improve`, and `verify` must see one dialect. Divergent local vs +remote output would fork all three downstream tools. + +### D7 — The skill `description` 1024-char ceiling is a gating check, not an afterthought + +The Agent Skills spec caps the skill `description` at 1024 characters. Reversing +the named-tool suppression and adding routing trigger language both consume that +budget. Every edit to the description SHALL be measured against 1024, and trigger +wording SHALL be tightened (reworded shorter) rather than appended when the budget +is tight. + +_Why:_ The description is the skill's trigger surface; silently overflowing 1024 +breaks loading or truncates triggers. Capturing the limit as a first-class +constraint now — before more capabilities pile trigger phrases onto the field — +prevents a later capability from being blocked by a full description. This is why +the constraint is surfaced in the proposal and carries an explicit length +scenario in the spec, not just a code comment. + +### D8 — When multiple paths fit, ask the user and explain trade-offs + +When more than one destination genuinely fits — most often `existing` and `static` +both viable — `route` SHALL present the options to the user with their trade-offs +rather than silently picking one. The trade-off framing SHALL include that the +service path (`remote`) consumes a generation and requires login, so it is the +right tool when something _cannot_ be solved locally, not a default. + +_Why:_ When two local paths both work, the choice is a user preference (which +toolchain owns the rule), not a technical fact the recipe should decide for them. +Surfacing it builds trust and keeps the user in control of where their rules live. +Naming the generation cost of `remote` makes the local-first bias legible: the +service is valuable precisely when local can't deliver, and spending a generation +should be a conscious choice. + +_Resolves open question:_ "should `route` offer `existing` and `static` in +parallel?" — yes, when both fit or the agent is unsure, ask and explain. + +## Risks / Trade-offs + +- **[`route` is over-confident and commits local when it shouldn't]** → The + `static` recipe verifies the authored rule against the user's success/failure + cases before reporting success, so a wrong-but-reasonable local bet surfaces as a + verified failure, not a silently bad rule. That failure escalates via try-verify- + escalate — the legitimate fallback — and the escalation prompts-and-confirms + before spending a generation, so an over-confident bet costs at most a confirm + dialog, never a silent service charge. +- **[`route` demands too much confidence and pushes solvable requests to the + service]** → The commit bar is _reasonable_ confidence, not certainty (D2), + precisely because the fallback backstops a wrong bet cheaply. Tuned with the + honesty eval fixtures across both failure directions. +- **[Reversing suppression pushes the skill `description` over the 1024-char Agent + Skills ceiling]** → The 1024 limit is a gating check on every description edit + (D7); trigger wording is tightened, not appended, and the spec carries an + explicit length scenario. +- **[`existing` author-only confuses users who expect `taskless check` to run + their linter]** → Recipe text must be explicit that the user's own toolchain + runs the authored rule; Taskless does not aggregate external linters in v1. +- **[Recipe drift vs the Runtime Rules canonical shape]** → D6 ties local output + to the upstream shape; verification (`rule verify`) catches shape regressions. +- **[Skill trigger reversal causes false engagement on tool-named requests]** → + The trigger still anchors on rule-authoring intent; naming a linter routes to + `route`, which can still conclude "this is a one-line config in your existing + tool" without heavy machinery. + +## Migration Plan + +1. Ship `detect` (additive command, no behavior change to existing commands). +2. Add the four recipes and register them in the help index (additive). +3. Update `skill-taskless` trigger/router text to engage `route`. This is the + only behavior change users perceive; it is reversible by restoring the + suppression clause. +4. No data migration; no change to on-disk rule shape or stored state. + +Rollback: revert the skill text and unregister the recipes; `detect` can remain +harmlessly as an unused command. + +## Resolved Questions + +- **Offer `existing` and `static` in parallel when both fit?** Resolved (D8): yes + — when both fit or the agent is unsure, ask the user and explain trade-offs, + including that `remote` consumes a generation and is for what can't be solved + locally. +- **How confident is "confident enough" to commit to `static`?** Resolved (D2): + _reasonably_ confident, not certain — because try-verify-escalate (with a + prompt-and-confirm before the service) backstops a wrong-but-reasonable bet, so + borderline-solvable requests are not pushed to the service unnecessarily. The + honesty eval fixtures calibrate the threshold in recipe language. +- **Where does the `static` candidate live and how is it cleaned up on fallback?** + Resolved: mirror the existing `.taskless/.tmp-*` + guaranteed-cleanup pattern + from `rule create` (cleanup on both success and failure). + +## Open Questions + +- None outstanding. diff --git a/openspec/changes/archive/2026-06-12-local-rule-routing/proposal.md b/openspec/changes/archive/2026-06-12-local-rule-routing/proposal.md new file mode 100644 index 00000000..10e1fd58 --- /dev/null +++ b/openspec/changes/archive/2026-06-12-local-rule-routing/proposal.md @@ -0,0 +1,90 @@ +## Why + +Authoring a rule today routes straight to `rule create`, which is login-gated and +runs generation off the developer's machine. There is no local front door that +asks "what kind of rule is this, and can I build it on-device first?" — so users +hit a login wall before Taskless has demonstrated it needs the service. At the +same time, the skill actively _suppresses_ itself when a user names a linter +(eslint, ruff, biome), stepping back from exactly the requests where Taskless +could help author the rule in that tool's own dialect. We want a local routing +layer that keeps authoring on-device whenever possible, engages other linters +instead of deferring, and suggests login only when it is not reasonably +confident a rule can be built locally — or when a confident local attempt has +failed and the user confirms spending a generation. + +## What Changes + +- **NEW `taskless detect --json`** — a deterministic, offline repo-signal scan: + which linters are configured, languages present, and the styles of the repo's + own existing rules. No LLM, no network. Feeds the routing recipe. +- **NEW routing recipe layer** under `help`, replacing the rule-type-agnostic + `rule create` entry as the front door for "author a rule": + - `route` — the lightweight **local classifier**. Biased to stay local; + suggests login when it is not reasonably confident the rule is locally + solvable, or when a confident local attempt fails and the user confirms. + - `existing` — author a rule in a linter already detected in the repo, in that + tool's dialect, sourced from the repo's own rules plus the agent's WebFetch. + - `static` — author a local ast-grep rule on-device, verified against the + user's success/failure cases (the lineage of `rule create --anonymous`). + - `remote` — collect inputs and call the Taskless service, which runs the heavy + classifier and returns either a static or a runtime rule (login required). +- **NEW confidence-gated routing contract** — `route` first writes an explicit + rationale (what `detect` shows, whether an existing linter covers it, whether + ast-grep can express it, resulting local-solvability confidence), and only then + names the destination it believes is correct, as a conclusion of that reasoning. + It commits to a local path on **reasonable** confidence (not certainty); when + that confidence is absent it selects `remote` directly. Try-verify-escalate is + the _failure fallback_: a local path committed to in good faith that fails + verification escalates — but escalation to the generation-consuming, login-gated + service **prompts and confirms with the user first**, never silently. When + multiple paths fit, `route` asks the user and explains the trade-offs. +- **MODIFIED skill trigger posture** — naming a specific linter should _engage_ + the authoring flow via `route`, not quiet the skill to a one-line offer. +- This change does **not** define static-vs-runtime classification. That cut is + owned upstream by the Runtime Rules project and lives behind `remote`; `route` + only decides local-vs-remote. + +## Capabilities + +### New Capabilities + +- `cli-detect`: A deterministic `taskless detect --json` command that scans the + working directory for linter configs, languages, and the repo's own + rule-authoring styles, emitting structured signals for downstream routing. No + inference, no network. +- `cli-rule-routing`: The `route` / `existing` / `static` / `remote` recipe layer + and the confidence-gated routing contract that determines the destination + upfront, keeps authoring local when there is confidence it is locally solvable, + and routes to the login-gated service directly when that confidence is absent — + avoiding developer-visible failed local attempts. + +### Modified Capabilities + +- `cli-help`: Register the four routing topics (`route`, `existing`, `static`, + `remote`) in the help index and emit their intent telemetry, consistent with + existing topic registration and embedding requirements. +- `skill-taskless`: Reverse the named-tool suppression clause — when a user names + a linter or asks to author a rule, the skill routes through + `npx @taskless/cli help route` instead of quieting. The skill stays a thin + router and adds no rule knowledge of its own. **Hard constraint:** the skill + `description` field has a + 1024-character ceiling (Agent Skills spec). Reversing the suppression clause and + adding routing trigger language compete for that budget, so trigger wording must + be _tightened_ as this and future capabilities land — not appended. Treat the + 1024-char limit as a gating check whenever the description changes. + +## Impact + +- **New command**: `packages/cli/src/commands/detect.ts`, registered in + `packages/cli/src/index.ts`; a `detect` output schema under `schemas/`. +- **New help recipes**: `packages/cli/src/help/{route,existing,static,remote}.txt`, + embedded via the existing `import.meta.glob` build step. +- **Skill body + description**: `skills/taskless/SKILL.md` routing/trigger text. +- **Reused, unchanged**: `rule create` (remote backend for `remote`), + `rule verify` (the local verification gate), the canonical on-disk rule + shape — remote and local paths already write the same files/paths, so + `check`/`improve`/`verify` see one dialect. +- **Upstream dependency**: the Runtime Rules project (TSKL `Runtime Rules`) owns + static-vs-runtime; this change references it and never redefines it. +- **No new network dependencies**; `detect` and the local authoring paths are + offline-capable. Only `remote` requires auth. diff --git a/openspec/changes/archive/2026-06-12-local-rule-routing/specs/cli-detect/spec.md b/openspec/changes/archive/2026-06-12-local-rule-routing/specs/cli-detect/spec.md new file mode 100644 index 00000000..f3e43ec2 --- /dev/null +++ b/openspec/changes/archive/2026-06-12-local-rule-routing/specs/cli-detect/spec.md @@ -0,0 +1,94 @@ +## ADDED Requirements + +### Requirement: Detect subcommand exists + +The CLI SHALL provide a `taskless detect` subcommand registered in the top-level +command list, with a `--json` flag and the standard `--dir`/`-d` working-directory +flag. + +#### Scenario: Detect is registered + +- **WHEN** `taskless detect --help` is run +- **THEN** the command SHALL be recognized and print its usage +- **AND** the command SHALL accept `--json` and `--dir`/`-d` + +### Requirement: Detect scans deterministic repo signals only + +The `detect` command SHALL emit only deterministic signals derived from files on +disk: configured linters, detected languages, and the styles of the repo's own +existing rules. It SHALL NOT perform any LLM inference and SHALL NOT match the +request against any catalog of known packaged linter rules. + +Detection follows a languages → linters flow: languages are inferred first, and +a linter's dependency evidence is then read from the manifest of that linter's +own language (a node dependency from `package.json`, a Python dependency from +`pyproject.toml`/`requirements.txt`) rather than conflating ecosystems. A +recognized linter config file on disk is honored regardless of the languages +inferred. + +#### Scenario: Linter configs are detected from disk + +- **WHEN** the working directory contains a recognized linter config (for + example `.eslintrc*`, `eslint.config.js`, `ruff.toml`, a `[tool.ruff]` block in + `pyproject.toml`, `.rubocop.yml`, `biome.json`, or `stylelint` config) +- **THEN** `detect --json` SHALL report each configured linter it found + +#### Scenario: Languages are reported + +- **WHEN** `detect --json` runs in a repository +- **THEN** the output SHALL include the languages inferred from manifest and + marker files present on disk and from the linters detected + +#### Scenario: A linter dependency is sourced from its own language's manifest + +- **WHEN** a dependency-evidenced linter (for example `ruff`) is named only in a + manifest belonging to a different language (for example `package.json`) +- **THEN** `detect --json` SHALL NOT report that linter from the mismatched + manifest + +#### Scenario: Configs in monorepo sub-packages are detected + +- **WHEN** a linter config or language manifest lives in a sub-package rather + than the repository root (for example `packages/api/.eslintrc.json`) +- **THEN** `detect --json` SHALL detect it and SHALL carry the path it was found + at in the linter's evidence +- **AND** the scan SHALL prune a curated set of ignored directories (for example + `node_modules`, `.git`, build output) and SHALL bound traversal depth + +#### Scenario: The repo's own rule styles are surfaced + +- **WHEN** the working directory contains existing rule definitions (for example + custom linter rules or `.taskless/rules/`) +- **THEN** `detect --json` SHALL surface a description of those existing rule + styles for downstream authoring + +#### Scenario: No packaged-rule catalog matching + +- **WHEN** `detect --json` runs +- **THEN** the output SHALL NOT claim a request maps to a specific named packaged + rule (such matching is left to the authoring recipe, not the command) + +### Requirement: Detect runs offline with no network or auth + +The `detect` command SHALL complete without network access and without +authentication. + +#### Scenario: Detect works without login or network + +- **WHEN** `detect --json` runs while logged out and offline +- **THEN** it SHALL produce its signal output successfully +- **AND** it SHALL NOT require or prompt for authentication + +### Requirement: Detect emits a stable JSON shape + +When `--json` is set, `detect` SHALL emit a single structured JSON object whose +shape is validated internally against a stable Zod output schema before being +printed, consistent with how other `--json` commands in the CLI (e.g. `info`, +`check`) validate their output. The schema is an internal contract, not a +published artifact, and `detect` does not expose a `--schema` mode. + +#### Scenario: JSON output validates against the internal schema + +- **WHEN** `detect --json` succeeds +- **THEN** stdout SHALL be a single JSON object that the command has validated + against its internal output schema (linters, languages, existing rule styles) diff --git a/openspec/changes/archive/2026-06-12-local-rule-routing/specs/cli-help/spec.md b/openspec/changes/archive/2026-06-12-local-rule-routing/specs/cli-help/spec.md new file mode 100644 index 00000000..926c0690 --- /dev/null +++ b/openspec/changes/archive/2026-06-12-local-rule-routing/specs/cli-help/spec.md @@ -0,0 +1,32 @@ +## ADDED Requirements + +### Requirement: Routing topics are registered in the help system + +The help system SHALL register the routing recipes `route`, `existing`, `static`, +and `remote` as embedded help topics, retrievable via `taskless help ` and +listed in the help index, consistent with the existing topic embedding and format +requirements. + +#### Scenario: Routing topics resolve + +- **WHEN** `taskless help route`, `taskless help existing`, + `taskless help static`, or `taskless help remote` is run +- **THEN** the corresponding recipe text SHALL be returned +- **AND** an unknown-topic error SHALL NOT be raised for any of the four + +#### Scenario: Routing topics appear in the index + +- **WHEN** `taskless help` (no arguments) is run +- **THEN** the topic index SHALL include the routing topics so an agent can + discover the authoring front door + +### Requirement: Routing topics emit intent telemetry + +Fetching a routing recipe SHALL emit a per-topic intent telemetry event, +consistent with the existing `help_` telemetry convention. + +#### Scenario: Help topic intent is captured for routing recipes + +- **WHEN** the agent fetches `route`, `existing`, `static`, or `remote` +- **THEN** the help command SHALL capture the corresponding `help_` intent + event with the topic name diff --git a/openspec/changes/archive/2026-06-12-local-rule-routing/specs/cli-rule-routing/spec.md b/openspec/changes/archive/2026-06-12-local-rule-routing/specs/cli-rule-routing/spec.md new file mode 100644 index 00000000..0c6cfce4 --- /dev/null +++ b/openspec/changes/archive/2026-06-12-local-rule-routing/specs/cli-rule-routing/spec.md @@ -0,0 +1,203 @@ +## ADDED Requirements + +### Requirement: Route is the local authoring classifier + +The CLI SHALL provide a `route` help recipe that instructs the agent to classify +a rule-authoring request into one of three destinations — `existing`, `static`, +or `remote` — using `taskless detect --json` signals plus the user's intent. The +`route` recipe SHALL be biased to stay local: it SHALL prefer `existing` or +`static` and SHALL treat `remote` as the escalation of last resort. + +#### Scenario: Route fetches detection before classifying + +- **WHEN** the agent fetches the `route` recipe to author a rule +- **THEN** the recipe SHALL direct the agent to run `taskless detect --json` and + use its signals as input to the classification + +#### Scenario: Route classifies into one of three destinations + +- **WHEN** the agent follows `route` +- **THEN** it SHALL select exactly one of `existing`, `static`, or `remote` +- **AND** it SHALL fetch the corresponding recipe to perform the authoring + +### Requirement: Route states reasoning before naming a destination + +The `route` recipe SHALL require the agent to write an explicit rationale before +naming a destination. The rationale SHALL cover what the `detect` signals show, +whether an existing linter plausibly already covers the request, whether the +pattern is expressible as a simple static ast-grep rule, and the resulting +confidence that the request is locally solvable. The destination SHALL be emitted +only after this rationale, and SHALL follow from it. + +#### Scenario: Rationale precedes the route decision + +- **WHEN** the agent follows `route` to classify a request +- **THEN** it SHALL produce a written rationale covering the detection signals, + existing-linter coverage, ast-grep expressibility, and local-solvability + confidence +- **AND** it SHALL name the destination (`existing`, `static`, or `remote`) only + after that rationale + +#### Scenario: Route is not named before reasoning + +- **WHEN** the agent has not yet articulated its reasoning +- **THEN** the recipe SHALL NOT permit committing to a destination +- **AND** the destination SHALL be a conclusion of the rationale, not asserted + ahead of it + +### Requirement: Route commits to the believed-correct path on reasonable confidence + +The `route` recipe SHALL determine the destination upfront from `detect` signals +and the user's intent, committing to the path it believes is correct. The bar to +commit to a local path SHALL be **reasonable confidence**, not certainty. Routing +distinguishes three states: reasonable confidence the request IS locally solvable +selects a local path; reasonable belief the request is NOT locally solvable selects +`remote` directly, without first attempting a local rule; genuine inability to +judge either way is uncertainty, which SHALL be resolved by asking the user (see +the clarifying-question scenario) and SHALL NOT by itself select `remote`. The +recipe SHALL NOT use a deliberate local attempt-and-fail with no genuine belief of +success as the mechanism for choosing `remote`. + +#### Scenario: Reasonably-confident-local commits locally without a justification probe + +- **WHEN** `route` is reasonably confident the request fits an existing linter or + a simple static ast-grep pattern +- **THEN** it SHALL select `existing` or `static` and proceed locally +- **AND** it SHALL NOT run a throwaway local attempt whose only purpose is to + justify the choice + +#### Scenario: Believed-not-local routes remote upfront + +- **WHEN** `route` reasonably believes the request cannot be solved locally — a + positive judgment, not mere inability to tell +- **THEN** it SHALL select `remote` directly +- **AND** it SHALL NOT manufacture a deliberate local failure to reach that + decision + +#### Scenario: Uncertainty biases toward asking, not toward login + +- **WHEN** `route` cannot reasonably place a request as local or remote +- **THEN** it SHALL prefer clarifying with the user over defaulting to `remote` +- **AND** uncertainty alone SHALL NOT be treated as a reason to consume a + generation via `remote` + +### Requirement: A believed-local path that fails escalates only after confirmation + +The `route` recipe SHALL treat try-verify-escalate as a legitimate failure +fallback: when it committed to a local path on reasonable confidence and the +authored rule then fails verification against the user's success/failure cases, it +SHALL surface the failure and SHALL obtain explicit user confirmation before +calling the Taskless service. The recipe SHALL NOT silently fall through from a +failed local attempt to a service call. + +#### Scenario: Failed local attempt prompts before spending a generation + +- **WHEN** a `static` rule the agent committed to fails verification +- **THEN** the recipe SHALL inform the user the local rule could not capture the + cases +- **AND** SHALL state that generating via the Taskless service uses a generation + and requires login +- **AND** SHALL call the service only after the user confirms + +#### Scenario: No silent fall-through to the service + +- **WHEN** a believed-local attempt fails +- **THEN** the recipe SHALL NOT invoke `remote` / the service without an explicit + user confirmation step + +### Requirement: Route asks the user when multiple paths fit + +The `route` recipe SHALL present the viable options to the user with their +trade-offs, rather than silently selecting one, whenever more than one destination +genuinely fits a request (most commonly both `existing` and `static`). The +trade-off framing SHALL note that `remote` consumes a generation and requires +login, so it is appropriate when a request cannot be solved locally rather than as +a default. + +#### Scenario: Both local paths viable surfaces a choice + +- **WHEN** the repository has a detected linter that fits AND the pattern is a + clean local static ast-grep rule +- **THEN** `route` SHALL present both `existing` and `static` with their + trade-offs and let the user choose + +#### Scenario: Trade-off framing names the generation cost of remote + +- **WHEN** `route` presents options that include `remote` +- **THEN** it SHALL state that `remote` consumes a generation and requires login +- **AND** SHALL frame `remote` as the path for what cannot be solved locally + +### Requirement: Existing recipe authors in the detected linter's dialect + +The CLI SHALL provide an `existing` help recipe that instructs the agent to +author a rule in a linter already detected in the repository, expressed in that +tool's own dialect. The recipe SHALL direct the agent to source authoring +knowledge first from the repository's own existing rules and only then from the +agent's own web research. The recipe SHALL NOT embed or rely on a Taskless- +maintained catalog of linter rules. + +#### Scenario: Repo-first knowledge sourcing + +- **WHEN** the agent follows `existing` for a detected linter +- **THEN** it SHALL first mine the repository's existing rules of that kind for + house style +- **AND** SHALL fall back to web research (WebFetch/WebSearch) only when the + repository signal is insufficient + +#### Scenario: Existing path is author-only + +- **WHEN** the agent authors a rule via `existing` +- **THEN** the recipe SHALL make clear the user's own toolchain runs the rule and + that `taskless check` does not execute the external linter + +### Requirement: Static recipe authors a verified local ast-grep rule + +The CLI SHALL provide a `static` help recipe that instructs the agent to author a +local ast-grep rule on-device, without calling the Taskless service, and to +verify it against the user's success and failure cases before reporting success. +The recipe SHALL produce the canonical on-disk rule shape and paths used by remote +generation so that `check`, `improve`, and `verify` see a single dialect. + +#### Scenario: Local authoring without the service + +- **WHEN** the agent follows `static` +- **THEN** it SHALL write the rule on-device without requiring login or the + Taskless API + +#### Scenario: Verification gates success + +- **WHEN** the agent authors a static rule +- **THEN** it SHALL verify the rule against the provided success/failure cases + before reporting the rule as complete + +#### Scenario: Canonical output shape + +- **WHEN** the agent writes a static rule to disk +- **THEN** the files, paths, and shape SHALL match those produced by remote + generation + +### Requirement: Remote recipe collects inputs and delegates to the service + +The CLI SHALL provide a `remote` help recipe that instructs the agent to gather +the inputs required to call the Taskless service and to invoke the existing rule +generation backend, which runs the service-side classifier and returns either a +static or a runtime rule. The `remote` recipe SHALL require authentication and +SHALL NOT itself decide static versus runtime. + +#### Scenario: Remote requires authentication + +- **WHEN** the agent follows `remote` while logged out +- **THEN** the recipe SHALL direct the agent to the authentication flow before + submitting the request + +#### Scenario: Static-versus-runtime is decided by the service + +- **WHEN** the agent submits an authored request via `remote` +- **THEN** the recipe SHALL rely on the service to classify static versus runtime +- **AND** SHALL NOT make that determination locally + +#### Scenario: Remote output matches local on-disk shape + +- **WHEN** the service returns a generated rule via `remote` +- **THEN** the written files and paths SHALL match the shape produced by the + local `static` path diff --git a/openspec/changes/archive/2026-06-12-local-rule-routing/specs/skill-taskless/spec.md b/openspec/changes/archive/2026-06-12-local-rule-routing/specs/skill-taskless/spec.md new file mode 100644 index 00000000..063ed1fd --- /dev/null +++ b/openspec/changes/archive/2026-06-12-local-rule-routing/specs/skill-taskless/spec.md @@ -0,0 +1,57 @@ +## MODIFIED Requirements + +### Requirement: Skill description anchors triggers on Taskless-specific phrases + +The consolidated skill's `description` frontmatter field SHALL anchor triggers on: + +1. An explicit reference to "Taskless" in the user's message, OR +2. A reference to the `.taskless/` directory or files within it (rules, rule-tests, rule-metadata), OR +3. A request to add/write/create a rule for code, including requests that name a specific lint/format/static-analysis tool (for example eslint, ruff, biome, stylelint, ast-grep). Naming such a tool SHALL engage the skill's routing flow rather than suppress it; the skill routes the request toward the appropriate authoring destination via `npx @taskless/cli help route`. + +The description SHALL NOT instruct the agent to suppress or quiet itself merely because a lint/format/static-analysis tool is named, and SHALL NOT contain a blanket "do NOT trigger on generic linting" instruction. + +#### Scenario: Description includes anchored trigger phrases + +- **WHEN** the skill `description` field is read +- **THEN** it SHALL include trigger phrases such as "create/add/write a taskless rule", "improve/fix/iterate on this taskless rule", "run taskless", "taskless login", "add taskless to CI" +- **AND** SHALL include a rule-authoring clause covering "add/write/create a rule" for code + +#### Scenario: Naming a linter engages routing rather than suppressing + +- **WHEN** the user asks to add/write/create a rule and names a specific lint/format/static-analysis tool (for example "write an eslint rule for X") +- **THEN** the skill SHALL trigger and route the request via `npx @taskless/cli help route` +- **AND** SHALL NOT quiet itself to a one-line offer on the basis of the named tool + +#### Scenario: Description omits suppression and the prior blanket carve-out + +- **WHEN** the skill `description` field is read +- **THEN** it SHALL NOT contain wording instructing the agent to suppress on a named lint/format/static-analysis tool +- **AND** it SHALL NOT contain wording instructing the agent to never trigger on generic ESLint/linting requests + +#### Scenario: Description is at most 1024 characters + +- **WHEN** the skill `description` field length is measured +- **THEN** it SHALL be at most 1024 characters (Agent Skills spec limit) + +## ADDED Requirements + +### Requirement: Skill routes authoring requests through the routing front door + +The skill body SHALL route rule-authoring requests through `npx @taskless/cli help route` +as the authoring front door, rather than fetching `rule create` directly. The +skill SHALL add no linter knowledge of its own and SHALL remain a thin router that +defers all authoring judgment to the fetched recipes. + +#### Scenario: Authoring requests are routed via route + +- **WHEN** the user asks to author a rule (with or without naming a tool) +- **THEN** the skill body SHALL direct the agent to fetch `npx @taskless/cli help route` + before any login-gated path +- **AND** SHALL NOT embed linter-specific rule knowledge in the skill body + +#### Scenario: Local-first before login + +- **WHEN** the routing recipe has not yet demonstrated that a rule cannot be built + locally +- **THEN** the skill SHALL NOT direct the agent toward a login-gated authoring + path diff --git a/openspec/changes/archive/2026-06-12-local-rule-routing/tasks.md b/openspec/changes/archive/2026-06-12-local-rule-routing/tasks.md new file mode 100644 index 00000000..68d2c09d --- /dev/null +++ b/openspec/changes/archive/2026-06-12-local-rule-routing/tasks.md @@ -0,0 +1,39 @@ +## 1. Detect command (cli-detect) + +- [x] 1.1 Add a `detect` output schema under `packages/cli/src/schemas/` (linters, languages, existing rule styles) +- [x] 1.2 Implement `packages/cli/src/commands/detect.ts`: deterministic, offline scan of linter configs, languages, and the repo's own rule styles — no LLM, no network, no auth; languages → linters flow with dependency evidence read from each linter's own manifest (`smol-toml` for `pyproject.toml`) +- [x] 1.3 Register `detect` in `packages/cli/src/index.ts` subCommands with `--json` and `--dir`/`-d` +- [x] 1.4 Emit `cli_detect` telemetry consistent with other commands +- [x] 1.5 Add unit tests covering: eslint/ruff/rubocop/biome/stylelint config detection, language inference, per-language dependency sourcing, graceful malformed-manifest handling, repo-rule-style surfacing, and JSON-shape validation against the schema +- [x] 1.6 Add a test asserting `detect` produces no packaged-rule-catalog claims and runs without network/auth + +## 2. Routing recipes (cli-rule-routing) + +- [x] 2.1 Author `packages/cli/src/help/route.txt`: run `detect`; require the agent to WRITE its rationale first (detect signals, existing-linter coverage, ast-grep expressibility, local-solvability confidence) and name a destination only as a conclusion of that rationale; commit to the believed-correct path on REASONABLE confidence (existing/static/remote), route remote directly when not reasonably confident; when multiple paths fit, ask the user and explain trade-offs (note `remote` consumes a generation + needs login); never use a deliberate fail-first probe to select `remote` +- [x] 2.1a In `route.txt`, specify the try-verify-escalate FALLBACK: when a believed-local `static` path fails verification, inform the user the local rule couldn't capture the cases and PROMPT-AND-CONFIRM before calling the service — never silently fall through to `remote` +- [x] 2.2 Author `packages/cli/src/help/existing.txt`: author in the detected linter's dialect; repo-first knowledge sourcing then WebFetch; explicit author-only (user's toolchain runs it; `taskless check` does not run the external linter) +- [x] 2.3 Author `packages/cli/src/help/static.txt`: local ast-grep authoring with verification against success/failure cases; canonical on-disk shape and paths (`.taskless/rules/.yml` + rule-tests, so `rule verify ` can read it); on verification failure into the escalation fallback, guarantee cleanup of the abandoned candidate rule + test files (mirrors the `rule create` cleanup intent); hand back to the `route` prompt-and-confirm fallback rather than escalating directly +- [x] 2.4 Author `packages/cli/src/help/remote.txt`: collect inputs, require auth, invoke the existing `rule create` backend; service decides static vs runtime; never decide that locally +- [x] 2.5 Ensure all four recipes follow the embedded help-text format (header, sprintf escaping) and reference `detect`/`route` consistently + +## 3. Help registration + telemetry (cli-help) + +- [x] 3.1 Confirm the four `.txt` recipes are picked up by the `import.meta.glob` embedding and resolve via `taskless help ` +- [x] 3.2 Ensure `route`, `existing`, `static`, `remote` appear in the `taskless help` (no-arg) topic index +- [x] 3.3 Verify `help_` intent telemetry fires for each routing topic +- [x] 3.4 Add tests for topic resolution, index listing, and telemetry capture + +## 4. Skill routing posture (skill-taskless) + +- [x] 4.1 Update `skills/taskless/SKILL.md` `description`: replace the named-tool suppression clause so naming a linter engages routing via `taskless help route`; tighten (reword shorter) rather than append trigger text +- [x] 4.1a Measure the resulting `description` length and assert it is ≤ 1024 chars (Agent Skills ceiling); treat overflow as a blocking failure and trim trigger wording until it fits — measured 835 chars +- [x] 4.2 Update the skill body to route authoring requests through `taskless help route` (not `rule create` directly); remove the "quiet suggestion" suppression path; keep the skill a thin router with no linter knowledge +- [x] 4.3 Bump the skill `metadata.version` per the file conventions — skill version is build-locked to the `@taskless/cli` package version (assertSkillVersions), which `scripts/sync-skill-versions.ts` treats as the source of truth; added a `minor` changeset targeting `@taskless/cli` (the root `@taskless/skills` package is private/unpublished) so the release path bumps the CLI version and syncs the skill version in lockstep, rather than hand-editing a mismatch +- [x] 4.4 Verify the skill change against the updated `skill-taskless` scenarios (routing on named tool, no suppression wording, local-first before login) + +## 5. Validation + quality gate + +- [x] 5.1 Run `pnpm openspec validate local-rule-routing` and resolve any issues +- [x] 5.2 Add/curate the honesty eval fixtures (labeled request → expected route) and assert the `route` heuristic against both failure directions: under-confident (escalating a locally-solvable request to login) and over-confident (claiming local for a request that needs the service). Use the fixtures to calibrate the "confident enough for local" threshold — dataset at `packages/cli/test/fixtures/route-eval.json` with a coverage test (`packages/cli/test/route-eval.test.ts`); route decision is agent-made (recipe-followed), so the test guards dataset balance across routes + every declared trap rather than running a code classifier +- [x] 5.3 Run `pnpm typecheck` and `pnpm lint`; fix all failures +- [x] 5.4 Manual smoke: `taskless detect --json`, then `taskless help route`/`existing`/`static`/`remote` resolve and read coherently end-to-end diff --git a/openspec/specs/cli-detect/spec.md b/openspec/specs/cli-detect/spec.md new file mode 100644 index 00000000..8fef84e5 --- /dev/null +++ b/openspec/specs/cli-detect/spec.md @@ -0,0 +1,100 @@ +# cli-detect Specification + +## Purpose + +TBD - created by archiving change local-rule-routing. Update Purpose after archive. + +## Requirements + +### Requirement: Detect subcommand exists + +The CLI SHALL provide a `taskless detect` subcommand registered in the top-level +command list, with a `--json` flag and the standard `--dir`/`-d` working-directory +flag. + +#### Scenario: Detect is registered + +- **WHEN** `taskless detect --help` is run +- **THEN** the command SHALL be recognized and print its usage +- **AND** the command SHALL accept `--json` and `--dir`/`-d` + +### Requirement: Detect scans deterministic repo signals only + +The `detect` command SHALL emit only deterministic signals derived from files on +disk: configured linters, detected languages, and the styles of the repo's own +existing rules. It SHALL NOT perform any LLM inference and SHALL NOT match the +request against any catalog of known packaged linter rules. + +Detection follows a languages → linters flow: languages are inferred first, and +a linter's dependency evidence is then read from the manifest of that linter's +own language (a node dependency from `package.json`, a Python dependency from +`pyproject.toml`/`requirements.txt`) rather than conflating ecosystems. A +recognized linter config file on disk is honored regardless of the languages +inferred. + +#### Scenario: Linter configs are detected from disk + +- **WHEN** the working directory contains a recognized linter config (for + example `.eslintrc*`, `eslint.config.js`, `ruff.toml`, a `[tool.ruff]` block in + `pyproject.toml`, `.rubocop.yml`, `biome.json`, or `stylelint` config) +- **THEN** `detect --json` SHALL report each configured linter it found + +#### Scenario: Languages are reported + +- **WHEN** `detect --json` runs in a repository +- **THEN** the output SHALL include the languages inferred from manifest and + marker files present on disk and from the linters detected + +#### Scenario: A linter dependency is sourced from its own language's manifest + +- **WHEN** a dependency-evidenced linter (for example `ruff`) is named only in a + manifest belonging to a different language (for example `package.json`) +- **THEN** `detect --json` SHALL NOT report that linter from the mismatched + manifest + +#### Scenario: Configs in monorepo sub-packages are detected + +- **WHEN** a linter config or language manifest lives in a sub-package rather + than the repository root (for example `packages/api/.eslintrc.json`) +- **THEN** `detect --json` SHALL detect it and SHALL carry the path it was found + at in the linter's evidence +- **AND** the scan SHALL prune a curated set of ignored directories (for example + `node_modules`, `.git`, build output) and SHALL bound traversal depth + +#### Scenario: The repo's own rule styles are surfaced + +- **WHEN** the working directory contains existing rule definitions (for example + custom linter rules or `.taskless/rules/`) +- **THEN** `detect --json` SHALL surface a description of those existing rule + styles for downstream authoring + +#### Scenario: No packaged-rule catalog matching + +- **WHEN** `detect --json` runs +- **THEN** the output SHALL NOT claim a request maps to a specific named packaged + rule (such matching is left to the authoring recipe, not the command) + +### Requirement: Detect runs offline with no network or auth + +The `detect` command SHALL complete without network access and without +authentication. + +#### Scenario: Detect works without login or network + +- **WHEN** `detect --json` runs while logged out and offline +- **THEN** it SHALL produce its signal output successfully +- **AND** it SHALL NOT require or prompt for authentication + +### Requirement: Detect emits a stable JSON shape + +When `--json` is set, `detect` SHALL emit a single structured JSON object whose +shape is validated internally against a stable Zod output schema before being +printed, consistent with how other `--json` commands in the CLI (e.g. `info`, +`check`) validate their output. The schema is an internal contract, not a +published artifact, and `detect` does not expose a `--schema` mode. + +#### Scenario: JSON output validates against the internal schema + +- **WHEN** `detect --json` succeeds +- **THEN** stdout SHALL be a single JSON object that the command has validated + against its internal output schema (linters, languages, existing rule styles) diff --git a/openspec/specs/cli-help/spec.md b/openspec/specs/cli-help/spec.md index f9ba03bc..f9fb6190 100644 --- a/openspec/specs/cli-help/spec.md +++ b/openspec/specs/cli-help/spec.md @@ -155,6 +155,37 @@ The help command's existing intent-telemetry requirement SHALL extend naturally - **WHEN** an agent runs `taskless help onboard` - **THEN** PostHog SHALL receive a `help_onboard` event +### Requirement: Routing topics are registered in the help system + +The help system SHALL register the routing recipes `route`, `existing`, `static`, +and `remote` as embedded help topics, retrievable via `taskless help ` and +listed in the help index, consistent with the existing topic embedding and format +requirements. + +#### Scenario: Routing topics resolve + +- **WHEN** `taskless help route`, `taskless help existing`, + `taskless help static`, or `taskless help remote` is run +- **THEN** the corresponding recipe text SHALL be returned +- **AND** an unknown-topic error SHALL NOT be raised for any of the four + +#### Scenario: Routing topics appear in the index + +- **WHEN** `taskless help` (no arguments) is run +- **THEN** the topic index SHALL include the routing topics so an agent can + discover the authoring front door + +### Requirement: Routing topics emit intent telemetry + +Fetching a routing recipe SHALL emit a per-topic intent telemetry event, +consistent with the existing `help_` telemetry convention. + +#### Scenario: Help topic intent is captured for routing recipes + +- **WHEN** the agent fetches `route`, `existing`, `static`, or `remote` +- **THEN** the help command SHALL capture the corresponding `help_` intent + event with the topic name + ## Goal diff --git a/openspec/specs/cli-rule-routing/spec.md b/openspec/specs/cli-rule-routing/spec.md new file mode 100644 index 00000000..27cca600 --- /dev/null +++ b/openspec/specs/cli-rule-routing/spec.md @@ -0,0 +1,209 @@ +# cli-rule-routing Specification + +## Purpose + +TBD - created by archiving change local-rule-routing. Update Purpose after archive. + +## Requirements + +### Requirement: Route is the local authoring classifier + +The CLI SHALL provide a `route` help recipe that instructs the agent to classify +a rule-authoring request into one of three destinations — `existing`, `static`, +or `remote` — using `taskless detect --json` signals plus the user's intent. The +`route` recipe SHALL be biased to stay local: it SHALL prefer `existing` or +`static` and SHALL treat `remote` as the escalation of last resort. + +#### Scenario: Route fetches detection before classifying + +- **WHEN** the agent fetches the `route` recipe to author a rule +- **THEN** the recipe SHALL direct the agent to run `taskless detect --json` and + use its signals as input to the classification + +#### Scenario: Route classifies into one of three destinations + +- **WHEN** the agent follows `route` +- **THEN** it SHALL select exactly one of `existing`, `static`, or `remote` +- **AND** it SHALL fetch the corresponding recipe to perform the authoring + +### Requirement: Route states reasoning before naming a destination + +The `route` recipe SHALL require the agent to write an explicit rationale before +naming a destination. The rationale SHALL cover what the `detect` signals show, +whether an existing linter plausibly already covers the request, whether the +pattern is expressible as a simple static ast-grep rule, and the resulting +confidence that the request is locally solvable. The destination SHALL be emitted +only after this rationale, and SHALL follow from it. + +#### Scenario: Rationale precedes the route decision + +- **WHEN** the agent follows `route` to classify a request +- **THEN** it SHALL produce a written rationale covering the detection signals, + existing-linter coverage, ast-grep expressibility, and local-solvability + confidence +- **AND** it SHALL name the destination (`existing`, `static`, or `remote`) only + after that rationale + +#### Scenario: Route is not named before reasoning + +- **WHEN** the agent has not yet articulated its reasoning +- **THEN** the recipe SHALL NOT permit committing to a destination +- **AND** the destination SHALL be a conclusion of the rationale, not asserted + ahead of it + +### Requirement: Route commits to the believed-correct path on reasonable confidence + +The `route` recipe SHALL determine the destination upfront from `detect` signals +and the user's intent, committing to the path it believes is correct. The bar to +commit to a local path SHALL be **reasonable confidence**, not certainty. Routing +distinguishes three states: reasonable confidence the request IS locally solvable +selects a local path; reasonable belief the request is NOT locally solvable selects +`remote` directly, without first attempting a local rule; genuine inability to +judge either way is uncertainty, which SHALL be resolved by asking the user (see +the clarifying-question scenario) and SHALL NOT by itself select `remote`. The +recipe SHALL NOT use a deliberate local attempt-and-fail with no genuine belief of +success as the mechanism for choosing `remote`. + +#### Scenario: Reasonably-confident-local commits locally without a justification probe + +- **WHEN** `route` is reasonably confident the request fits an existing linter or + a simple static ast-grep pattern +- **THEN** it SHALL select `existing` or `static` and proceed locally +- **AND** it SHALL NOT run a throwaway local attempt whose only purpose is to + justify the choice + +#### Scenario: Believed-not-local routes remote upfront + +- **WHEN** `route` reasonably believes the request cannot be solved locally — a + positive judgment, not mere inability to tell +- **THEN** it SHALL select `remote` directly +- **AND** it SHALL NOT manufacture a deliberate local failure to reach that + decision + +#### Scenario: Uncertainty biases toward asking, not toward login + +- **WHEN** `route` cannot reasonably place a request as local or remote +- **THEN** it SHALL prefer clarifying with the user over defaulting to `remote` +- **AND** uncertainty alone SHALL NOT be treated as a reason to consume a + generation via `remote` + +### Requirement: A believed-local path that fails escalates only after confirmation + +The `route` recipe SHALL treat try-verify-escalate as a legitimate failure +fallback: when it committed to a local path on reasonable confidence and the +authored rule then fails verification against the user's success/failure cases, it +SHALL surface the failure and SHALL obtain explicit user confirmation before +calling the Taskless service. The recipe SHALL NOT silently fall through from a +failed local attempt to a service call. + +#### Scenario: Failed local attempt prompts before spending a generation + +- **WHEN** a `static` rule the agent committed to fails verification +- **THEN** the recipe SHALL inform the user the local rule could not capture the + cases +- **AND** SHALL state that generating via the Taskless service uses a generation + and requires login +- **AND** SHALL call the service only after the user confirms + +#### Scenario: No silent fall-through to the service + +- **WHEN** a believed-local attempt fails +- **THEN** the recipe SHALL NOT invoke `remote` / the service without an explicit + user confirmation step + +### Requirement: Route asks the user when multiple paths fit + +The `route` recipe SHALL present the viable options to the user with their +trade-offs, rather than silently selecting one, whenever more than one destination +genuinely fits a request (most commonly both `existing` and `static`). The +trade-off framing SHALL note that `remote` consumes a generation and requires +login, so it is appropriate when a request cannot be solved locally rather than as +a default. + +#### Scenario: Both local paths viable surfaces a choice + +- **WHEN** the repository has a detected linter that fits AND the pattern is a + clean local static ast-grep rule +- **THEN** `route` SHALL present both `existing` and `static` with their + trade-offs and let the user choose + +#### Scenario: Trade-off framing names the generation cost of remote + +- **WHEN** `route` presents options that include `remote` +- **THEN** it SHALL state that `remote` consumes a generation and requires login +- **AND** SHALL frame `remote` as the path for what cannot be solved locally + +### Requirement: Existing recipe authors in the detected linter's dialect + +The CLI SHALL provide an `existing` help recipe that instructs the agent to +author a rule in a linter already detected in the repository, expressed in that +tool's own dialect. The recipe SHALL direct the agent to source authoring +knowledge first from the repository's own existing rules and only then from the +agent's own web research. The recipe SHALL NOT embed or rely on a Taskless- +maintained catalog of linter rules. + +#### Scenario: Repo-first knowledge sourcing + +- **WHEN** the agent follows `existing` for a detected linter +- **THEN** it SHALL first mine the repository's existing rules of that kind for + house style +- **AND** SHALL fall back to web research (WebFetch/WebSearch) only when the + repository signal is insufficient + +#### Scenario: Existing path is author-only + +- **WHEN** the agent authors a rule via `existing` +- **THEN** the recipe SHALL make clear the user's own toolchain runs the rule and + that `taskless check` does not execute the external linter + +### Requirement: Static recipe authors a verified local ast-grep rule + +The CLI SHALL provide a `static` help recipe that instructs the agent to author a +local ast-grep rule on-device, without calling the Taskless service, and to +verify it against the user's success and failure cases before reporting success. +The recipe SHALL produce the canonical on-disk rule shape and paths used by remote +generation so that `check`, `improve`, and `verify` see a single dialect. + +#### Scenario: Local authoring without the service + +- **WHEN** the agent follows `static` +- **THEN** it SHALL write the rule on-device without requiring login or the + Taskless API + +#### Scenario: Verification gates success + +- **WHEN** the agent authors a static rule +- **THEN** it SHALL verify the rule against the provided success/failure cases + before reporting the rule as complete + +#### Scenario: Canonical output shape + +- **WHEN** the agent writes a static rule to disk +- **THEN** the files, paths, and shape SHALL match those produced by remote + generation + +### Requirement: Remote recipe collects inputs and delegates to the service + +The CLI SHALL provide a `remote` help recipe that instructs the agent to gather +the inputs required to call the Taskless service and to invoke the existing rule +generation backend, which runs the service-side classifier and returns either a +static or a runtime rule. The `remote` recipe SHALL require authentication and +SHALL NOT itself decide static versus runtime. + +#### Scenario: Remote requires authentication + +- **WHEN** the agent follows `remote` while logged out +- **THEN** the recipe SHALL direct the agent to the authentication flow before + submitting the request + +#### Scenario: Static-versus-runtime is decided by the service + +- **WHEN** the agent submits an authored request via `remote` +- **THEN** the recipe SHALL rely on the service to classify static versus runtime +- **AND** SHALL NOT make that determination locally + +#### Scenario: Remote output matches local on-disk shape + +- **WHEN** the service returns a generated rule via `remote` +- **THEN** the written files and paths SHALL match the shape produced by the + local `static` path diff --git a/openspec/specs/skill-taskless/spec.md b/openspec/specs/skill-taskless/spec.md index 09d776de..651c3664 100644 --- a/openspec/specs/skill-taskless/spec.md +++ b/openspec/specs/skill-taskless/spec.md @@ -32,22 +32,27 @@ The consolidated skill's `description` frontmatter field SHALL anchor triggers o 1. An explicit reference to "Taskless" in the user's message, OR 2. A reference to the `.taskless/` directory or files within it (rules, rule-tests, rule-metadata), OR -3. A request to add/write/create a rule where the user has NOT named a specific lint/format/static-analysis tool. The description SHALL include four illustrative example tools whose presence in the user's message suppresses this trigger: eslint, ruff, biome, ast-grep. The wording SHALL make clear the list is illustrative — any named lint/format/static-analysis tool suppresses the trigger. +3. A request to add/write/create a rule for code, including requests that name a specific lint/format/static-analysis tool (for example eslint, ruff, biome, stylelint, ast-grep). Naming such a tool SHALL engage the skill's routing flow rather than suppress it; the skill routes the request toward the appropriate authoring destination via `npx @taskless/cli help route`. -The description SHALL NOT contain a blanket "do NOT trigger on generic linting" instruction; that prior carve-out is replaced by the named-tool suppression in clause 3. +The description SHALL NOT instruct the agent to suppress or quiet itself merely because a lint/format/static-analysis tool is named, and SHALL NOT contain a blanket "do NOT trigger on generic linting" instruction. #### Scenario: Description includes anchored trigger phrases - **WHEN** the skill `description` field is read - **THEN** it SHALL include trigger phrases such as "create/add/write a taskless rule", "improve/fix/iterate on this taskless rule", "run taskless", "taskless login", "add taskless to CI" -- **AND** SHALL include the unspecified-tool clause covering "add/write/create a rule" with no tool named -- **AND** SHALL list at least the four illustrative suppressing tool names: eslint, ruff, biome, ast-grep +- **AND** SHALL include a rule-authoring clause covering "add/write/create a rule" for code -#### Scenario: Description omits the prior blanket carve-out +#### Scenario: Naming a linter engages routing rather than suppressing + +- **WHEN** the user asks to add/write/create a rule and names a specific lint/format/static-analysis tool (for example "write an eslint rule for X") +- **THEN** the skill SHALL trigger and route the request via `npx @taskless/cli help route` +- **AND** SHALL NOT quiet itself to a one-line offer on the basis of the named tool + +#### Scenario: Description omits suppression and the prior blanket carve-out - **WHEN** the skill `description` field is read -- **THEN** it SHALL NOT contain wording instructing the agent to never trigger on generic ESLint/linting requests -- **AND** any suppression wording SHALL be expressed via the named-tool clause +- **THEN** it SHALL NOT contain wording instructing the agent to suppress on a named lint/format/static-analysis tool +- **AND** it SHALL NOT contain wording instructing the agent to never trigger on generic ESLint/linting requests #### Scenario: Description is at most 1024 characters @@ -129,3 +134,24 @@ The consolidated skill's frontmatter SHALL include `metadata.commandName: tskl` - **WHEN** a user invokes `/tskl` with no arguments - **THEN** the command body SHALL instruct the agent to ask the user what they want to do with Taskless before proceeding + +### Requirement: Skill routes authoring requests through the routing front door + +The skill body SHALL route rule-authoring requests through `npx @taskless/cli help route` +as the authoring front door, rather than fetching `rule create` directly. The +skill SHALL add no linter knowledge of its own and SHALL remain a thin router that +defers all authoring judgment to the fetched recipes. + +#### Scenario: Authoring requests are routed via route + +- **WHEN** the user asks to author a rule (with or without naming a tool) +- **THEN** the skill body SHALL direct the agent to fetch `npx @taskless/cli help route` + before any login-gated path +- **AND** SHALL NOT embed linter-specific rule knowledge in the skill body + +#### Scenario: Local-first before login + +- **WHEN** the routing recipe has not yet demonstrated that a rule cannot be built + locally +- **THEN** the skill SHALL NOT direct the agent toward a login-gated authoring + path diff --git a/packages/cli/README.md b/packages/cli/README.md index 15e05f6c..db8635e2 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -176,6 +176,30 @@ All commands output structured JSON to stdout by default. Parse with `JSON.parse ## Developing +### Testing + +```bash +pnpm --filter @taskless/cli test # run the suite once +pnpm --filter @taskless/cli exec vitest # watch mode +``` + +The suite runs entirely locally under vitest — no network, no auth, no agent. +Integration tests that exercise the built binary (for example `detect`) run +against `dist/`, so run `pnpm --filter @taskless/cli build` first (or after any +source change) before invoking them directly. + +**Two kinds of test, one of which is not fully automatable.** Most tests are +deterministic unit/integration checks. The route-honesty dataset +(`test/fixtures/route-eval.json`) is different: the actual routing decision is +made by an _agent_ following `help/route.txt`, so it cannot be asserted by a +code classifier. The automated test (`test/route-eval.test.ts`) therefore only +**guards the dataset** — that it stays structurally valid and balanced across +every route and both failure directions (over-claim / over-escalate). Running +the dataset _as an evaluation_ — feeding each case to an agent and scoring its +chosen destination — is a separate, manual calibration step with more setup; it +is not part of `pnpm test`. Keep the two distinct: the suite proves the fixtures +are well-formed; an agent run proves the recipe routes honestly. + ### API base URL The CLI resolves the API base URL in this order: diff --git a/packages/cli/package.json b/packages/cli/package.json index b0fdddd4..20e80f4d 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -25,7 +25,7 @@ "dist" ], "engines": { - "node": "^20.20.0 || >=22.22.0" + "node": ">=22.22.0" }, "dependencies": { "@ast-grep/cli": "^0.41.0", @@ -36,6 +36,7 @@ "openapi-fetch": "^0.17.0", "picocolors": "^1.1.1", "posthog-node": "^5.28.11", + "smol-toml": "^1.6.1", "sprintf-js": "^1.1.3", "yaml": "^2.8.2", "zod": "^4.3.6" diff --git a/packages/cli/src/commands/detect.ts b/packages/cli/src/commands/detect.ts new file mode 100644 index 00000000..f0be6b32 --- /dev/null +++ b/packages/cli/src/commands/detect.ts @@ -0,0 +1,77 @@ +import { resolve } from "node:path"; + +import { defineCommand } from "citty"; + +import { detectRepository } from "../detect/scan"; +import { outputSchema as detectOutputSchema } from "../schemas/detect"; +import { makeErrorEnvelope } from "../types/errors"; + +export const detectCommand = defineCommand({ + meta: { + name: "detect", + description: + "Scan the repo for configured linters, languages, and existing rule styles (offline, deterministic)", + }, + args: { + dir: { + type: "string", + alias: "d", + description: "Working directory", + }, + json: { + type: "boolean", + description: "Output as JSON", + default: false, + }, + }, + async run({ args }) { + const cwd = resolve(args.dir ?? process.cwd()); + // detect is read-only with no state transition, so it emits no bespoke + // event — the per-invocation cli_run denominator (emitted by the runner) + // covers it, consistent with info under the cli_ telemetry taxonomy. + + const result = { + success: true as const, + ...(await detectRepository(cwd)), + }; + + if (args.json) { + const parsed = detectOutputSchema.safeParse(result); + if (!parsed.success) { + console.log( + JSON.stringify( + makeErrorEnvelope( + "INTERNAL_ERROR", + "Internal schema validation failed" + ) + ) + ); + process.exitCode = 1; + return; + } + console.log(JSON.stringify(parsed.data)); + return; + } + + // Human-readable output + if (result.linters.length === 0) { + console.log("Linters: none detected"); + } else { + console.log("Linters:"); + for (const linter of result.linters) { + console.log(` ${linter.name}: ${linter.evidence.join(", ")}`); + } + } + + console.log( + `\nLanguages: ${result.languages.length > 0 ? result.languages.join(", ") : "none detected"}` + ); + + if (result.ruleStyles.length > 0) { + console.log("\nExisting rule styles:"); + for (const style of result.ruleStyles) { + console.log(` ${style.source}: ${style.description}`); + } + } + }, +}); diff --git a/packages/cli/src/commands/help.ts b/packages/cli/src/commands/help.ts index f6e12938..edc80db5 100644 --- a/packages/cli/src/commands/help.ts +++ b/packages/cli/src/commands/help.ts @@ -50,6 +50,16 @@ function buildHelpMaps(): { const { helpMap, anonymousMap } = buildHelpMaps(); +// Help-only recipe topics (no backing subcommand) that should still be +// discoverable from the `taskless help` index. The rule-authoring front +// door (`route`) and its destinations live here so an agent can find them. +const RECIPE_TOPICS: ReadonlyArray<[string, string]> = [ + ["route", "Decide where to author a rule (existing/static/remote)"], + ["existing", "Author a rule in a linter the repo already uses"], + ["static", "Author a local ast-grep rule on this machine (no login)"], + ["remote", "Generate a rule via the Taskless service (login)"], +]; + // Topic → Zod input schema. When a recipe contains the %(INPUT_SCHEMA)s // placeholder, the help command substitutes the JSON Schema rendered // from this Zod source. @@ -173,11 +183,21 @@ export function createHelpCommand(subCommands: SubCommandsDef) { entries.push([name, description]); } - const maxLength = Math.max(...entries.map(([name]) => name.length)); + // Pad commands and recipe topics against a shared width so the two + // sections line up. + const maxLength = Math.max( + ...entries.map(([name]) => name.length), + ...RECIPE_TOPICS.map(([name]) => name.length) + ); for (const [name, description] of entries) { console.log(` ${name.padEnd(maxLength + 2)}${description}`); } + console.log("\nAuthoring recipes:"); + for (const [name, description] of RECIPE_TOPICS) { + console.log(` ${name.padEnd(maxLength + 2)}${description}`); + } + console.log( "\nAppend `--anonymous` to any rule/check command to skip the Taskless API" ); diff --git a/packages/cli/src/detect/scan.ts b/packages/cli/src/detect/scan.ts new file mode 100644 index 00000000..8577d32b --- /dev/null +++ b/packages/cli/src/detect/scan.ts @@ -0,0 +1,606 @@ +import { existsSync, globSync } from "node:fs"; +import { readFile as readFileNode } from "node:fs/promises"; +import { resolve } from "node:path"; + +import { parse as parseToml } from "smol-toml"; + +export interface DetectedLinter { + name: string; + /** + * On-disk evidence for this linter: a config-file path, a `pyproject.toml` + * table marker, or a dependency marker from the language's package file. Each + * entry carries the path it was found at, so monorepo evidence + * (`packages/api/.eslintrc.json`) is attributable. Not all entries are file + * paths. + */ + evidence: string[]; +} + +export interface RuleStyle { + source: string; + description: string; +} + +export interface DetectResult { + linters: DetectedLinter[]; + languages: string[]; + ruleStyles: RuleStyle[]; +} + +/** + * Directory names pruned from the repo walk. Held as a curated list so the scan + * never descends into dependency trees, build output, or VCS metadata — the + * places a real linter config never lives and where traversal cost explodes. + * Recursive `fs.glob` does not honor `.gitignore`, so we prune explicitly; an + * explicit list is also more deterministic than whatever each repo ignores. + */ +const IGNORED_DIRECTORIES: ReadonlySet = new Set([ + "node_modules", + ".git", + ".hg", + ".svn", + "dist", + "build", + "out", + "coverage", + "vendor", + "target", + ".next", + ".nuxt", + ".svelte-kit", + ".turbo", + ".cache", + ".parcel-cache", + ".venv", + "venv", + "__pycache__", + ".mypy_cache", + ".pytest_cache", + ".tox", + ".gradle", +]); + +/** + * Maximum directory depth (levels below the scan root) the walk descends. + * Bounds traversal on pathological trees; monorepo manifests live well within + * this — root → workspace group → package → nested package is four. + */ +const MAX_DIRECTORY_DEPTH = 8; + +/** + * A linter is curated, deterministic signal — never inference. detect never + * matches a request against a catalog of packaged rules; that judgment lives in + * the `existing` recipe. + * + * Each linter is tagged with the language(s) it serves so dependency evidence + * is read from the right package file: a node dependency lives in + * `package.json`, a Python dependency in `pyproject.toml`/`requirements.txt`. + * Tagging by language lets the scan look for a linter's dependency only in its + * own ecosystem's manifest instead of conflating the two. Config-file presence + * is honored unconditionally, per the detect spec ("a recognized linter config + * ... SHALL report each configured linter it found"), so a lone `.eslintrc.json` + * still detects eslint. + */ +interface LinterSignal { + name: string; + /** Languages this linter serves; a detected linter contributes these. */ + languages: string[]; + /** Fixed config filenames; presence on disk is direct evidence. */ + configFiles?: string[]; + /** `[tool.]` tables in `pyproject.toml` (Python linters). */ + pyprojectTables?: string[]; + /** Dependency names, matched against the manifest of this linter's language. */ + deps?: string[]; +} + +const LINTER_SIGNALS: readonly LinterSignal[] = [ + { + name: "eslint", + languages: ["JavaScript", "TypeScript"], + configFiles: [ + ".eslintrc", + ".eslintrc.js", + ".eslintrc.cjs", + ".eslintrc.mjs", + ".eslintrc.json", + ".eslintrc.yml", + ".eslintrc.yaml", + "eslint.config.js", + "eslint.config.mjs", + "eslint.config.cjs", + "eslint.config.ts", + ], + deps: ["eslint"], + }, + { + name: "biome", + languages: ["JavaScript", "TypeScript"], + configFiles: ["biome.json", "biome.jsonc"], + deps: ["@biomejs/biome"], + }, + { + name: "stylelint", + languages: ["JavaScript", "TypeScript"], + configFiles: [ + ".stylelintrc", + ".stylelintrc.js", + ".stylelintrc.cjs", + ".stylelintrc.json", + ".stylelintrc.yml", + ".stylelintrc.yaml", + "stylelint.config.js", + "stylelint.config.cjs", + "stylelint.config.mjs", + ], + deps: ["stylelint"], + }, + { + name: "prettier", + languages: ["JavaScript", "TypeScript"], + configFiles: [ + ".prettierrc", + ".prettierrc.js", + ".prettierrc.cjs", + ".prettierrc.json", + ".prettierrc.yml", + ".prettierrc.yaml", + "prettier.config.js", + "prettier.config.cjs", + "prettier.config.mjs", + ], + deps: ["prettier"], + }, + { + name: "ruff", + languages: ["Python"], + configFiles: ["ruff.toml", ".ruff.toml"], + pyprojectTables: ["ruff"], + deps: ["ruff"], + }, + { + name: "flake8", + languages: ["Python"], + configFiles: [".flake8"], + deps: ["flake8"], + }, + { + name: "pylint", + languages: ["Python"], + configFiles: [".pylintrc", "pylintrc"], + pyprojectTables: ["pylint"], + deps: ["pylint"], + }, + { + name: "black", + languages: ["Python"], + pyprojectTables: ["black"], + deps: ["black"], + }, + { + name: "rubocop", + languages: ["Ruby"], + configFiles: [".rubocop.yml", ".rubocop.yaml"], + }, + { + name: "golangci-lint", + languages: ["Go"], + configFiles: [ + ".golangci.yml", + ".golangci.yaml", + ".golangci.toml", + ".golangci.json", + ], + }, + { + name: "clippy", + languages: ["Rust"], + configFiles: ["clippy.toml", ".clippy.toml"], + }, + { + name: "phpstan", + languages: ["PHP"], + configFiles: ["phpstan.neon", "phpstan.neon.dist", "phpstan.dist.neon"], + }, + { + name: "php_codesniffer", + languages: ["PHP"], + configFiles: [ + "phpcs.xml", + "phpcs.xml.dist", + ".phpcs.xml", + ".phpcs.xml.dist", + ], + }, + { + name: "psalm", + languages: ["PHP"], + configFiles: ["psalm.xml", "psalm.xml.dist"], + }, + { name: "clang-tidy", languages: ["C", "C++"], configFiles: [".clang-tidy"] }, + { + name: "swiftlint", + languages: ["Swift"], + configFiles: [".swiftlint.yml", ".swiftlint.yaml"], + }, + { name: "checkstyle", languages: ["Java"], configFiles: ["checkstyle.xml"] }, +]; + +/** + * Languages inferred from the presence of a manifest or marker file anywhere in + * the tree. JavaScript and TypeScript are resolved separately (they share + * `package.json`). + */ +const LANGUAGE_MARKERS: ReadonlyArray<{ language: string; files: string[] }> = [ + { + language: "Python", + files: [ + "pyproject.toml", + "requirements.txt", + "setup.cfg", + "setup.py", + "Pipfile", + ], + }, + { language: "Ruby", files: ["Gemfile"] }, + { language: "Go", files: ["go.mod"] }, + { language: "Rust", files: ["Cargo.toml"] }, + { language: "PHP", files: ["composer.json"] }, + { language: "Java", files: ["pom.xml", "build.gradle", "build.gradle.kts"] }, + { language: "Swift", files: ["Package.swift"] }, +]; + +/** Every basename the walk needs to find, deduped for a single glob pass. */ +const DISCOVERABLE_FILES: readonly string[] = [ + ...new Set([ + "package.json", + "tsconfig.json", + ...LANGUAGE_MARKERS.flatMap((marker) => marker.files), + ...LINTER_SIGNALS.flatMap((signal) => signal.configFiles ?? []), + ]), +]; + +async function readFileSafe(path: string): Promise { + try { + return await readFileNode(path, "utf8"); + } catch { + return undefined; + } +} + +interface PackageJson { + dependencies?: Record; + devDependencies?: Record; + peerDependencies?: Record; + eslintConfig?: unknown; +} + +/** Object keys, but only for a plain object (a malformed manifest field that + * is an array, string, or null contributes no dependency names). */ +function plainObjectKeys(value: unknown): string[] { + if (typeof value !== "object" || value === null || Array.isArray(value)) { + return []; + } + return Object.keys(value as Record); +} + +/** Collect all declared dependency names from a parsed package.json. */ +function nodeDependencyNames(packageJson: PackageJson): Set { + return new Set([ + ...plainObjectKeys(packageJson.dependencies), + ...plainObjectKeys(packageJson.devDependencies), + ...plainObjectKeys(packageJson.peerDependencies), + ]); +} + +/** + * Parse `pyproject.toml` with a real TOML parser. A malformed file yields + * `undefined` rather than throwing — detect degrades gracefully, losing only + * the pyproject-derived signals (the file's mere presence still marks Python, + * and config files like `ruff.toml` are independent tells). + */ +function parsePyproject( + raw: string | undefined +): Record | undefined { + if (raw === undefined) return undefined; + try { + return parseToml(raw) as Record; + } catch { + return undefined; + } +} + +function asRecord(value: unknown): Record | undefined { + if (typeof value !== "object" || value === null || Array.isArray(value)) { + return undefined; + } + return value as Record; +} + +/** + * Whether a parsed `pyproject.toml` declares a `[tool.]` table (or a + * nested table under it, e.g. `[tool.ruff.lint]`). A real parser makes this + * exact: a similarly-named sibling like `[tool.ruff-lsp]` is a distinct key and + * does not match. + */ +function pyprojectHasTable( + pyproject: Record | undefined, + table: string +): boolean { + const tool = asRecord(pyproject?.tool); + return tool !== undefined && Object.hasOwn(tool, table); +} + +/** Strip a PEP 508 requirement string down to its package name. */ +function requirementName(requirement: string): string { + return requirement + .trim() + .split(/[\s<>=!~;[\],()]/)[0]! + .toLowerCase(); +} + +/** Python dependency names declared in a parsed `pyproject.toml` (PEP 621 + Poetry). */ +function pythonDepsFromPyproject( + pyproject: Record | undefined +): string[] { + if (pyproject === undefined) return []; + const names: string[] = []; + + const project = asRecord(pyproject.project); + const projectDeps = project?.dependencies; + if (Array.isArray(projectDeps)) { + for (const entry of projectDeps) { + if (typeof entry === "string") names.push(requirementName(entry)); + } + } + const optional = asRecord(project?.["optional-dependencies"]); + for (const group of Object.values(optional ?? {})) { + if (!Array.isArray(group)) continue; + for (const entry of group) { + if (typeof entry === "string") names.push(requirementName(entry)); + } + } + + const poetry = asRecord(asRecord(pyproject.tool)?.poetry); + for (const key of ["dependencies", "dev-dependencies", "group"]) { + const table = asRecord(poetry?.[key]); + if (table) { + names.push(...Object.keys(table).map((name) => name.toLowerCase())); + } + } + + return names; +} + +/** Python dependency names declared in a `requirements.txt`. */ +function pythonDepsFromRequirements(raw: string | undefined): string[] { + if (raw === undefined) return []; + const names: string[] = []; + for (const line of raw.split("\n")) { + const trimmed = line.trim(); + if (trimmed === "" || trimmed.startsWith("#") || trimmed.startsWith("-")) { + continue; + } + names.push(requirementName(trimmed)); + } + return names; +} + +/** The basename (last path segment) of a `/`-or-`\`-separated relative path. */ +function basenameOf(relativePath: string): string { + return relativePath.split(/[/\\]/).at(-1) ?? ""; +} + +/** + * Prune the walk: skip the curated ignore directories and anything past the + * depth cap. `fs.glob` calls this on each candidate as it descends, so a `true` + * here stops traversal into that directory. + */ +function shouldExclude(relativePath: string): boolean { + const segments = relativePath.split(/[/\\]/); + if (segments.length > MAX_DIRECTORY_DEPTH) return true; + return IGNORED_DIRECTORIES.has(segments.at(-1) ?? ""); +} + +interface NodeManifest { + path: string; + deps: Set; + hasEslintConfigKey: boolean; +} + +interface PythonManifest { + path: string; + parsed?: Record; + deps: Set; +} + +/** + * Surface the styles of the repo's own existing rules so the authoring recipe + * can match house conventions. `.taskless/rules` is the repo-root, polyglot + * Taskless convention, so it is read at the scan root; the custom-ESLint-rule + * tells (house rule directories and the local-rules plugin dependency) describe + * how this repo already authors lint rules. + */ +function detectRuleStyles( + root: string, + nodeManifests: NodeManifest[] +): RuleStyle[] { + const ruleStyles: RuleStyle[] = []; + if (existsSync(resolve(root, ".taskless", "rules"))) { + ruleStyles.push({ + source: ".taskless/rules", + description: + "Existing Taskless ast-grep rules — match their structure and conventions.", + }); + } + for (const directory of [ + "eslint-rules", + "eslint-local-rules", + "tools/eslint-rules", + ]) { + if (existsSync(resolve(root, directory))) { + ruleStyles.push({ + source: directory, + description: + "Custom ESLint rules — follow the house style when authoring new ones.", + }); + } + } + const localRulesManifest = nodeManifests.find( + (manifest) => + manifest.deps.has("eslint-plugin-local") || + manifest.deps.has("eslint-local-rules") + ); + if (localRulesManifest) { + ruleStyles.push({ + source: localRulesManifest.path, + description: + "Local ESLint rule plugin in use — author new rules to match it.", + }); + } + return ruleStyles; +} + +/** + * Deterministically scan `cwd` for the languages present, the linters + * configured for those languages, and the repo's own rule styles. Pure + * filesystem reads — no network, no auth, no LLM. Unreadable or malformed files + * are skipped rather than failing the scan. + * + * The scan is monorepo-aware: a single bounded `fs.glob` walk (curated ignore + * list + depth cap) finds manifests and configs anywhere in the tree, so a + * linter configured in a sub-package is detected with its path as evidence. The + * flow is languages → linters: a linter's dependency is looked up only in its + * own language's manifests. + */ +export async function detectRepository(cwd: string): Promise { + const root = resolve(cwd); + + const foundPaths = globSync(`**/{${DISCOVERABLE_FILES.join(",")}}`, { + cwd: root, + exclude: shouldExclude, + }); + + const pathsByBasename = new Map(); + for (const relativePath of foundPaths) { + const basename = basenameOf(relativePath); + const list = pathsByBasename.get(basename); + if (list) list.push(relativePath); + else pathsByBasename.set(basename, [relativePath]); + } + const pathsFor = (basename: string): string[] => + pathsByBasename.get(basename) ?? []; + + // Node manifests → JS/TS dependency names, per location. + const nodeManifests: NodeManifest[] = []; + for (const relativePath of pathsFor("package.json")) { + const raw = await readFileSafe(resolve(root, relativePath)); + if (raw === undefined) continue; + let parsed: PackageJson; + try { + parsed = JSON.parse(raw) as PackageJson; + } catch { + continue; + } + nodeManifests.push({ + path: relativePath, + deps: nodeDependencyNames(parsed), + hasEslintConfigKey: parsed.eslintConfig !== undefined, + }); + } + + // Python manifests → Python dependency names, per location. + const pythonManifests: PythonManifest[] = []; + for (const relativePath of pathsFor("pyproject.toml")) { + const parsed = parsePyproject( + await readFileSafe(resolve(root, relativePath)) + ); + pythonManifests.push({ + path: relativePath, + parsed, + deps: new Set(pythonDepsFromPyproject(parsed)), + }); + } + for (const relativePath of pathsFor("requirements.txt")) { + pythonManifests.push({ + path: relativePath, + deps: new Set( + pythonDepsFromRequirements( + await readFileSafe(resolve(root, relativePath)) + ) + ), + }); + } + + // Languages: manifest/marker files first, then JS/TS from node manifests. + const languages = new Set(); + if ( + pathsFor("package.json").length > 0 || + pathsFor("tsconfig.json").length > 0 + ) { + languages.add("JavaScript"); + } + if ( + pathsFor("tsconfig.json").length > 0 || + nodeManifests.some((manifest) => manifest.deps.has("typescript")) + ) { + languages.add("TypeScript"); + } + for (const marker of LANGUAGE_MARKERS) { + if (marker.files.some((file) => pathsFor(file).length > 0)) { + languages.add(marker.language); + } + } + + // Linters: config files unconditionally; dependency evidence from the + // manifests of the linter's own language. A detected linter contributes its + // language(s). + const linters: DetectedLinter[] = []; + for (const signal of LINTER_SIGNALS) { + const evidence: string[] = []; + const servesPython = signal.languages.includes("Python"); + const servesNode = + signal.languages.includes("JavaScript") || + signal.languages.includes("TypeScript"); + + for (const configFile of signal.configFiles ?? []) { + evidence.push(...pathsFor(configFile)); + } + for (const table of signal.pyprojectTables ?? []) { + for (const manifest of pythonManifests) { + if (pyprojectHasTable(manifest.parsed, table)) { + evidence.push(`${manifest.path} [tool.${table}]`); + } + } + } + for (const dep of signal.deps ?? []) { + const manifests = servesPython + ? pythonManifests + : servesNode + ? nodeManifests + : []; + for (const manifest of manifests) { + if (manifest.deps.has(dep)) { + evidence.push(`dependency ${dep} (${manifest.path})`); + } + } + } + if (signal.name === "eslint") { + for (const manifest of nodeManifests) { + if (manifest.hasEslintConfigKey) { + evidence.push(`${manifest.path} (eslintConfig)`); + } + } + } + + if (evidence.length > 0) { + linters.push({ name: signal.name, evidence }); + for (const language of signal.languages) languages.add(language); + } + } + + return { + linters, + languages: [...languages], + ruleStyles: detectRuleStyles(root, nodeManifests), + }; +} diff --git a/packages/cli/src/help/existing.txt b/packages/cli/src/help/existing.txt new file mode 100644 index 00000000..efc00e53 --- /dev/null +++ b/packages/cli/src/help/existing.txt @@ -0,0 +1,57 @@ +# Topic: existing (CLI v%(CLI_VERSION)s / topic v1) + +## Goal +Author a rule in a linter the repository ALREADY uses, expressed in that +tool's own dialect (an ESLint rule, a Ruff rule selection, a RuboCop cop, +a Stylelint rule, etc.). Taskless does not maintain a catalog of linter +rules — you source the knowledge from the repo first and the web second, +then write the rule where that tool expects it. + +## Preconditions +- The repo has a detected linter (confirm via `taskless detect --json`). +- The agent can read/write files and fetch web pages. +- No auth required. + +## Steps + +1. **Confirm the target tool.** Use `taskless detect --json` to identify + which linter is configured. If more than one could host this rule, + ask the user which tool should own it. + +2. **Mine the repo's own rules first.** This is the highest-signal + source for house style. Look at how the repo already writes rules of + this kind: + - existing config (e.g. `.eslintrc*`/`eslint.config.*`, `ruff.toml` + or `[tool.ruff]`, `.rubocop.yml`, `.stylelintrc*`); + - any custom/local rules the repo authored (the `detect` output's + rule styles point at these); + - the conventions, severity choices, and naming they use. + Match that style. + +3. **Fall back to the web only if the repo signal is thin.** If the repo + doesn't show how to express this rule, fetch the linter's CURRENT + documentation (WebFetch/WebSearch). Prefer the latest version's docs + and, when present, an `llms.txt` on the tool's docs site. Confirm the + installed version where it matters so you target the right syntax. + +4. **Author the rule in the tool's dialect.** Write or extend the + linter's config / custom-rule file the way that tool expects. Keep it + consistent with the repo's existing entries from step 2. + +5. **Report, and be explicit about who runs it.** Show the file(s) you + changed. Make clear that the user's OWN toolchain runs this rule — + `taskless check` does NOT execute external linters. Tell the user how + to run their linter to see it fire (e.g. their existing lint script). + +## Important Notes + +- Do not invent linter rules from memory — verify against the repo's + usage and the tool's current docs. +- This path is author-only. Taskless does not aggregate or run external + linters; it writes the rule in the tool's dialect and hands off. + +## See Also + +- `taskless help route` — re-decide the destination if this no longer fits +- `taskless help static` — author a local ast-grep rule instead +- `taskless help remote` — generate via the Taskless service (login) diff --git a/packages/cli/src/help/remote.txt b/packages/cli/src/help/remote.txt new file mode 100644 index 00000000..823cffc2 --- /dev/null +++ b/packages/cli/src/help/remote.txt @@ -0,0 +1,60 @@ +# Topic: remote (CLI v%(CLI_VERSION)s / topic v1) + +## Goal +Generate a rule using the Taskless service. Your job on this path is to +gather the inputs and hand off to `rule create`; the service generates +the rule and writes the standard rule files. This path consumes a +generation and requires login. Reach it when a rule is not reasonably +solvable locally, or when a believed-local attempt failed and the user +confirmed. + +## Preconditions +- `.taskless/` directory exists. +- The user is logged in (this path requires auth). +- The request has been routed here (see `taskless help route`), not + reached by skipping the local-first decision. + +## Steps + +1. **Confirm this is the right path.** You should be here because the + request is not reasonably solvable locally, OR a local `static` + attempt failed and the user confirmed spending a generation. If the + user has not confirmed an escalation from a failed local attempt, get + that confirmation first — the service costs a generation and login. + +2. **Confirm auth.** Run: + ``` + npx @taskless/cli info --json + ``` + Check `loggedIn`. If false, fetch `taskless help auth` and follow the + login recipe before continuing. + +3. **Gather the request.** Collect the rule description plus concrete + success and failure cases. If you arrived here from `static`, reuse + the cases you already gathered. The service uses these to generate the + rule. + +4. **Delegate to the generation backend.** Fetch `taskless help rule + create` and follow it. That recipe builds the request payload and + invokes `rule create`, which submits to the service and writes the + result. Do not re-implement the submission here. + +5. **Report results.** The service writes the generated rule to the same + on-disk paths and shape as a locally authored rule, so `check`, + `improve`, and `verify` treat it identically. Show the user the file + paths and suggest `taskless help check` to validate. + +## Important Notes + +- Do NOT pre-build the rule yourself on this path — submit the request + and let the service generate it. +- The service owns rule-type selection. Today it generates ast-grep + rules written under `.taskless/rules/`, the same shape the local + `static` path produces. + +## See Also + +- `taskless help route` — the local-first routing decision +- `taskless help rule create` — the generation backend this path uses +- `taskless help auth` — log in before generating +- `taskless help check` — validate the generated rule diff --git a/packages/cli/src/help/route.txt b/packages/cli/src/help/route.txt new file mode 100644 index 00000000..b1ece4ed --- /dev/null +++ b/packages/cli/src/help/route.txt @@ -0,0 +1,95 @@ +# Topic: route (CLI v%(CLI_VERSION)s / topic v1) + +## Goal +Decide where a rule-authoring request should be built: in a linter the +repo ALREADY uses (`existing`), as a local ast-grep rule on this machine +(`static`), or by the Taskless service (`remote`). This is the front +door for "author/write/create a rule" requests. You stay local whenever +you can reasonably build the rule on-device; you only send the user to +the login-gated service when you are not reasonably confident the rule +is locally solvable — or when a believed-local attempt has genuinely +failed and the user confirms. + +## Preconditions +- A working repository the agent can read. +- No auth required to route. (The `remote` destination requires login; + the `existing` and `static` destinations do not.) + +## Steps + +1. **Scan the repo.** Run: + ``` + npx @taskless/cli detect --json + ``` + This returns the configured linters, languages, and the repo's own + rule styles. It is deterministic and offline — use it as ground truth + instead of guessing the repo's tooling. The scan is monorepo-aware, so + evidence may carry a sub-package path. The output shape: + ```json + { + "success": true, + "linters": [{ "name": "eslint", "evidence": ["packages/api/.eslintrc.json"] }], + "languages": ["JavaScript", "TypeScript"], + "ruleStyles": [ + { "source": ".taskless/rules", "description": "Existing Taskless ast-grep rules." } + ] + } + ``` + +2. **Write your reasoning BEFORE naming a destination.** Do not pick a + route first and justify it after. Write a short rationale covering: + - what the `detect` signals show (linters, languages, repo rule styles); + - whether an existing linter plausibly already covers this request; + - whether the pattern is expressible as a simple static ast-grep rule; + - your resulting confidence that the request is locally solvable. + The destination you choose in step 3 MUST follow from this rationale. + +3. **Choose the destination as a conclusion of the rationale:** + + - **Both an existing linter AND a local static rule fit** → do NOT + silently pick one. Present both to the user with trade-offs and let + them choose. Note that `remote` consumes a Taskless generation and + requires login, so it is for what cannot be solved locally — not a + default. + - **An existing linter clearly fits** (the repo uses it and it can + express this) → fetch `taskless help existing`. + - **You are reasonably confident it is a simple static ast-grep + pattern** → fetch `taskless help static`. Reasonable confidence is + enough here; you do not need certainty, because step 4 backstops a + wrong-but-reasonable bet. + - **You reasonably believe it CANNOT be solved locally** (a positive + judgment — e.g. it needs cross-file or semantic checks ast-grep can't + express) → fetch `taskless help remote`. Do not manufacture a + deliberate local failure to reach this; route here directly. + + These are three distinct states. If you genuinely cannot tell whether + the request is local or remote — mere uncertainty, not a belief that it + needs the service — ASK the user a clarifying question. Uncertainty is a + reason to ask, never a reason to spend a generation on `remote`. + +4. **Failure fallback (try-verify-escalate).** If you committed to a + local `static` rule on reasonable confidence and it then FAILS + verification against the user's success/failure cases: + - Tell the user the local rule could not capture the cases. + - State that generating via the Taskless service uses a generation + and requires login. + - Call the service only after the user confirms. On yes, fetch + `taskless help remote`. + Never silently fall through from a failed local attempt to a service + call — the confirmation step is required. + +## Important Notes + +- Reason first, route second. Do not name a destination before the + rationale is written. +- Stay local when you reasonably can. `remote` is the path for what + cannot be solved on-device, and it costs a generation plus login. +- A developer who watches a local attempt fail reads it as a Taskless + failure. Only attempt local when you reasonably believe it will work; + otherwise route `remote` upfront. + +## See Also + +- `taskless help existing` — author in a linter the repo already uses +- `taskless help static` — author a local ast-grep rule (no login) +- `taskless help remote` — generate via the Taskless service (login) diff --git a/packages/cli/src/help/static.txt b/packages/cli/src/help/static.txt new file mode 100644 index 00000000..2332e570 --- /dev/null +++ b/packages/cli/src/help/static.txt @@ -0,0 +1,74 @@ +# Topic: static (CLI v%(CLI_VERSION)s / topic v1) + +## Goal +Author a Taskless ast-grep rule **locally**, on this machine, without +contacting the Taskless service. You derive the rule yourself, write it +in the canonical on-disk shape, and validate it with `rule verify` in a +feedback loop. The files you produce match exactly what the service +writes, so `check`, `improve`, and `verify` treat them identically. + +## Preconditions +- `.taskless/` directory exists. +- The agent can read/write files and run shell commands. +- No auth required. + +## Steps + +1. **Learn the ast-grep rule format.** Consult the ast-grep rule + reference at https://ast-grep.github.io/guide/rule-config.html for + valid fields and operators (`pattern`, `kind`, `regex`, + `any`/`all`/`has`/`inside`/`not`) and meta-variable syntax. + +2. **Gather and confirm the pattern.** Make sure you have concrete + success cases (code that should pass) and failure cases (code that + should be flagged), the target language, and any exceptions. Search + the codebase for real instances and confirm exclusions with the user. + +3. **Author the rule in the canonical shape.** Write the rule to + `.taskless/rules/.yml` with at minimum `id` (kebab-case), + `language`, `severity` (`error`/`warning`/`info`/`hint`), `message`, + and the `rule` object. Write tests to + `.taskless/rule-tests/-YYYYMMDD-test.yml` with the matching `id` + field plus `valid` and `invalid` arrays (at least two of each). The + `id` must match the rule's `id` so ast-grep test filtering pairs them. + These paths and shape are the same ones the service writes — do not + invent a different layout. + +4. **Run the verify feedback loop.** Run: + ``` + npx @taskless/cli rule verify --json + ``` + - `success: true` → the rule passes. Go to step 5. + - `success: false` → read the per-layer errors (`schema`, + `requirements`, `tests`), fix the rule or tests, and re-run. Repeat + up to 3 times. + +5. **On success, report.** Show the rule and test file paths and a + one-line summary of what the rule detects. Suggest `taskless help + check` to validate against the broader codebase. + +6. **On failure, escalate via the route fallback — with confirmation.** + If after the feedback loop the rule still cannot capture the user's + cases, this is the try-verify-escalate fallback: + - Delete the candidate `.taskless/rules/.yml` and its test file so + the repo is not left with a broken rule (guaranteed cleanup of the + abandoned candidate). + - Tell the user the local rule could not capture the cases, and that + generating via the Taskless service uses a generation and requires + login. + - Only after the user confirms, fetch `taskless help remote` and + follow it. Do not call the service silently. + +## Important Notes + +- Do NOT make any HTTP requests to taskless.io on this path. +- Do NOT write to `.taskless/rule-metadata/` — local rules have no + metadata sidecar; they iterate via file edits. +- The verify loop is the quality gate. A clean failure is a legitimate + reason to escalate, but only with the user's confirmation (step 6). + +## See Also + +- `taskless help route` — re-decide the destination +- `taskless help remote` — generate via the Taskless service (login) +- `taskless help check` — validate the new rule against the codebase diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts index ae895704..418eedbc 100644 --- a/packages/cli/src/index.ts +++ b/packages/cli/src/index.ts @@ -2,6 +2,7 @@ import { defineCommand, runCommand, showUsage } from "citty"; import { authCommand } from "./commands/auth"; import { checkCommand } from "./commands/check"; +import { detectCommand } from "./commands/detect"; import { initCommand, updateCommand } from "./commands/init"; import { infoCommand } from "./commands/info"; import { createHelpCommand } from "./commands/help"; @@ -19,6 +20,7 @@ const subCommands = { init: initCommand, update: updateCommand, info: infoCommand, + detect: detectCommand, check: checkCommand, auth: authCommand, onboard: onboardCommand, diff --git a/packages/cli/src/schemas/detect.ts b/packages/cli/src/schemas/detect.ts new file mode 100644 index 00000000..1fd59e76 --- /dev/null +++ b/packages/cli/src/schemas/detect.ts @@ -0,0 +1,39 @@ +import { z } from "zod"; + +/** A linter detected from configuration on disk */ +const detectedLinterSchema = z.object({ + name: z.string().describe("Linter identifier, e.g. eslint, ruff, rubocop"), + evidence: z + .array(z.string()) + .describe( + "On-disk evidence: config-file paths, a pyproject table marker, or a dependency marker from the language's package file (not all entries are file paths)" + ), +}); + +/** A surfaced style of the repo's own existing rules */ +const ruleStyleSchema = z.object({ + source: z + .string() + .describe("Where the existing rules live, e.g. .taskless/rules"), + description: z + .string() + .describe("How the repo authors rules of this kind, for downstream reuse"), +}); + +/** Output schema for `taskless detect --json` on success */ +export const outputSchema = z.object({ + success: z.literal(true), + linters: z + .array(detectedLinterSchema) + .describe("Linters configured in the working directory"), + languages: z + .array(z.string()) + .describe("Languages inferred from manifests and detected linters"), + ruleStyles: z + .array(ruleStyleSchema) + .describe("Styles of the repo's own existing rules"), +}); + +// On the (internal-only) error path `detect` emits the standard +// `{ ok: false, code, message }` envelope via makeErrorEnvelope — there is no +// command-specific error schema to keep in sync. diff --git a/packages/cli/test/detect.test.ts b/packages/cli/test/detect.test.ts new file mode 100644 index 00000000..924d44bf --- /dev/null +++ b/packages/cli/test/detect.test.ts @@ -0,0 +1,289 @@ +import { execFile } from "node:child_process"; +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { promisify } from "node:util"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +const execFileAsync = promisify(execFile); +const binPath = resolve(import.meta.dirname, "../dist/index.js"); + +interface ExecError extends Error { + stdout?: string; + stderr?: string; + code?: number; +} + +async function runCli( + args: string[], + cwd: string +): Promise<{ stdout: string; stderr: string; exitCode: number }> { + try { + const { stdout, stderr } = await execFileAsync("node", [binPath, ...args], { + cwd, + // Keep detect hermetic: telemetry is best-effort and would otherwise + // write an anonymous-id file and attempt network I/O on shutdown, which + // must never be part of detect's offline scan path. + env: { + ...process.env, + DO_NOT_TRACK: "1", + TASKLESS_TELEMETRY_DISABLED: "1", + }, + }); + return { stdout, stderr, exitCode: 0 }; + } catch (error) { + const error_ = error as ExecError; + return { + stdout: error_.stdout ?? "", + stderr: error_.stderr ?? "", + exitCode: error_.code ?? 1, + }; + } +} + +interface DetectJson { + success: boolean; + linters: { name: string; evidence: string[] }[]; + languages: string[]; + ruleStyles: { source: string; description: string }[]; +} + +async function detect(cwd: string): Promise { + const { stdout, exitCode } = await runCli( + ["detect", "--json", "-d", cwd], + cwd + ); + expect(exitCode).toBe(0); + return JSON.parse(stdout.trim()) as DetectJson; +} + +function linterNames(result: DetectJson): string[] { + return result.linters.map((l) => l.name); +} + +describe("taskless detect", () => { + let cwd: string; + + beforeEach(async () => { + cwd = await mkdtemp(join(tmpdir(), "taskless-detect-")); + }); + + afterEach(async () => { + await rm(cwd, { recursive: true, force: true }); + }); + + it("detects eslint from a config file", async () => { + await writeFile(join(cwd, ".eslintrc.json"), "{}", "utf8"); + const result = await detect(cwd); + expect(linterNames(result)).toContain("eslint"); + }); + + it("detects ruff from a pyproject [tool.ruff] table", async () => { + await writeFile( + join(cwd, "pyproject.toml"), + "[tool.ruff]\nline-length = 88\n", + "utf8" + ); + const result = await detect(cwd); + expect(linterNames(result)).toContain("ruff"); + expect(result.languages).toContain("Python"); + }); + + it("detects rubocop from .rubocop.yml", async () => { + await writeFile(join(cwd, ".rubocop.yml"), "", "utf8"); + await writeFile(join(cwd, "Gemfile"), "", "utf8"); + const result = await detect(cwd); + expect(linterNames(result)).toContain("rubocop"); + expect(result.languages).toContain("Ruby"); + }); + + it("detects golangci-lint for a Go repo", async () => { + await writeFile(join(cwd, "go.mod"), "module example.com/x\n", "utf8"); + await writeFile(join(cwd, ".golangci.yml"), "", "utf8"); + const result = await detect(cwd); + expect(linterNames(result)).toContain("golangci-lint"); + expect(result.languages).toContain("Go"); + }); + + it("detects phpstan for a PHP repo", async () => { + await writeFile(join(cwd, "composer.json"), "{}", "utf8"); + await writeFile(join(cwd, "phpstan.neon"), "", "utf8"); + const result = await detect(cwd); + expect(linterNames(result)).toContain("phpstan"); + expect(result.languages).toContain("PHP"); + }); + + it("detects clippy for a Rust repo", async () => { + await writeFile(join(cwd, "Cargo.toml"), '[package]\nname = "x"\n', "utf8"); + await writeFile(join(cwd, "clippy.toml"), "", "utf8"); + const result = await detect(cwd); + expect(linterNames(result)).toContain("clippy"); + expect(result.languages).toContain("Rust"); + }); + + it("detects biome from biome.json", async () => { + await writeFile(join(cwd, "biome.json"), "{}", "utf8"); + const result = await detect(cwd); + expect(linterNames(result)).toContain("biome"); + }); + + it("detects stylelint from a package.json devDependency", async () => { + await writeFile( + join(cwd, "package.json"), + JSON.stringify({ devDependencies: { stylelint: "^16.0.0" } }), + "utf8" + ); + const result = await detect(cwd); + expect(linterNames(result)).toContain("stylelint"); + }); + + it("infers languages from package.json", async () => { + await writeFile( + join(cwd, "package.json"), + JSON.stringify({ + dependencies: { react: "^18", next: "^14" }, + devDependencies: { typescript: "^5" }, + }), + "utf8" + ); + const result = await detect(cwd); + expect(result.languages).toEqual( + expect.arrayContaining(["JavaScript", "TypeScript"]) + ); + }); + + it("detects ruff from a pyproject [project] dependency", async () => { + await writeFile( + join(cwd, "pyproject.toml"), + '[project]\nname = "x"\ndependencies = ["ruff>=0.4", "requests"]\n', + "utf8" + ); + const result = await detect(cwd); + expect(linterNames(result)).toContain("ruff"); + expect(result.languages).toContain("Python"); + }); + + it("detects flake8 from a requirements.txt entry", async () => { + await writeFile( + join(cwd, "requirements.txt"), + "# linting\nflake8==7.0.0\nrequests>=2\n", + "utf8" + ); + const result = await detect(cwd); + expect(linterNames(result)).toContain("flake8"); + expect(result.languages).toContain("Python"); + }); + + it("does not look up a Python linter dependency in package.json", async () => { + // A node manifest naming `ruff` must not register the Python linter — deps + // are sourced from the language's own manifest, never conflated. + await writeFile( + join(cwd, "package.json"), + JSON.stringify({ dependencies: { ruff: "^1.0.0" } }), + "utf8" + ); + const result = await detect(cwd); + expect(linterNames(result)).not.toContain("ruff"); + }); + + it("degrades gracefully on a malformed pyproject.toml", async () => { + // A TOML parse failure drops only the pyproject-derived signal; the file's + // presence still marks Python and a config file is an independent tell. + await writeFile( + join(cwd, "pyproject.toml"), + "this is = = not valid toml [[[\n", + "utf8" + ); + await writeFile(join(cwd, "ruff.toml"), "line-length = 88\n", "utf8"); + const result = await detect(cwd); + expect(result.success).toBe(true); + expect(result.languages).toContain("Python"); + expect(linterNames(result)).toContain("ruff"); + }); + + it("detects a linter configured in a sub-package (monorepo)", async () => { + await mkdir(join(cwd, "packages", "api"), { recursive: true }); + await writeFile( + join(cwd, "packages", "api", ".eslintrc.json"), + "{}", + "utf8" + ); + const result = await detect(cwd); + expect(linterNames(result)).toContain("eslint"); + const eslint = result.linters.find((l) => l.name === "eslint"); + expect(eslint?.evidence).toContain("packages/api/.eslintrc.json"); + }); + + it("ignores linter configs inside node_modules", async () => { + await mkdir(join(cwd, "node_modules", "some-dep"), { recursive: true }); + await writeFile( + join(cwd, "node_modules", "some-dep", ".eslintrc.json"), + "{}", + "utf8" + ); + const result = await detect(cwd); + expect(linterNames(result)).not.toContain("eslint"); + }); + + it("surfaces the repo's own Taskless rule styles", async () => { + await mkdir(join(cwd, ".taskless", "rules"), { recursive: true }); + const result = await detect(cwd); + expect(result.ruleStyles.some((s) => s.source === ".taskless/rules")).toBe( + true + ); + }); + + it("emits a stable JSON shape with only signal keys (no packaged-rule claims)", async () => { + await writeFile(join(cwd, ".eslintrc.json"), "{}", "utf8"); + const result = await detect(cwd); + expect(Object.keys(result).toSorted()).toEqual( + ["languages", "linters", "ruleStyles", "success"].toSorted() + ); + // A linter entry exposes only name + evidence, never a rule-name claim. + for (const linter of result.linters) { + expect(Object.keys(linter).toSorted()).toEqual( + ["evidence", "name"].toSorted() + ); + } + }); + + it("does not false-positive a pyproject table on a similarly-prefixed sibling", async () => { + // `[tool.ruff-lsp]` must NOT be read as the `ruff` tool table. + await writeFile( + join(cwd, "pyproject.toml"), + "[tool.ruff-lsp]\nfoo = 1\n", + "utf8" + ); + const result = await detect(cwd); + expect(linterNames(result)).not.toContain("ruff"); + }); + + it("detects ruff from a nested pyproject table ([tool.ruff.lint])", async () => { + await writeFile( + join(cwd, "pyproject.toml"), + '[tool.ruff.lint]\nselect = ["E"]\n', + "utf8" + ); + const result = await detect(cwd); + expect(linterNames(result)).toContain("ruff"); + }); + + it("ignores a malformed package.json dependency field without crashing", async () => { + // `dependencies` as an array (not an object) must not yield bogus deps. + await writeFile( + join(cwd, "package.json"), + JSON.stringify({ dependencies: ["eslint"], devDependencies: null }), + "utf8" + ); + const result = await detect(cwd); + expect(result.success).toBe(true); + // The array form yields no dependency names, so no dep-based linter. + expect(linterNames(result)).not.toContain("eslint"); + }); + + it("runs successfully with no linters, no network, and no auth", async () => { + const result = await detect(cwd); + expect(result.success).toBe(true); + expect(result.linters).toEqual([]); + }); +}); diff --git a/packages/cli/test/fixtures/route-eval.json b/packages/cli/test/fixtures/route-eval.json new file mode 100644 index 00000000..38813b2c --- /dev/null +++ b/packages/cli/test/fixtures/route-eval.json @@ -0,0 +1,79 @@ +{ + "description": "Honesty eval fixtures for the `route` recipe. Each case is a labeled request with the destination route SHOULD reach. `trap` names the failure direction the case guards against (or null for a clean case). These are a calibration dataset for evaluating the route recipe's decisions; the route decision itself is made by an agent following help/route.txt, so these are not asserted against a code classifier — the coverage test only checks the dataset stays balanced across routes and both failure directions.", + "routes": ["existing", "static", "remote"], + "traps": ["over-claim", "over-escalate", "under-engage"], + "cases": [ + { + "request": "warn on console.log", + "expected": "existing", + "trap": "over-claim", + "reason": "ESLint's packaged no-console already covers this; claiming it for a Taskless rule erodes trust where Taskless would never win." + }, + { + "request": "sort our imports", + "expected": "existing", + "trap": "over-claim", + "reason": "Formatter / eslint-plugin-import territory, not a semantic rule." + }, + { + "request": "disallow `any` in TypeScript", + "expected": "existing", + "trap": "over-claim", + "reason": "@typescript-eslint/no-explicit-any is a packaged rule; enable it rather than authoring." + }, + { + "request": "write a Ruff rule for X (repo has ruff.toml)", + "expected": "existing", + "trap": "under-engage", + "reason": "Naming a tool must ENGAGE routing into that tool, not suppress the skill." + }, + { + "request": "add an ESLint rule banning direct process.env reads outside config/", + "expected": "existing", + "trap": "under-engage", + "reason": "User named ESLint and wants a custom rule; author it in ESLint's dialect, mined from the repo's own rules." + }, + { + "request": "controllers must not import the db module directly", + "expected": "static", + "trap": null, + "reason": "Bespoke syntactic import ban, no packaged rule, cleanly expressible as a local ast-grep rule." + }, + { + "request": "never call fetch directly — use our http wrapper", + "expected": "static", + "trap": null, + "reason": "Call/import ban; classic local ast-grep pattern." + }, + { + "request": "enforce our feature-flag naming convention ff_*", + "expected": "static", + "trap": "over-escalate", + "reason": "Repo-specific identifier pattern is a simple local ast-grep rule; escalating to the service here wastes a generation and a login." + }, + { + "request": "flag string literals that look like API keys in source files", + "expected": "static", + "trap": "over-escalate", + "reason": "A regex-shaped syntactic match is locally solvable; do not route to remote just because it sounds security-ish." + }, + { + "request": "every API handler must parse its request body through a Zod schema before using it", + "expected": "remote", + "trap": null, + "reason": "Cross-statement intent (parse-before-use) that a single ast-grep pattern strains to express; let the service classify static vs runtime." + }, + { + "request": "flag any PR that adds a new env var without documenting it in the README", + "expected": "remote", + "trap": null, + "reason": "Cross-file, config-vs-docs consistency — needs a runtime check.ts the service generates." + }, + { + "request": "ensure every exported function in a route file has a matching test in __tests__", + "expected": "remote", + "trap": null, + "reason": "Cross-file correlation no static syntactic matcher expresses; service/runtime territory." + } + ] +} diff --git a/packages/cli/test/help-extensions.test.ts b/packages/cli/test/help-extensions.test.ts index 998dd553..10688669 100644 --- a/packages/cli/test/help-extensions.test.ts +++ b/packages/cli/test/help-extensions.test.ts @@ -58,6 +58,37 @@ describe("taskless help (no args)", () => { const result = await runCli(["help", "-d", cwd]); expect(result.stdout).toContain("--anonymous"); }); + + it("lists the routing recipe topics under Authoring recipes", async () => { + const result = await runCli(["help", "-d", cwd]); + expect(result.stdout).toContain("Authoring recipes:"); + for (const topic of ["route", "existing", "static", "remote"]) { + expect(result.stdout).toContain(topic); + } + }); +}); + +describe("taskless help ", () => { + let cwd: string; + + beforeEach(async () => { + cwd = await mkdtemp(join(tmpdir(), "taskless-help-routing-")); + }); + + afterEach(async () => { + await rm(cwd, { recursive: true, force: true }); + }); + + it.each(["route", "existing", "static", "remote"])( + "resolves the %s recipe without an unknown-topic error", + async (topic) => { + const result = await runCli(["help", topic, "-d", cwd]); + expect(result.exitCode).toBe(0); + expect(result.stdout).toContain(`# Topic: ${topic}`); + expect(result.stdout).toContain("## Goal"); + expect(result.stderr).not.toContain("Unknown command"); + } + ); }); describe("taskless help ", () => { diff --git a/packages/cli/test/help-routing-telemetry.test.ts b/packages/cli/test/help-routing-telemetry.test.ts new file mode 100644 index 00000000..ece49ffe --- /dev/null +++ b/packages/cli/test/help-routing-telemetry.test.ts @@ -0,0 +1,55 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +// Spy on the telemetry capture by mocking the telemetry module the help +// command imports. The factory is invoked lazily at import time, so the +// closure over `capture` resolves after initialization (same pattern as +// telemetry.test.ts mocking posthog-node). +const capture = vi.fn(); +vi.mock("../src/telemetry", () => ({ + getTelemetry: vi.fn(() => + Promise.resolve({ + capture, + shutdown: () => Promise.resolve(), + }) + ), + shutdownTelemetry: () => Promise.resolve(), +})); + +const { createHelpCommand } = await import("../src/commands/help"); + +interface RunnableCommand { + run: (context: { + args: { dir: string; anonymous: boolean }; + rawArgs: string[]; + }) => Promise; +} + +describe("help routing topics emit cli_help intent telemetry", () => { + let logSpy: ReturnType; + + beforeEach(() => { + capture.mockClear(); + // Suppress the recipe text the command prints to stdout. + logSpy = vi.spyOn(console, "log").mockImplementation(() => {}); + }); + + afterEach(() => { + logSpy.mockRestore(); + }); + + it.each(["route", "existing", "static", "remote"])( + "captures cli_help for %s", + async (topic) => { + const command = createHelpCommand({}) as unknown as RunnableCommand; + await command.run({ + args: { dir: process.cwd(), anonymous: false }, + rawArgs: ["help", topic], + }); + + expect(capture).toHaveBeenCalledWith( + "cli_help", + expect.objectContaining({ topic }) + ); + } + ); +}); diff --git a/packages/cli/test/route-eval.test.ts b/packages/cli/test/route-eval.test.ts new file mode 100644 index 00000000..99f5256d --- /dev/null +++ b/packages/cli/test/route-eval.test.ts @@ -0,0 +1,90 @@ +import { readFileSync } from "node:fs"; +import { resolve } from "node:path"; +import { describe, expect, it } from "vitest"; + +// The route decision is made by an agent following help/route.txt, so this +// dataset is not run against a code classifier. The test guards the dataset +// itself: it must stay structurally valid and balanced across every route and +// both failure directions, so it remains a usable calibration set. + +interface EvalCase { + request: string; + expected: string; + trap: string | null; + reason: string; +} + +interface EvalFixtures { + routes: string[]; + traps: string[]; + cases: EvalCase[]; +} + +const fixtures = JSON.parse( + readFileSync(resolve(import.meta.dirname, "fixtures/route-eval.json"), "utf8") +) as EvalFixtures; + +const casesForTrap = (trap: string): EvalCase[] => + fixtures.cases.filter((c) => c.trap === trap); + +describe("route honesty eval fixtures", () => { + it("declares the three routes and both failure-direction traps", () => { + expect(fixtures.routes).toEqual(["existing", "static", "remote"]); + // Over-claim = Taskless grabbing a packaged/formatter job; over-escalate = + // sending a locally-solvable request to the login-gated service. + expect(fixtures.traps).toEqual( + expect.arrayContaining(["over-claim", "over-escalate", "under-engage"]) + ); + }); + + it("every case is well-formed and uses a declared route/trap", () => { + expect(fixtures.cases.length).toBeGreaterThanOrEqual(10); + for (const c of fixtures.cases) { + expect(c.request.length).toBeGreaterThan(0); + expect(c.reason.length).toBeGreaterThan(0); + expect(fixtures.routes).toContain(c.expected); + if (c.trap !== null) { + expect(fixtures.traps).toContain(c.trap); + } + } + }); + + it("covers every route at least twice", () => { + for (const route of fixtures.routes) { + const count = fixtures.cases.filter((c) => c.expected === route).length; + expect(count, `route ${route} needs >= 2 cases`).toBeGreaterThanOrEqual( + 2 + ); + } + }); + + it("guards every declared trap with at least one correctly-routed case", () => { + // Each declared trap must have at least one case, so the dataset can't + // silently stop covering a failure direction while the test still passes. + for (const trap of fixtures.traps) { + expect( + casesForTrap(trap).length, + `trap ${trap} needs >= 1 case` + ).toBeGreaterThan(0); + } + + // Over-claim: Taskless should not have grabbed it → expect `existing`. + for (const c of casesForTrap("over-claim")) + expect(c.expected).toBe("existing"); + // Over-escalate: locally solvable → expect `static`, not remote. + for (const c of casesForTrap("over-escalate")) + expect(c.expected).toBe("static"); + // Under-engage: naming a tool must engage routing into that linter, not + // suppress → expect `existing`. + for (const c of casesForTrap("under-engage")) + expect(c.expected).toBe("existing"); + }); + + it("includes genuine remote cases that are not locally solvable", () => { + const remote = fixtures.cases.filter((c) => c.expected === "remote"); + expect(remote.length).toBeGreaterThanOrEqual(2); + // Genuine remote cases are clean (no trap) — they legitimately need the + // service, not a misroute being corrected. + expect(remote.every((c) => c.trap === null)).toBe(true); + }); +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 1ae84903..bbbc1243 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -92,6 +92,9 @@ importers: posthog-node: specifier: ^5.28.11 version: 5.28.11 + smol-toml: + specifier: ^1.6.1 + version: 1.6.1 sprintf-js: specifier: ^1.1.3 version: 1.1.3 @@ -2024,6 +2027,10 @@ packages: resolution: {integrity: sha512-iOBWFgUX7caIZiuutICxVgX1SdxwAVFFKwt1EvMYYec/NWO5meOJ6K5uQxhrYBdQJne4KxiqZc+KptFOWFSI9w==} engines: {node: '>=18'} + smol-toml@1.6.1: + resolution: {integrity: sha512-dWUG8F5sIIARXih1DTaQAX4SsiTXhInKf1buxdY9DIg4ZYPZK5nGM1VRIYmEbDbsHt7USo99xSLFu5Q1IqTmsg==} + engines: {node: '>= 18'} + source-map-js@1.2.1: resolution: {integrity: sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==} engines: {node: '>=0.10.0'} @@ -4195,6 +4202,8 @@ snapshots: ansi-styles: 6.2.3 is-fullwidth-code-point: 5.1.0 + smol-toml@1.6.1: {} + source-map-js@1.2.1: {} spawndamnit@3.0.1: diff --git a/skills/taskless/SKILL.md b/skills/taskless/SKILL.md index b5e56174..2b26955c 100644 --- a/skills/taskless/SKILL.md +++ b/skills/taskless/SKILL.md @@ -14,12 +14,10 @@ description: | - "add taskless to CI", "wire taskless into github actions" - "onboard with taskless", "set up taskless for this project" - Also trigger when the user asks to add/write/create a rule and has NOT - named a specific lint/format/static-analysis tool. Examples that suppress - this trigger (illustrative — any named tool of this kind suppresses): - eslint, ruff, biome, ast-grep. On this trigger, surface a quiet - single-line offer rather than launching a recipe; see the body's quiet - suggestion section. + Also trigger on any request to add/write/create a lint or code rule, + including ones that name a specific tool (eslint, ruff, biome, stylelint, + ast-grep). Naming a tool ENGAGES this skill's routing flow via + `taskless help route`; it does NOT suppress the skill. metadata: author: taskless version: 0.8.1 @@ -34,20 +32,33 @@ canonical recipes live behind `npx @taskless/cli help `. Always fetch the recipe first; do not improvise from prior knowledge — recipes change with each CLI version. -## First step: confirm Taskless is installed here +## Authoring a rule: always start at route -If the working directory does not contain a `.taskless/` directory, ask the -user to confirm they meant Taskless (vs. ESLint or another tool). If they -confirm, offer to run `npx @taskless/cli` to install. Otherwise, stop. +For any request to add/write/create a rule — whether or not the user names a +tool (eslint, ruff, biome, stylelint, ast-grep) — fetch `npx @taskless/cli help route` +and follow it. Do NOT fetch `rule create` directly, and do NOT author from +your own linter knowledge. `route` runs `detect`, reasons about the request, +and decides whether the rule is built in an existing linter (`existing`), as a +local ast-grep rule (`static`), or via the Taskless service (`remote`) — and it +keeps the work local before any login. This skill is a thin router: all +authoring judgment lives in the fetched recipes. + +## Confirm Taskless is installed when a path needs it + +`route` and the `existing` path only read the repo, so they need no install. If +routing lands on a local Taskless rule (`static`) or the service (`remote`) and +the working directory has no `.taskless/` directory, offer to run +`npx @taskless/cli` to install. If the user only wanted help with their own +linter, the `existing` path needs nothing installed. ## Topics | User wants | Topic | | -------------------------- | ------------------------------------- | +| Author/create a rule | `npx @taskless/cli help route` | | First-time install | tell user to run `npx @taskless/cli` | | Update existing install | `npx @taskless/cli update` | | Discover candidate rules | `npx @taskless/cli help onboard` | -| Create a new rule | `npx @taskless/cli help rule create` | | Improve an existing rule | `npx @taskless/cli help rule improve` | | Delete a rule | `npx @taskless/cli help rule delete` | | Check code against rules | `npx @taskless/cli help check` | @@ -58,19 +69,6 @@ If the user's intent is ambiguous between two topics, run `npx @taskless/cli help` (no args) to see the disambiguation table, or ask the user. -## Quiet suggestion - -When you trigger because the user asked to add/write/create a rule and did -NOT name a specific tool (per the description's named-tool clause), DO NOT -launch into a full recipe. Instead surface a one-line offer such as: "I can -capture this as a Taskless rule if you want — say so, or I'll proceed with -." If the user accepts, fetch -`npx @taskless/cli help rule create` and follow it. If the user declines or -ignores the offer, proceed with what you would have done without this skill, -and DO NOT re-offer Taskless in the same conversation. The decline is sticky -within the conversation only — do NOT write any persistent decline state to -disk or to `.taskless/taskless.json`. - ## --anonymous Any rule/check command accepts `--anonymous` to skip the Taskless API and