diff --git a/.changeset/legacy-linter-recipes.md b/.changeset/legacy-linter-recipes.md new file mode 100644 index 00000000..8020d930 --- /dev/null +++ b/.changeset/legacy-linter-recipes.md @@ -0,0 +1,7 @@ +--- +"@taskless/cli": patch +--- + +Rules for a linter the repository already runs now follow a fixed order. The agent first looks for a rule the linter already ships, checking each tool's catch-all rules by name (`no-restricted-syntax`, `banned-api`, `noRestrictedImports`, and others). It writes a new custom rule in that linter only when the repository already has five or more of its own to model it on. Otherwise the request becomes a Taskless rule. `route` (topic v8) no longer asks the user to choose between a linter rule and a Taskless rule, and it describes each linter `detect` found in the repository. During onboarding (topic v11), the agent only enables rules the repository's installed packages already ship. + +`detect --json` reports `customRules` for ESLint, Biome, pylint, flake8, and Ruff: how many rules the repository wrote itself for that linter, and where they are. A count that cannot be read without running a JavaScript config is reported as `"unknown"`. diff --git a/openspec/changes/legacy-linter-recipes/.openspec.yaml b/openspec/changes/legacy-linter-recipes/.openspec.yaml new file mode 100644 index 00000000..7cc41bb5 --- /dev/null +++ b/openspec/changes/legacy-linter-recipes/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-10-22 diff --git a/openspec/changes/legacy-linter-recipes/design.md b/openspec/changes/legacy-linter-recipes/design.md new file mode 100644 index 00000000..76389aee --- /dev/null +++ b/openspec/changes/legacy-linter-recipes/design.md @@ -0,0 +1,78 @@ +## Context + +`route` already names `create-legacy-rule` for "a tool the repo already runs", onboarding lists `legacy` as a candidate destination, and the report has a `legacy` section that counts toward its three-rule gate. The gap is the recipe, which is generic and teaches nothing tool-specific. See proposal.md for motivation. + +Constraints that shape the approach: + +- Recipes are embedded at build time (`src/prompts/recipes.ts`). Variants already exist as `.anonymous.md`, selected by `--anonymous`. +- `route` already has runtime-rendered slots (`%(DETECT_EVIDENCE)s`, `%(HOST_TOOLS)s`), so a per-linter paragraph is the same mechanism with a new slot. +- `detectRuleStyles` (`src/detect/scan.ts`) only recognizes ESLint local rule directories and plugin packages, and reports styles, not counts. +- Unlike Vale and ast-grep, Taskless does not pin these linters. The installed version is whatever the repository has, so recipe claims must hold across the versions a repository is likely to run, or say where they diverge. + +## Goals / Non-Goals + +**Goals:** + +- An agent that reaches a legacy destination finds and enables an existing rule correctly, including the traps for its tool and version. +- The decision to write a new custom rule rests on a count the CLI computes, not on the agent's impression. + +**Non-Goals:** + +- Teaching rule authoring for any linter. Step 2 learns from the repository's examples only. +- Running external linters from `taskless check`. +- Recipes for linters beyond the five launch tools. The generic recipe still covers them. +- Matching a request to a named rule inside the CLI. That stays the agent's job, so `detect`'s "no packaged-rule catalog matching" requirement is unchanged. + +## Decisions + +**Per-tool variants reuse the `.anonymous` mechanism, keyed by `detect`'s linter name.** `create-legacy-rule..md` files are collected by the same glob. `--tool` takes precedence over `--anonymous`, since no legacy recipe has an anonymous variant. Keying on `detect`'s `name` means the agent copies a value from JSON into a flag with no interpretation. Keying on language was rejected: Python alone maps to Ruff, pylint, and flake8, which have different dialects. `--tool=anonymous` is refused so it cannot collide with the anonymous variant file. + +**`route` paragraphs render at serve time from a scan.** `route` is served with a fresh `detect` scan, the same way `HOST_TOOLS` is, and gains one paragraph per detected linter that has a legacy recipe. A static table of every tool was rejected: it grows with every linter we add, and an agent in a Ruff repository gains nothing from reading about Biome. With `mechanics === false` (recipes served without a CLI), the slot renders the full table, because there is no repository to scan. + +**The five-rule threshold is a `detect` count, not recipe prose.** The threshold decides whether an agent may write code into a linter, so it should be computed rather than judged. `detect` already reads the config files involved. Counting rules means: + +- **ESLint:** rule files and rule folders in a house rule directory (`eslint-rules`, `eslint-local-rules`, `tools/eslint-rules`). Index files, tests, type declarations, and helper folders are not counted. A flat config that imports a plugin from a relative path or defines rules inline, and a local-rules loader package, are custom rules whose count cannot be read without executing JavaScript, so they are reported as evidence with an `"unknown"` count. A counted directory of five or more settles it either way. Otherwise unknown stays unknown, and the recipe treats it as below the threshold. +- **Biome:** the length of `plugins` across the root and nested configs. +- **pylint:** `load-plugins` entries that resolve to a module inside the repository. +- **flake8:** checkers under `[flake8:local-plugins]`. + +**Discovery for Ruff and Biome is web-first, then confirmed locally.** Both tools ship too many rules to match a request against: `ruff rule --all --output-format json` returned 972 rules on 0.17.2. External testing found no reliable way for an agent to search that list for a request. The web finds the candidate. `ruff rule ` and `biome explain ` confirm that it exists in the installed version and reveal preview status. + +**The ESLint plugin list is curated and linked, and the recipe never lists individual rules.** Eight plugins, each with a summary and a docs link. This narrows the standing "no catalog" line instead of removing it, because a catalog of rules goes stale with every plugin release, while a list of plugins changes rarely. + +**Vendor tests pin recipe claims against real installs.** Each claim a recipe makes about tool behavior gets a test that runs the tool, in the same spirit as the existing Vale and ast-grep vendor-contract tests: preview status in the `ruff rule` JSON, `--help-msg` failing without `--load-plugins`, `biome explain` output, and `banned-modules` in `flake8-tidy-imports --help`. They run through `uvx` and `pnpm dlx` at pinned versions, so a version bump that breaks a claim fails a test instead of shipping a wrong recipe. + +## Research record + +Measured on 2026-10-22 with Ruff 0.17.2, pylint 4.1.3, flake8 7.4.1, Biome 2.6.2, and ESLint 10.13.0 current on npm. Doc quotes were fetched from official pages the same day. + +| Tool | Claim | Source | +| ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------- | +| Ruff | Default set grew to 413 rules in 0.16.0, so `select` cannot be compared against `E4/E7/E9/F` | https://github.com/astral-sh/ruff/blob/main/changelogs/0.16.x.md | +| Ruff | `banned-api` is TID251, configured under `flake8-tidy-imports` | https://docs.astral.sh/ruff/settings/ and `ruff rule TID251` | +| Ruff | `unnecessary-regular-expression-compile` is preview-only and needs `--preview` | `ruff rule unnecessary-regular-expression-compile` | +| Ruff | No custom-rule API exists | no official page describes one | +| ESLint | v10 removed `.eslintrc*`; config lookup starts from each linted file | https://eslint.org/docs/latest/use/migrate-to-10.0.0 | +| ESLint | Local and inline plugins are supported in flat config | https://eslint.org/docs/latest/use/configure/plugins | +| ESLint | `eslint-plugin-import` 2.32.0, `eslint-plugin-react` 7.37.5, `eslint-plugin-jsx-a11y` 6.10.2 declare `eslint` peers only up to 9. `eslint-plugin-import-x` 4.17.1 declares `^10.0.0` | `npm view peerDependencies.eslint` | +| Biome | GritQL plugins since 2.0; JS plugins a developer preview since 2.6 | https://biomejs.dev/linter/plugins/, https://biomejs.dev/blog/biome-v2-6/ | +| Biome | `biome explain ` prints a rule's documentation | `biome explain noRestrictedImports` on 2.6.2 | +| pylint | `--list-msgs` lists 393 messages; `--help-msg=bad-builtin` fails without `--load-plugins=pylint.extensions.bad_builtin` | measured | +| pylint | `load-plugins`, `deprecated-modules`, `preferred-modules` | https://pylint.readthedocs.io/en/stable/user_guide/configuration/all-options.html | +| flake8 | Reads only `setup.cfg`, `tox.ini`, `.flake8`; `[flake8:local-plugins]` since 3.5.0 | https://flake8.pycqa.org/en/latest/user/configuration.html | +| flake8 | No rule-listing command; `--bug-report` lists installed plugins | measured | +| flake8 | `flake8-tidy-imports` provides `--banned-modules` | `flake8 --help` with the plugin installed | + +Two claims are inferred, not quoted, and are not used in recipe text until verified: that flake8 has no native `pyproject.toml` support, and that `--rulesdir` is dead in ESLint 10. + +## Risks / Trade-offs + +- **A recipe claim drifts when a linter releases.** → The vendor tests pin each claim at a version. Bumping that version is a deliberate step that reruns them. +- **The repository runs an older linter than the recipe was measured against.** → Recipes state version boundaries where they matter (ESLint 9 against 10, Biome 1 against 2, Ruff before and after 0.16), and confirm discovery against the installed version. +- **The ESLint count is undercounted when config is computed.** → Unknown counts are treated as below the threshold, which sends the request to Taskless, the safe direction. +- **Web discovery is unavailable on the host.** → The recipe falls back to the local confirm command plus the catch-all rules, and says the search was skipped. +- **Recommending an `eslint-plugin-import` to an ESLint 10 repository installs with a peer warning.** → The recipe names `import-x` for that case and prefers whatever is already installed. + +## Migration Plan + +Recipe and additive JSON changes only. The `detect --json` schema gains a field, which existing readers ignore. Rollback is a revert of the recipe files. There is no state to migrate. diff --git a/openspec/changes/legacy-linter-recipes/proposal.md b/openspec/changes/legacy-linter-recipes/proposal.md new file mode 100644 index 00000000..c7fbdd34 --- /dev/null +++ b/openspec/changes/legacy-linter-recipes/proposal.md @@ -0,0 +1,53 @@ +## Why + +When a repository already runs a linter, `create-legacy-rule` hands the agent 64 generic lines that say "mine the repo, then search the web". Each agent works out every linter's dialect from scratch, so it misses rules the tool already ships, gets traps like preview-only rules wrong, and writes rules into tools the team does not maintain. We are adding linters, and teaching an agent to author rules for hundreds of them is not sustainable. What is sustainable is teaching it to use what each tool already has. + +## What Changes + +- **A legacy ladder replaces the open-ended legacy destination.** For a request in a repository with a detected linter, `route` applies these steps in order: + 1. **Always: would a rule the tool already ships solve this?** Each tool has a step it always takes. Ruff always tries `banned-api` (TID251). ESLint always tries a built-in rule, then a curated plugin list. Biome always checks whether an existing rule makes the problem go away. + 2. **Only when the repository already has five or more of its own custom rules for that linter:** author another one, using those rules as the reference examples, not the tool's documentation. + 3. A Taskless `sg` or `vale` rule. + 4. A Taskless runtime rule. +- **Taskless does not teach linter rule authoring.** The legacy recipes carry no instructions for writing a custom ESLint rule, a pylint checker, a flake8 plugin, a Biome GritQL or JS plugin, or anything else. Step 2 learns only from the repository's own examples. A check that needs custom code and has no such examples belongs to Taskless. +- **Onboarding skips step 2 and never adds a dependency.** Onboarding runs logged out and writes no executable code, so it uses only step 1, and only with rules from packages the repository already has installed. A plugin that is not installed is something to recommend outside onboarding, after the user agrees to add it. +- **`taskless agent create-legacy-rule --tool=`** returns a per-tool recipe. `` is the linter `name` that `detect --json` reports. An unknown tool falls back to the generic recipe. Tools that cannot hold a rule of this kind (`ast-grep` and `vale`, which have their own recipes, and the formatters `prettier` and `black`) are refused with a pointer to the right recipe. +- **Per-tool recipes at launch: Ruff, ESLint (including typescript-eslint), Biome, pylint, flake8.** TSLint is not supported. Each recipe covers how to find an existing rule for a request, the tool's catch-all rules, version-specific traps, and how to run the tool so the user can see the rule fire: + - **Ruff and Biome:** find a rule by web search of the rules page, then confirm it against the installed version with `ruff rule ` or `biome explain `. Both tools have too many rules to match a request against locally. Ruff marks preview-only rules, and enabling one without `preview = true` means it never fires. + - **ESLint:** core rules first, with `no-restricted-syntax` and `no-restricted-imports` (and their `@typescript-eslint/` versions) as the catch-alls. Then a curated list of eight plugins, each with a one-line summary and a docs link. + - **pylint:** `--list-msgs` and `--help-msg`, with `deprecated-modules`, `preferred-modules`, and `bad-functions` as the catch-alls. + - **flake8:** no discovery step. The catch-all is `banned-modules` from `flake8-tidy-imports`. +- **`detect` counts the repository's own rules per linter.** `detect --json` adds a count of custom rules for each linter, which is the signal step 2 reads. The count covers local ESLint plugins and rule directories, Biome `plugins` entries, pylint `load-plugins` modules inside the repository, and flake8 `[flake8:local-plugins]`. Built-in extensions and third-party packages do not count. Neither does a config file on its own. +- **`route` gives a paragraph to each linter it detects.** The CLI renders each paragraph at runtime, and only for linters `detect` found. Each one says what the tool is for, what it always tries first, and the `--tool` command to fetch. +- **The catalog prohibition is lifted, narrowly.** The standing requirement says `create-legacy-rule` SHALL NOT embed a Taskless-maintained catalog of linter rules. The curated ESLint plugin list and each tool's named catch-all rules are a catalog, so that line changes to allow a short, curated, linked list of rule sources. It still forbids a catalog of individual rules. +- **`route` stops asking the user when both legacy and a Taskless rule fit.** The ladder's order decides instead. This also resolves a conflict with onboarding, which told the agent not to ask. +- Topic versions bump: `route` v7 → v8, `create-legacy-rule` v2 → v3, `onboard` v10 → v11. + +## Capabilities + +### New Capabilities + +- `cli-legacy-rule-recipes`: the `--tool` per-tool variants, the five launch recipes and what each must cover, the generic fallback, refused tools, and the rule that legacy recipes do not teach rule authoring. + +### Modified Capabilities + +- `cli-rule-routing`: "Existing recipe authors in the detected linter's dialect" is restated for the ladder and the narrowed catalog line. "Route asks the user when multiple paths fit" no longer poses legacy against static as a choice. A new requirement covers the ladder and the per-linter paragraphs. +- `cli-detect`: a new requirement covers the per-linter custom-rule count. "No packaged-rule catalog matching" is unchanged, because the curated list lives in recipes and is not part of detection. +- `cli-agent`: "Agent subcommand serves recipes for commands" gains the `--tool` flag and its resolution order. +- `cli-onboard`: a new requirement limits onboarding to step 1, with installed packages only. + +## Delivery Shape + +**Stacked, merging forward.** Each PR is safe in production on its own: + +1. **The `detect` custom-rule count, the ladder in `route`, the per-linter paragraphs, and the generic `create-legacy-rule` v3 and onboard v11 that follow the ladder.** The paragraphs name only `create-legacy-rule` here, so nothing references a variant that does not exist yet. The changeset lands on this PR. +2. **`--tool`, the five per-tool recipes and their vendor tests.** The `route` paragraphs start naming `--tool=`. This PR archives the change. + +The package is pre-1.0 and every surface here is additive or recipe text, so the bump is `patch`. + +## Impact + +- **Code**: `packages/cli/src/detect/scan.ts` and the detect schema; `packages/cli/src/commands/agent.ts` (the `--tool` flag); `packages/cli/src/prompts/recipes.ts` (variant lookup, runtime-rendered linter paragraphs); `packages/cli/src/agent/` (`route.md`, `create-legacy-rule.md`, `onboard.md`, five new `create-legacy-rule..md` files). +- **Tests**: vendor tests that pin each recipe's claims against a real install of that linter (preview status in `ruff rule --output-format json`, `--help-msg` needing `--load-plugins`, `biome explain` output). These need Ruff, Biome, pylint, flake8, and ESLint available in CI. +- **No new runtime dependency** for the CLI. Recipes recommend plugins; the CLI installs nothing. +- **Research**: the per-tool signals and traps come from official docs and measured runs, recorded in `design.md` with the source for each. diff --git a/openspec/changes/legacy-linter-recipes/specs/cli-agent/spec.md b/openspec/changes/legacy-linter-recipes/specs/cli-agent/spec.md new file mode 100644 index 00000000..973dece6 --- /dev/null +++ b/openspec/changes/legacy-linter-recipes/specs/cli-agent/spec.md @@ -0,0 +1,54 @@ +## MODIFIED Requirements + +### Requirement: Agent subcommand serves recipes for commands + +The CLI SHALL support an `agent` subcommand that accepts at most one positional argument identifying a topic, an optional `--anonymous` boolean flag, AND an optional `--tool ` value flag. Topics SHALL be addressed by a single token; the subcommand SHALL NOT join multiple positionals into a topic key. When a topic is provided, the subcommand SHALL look up a matching recipe file embedded at build time using the following resolution order: + +1. If `--tool` is set AND `..md` exists in the embedded map, return that file. +2. If `--anonymous` is set AND `.anonymous.md` exists in the embedded map, return that file. +3. Otherwise, return `.md`. +4. If none of these exists, exit with code 1 and an error message suggesting `taskless agent` for the topic index. + +`--tool` SHALL refuse, with a non-zero exit and a message naming the recipe to use instead, a tool that has its own authoring recipe (`ast-grep`, `vale`) or that cannot hold a rule (`prettier`, `black`). An unrecognized tool name SHALL fall back to `.md`. + +When no positional argument is provided, the subcommand SHALL print a topic index containing a one-paragraph human slug followed by a topic disambiguation table mapping topic names to their summaries. + +The subcommand is named for its reader. It serves agents fetching a procedure, not humans asking for help, and single-token addressing exists so a topic name is a literal string an agent copies rather than a phrase it can reorder or paraphrase. + +#### Scenario: Agent subcommand for a topic returns the recipe + +- **WHEN** a user runs `taskless agent check` +- **THEN** the CLI SHALL print the contents of `check.md` to stdout + +#### Scenario: Multi-word topic paths are not resolved + +- **WHEN** a user runs `taskless agent rule create` +- **THEN** the CLI SHALL NOT look up `rule-create.md` by joining the positionals +- **AND** it SHALL exit non-zero rather than guessing a topic + +#### Scenario: Formerly nested topics are addressed by one token + +- **WHEN** a user runs `taskless agent improve-rule` +- **THEN** the CLI SHALL look up `improve-rule.md` and print its contents + +#### Scenario: The former command name is gone + +- **WHEN** a user runs `taskless help check` +- **THEN** the CLI SHALL NOT print recipe text for `check` + +#### Scenario: A tool variant is served + +- **WHEN** a user runs `taskless agent create-legacy-rule --tool=ruff` +- **AND** `create-legacy-rule.ruff.md` exists +- **THEN** the CLI SHALL print that file's contents + +#### Scenario: An unknown tool falls back to the generic recipe + +- **WHEN** a user runs `taskless agent create-legacy-rule --tool=rubocop` +- **AND** no `create-legacy-rule.rubocop.md` exists +- **THEN** the CLI SHALL print `create-legacy-rule.md` + +#### Scenario: A tool with its own recipe is refused + +- **WHEN** a user runs `taskless agent create-legacy-rule --tool=vale` +- **THEN** the CLI SHALL exit non-zero and name `create-vale-rule` as the recipe to use diff --git a/openspec/changes/legacy-linter-recipes/specs/cli-detect/spec.md b/openspec/changes/legacy-linter-recipes/specs/cli-detect/spec.md new file mode 100644 index 00000000..45845ef8 --- /dev/null +++ b/openspec/changes/legacy-linter-recipes/specs/cli-detect/spec.md @@ -0,0 +1,27 @@ +## ADDED Requirements + +### Requirement: Detect counts the repository's own rules per linter + +`detect --json` SHALL report, for each detected linter that can host custom rules, how many custom rules the repository itself authors for it, and the evidence for that count. A custom rule is one whose logic the repository wrote: + +- ESLint: rules in a local plugin or rule directory, including plugins defined inline in the config or imported from a file in the repository. +- Biome: entries in the configuration's `plugins` list. +- pylint: modules named by `load-plugins` that live in the repository. +- flake8: checkers registered under `[flake8:local-plugins]`, or a `flake8.extension` entry point the repository publishes. + +Optional extensions shipped with the tool, third-party plugin packages, presets, and rule selection SHALL NOT count. Ruff has no custom-rule mechanism and SHALL report a count of zero. The count SHALL be derived from files on disk without executing anything, consistent with detect's other signals. + +#### Scenario: Local ESLint rules are counted + +- **WHEN** the repository has a local ESLint plugin defining six rules +- **THEN** `detect --json` SHALL report a custom-rule count of six for ESLint with the plugin's path as evidence + +#### Scenario: A built-in pylint extension does not count + +- **WHEN** `load-plugins` names only `pylint.extensions.*` modules +- **THEN** `detect --json` SHALL report a custom-rule count of zero for pylint + +#### Scenario: Ruff reports no custom rules + +- **WHEN** the repository configures Ruff, however extensively +- **THEN** `detect --json` SHALL report a custom-rule count of zero for Ruff diff --git a/openspec/changes/legacy-linter-recipes/specs/cli-legacy-rule-recipes/spec.md b/openspec/changes/legacy-linter-recipes/specs/cli-legacy-rule-recipes/spec.md new file mode 100644 index 00000000..9ba18149 --- /dev/null +++ b/openspec/changes/legacy-linter-recipes/specs/cli-legacy-rule-recipes/spec.md @@ -0,0 +1,107 @@ +## Purpose + +Per-tool recipes that teach an agent to solve a request with the rules a linter the repository already runs ships, and to write a new rule in that linter only when the repository's own rules show how. + +## ADDED Requirements + +### Requirement: Legacy recipes use what a tool ships before anything is written + +Every `create-legacy-rule` recipe, generic or per-tool, SHALL direct the agent to first establish whether a rule the tool already ships solves the request, including the tool's configurable catch-all rules. A recipe SHALL name the catch-all rules for its tool explicitly, because a search for the request's subject does not surface a catch-all rule. + +A per-tool recipe SHALL state the check its tool always performs first: + +- Ruff: whether `banned-api` (TID251) covers the request. +- ESLint: whether a core rule covers it, with `no-restricted-syntax` and `no-restricted-imports` (and their `@typescript-eslint/` counterparts) named as the catch-alls, before any plugin. +- Biome: whether an existing rule makes the problem go away, with `noRestrictedImports` and `noRestrictedGlobals` named as the catch-alls. +- pylint: whether a shipped message or the `deprecated-modules`, `preferred-modules`, or `bad-functions` options cover it. +- flake8: whether `banned-modules` from `flake8-tidy-imports` covers it. The flake8 recipe SHALL NOT include a discovery step. + +#### Scenario: Ruff tries banned-api first + +- **WHEN** the agent follows the Ruff recipe for a request to forbid an API +- **THEN** the recipe SHALL direct it to check `banned-api` before any other step + +#### Scenario: ESLint tries a core rule before a plugin + +- **WHEN** the agent follows the ESLint recipe +- **THEN** it SHALL check core rules and the named catch-alls before considering any plugin from the curated list + +#### Scenario: flake8 goes straight to its catch-all + +- **WHEN** the agent follows the flake8 recipe +- **THEN** it SHALL be directed to `banned-modules` without a rule-discovery step + +### Requirement: Rule discovery is confirmed against the installed version + +Where a tool's rule set is too large to match a request against locally, the recipe SHALL direct the agent to find candidate rules on the tool's rules page by web search or fetch, then confirm each candidate against the installed version with the tool's own command before enabling it. The Ruff recipe SHALL use `ruff rule ` and the Biome recipe `biome explain `. + +The Ruff recipe SHALL state that a preview-only rule does not fire unless preview mode is enabled, and SHALL direct the agent to check the rule's preview status before enabling it. The pylint recipe SHALL state that `--help-msg` reports a message from an optional extension as nonexistent unless that extension is passed with `--load-plugins`. + +#### Scenario: A Ruff rule is confirmed locally + +- **WHEN** the agent finds a candidate Ruff rule on the web +- **THEN** it SHALL confirm it with `ruff rule` against the installed version before enabling it + +#### Scenario: A preview-only Ruff rule is not enabled silently + +- **WHEN** the confirmed Ruff rule is preview-only +- **THEN** the agent SHALL either enable preview mode for it with the user's knowledge or choose another rule, and SHALL NOT enable it in a configuration where it cannot fire + +#### Scenario: A pylint extension message is looked up with its extension loaded + +- **WHEN** the agent looks up a message that belongs to a pylint optional extension +- **THEN** it SHALL pass that extension with `--load-plugins` before concluding the message does not exist + +### Requirement: The ESLint recipe carries a curated plugin list + +The ESLint recipe SHALL carry a curated list of plugins, each with a one-line summary of what it covers and a link to its documentation, so the agent can choose whether to investigate further. At launch the list SHALL be `eslint-plugin-import`, `eslint-plugin-regexp`, `eslint-plugin-promise`, `eslint-plugin-n`, `eslint-plugin-react`, `eslint-plugin-react-hooks`, `eslint-plugin-jsx-a11y`, and `eslint-plugin-sonarjs`. + +The recipe SHALL direct the agent to prefer a plugin the repository already has installed. Where a listed plugin does not declare support for the repository's installed ESLint major version, the recipe SHALL say so, and for imports SHALL name `eslint-plugin-import-x` as the alternative. + +The list names rule sources, not individual rules. The recipe SHALL NOT embed a catalog of individual plugin rules. + +#### Scenario: An installed plugin is preferred + +- **WHEN** a request is covered by a plugin the repository already depends on +- **THEN** the agent SHALL use that plugin's rule rather than recommending another + +#### Scenario: Each listed plugin links to its docs + +- **WHEN** the ESLint recipe is rendered +- **THEN** every curated plugin SHALL appear with a one-line summary and a documentation URL + +### Requirement: A new legacy rule is written only from the repository's own examples + +A legacy recipe SHALL permit the agent to write a new custom rule in a linter only when the repository already has at least five custom rules for that linter, as counted by `detect --json`. The agent SHALL use those rules as its reference for structure and conventions. + +The recipes SHALL NOT teach custom-rule authoring for any tool. They SHALL NOT carry instructions for writing an ESLint rule, a pylint checker, a flake8 plugin, or a Biome plugin of any kind. Below the threshold, a request no shipped rule solves SHALL return to `route`, which sends it to a Taskless rule. + +#### Scenario: Five examples permit a new rule + +- **WHEN** no shipped rule solves the request AND `detect` counts five or more custom rules for the linter +- **THEN** the agent MAY write a new custom rule modeled on those rules + +#### Scenario: Fewer examples send the request back to Taskless + +- **WHEN** no shipped rule solves the request AND `detect` counts fewer than five custom rules for the linter +- **THEN** the agent SHALL NOT write a custom rule in that linter +- **AND** it SHALL continue down the `route` ladder to a Taskless rule + +### Requirement: A plugin the repository lacks is added only with consent + +A legacy recipe SHALL treat recommending a plugin the repository does not have installed as adding a dependency. The agent SHALL ask the user before adding it, and SHALL state the package it would add. + +#### Scenario: A missing plugin is proposed, not installed + +- **WHEN** the best existing rule belongs to a plugin the repository does not depend on +- **THEN** the agent SHALL name the package and ask before adding it + +### Requirement: A legacy rule hand-off says who runs it + +Every legacy recipe SHALL end by naming the files it changed and telling the user how to run the linter to see the rule fire. It SHALL state that `taskless check` does not run external linters. + +#### Scenario: The user is told how to see the rule fire + +- **WHEN** the agent finishes a legacy rule +- **THEN** it SHALL name the changed files and the command that runs the linter +- **AND** it SHALL state that `taskless check` does not run that linter diff --git a/openspec/changes/legacy-linter-recipes/specs/cli-onboard/spec.md b/openspec/changes/legacy-linter-recipes/specs/cli-onboard/spec.md new file mode 100644 index 00000000..b3b45a27 --- /dev/null +++ b/openspec/changes/legacy-linter-recipes/specs/cli-onboard/spec.md @@ -0,0 +1,18 @@ +## ADDED Requirements + +### Requirement: Onboarding uses only shipped rules from installed packages + +When onboarding sends a candidate to a legacy destination, the recipe SHALL limit the agent to enabling or configuring a rule that the linter, or a plugin package the repository already has installed, already ships. Onboarding SHALL NOT write a custom rule in any linter, regardless of how many custom rules the repository has, and SHALL NOT add a dependency. + +A candidate that no installed rule solves SHALL continue down the `route` ladder to a Taskless rule, or to the report's `cloud` tier when only a runtime rule could solve it. + +#### Scenario: A plugin that is not installed is not added + +- **WHEN** the best existing rule for a candidate belongs to a plugin the repository does not depend on +- **THEN** onboarding SHALL NOT add that plugin +- **AND** the candidate SHALL continue to a Taskless destination + +#### Scenario: Custom-rule examples do not unlock authoring during onboarding + +- **WHEN** the repository has five or more custom ESLint rules AND no shipped rule solves a candidate +- **THEN** onboarding SHALL NOT write a custom ESLint rule for it diff --git a/openspec/changes/legacy-linter-recipes/specs/cli-rule-routing/spec.md b/openspec/changes/legacy-linter-recipes/specs/cli-rule-routing/spec.md new file mode 100644 index 00000000..40dbc326 --- /dev/null +++ b/openspec/changes/legacy-linter-recipes/specs/cli-rule-routing/spec.md @@ -0,0 +1,86 @@ +## ADDED Requirements + +### Requirement: Route applies the legacy ladder in order + +For a request in a repository where `detect --json` reports a linter, the `route` recipe SHALL apply these steps in order and stop at the first that solves the request: + +1. A rule the detected linter already ships, including its catch-all rules, solves it. +2. The repository has five or more of its own custom rules for that linter, and a new one modeled on them is maintainable. +3. A Taskless static rule. +4. A Taskless runtime rule. + +Step 1 SHALL be taken whenever a detected linter could host the rule, regardless of how much custom configuration the repository has. Step 2 SHALL NOT be taken below the five-rule threshold. + +#### Scenario: A shipped rule wins over a Taskless rule + +- **WHEN** a detected linter ships a rule that solves the request +- **THEN** `route` SHALL name `create-legacy-rule` for that linter + +#### Scenario: No shipped rule and no examples falls to Taskless + +- **WHEN** no shipped rule solves the request AND the repository has fewer than five custom rules for any linter that could host it +- **THEN** `route` SHALL continue to a Taskless destination + +### Requirement: Route describes each detected linter + +The `route` recipe SHALL include, for each linter `detect` reports that can host a legacy rule, one paragraph saying what the tool is for, what the agent always checks first in it, and the recipe command to fetch for it. The paragraphs SHALL be rendered when the recipe is served, and SHALL appear only for linters detected in the repository. + +#### Scenario: Only detected linters are described + +- **WHEN** `detect` reports Ruff and no other linter +- **THEN** the served `route` recipe SHALL contain the Ruff paragraph +- **AND** SHALL NOT contain a paragraph for any other linter + +#### Scenario: A tool with no legacy recipe gets no paragraph + +- **WHEN** `detect` reports only `prettier` +- **THEN** the served `route` recipe SHALL contain no linter paragraph + +### Requirement: Route asks the user only when the ladder does not decide + +The `route` recipe SHALL present the viable options to the user with their +trade-offs, rather than silently selecting one, whenever more than one destination +genuinely fits a request. The order of the legacy ladder SHALL decide between a +legacy rule and a Taskless rule, so that pair SHALL NOT be posed as a choice. The +trade-off framing SHALL note that `remote` consumes a generation and requires +login, so it is appropriate when a request cannot be solved locally rather than as +a default. + +#### Scenario: A legacy rule and a static rule are not posed as a choice + +- **WHEN** the repository has a detected linter that ships a rule solving the + request AND the pattern is also a clean local static ast-grep rule +- **THEN** `route` SHALL name the legacy destination without asking the user to + choose between it and `create-sg-rule` + +#### Scenario: Trade-off framing names the generation cost of remote + +- **WHEN** `route` presents options that include `remote` +- **THEN** it SHALL state that `remote` consumes a generation and requires login +- **AND** SHALL frame `remote` as the path for what cannot be solved locally + +## MODIFIED Requirements + +### Requirement: Existing recipe authors in the detected linter's dialect + +The CLI SHALL provide a `create-legacy-rule` help recipe that instructs the agent to solve a request with a linter already detected in the repository, in that tool's own dialect. The recipe SHALL direct the agent to look first for a rule the tool already ships, and to write a new custom rule only from the repository's own existing rules for that linter. The recipe MAY carry a short, curated, linked list of rule sources, such as a tool's catch-all rules or a set of well-known plugins. It SHALL NOT embed a catalog of individual linter rules. + +The recipe is named for the artifact it produces. "Existing" described the repository's state rather than the rule being written, which is not something an agent can address by name. + +#### Scenario: Repo-first knowledge sourcing + +- **WHEN** the agent follows `create-legacy-rule` to write a new custom rule +- **THEN** it SHALL read the repository's own rules for that linter before consulting any external source + +#### Scenario: A shipped rule is looked for first + +- **WHEN** the agent follows `create-legacy-rule` +- **THEN** it SHALL establish whether a rule the tool already ships solves the request before writing anything new + +## REMOVED Requirements + +### Requirement: Route asks the user when multiple paths fit + +**Reason**: It required `route` to pose a legacy rule and a Taskless static rule as a choice whenever both fit. The legacy ladder now decides that pair by order, and onboarding, which follows `route`, is not allowed to ask per candidate. + +**Migration**: Replaced by "Route asks the user only when the ladder does not decide", which keeps the remote trade-off framing and asks only where more than one destination fits after the ladder is applied. diff --git a/openspec/changes/legacy-linter-recipes/tasks.md b/openspec/changes/legacy-linter-recipes/tasks.md new file mode 100644 index 00000000..7cda3a14 --- /dev/null +++ b/openspec/changes/legacy-linter-recipes/tasks.md @@ -0,0 +1,35 @@ +## 1. Detect counts custom rules (PR 1) + +- [x] 1.1 Add the per-linter custom-rule count and its evidence to the detect output schema. Verify the schema test validates a payload with and without counts, and that existing fields are unchanged. +- [x] 1.2 Count ESLint custom rules statically: rules in a house rule directory, and an unknown count for a plugin imported from a local file, an inline plugin, or a local-rules loader package. Verify fixture-repo tests for a rule directory, an imported plugin, an inline plugin, and a preset-only config (zero). +- [x] 1.3 Count Biome `plugins` entries across root and nested configs, pylint `load-plugins` modules that resolve inside the repository (excluding `pylint.extensions.*`), and flake8 `[flake8:local-plugins]` checkers. Report Ruff as zero. Verify a fixture-repo test per tool, including a pylint config naming only built-in extensions (zero). +- [x] 1.4 Confirm detect still executes nothing and makes no network call with the new scan. Verify the existing offline test passes. + +## 2. Route ladder and linter paragraphs (PR 1) + +- [x] 2.1 Add a `%(LEGACY_LINTERS)s` slot to `route`, rendered at serve time from a detect scan with one paragraph per detected linter that has a legacy recipe, and the full set when `mechanics` is off. Verify recipe tests: a Ruff-only fixture renders only the Ruff paragraph, a `prettier`-only fixture renders none, and the mechanics-off path renders all five. +- [x] 2.2 Rewrite `route.md` as topic v8: the four-step ladder, step 1 always taken, step 2 gated on the five-rule count, legacy and static no longer posed as a choice. Verify the recipe tests for each scenario in the `cli-rule-routing` delta, and that "Every destination resolves to a recipe" still passes. +- [x] 2.3 Rewrite generic `create-legacy-rule.md` as topic v3: shipped rule first, custom rule only with five or more repository examples and modeled on them, a missing plugin added only with consent, and the hand-off naming who runs the rule. Verify the recipe tests for each `cli-legacy-rule-recipes` scenario that applies to the generic recipe. +- [x] 2.4 Update `onboard.md` to topic v11: legacy candidates use shipped rules from installed packages only, with no custom rules and no new dependencies. Verify the `cli-onboard` delta scenarios and the onboard byte-parity tests. +- [x] 2.5 Add the changeset (`patch`) describing the ladder and the detect count. Run `pnpm typecheck` and `pnpm lint`; both pass. + +## 3. The `--tool` flag (PR 2) + +- [ ] 3.1 Collect `create-legacy-rule..md` variants at build time with the same glob mechanism as `.anonymous`, and add `--tool` to the agent command's value flags. Resolve `--tool` before `--anonymous`. Verify tests: a known tool serves its variant, an unknown tool falls back to the generic recipe, `--tool` as `--tool x` and `--tool=x` both parse, and the positional parser does not read the value as a topic. +- [ ] 3.2 Refuse `ast-grep`, `vale`, `prettier`, `black`, and `anonymous` with a non-zero exit that names the right recipe. Verify a test per refused name. +- [ ] 3.3 Make the `route` paragraphs name `create-legacy-rule --tool=`. Verify the "Shipped recipes name only commands that exist" check passes for every rendered paragraph. + +## 4. Per-tool recipes (PR 2) + +- [ ] 4.1 Write `create-legacy-rule.ruff.md`: `banned-api` first, web discovery confirmed with `ruff rule`, preview status checked, the 0.16 default-set change, `extend` and nested configs, and how to run Ruff. Verify recipe tests for the Ruff scenarios. +- [ ] 4.2 Write `create-legacy-rule.eslint.md`: core rules and the `no-restricted-*` catch-alls first (including `@typescript-eslint/`), the eight curated plugins each with a summary and docs link, installed plugins preferred, the ESLint 10 peer gap and `import-x`, flat config against `.eslintrc`, and how to run ESLint. Verify recipe tests that every plugin has a summary and URL and that no individual plugin rules are listed. +- [ ] 4.3 Write `create-legacy-rule.biome.md`: an existing rule first, `noRestrictedImports`/`noRestrictedGlobals` as the catch-alls, web discovery confirmed with `biome explain`, nested configs and `extends`, and how to run Biome. Verify the recipe tests. +- [ ] 4.4 Write `create-legacy-rule.pylint.md`: `--list-msgs`/`--help-msg` with the `--load-plugins` trap, `deprecated-modules`/`preferred-modules`/`bad-functions`, config search order, and how to run pylint. Verify the recipe tests. +- [ ] 4.5 Write `create-legacy-rule.flake8.md`: no discovery step, `banned-modules` from `flake8-tidy-imports` (a dependency that needs consent), the three config files, and how to run flake8. Verify the recipe tests. +- [ ] 4.6 Add vendor tests that run each tool at a pinned version and assert every behavioral claim the recipes make: the `preview` field in `ruff rule --output-format json`, `ruff rule TID251`, `biome explain` output, `pylint --help-msg` failing without and succeeding with `--load-plugins`, and `--banned-modules` in `flake8 --help` with the plugin. Verify they pass locally and in CI, and fail when a claim is edited to be false. +- [ ] 4.7 Extend the changeset with `--tool` and the five recipes. Run `pnpm typecheck` and `pnpm lint`; both pass. + +## 5. Archive (PR 2, tip) + +- [ ] 5.1 Run the pre-archive check from CLAUDE.md (commit, keep the SHA, `pnpm openspec archive legacy-linter-recipes -y`, confirm every prior scenario of `cli-rule-routing`, `cli-agent`, `cli-detect`, and `cli-onboard` survives, then reset to the SHA). +- [ ] 5.2 Archive for real. Verify `pnpm openspec validate --strict` passes on the result. diff --git a/packages/cli/src/agent/create-legacy-rule.md b/packages/cli/src/agent/create-legacy-rule.md index 54bf30d2..78bd6fea 100644 --- a/packages/cli/src/agent/create-legacy-rule.md +++ b/packages/cli/src/agent/create-legacy-rule.md @@ -1,51 +1,60 @@ -# Topic: create-legacy-rule (CLI v%(CLI_VERSION)s / topic v2) +# Topic: create-legacy-rule (CLI v%(CLI_VERSION)s / topic v3) %(ONE_COMMAND_PER_CALL)s ## You are here -This is `create-legacy-rule`. It helps you write a rule for a linter the -repository already runs (ESLint, Ruff, RuboCop, Stylelint) in that -tool's own dialect, so that tool enforces it. +This is `create-legacy-rule`. It solves a request with a linter the +repository already runs (ESLint, Ruff, Biome, pylint, flake8, and +others), in that tool's own configuration, so that tool enforces it. If that is not the kind of check you need, re-run `%(TASKLESS_CLI)s agent route` and follow its decision rather than adapting this recipe. ## Goal -Author a rule in a linter the repository ALREADY uses, expressed in that -tool's own dialect (an ESLint rule, a Ruff rule selection, a RuboCop cop, -a Stylelint rule, etc.). Taskless does not maintain a catalog of linter -rules, you source the knowledge from the repo first and the web second, -then write the rule where that tool expects it. +Find a rule the linter already ships that solves the request, and enable +or configure it. Write a new custom rule in that linter only when the +repository already has five or more of its own to model it on. Taskless +does not teach custom-rule authoring for any linter: the repository's +own rules are the only reference. ## Preconditions - The repo has a detected linter (confirm via `%(TASKLESS_CLI)s detect --json`). -- The agent can read/write files and fetch web pages. +- The agent can read and write files. Web access helps find a shipped + rule, but is not required. - No auth required. ## Steps -1. **Confirm the target tool.** Use `%(TASKLESS_CLI)s detect --json` to identify - which linter is configured. If more than one could host this rule, +1. **Confirm the target tool.** Use `%(TASKLESS_CLI)s detect --json` to see + which linters are configured. If more than one could host this rule, ask the user which tool should own it. -2. **Mine the repo's own rules first.** This is the highest-signal - source for house style. Look at how the repo already writes rules of - this kind: - - existing config (e.g. `.eslintrc*`/`eslint.config.*`, `ruff.toml` - or `[tool.ruff]`, `.rubocop.yml`, `.stylelintrc*`); - - any custom/local rules the repo authored (the `detect` output's - rule styles point at these); - - the conventions, severity choices, and naming they use. - Match that style. +2. **Look for a rule the tool already ships.** Always do this first, + however little the repo has configured. + - **Start with the tool's catch-all rules**, the ones that take your + own pattern or list and a message: ESLint's `no-restricted-syntax` + and `no-restricted-imports`, Ruff's `banned-api`, Biome's + `noRestrictedImports`, pylint's `deprecated-modules`. A search for + the request's subject does not find these, so check them by name. + - **Then the tool's own rule list and the plugins the repo already + has installed.** Prefer the tool's current documentation, and an + `llms.txt` on its docs site when there is one. Confirm the rule + exists in the version the repo has installed before you rely on it. + - **A plugin the repo does not have is a new dependency.** Name the + package and ask the user before adding it. -3. **Fall back to the web only if the repo signal is thin.** If the repo - doesn't show how to express this rule, fetch the linter's CURRENT - documentation (WebFetch/WebSearch). Prefer the latest version's docs - and, when present, an `llms.txt` on the tool's docs site. Confirm the - installed version where it matters so you target the right syntax. +3. **Enable or configure that rule** where the tool expects its + configuration, matching how the repo's existing configuration is + written: its file, its severity choices, its ordering. -4. **Author the rule in the tool's dialect.** Write or extend the - linter's config / custom-rule file the way that tool expects. Keep it - consistent with the repo's existing entries from step 2. +4. **No shipped rule fits: check the repo's own rules.** Read + `customRules` for this linter in `detect --json`. + - **A `count` of 5 or more:** you may write a new rule. Read the repo's + rules listed in `evidence` first and model yours on them: their + structure, naming, tests, and how they are registered. Do not reach + for the tool's authoring documentation instead. + - **Fewer than 5, `"unknown"`, or no `customRules` at all:** do not + write a custom rule. Return to `%(TASKLESS_CLI)s agent route` and + continue to step 3 of its legacy ladder, a Taskless rule. 5. **Report, and be explicit about who runs it.** Show the file(s) you changed. Make clear that the user's OWN toolchain runs this rule: @@ -54,10 +63,10 @@ then write the rule where that tool expects it. ## Important Notes -- Do not invent linter rules from memory: verify against the repo's - usage and the tool's current docs. -- This path is author-only. Taskless does not aggregate or run external - linters; it writes the rule in the tool's dialect and hands off. +- Do not invent linter rules from memory: confirm a rule against the + installed version before enabling it. +- This path writes the rule in the tool's configuration and hands off. + Taskless does not aggregate or run external linters. ## See Also diff --git a/packages/cli/src/agent/onboard.md b/packages/cli/src/agent/onboard.md index 43e5337d..96170814 100644 --- a/packages/cli/src/agent/onboard.md +++ b/packages/cli/src/agent/onboard.md @@ -1,4 +1,4 @@ -# Topic: onboard (CLI v%(CLI_VERSION)s / topic v10) +# Topic: onboard (CLI v%(CLI_VERSION)s / topic v11) %(ONE_COMMAND_PER_CALL)s @@ -155,6 +155,13 @@ making the case for running them in CI. recipe you already fetched, with the candidate as the rule description, and write it. Do not ask before each one. + A `legacy` candidate here only ever enables or configures a rule + the linter, or a plugin the repo already has installed, already + ships. During onboarding do not write a custom rule in any linter, + however many the repo has, and do not add a dependency. When no + installed rule solves it, continue down `route`'s legacy ladder to + a Taskless rule. + A candidate you judge better solved by a runtime rule is not written: runtime rules need an account, and onboarding stays logged out. If `route` sends a rule toward a runtime or diff --git a/packages/cli/src/agent/route.md b/packages/cli/src/agent/route.md index e2b1c252..07edaf74 100644 --- a/packages/cli/src/agent/route.md +++ b/packages/cli/src/agent/route.md @@ -1,4 +1,4 @@ -# Topic: route (CLI v%(CLI_VERSION)s / topic v7) +# Topic: route (CLI v%(CLI_VERSION)s / topic v8) %(ONE_COMMAND_PER_CALL)s @@ -63,9 +63,31 @@ answered together. and only here. The destination recipes describe their own scope and deliberately do not restate this table. + **The legacy ladder comes first when `detect` reports a linter.** Take + these in order and stop at the first that solves the request: + + 1. **A rule the linter already ships solves it.** Always check this, + however little the repo has configured: enabling a shipped rule adds + nothing new for the team to maintain. Each linter's catch-all rules + count, and are named below because a search for the request's + subject will not surface them. + 2. **The repo already writes its own rules for that linter.** Only when + `detect` reports a `customRules.count` of 5 or more, so there are + enough of the repo's own examples to model a new one on. An + `"unknown"` count is below the threshold. Taskless does not teach + custom-rule authoring for any linter; the repo's examples are the + only reference. + 3. **A Taskless `sg` or `vale` rule**, from the table below. + 4. **A Taskless runtime rule**, from the table below. + + Steps 1 and 2 go to `create-legacy-rule`. When neither applies, the + request is a Taskless rule even though the linter is installed. + +%(LEGACY_LINTERS)s + | The rule is decided by… | Destination | Login | |------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------|------------| -| a tool the repo already runs, in that tool's dialect | `create-legacy-rule` | no | +| a rule a linter the repo already runs ships, or a new one there (the legacy ladder above) | `create-legacy-rule` | no | | **one file's syntax tree**: a call, an import, a JSX attribute, a type annotation | `create-sg-rule` | no | | **a document's words**: docs, README, comments, commit bodies | `create-vale-rule` | no | | **more than one file, or something outside the files**: the repo graph, git metadata, build output, a resolved config chain | `create-runtime-rule` / `create-remote-rule` | see step 5 | @@ -186,8 +208,8 @@ answered together. **A language on neither list does not route to runtime by default.** Check `create-legacy-rule` first: the repo may already run a linter - that speaks it, and that linter's own dialect is a local destination - with no login. Escalate only once nothing local can see the evidence. + that ships a rule for it, which is a local destination with no login. + Escalate only once nothing local can see the evidence. Worked examples: @@ -248,8 +270,9 @@ answered together. ## When two destinations both fit - **A linter the repo already runs AND a local Taskless rule both fit** → - do not silently pick. Present both with trade-offs and let the user - choose. + do not ask. The legacy ladder in step 4 decides: a rule the linter + already ships wins, and so does a new one where the repo has 5 or more + of its own to model it on. Otherwise it is the Taskless rule. - **A rule that seems to need two engines is usually two rules.** Split it and route each half, rather than forcing one engine to cover evidence it cannot see. @@ -300,7 +323,7 @@ will work. ## See Also -- `%(RECIPE_FETCH)screate-legacy-rule`: author in a linter the repo already uses +- `%(RECIPE_FETCH)screate-legacy-rule`: use a rule a linter the repo already runs ships - `%(RECIPE_FETCH)screate-sg-rule`: author a local ast-grep rule (no login) - `%(RECIPE_FETCH)screate-vale-rule`: author a local Vale rule (no login) - `%(RECIPE_FETCH)screate-runtime-rule`: the runtime tier, logged out diff --git a/packages/cli/src/commands/agent.ts b/packages/cli/src/commands/agent.ts index 74396738..859e2d8b 100644 --- a/packages/cli/src/commands/agent.ts +++ b/packages/cli/src/commands/agent.ts @@ -10,8 +10,10 @@ import { import { getTelemetry } from "../telemetry"; import { detectHostTools } from "../detect/host-tools"; +import { detectRepository } from "../detect/scan"; import { HOST_TOOL_VARIABLES, + LINTER_VARIABLES, getRawRecipe, getRecipe, } from "../prompts/recipes"; @@ -262,10 +264,19 @@ export function createAgentCommand(subCommands: SubCommandsDef) { ) ? await detectHostTools(cwd) : undefined; + // The same question for the per-linter passage: a repository scan is + // filesystem work, so only the topics that describe linters pay for it. + const scan = template?.variables.some((name) => + LINTER_VARIABLES.has(name) + ) + ? await detectRepository(cwd) + : undefined; + const linters = scan?.linters.map((linter) => linter.name); const recipe = getRecipe(key, { anonymous: args.anonymous, invocation, hostTools, + linters, // What this command serves IS a fetch, so the served text says so: // resolved now, fetch again next task, and a session that installed // or upgraded mid-way holds a stale skill. The prompts export leaves diff --git a/packages/cli/src/detect/custom-rules.ts b/packages/cli/src/detect/custom-rules.ts new file mode 100644 index 00000000..50157dad --- /dev/null +++ b/packages/cli/src/detect/custom-rules.ts @@ -0,0 +1,404 @@ +import { existsSync, readdirSync, statSync } from "node:fs"; +import { readFile } from "node:fs/promises"; +import { dirname, join, resolve } from "node:path"; + +/** + * How many rules the repository wrote itself for one linter, and where they + * were found. The legacy ladder reads this to decide whether an agent may write + * another custom rule, which it may only do with five or more examples to model + * it on. + * + * `"unknown"` means a custom rule exists but its count cannot be read without + * executing configuration (an ESLint plugin imported into a JavaScript config). + * Recipes treat it as below the threshold, which sends the request to a + * Taskless rule, the safe direction. + */ +export interface CustomRules { + count: number | "unknown"; + evidence: string[]; +} + +/** What the scan already found, so this module reads nothing twice. */ +export interface CustomRuleContext { + root: string; + pathsFor: (basename: string) => string[]; + pyprojects: Array<{ path: string; parsed?: Record }>; + nodeDeps: Array<{ path: string; deps: Set }>; +} + +export async function countCustomRules( + linter: string, + context: CustomRuleContext +): Promise { + switch (linter) { + case "eslint": { + return countEslint(context); + } + case "biome": { + return countBiome(context); + } + case "pylint": { + return countPylint(context); + } + case "flake8": { + return countFlake8(context); + } + // Ruff has no custom-rule mechanism. A fixed zero rather than nothing, + // because an absent count would read as "not checked". + case "ruff": { + return { count: 0, evidence: [] }; + } + default: { + return undefined; + } + } +} + +async function readText(path: string): Promise { + try { + return await readFile(path, "utf8"); + } catch { + return undefined; + } +} + +function asRecord(value: unknown): Record | undefined { + if (typeof value !== "object" || value === null || Array.isArray(value)) { + return undefined; + } + return value as Record; +} + +// --- ESLint ------------------------------------------------------------------ + +/** + * House directories that hold one ESLint rule per entry. The scan's rule-style + * hints read the same list, so the two cannot drift. + */ +export const ESLINT_RULE_DIRECTORIES = [ + "eslint-rules", + "eslint-local-rules", + "tools/eslint-rules", +]; + +const FLAT_CONFIGS = [ + "eslint.config.js", + "eslint.config.mjs", + "eslint.config.cjs", + "eslint.config.ts", + "eslint.config.mts", + "eslint.config.cts", +]; + +/** Packages that load rules from a file in the repository. */ +const ESLINT_LOCAL_RULE_PACKAGES = [ + "eslint-plugin-local-rules", + "eslint-plugin-local", + "eslint-local-rules", +]; + +const RULE_SOURCE = /\.(?:[cm]?[jt]s)$/; +const NOT_A_RULE_FILE = /(?:^index\.|\.test\.|\.spec\.|\.d\.[cm]?ts$)/; +const NOT_A_RULE_DIRECTORY = new Set([ + "__tests__", + "tests", + "test", + "fixtures", + "utils", + "util", + "lib", + "helpers", +]); + +/** + * Rules in a house rule directory: each source file is one rule, and so is + * each subdirectory (a rule with its own folder). Index files, tests, type + * declarations, and helper folders are not rules. + * + * An unreadable directory returns `undefined` rather than throwing: the scan + * skips what it cannot read, and one bad directory must not fail `detect`. + */ +function countRuleDirectory(path: string): number | undefined { + let entries: string[]; + try { + entries = readdirSync(path); + } catch { + return undefined; + } + let count = 0; + for (const entry of entries) { + let isDirectory: boolean; + try { + isDirectory = statSync(join(path, entry)).isDirectory(); + } catch { + continue; + } + if (isDirectory) { + if (!NOT_A_RULE_DIRECTORY.has(entry)) count++; + } else if (RULE_SOURCE.test(entry) && !NOT_A_RULE_FILE.test(entry)) { + count++; + } + } + return count; +} + +/** A relative import or require, the shape of a plugin loaded from a file. */ +const RELATIVE_IMPORT = /(?:from\s*|require\(\s*)["']\.\.?\//; +/** A rule defined inside the config itself. */ +const INLINE_RULE = /\bcreate\s*\(\s*context\b/; + +async function countEslint(context: CustomRuleContext): Promise { + let known = 0; + let unknown = false; + const evidence: string[] = []; + + for (const directory of ESLINT_RULE_DIRECTORIES) { + const path = resolve(context.root, directory); + if (!existsSync(path)) continue; + const count = countRuleDirectory(path); + if (count === undefined) continue; + known += count; + evidence.push(`${directory} (${count} rules)`); + } + + for (const name of FLAT_CONFIGS) { + for (const relativePath of context.pathsFor(name)) { + const source = await readText(resolve(context.root, relativePath)); + if (source === undefined || !/\bplugins\s*:/.test(source)) continue; + if (INLINE_RULE.test(source)) { + unknown = true; + evidence.push(`${relativePath} (inline plugin rules)`); + } else if (RELATIVE_IMPORT.test(source)) { + unknown = true; + evidence.push(`${relativePath} (plugin imported from a local file)`); + } + } + } + + for (const manifest of context.nodeDeps) { + for (const name of ESLINT_LOCAL_RULE_PACKAGES) { + if (manifest.deps.has(name)) { + unknown = true; + evidence.push(`dependency ${name} (${manifest.path})`); + } + } + } + + // A counted directory that already clears the threshold settles it; an + // uncountable source can only add to it. + const count = unknown && known < 5 ? "unknown" : known; + return { count, evidence }; +} + +// --- Biome ------------------------------------------------------------------- + +/** + * Parse JSONC: comments and trailing commas are dropped outside strings, so a + * `$schema` URL keeps its `//`. Returns `undefined` for anything malformed. + */ +function parseJsonc(source: string): unknown { + let out = ""; + let index = 0; + while (index < source.length) { + const char = source[index]!; + if (char === '"') { + let end = index + 1; + while (end < source.length && source[end] !== '"') { + end += source[end] === "\\" ? 2 : 1; + } + out += source.slice(index, end + 1); + index = end + 1; + } else if (char === "/" && source[index + 1] === "/") { + while (index < source.length && source[index] !== "\n") index++; + } else if (char === "/" && source[index + 1] === "*") { + const end = source.indexOf("*/", index + 2); + index = end === -1 ? source.length : end + 2; + } else { + out += char; + index++; + } + } + try { + return JSON.parse(out.replaceAll(/,(\s*[}\]])/g, "$1")); + } catch { + return undefined; + } +} + +async function countBiome(context: CustomRuleContext): Promise { + let count = 0; + const evidence: string[] = []; + for (const name of ["biome.json", "biome.jsonc"]) { + for (const relativePath of context.pathsFor(name)) { + const source = await readText(resolve(context.root, relativePath)); + const plugins = asRecord( + source === undefined ? undefined : parseJsonc(source) + )?.plugins; + if (!Array.isArray(plugins) || plugins.length === 0) continue; + count += plugins.length; + evidence.push(`${relativePath} (${plugins.length} plugins)`); + } + } + return { count, evidence }; +} + +// --- Python INI -------------------------------------------------------------- + +/** + * Read an INI file the way Python's `configparser` does for these keys: + * `[section]` headers, `key = value` or `key: value`, and indented + * continuation lines appended to the previous value. Comments start with `#` + * or `;`. Keys are lowercased, as configparser does. + */ +function readIniSections(source: string): Map> { + const sections = new Map>(); + let current: Map | undefined; + let lastKey: string | undefined; + for (const raw of source.split(/\r?\n/)) { + const trimmed = raw.trim(); + if (trimmed === "" || trimmed.startsWith("#") || trimmed.startsWith(";")) { + continue; + } + if (/^\s/.test(raw) && current && lastKey !== undefined) { + current.set(lastKey, `${current.get(lastKey)}\n${trimmed}`); + continue; + } + const header = /^\[(.+)\]$/.exec(trimmed); + if (header) { + current = new Map(); + sections.set(header[1]!.trim(), current); + lastKey = undefined; + continue; + } + const property = /^([^=:]+)[=:](.*)$/.exec(trimmed); + if (property && current) { + lastKey = property[1]!.trim().toLowerCase(); + current.set(lastKey, property[2]!.trim()); + } + } + return sections; +} + +function splitList(value: string): string[] { + return value + .split(/[,\n]/) + .map((item) => item.trim()) + .filter((item) => item !== ""); +} + +// --- pylint ------------------------------------------------------------------ + +/** + * Whether a dotted module name resolves to a file in the repository, looked + * for beside the config, at the root, and under `src/`. + */ +function moduleInRepository( + root: string, + configDirectory: string, + module: string +): boolean { + const relative = module.replaceAll(".", "/"); + for (const base of [configDirectory, root, join(root, "src")]) { + if ( + existsSync(join(base, `${relative}.py`)) || + existsSync(join(base, relative, "__init__.py")) + ) { + return true; + } + } + return false; +} + +/** `load-plugins` values from a parsed `pyproject.toml`'s `[tool.pylint.*]`. */ +function pyprojectLoadPlugins(parsed: Record): string[] { + const pylint = asRecord(asRecord(parsed.tool)?.pylint); + if (!pylint) return []; + const plugins: string[] = []; + for (const section of [pylint, ...Object.values(pylint)]) { + const value = asRecord(section)?.["load-plugins"]; + if (typeof value === "string") plugins.push(...splitList(value)); + if (Array.isArray(value)) { + plugins.push( + ...value.filter((item): item is string => typeof item === "string") + ); + } + } + return plugins; +} + +async function countPylint(context: CustomRuleContext): Promise { + const named: Array<{ config: string; module: string }> = []; + + for (const name of [".pylintrc", "pylintrc"]) { + for (const relativePath of context.pathsFor(name)) { + const source = await readText(resolve(context.root, relativePath)); + if (source === undefined) continue; + for (const section of readIniSections(source).values()) { + const value = section.get("load-plugins"); + if (value === undefined) continue; + for (const module of splitList(value)) { + named.push({ config: relativePath, module }); + } + } + } + } + for (const pyproject of context.pyprojects) { + if (!pyproject.parsed) continue; + for (const module of pyprojectLoadPlugins(pyproject.parsed)) { + named.push({ config: pyproject.path, module }); + } + } + + const evidence: string[] = []; + for (const { config, module } of named) { + // Optional checkers that ship with pylint are first-party, not the + // repository's own; third-party packages do not resolve to a file here. + if (module.startsWith("pylint.")) continue; + const configDirectory = dirname(resolve(context.root, config)); + if (moduleInRepository(context.root, configDirectory, module)) { + evidence.push(`${config} load-plugins ${module}`); + } + } + return { count: evidence.length, evidence }; +} + +// --- flake8 ------------------------------------------------------------------ + +async function countFlake8(context: CustomRuleContext): Promise { + let count = 0; + const evidence: string[] = []; + + for (const name of [".flake8", "setup.cfg", "tox.ini"]) { + for (const relativePath of context.pathsFor(name)) { + const source = await readText(resolve(context.root, relativePath)); + if (source === undefined) continue; + const extension = readIniSections(source) + .get("flake8:local-plugins") + ?.get("extension"); + if (extension === undefined) continue; + // One `CODE = module:Class` per line registers one checker. + const checkers = extension + .split("\n") + .filter((line) => line.includes("=")).length; + if (checkers === 0) continue; + count += checkers; + evidence.push(`${relativePath} [flake8:local-plugins] (${checkers})`); + } + } + + for (const pyproject of context.pyprojects) { + const entryPoints = asRecord( + asRecord(asRecord(pyproject.parsed?.project)?.["entry-points"])?.[ + "flake8.extension" + ] + ); + const published = Object.keys(entryPoints ?? {}).length; + if (published === 0) continue; + count += published; + evidence.push( + `${pyproject.path} flake8.extension entry points (${published})` + ); + } + + return { count, evidence }; +} diff --git a/packages/cli/src/detect/scan.ts b/packages/cli/src/detect/scan.ts index eab90341..27ffe9e0 100644 --- a/packages/cli/src/detect/scan.ts +++ b/packages/cli/src/detect/scan.ts @@ -11,6 +11,11 @@ import { detectAutomation, type DetectedAutomation, } from "./automation"; +import { + countCustomRules, + type CustomRules, + ESLINT_RULE_DIRECTORIES, +} from "./custom-rules"; export interface DetectedLinter { name: string; @@ -23,6 +28,11 @@ export interface DetectedLinter { * paths. */ evidence: string[]; + /** + * Rules the repository wrote itself for this linter, for the linters that + * can hold one. Absent for every other linter. + */ + customRules?: CustomRules; } export interface RuleStyle { @@ -735,11 +745,7 @@ function detectRuleStyles( }); break; } - for (const directory of [ - "eslint-rules", - "eslint-local-rules", - "tools/eslint-rules", - ]) { + for (const directory of ESLINT_RULE_DIRECTORIES) { if (existsSync(resolve(root, directory))) { ruleStyles.push({ source: directory, @@ -962,7 +968,17 @@ export async function detectRepository(cwd: string): Promise { } if (evidence.length > 0) { - linters.push({ name: signal.name, evidence }); + const customRules = await countCustomRules(signal.name, { + root, + pathsFor, + pyprojects: tomlManifests.get("pyproject.toml") ?? [], + nodeDeps: nodeManifests, + }); + linters.push({ + name: signal.name, + evidence, + ...(customRules && { customRules }), + }); for (const language of signal.languages) languages.add(language); } } diff --git a/packages/cli/src/prompts/recipes.ts b/packages/cli/src/prompts/recipes.ts index 6a0a7b55..15737eb1 100644 --- a/packages/cli/src/prompts/recipes.ts +++ b/packages/cli/src/prompts/recipes.ts @@ -267,6 +267,15 @@ export interface RecipeOptions { * told to install `gh`, which is the one instruction that cannot help it. */ hostTools?: HostTool[]; + /** + * The linter names `detect` found in the repository, for the `route` + * passage that describes each one. + * + * An argument for the same reason as {@link hostTools}: scanning needs the + * filesystem. Omitting it renders every linter that has a legacy recipe, + * since nothing was measured; an empty array is a measured "none". + */ + linters?: string[]; } /** @@ -420,6 +429,91 @@ export const HOST_TOOL_VARIABLES: ReadonlySet = new Set([ "HOST_TOOLS", ]); +/** + * The variable names whose value depends on {@link RecipeOptions.linters}. + * Exported for the same reason as {@link HOST_TOOL_VARIABLES}: the `agent` + * command asks the template whether a scan is worth running. + */ +export const LINTER_VARIABLES: ReadonlySet = new Set([ + "LEGACY_LINTERS", +]); + +/** + * One paragraph per linter that has a legacy recipe, in the order `route` + * prints them. Each says what the tool is for and what step 1 always checks + * there, which is the part an agent gets wrong without being told: a + * catch-all rule is not found by searching for the request's subject. + */ +const LEGACY_LINTER_GUIDE: ReadonlyArray<[string, string]> = [ + [ + "eslint", + `**ESLint** lints JavaScript and TypeScript and runs plugins. Always +try a core rule first. \`no-restricted-syntax\` (an AST selector with +your message) and \`no-restricted-imports\` are the catch-alls, with +their \`@typescript-eslint/\` counterparts in a TypeScript repo. Then a +plugin the repo already has installed.`, + ], + [ + "biome", + `**Biome** lints JavaScript, TypeScript, JSON, and CSS with a large +built-in rule set. Always check whether an existing rule makes the +problem go away. \`noRestrictedImports\` and \`noRestrictedGlobals\` +are the catch-alls.`, + ], + [ + "ruff", + `**Ruff** lints Python with a large built-in rule set and has no +custom-rule mechanism, so it never takes step 2. Always check +\`banned-api\` (TID251) first: it bans an import or attribute with a +message of your choosing, which covers most "never call X" requests.`, + ], + [ + "pylint", + `**pylint** lints Python with a broad set of built-in messages. Always +check them first, with the \`deprecated-modules\`, +\`preferred-modules\`, and \`bad-functions\` options as the catch-alls.`, + ], + [ + "flake8", + `**flake8** lints Python through plugins. Always check \`banned-modules\` +from \`flake8-tidy-imports\` first, which bans a module with a message. +It is a plugin package, so the repo may need to add it.`, + ], +]; + +/** + * The linters `route` describes, as one block. + * + * `undefined` means nobody scanned, which renders every paragraph: a + * consumer of `@taskless/cli/prompts` has no repository to describe, and + * omitting a linter there would read as "this one does not apply". An empty + * match is a measured answer and says so in one line. + */ +function legacyLintersBlock( + linters: string[] | undefined, + fetchPrefix: string +): string { + const described = LEGACY_LINTER_GUIDE.filter( + ([name]) => linters === undefined || linters.includes(name) + ); + if (described.length === 0) { + return ` \`detect\` found no linter here with a legacy recipe, so step 1 and + step 2 do not apply. Continue from step 3.`; + } + const heading = + linters === undefined + ? " Linters with a legacy recipe, and what step 1 always checks:" + : " The linters `detect` found here, and what step 1 always checks:"; + const items = described.map( + ([, paragraph]) => ` - ${indentBlock(paragraph, " ")}` + ); + return `${heading} + +${items.join("\n")} + + For any of them, fetch \`${fetchPrefix}create-legacy-rule\`.`; +} + /** The state a passage conditioned on one tool renders from. */ type ToolState = "unknown" | "present" | "absent" | "not-applicable"; @@ -674,6 +768,10 @@ export function buildVariables( " " ), HOST_TOOLS: indentBlock(hostToolsStep(options.hostTools), " "), + LEGACY_LINTERS: legacyLintersBlock( + options.linters, + options.mechanics === false ? "" : `${resolveInvocation(options)} agent ` + ), }; if (content.includes("%(INPUT_SCHEMA)s")) { const schema = TOPIC_INPUT_SCHEMAS[topic]; diff --git a/packages/cli/src/schemas/detect.ts b/packages/cli/src/schemas/detect.ts index a1d4be12..1909b375 100644 --- a/packages/cli/src/schemas/detect.ts +++ b/packages/cli/src/schemas/detect.ts @@ -8,6 +8,19 @@ const detectedLinterSchema = z.object({ .describe( "On-disk evidence: config-file paths, a manifest table or section marker, a dependency marker from the language's package file, or a root CI or hook config that runs the linter (not all entries are file paths)" ), + customRules: z + .object({ + count: z + .union([z.number().int().nonnegative(), z.literal("unknown")]) + .describe( + "Rules the repository wrote itself for this linter; unknown when one exists but cannot be counted without executing config" + ), + evidence: z.array(z.string()).describe("Where those rules were found"), + }) + .optional() + .describe( + "Present only for linters that can hold a custom rule (eslint, biome, pylint, flake8) and for ruff, which always reports zero" + ), }); /** A surfaced style of the repo's own existing rules */ diff --git a/packages/cli/test/detect.test.ts b/packages/cli/test/detect.test.ts index f1a5913d..d86ac4b2 100644 --- a/packages/cli/test/detect.test.ts +++ b/packages/cli/test/detect.test.ts @@ -1,5 +1,5 @@ import { execFile } from "node:child_process"; -import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { chmod, mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { promisify } from "node:util"; @@ -45,7 +45,11 @@ async function runCli( interface DetectJson { success: boolean; - linters: { name: string; evidence: string[] }[]; + linters: { + name: string; + evidence: string[]; + customRules?: { count: number | "unknown"; evidence: string[] }; + }[]; languages: string[]; ruleStyles: { source: string; description: string }[]; ci: { name: string; evidence: string[] }[]; @@ -73,6 +77,10 @@ function linterNames(result: DetectJson): string[] { return result.linters.map((l) => l.name); } +function customRules(result: DetectJson, name: string) { + return result.linters.find((l) => l.name === name)?.customRules; +} + describe("taskless detect", () => { let cwd: string; @@ -553,11 +561,12 @@ describe("taskless detect", () => { "success", ].toSorted() ); - // A linter entry exposes only name + evidence, never a rule-name claim. + // A linter entry exposes name, evidence, and a count of the repository's + // own rules, never a rule-name claim. for (const linter of result.linters) { - expect(Object.keys(linter).toSorted()).toEqual( - ["evidence", "name"].toSorted() - ); + for (const key of Object.keys(linter)) { + expect(["customRules", "evidence", "name"]).toContain(key); + } } }); @@ -657,6 +666,157 @@ describe("taskless detect", () => { expect(result.linters).toEqual([]); }); + describe("custom-rule counts", () => { + it("counts the rules in a house ESLint rule directory", async () => { + await writeFile(join(cwd, "eslint.config.js"), "export default [];"); + await mkdir(join(cwd, "eslint-rules", "no-foo"), { recursive: true }); + for (const file of [ + "a.js", + "b.ts", + "c.mjs", + "d.cjs", + "e.js", + "f.js", + "index.js", + "a.test.js", + "types.d.ts", + "README.md", + ]) { + await writeFile(join(cwd, "eslint-rules", file), ""); + } + await mkdir(join(cwd, "eslint-rules", "__tests__")); + const rules = customRules(await detect(cwd), "eslint"); + // Six files and the no-foo rule folder; index, tests, declarations, + // docs, and the tests folder are not rules. + expect(rules?.count).toBe(7); + expect(rules?.evidence).toEqual(["eslint-rules (7 rules)"]); + }); + + // Permissions do not restrict root, and Windows ignores the mode bits. + it.skipIf(process.platform === "win32" || process.getuid?.() === 0)( + "skips an unreadable rule directory instead of failing the scan", + async () => { + await writeFile(join(cwd, "eslint.config.js"), "export default [];"); + const directory = join(cwd, "eslint-rules"); + await mkdir(directory); + await writeFile(join(directory, "a.js"), ""); + await chmod(directory, 0o000); + try { + const result = await detect(cwd); + expect(result.success).toBe(true); + expect(customRules(result, "eslint")).toEqual({ + count: 0, + evidence: [], + }); + } finally { + await chmod(directory, 0o755); + } + } + ); + + it("reports a preset-only ESLint config as zero", async () => { + await writeFile( + join(cwd, "eslint.config.js"), + 'import js from "@eslint/js";\nexport default [js.configs.recommended];\n' + ); + expect(customRules(await detect(cwd), "eslint")).toEqual({ + count: 0, + evidence: [], + }); + }); + + it("reports a plugin imported from a local file as unknown", async () => { + await writeFile( + join(cwd, "eslint.config.js"), + 'import local from "./tools/lint-plugin.js";\nexport default [{ plugins: { local } }];\n' + ); + const rules = customRules(await detect(cwd), "eslint"); + expect(rules?.count).toBe("unknown"); + expect(rules?.evidence).toEqual([ + "eslint.config.js (plugin imported from a local file)", + ]); + }); + + it("reports an inline plugin as unknown", async () => { + await writeFile( + join(cwd, "eslint.config.mjs"), + "export default [{ plugins: { house: { rules: { a: { create(context) { return {}; } } } } } }];\n" + ); + expect(customRules(await detect(cwd), "eslint")?.count).toBe("unknown"); + }); + + it("counts Biome plugins across nested configs, reading JSONC", async () => { + await writeFile( + join(cwd, "biome.jsonc"), + '{\n // house rules\n "$schema": "https://biomejs.dev/schemas/2.6.2/schema.json",\n "plugins": ["./a.grit", "./b.grit",],\n}\n' + ); + await mkdir(join(cwd, "packages", "web"), { recursive: true }); + await writeFile( + join(cwd, "packages", "web", "biome.json"), + '{ "root": false, "plugins": ["./c.grit"] }' + ); + const rules = customRules(await detect(cwd), "biome"); + expect(rules?.count).toBe(3); + expect(rules?.evidence.toSorted()).toEqual( + [ + "biome.jsonc (2 plugins)", + "packages/web/biome.json (1 plugins)", + ].toSorted() + ); + }); + + it("counts pylint plugins that live in the repository", async () => { + await writeFile( + join(cwd, ".pylintrc"), + "[MAIN]\nload-plugins = house.checkers,\n pylint.extensions.docparams,\n pylint_django\n" + ); + await mkdir(join(cwd, "house"), { recursive: true }); + await writeFile(join(cwd, "house", "checkers.py"), ""); + const rules = customRules(await detect(cwd), "pylint"); + expect(rules).toEqual({ + count: 1, + evidence: [".pylintrc load-plugins house.checkers"], + }); + }); + + it("does not count pylint's own extensions", async () => { + await writeFile( + join(cwd, "pyproject.toml"), + '[tool.pylint.main]\nload-plugins = ["pylint.extensions.bad_builtin"]\n' + ); + expect(customRules(await detect(cwd), "pylint")?.count).toBe(0); + }); + + it("counts flake8 local plugins and published entry points", async () => { + await writeFile( + join(cwd, ".flake8"), + "[flake8]\nmax-line-length = 100\n\n[flake8:local-plugins]\nextension =\n HSE1 = house.flake8:NoFoo\n HSE2 = house.flake8:NoBar\npaths = ./tools\n" + ); + await writeFile( + join(cwd, "pyproject.toml"), + '[project]\nname = "house"\n\n[project.entry-points."flake8.extension"]\nHSE3 = "house.plugin:NoBaz"\n' + ); + const rules = customRules(await detect(cwd), "flake8"); + expect(rules?.count).toBe(3); + }); + + it("reports Ruff as zero however it is configured", async () => { + await writeFile( + join(cwd, "ruff.toml"), + '[lint]\nextend-select = ["TID"]\n[lint.flake8-tidy-imports.banned-api]\n"requests.get".msg = "use the client"\n' + ); + expect(customRules(await detect(cwd), "ruff")).toEqual({ + count: 0, + evidence: [], + }); + }); + + it("gives no count to a linter that cannot hold a custom rule", async () => { + await writeFile(join(cwd, ".prettierrc"), "{}"); + expect(customRules(await detect(cwd), "prettier")).toBeUndefined(); + }); + }); + describe("CI systems and commit hooks", () => { it("reports both as empty arrays on a bare repository", async () => { const result = await detect(cwd); diff --git a/packages/cli/test/legacy-ladder.test.ts b/packages/cli/test/legacy-ladder.test.ts new file mode 100644 index 00000000..62e2e698 --- /dev/null +++ b/packages/cli/test/legacy-ladder.test.ts @@ -0,0 +1,101 @@ +import { describe, expect, it } from "vitest"; + +import { getRecipe } from "../src/prompts/recipes"; + +function recipe(topic: string): string { + const text = getRecipe(topic); + if (text === undefined) throw new Error(`no recipe for ${topic}`); + return text; +} + +describe("route's legacy ladder", () => { + const route = recipe("route"); + + it("is topic v8", () => { + expect(route).toMatch(/^# Topic: route {5}\(CLI v.+ \/ topic v8\)/m); + }); + + it("orders shipped rule, repo examples, Taskless static, Taskless runtime", () => { + const steps = [ + "**A rule the linter already ships solves it.**", + "**The repo already writes its own rules for that linter.**", + "**A Taskless `sg` or `vale` rule**", + "**A Taskless runtime rule**", + ].map((step) => route.indexOf(step)); + expect(steps.every((index) => index !== -1)).toBe(true); + expect(steps).toEqual(steps.toSorted((a, b) => a - b)); + }); + + it("gates step 2 on five custom rules and treats unknown as below it", () => { + expect(route).toContain("`customRules.count` of 5 or more"); + expect(route).toContain( + 'An\n `"unknown"` count is below the threshold.' + ); + }); + + it("does not pose a legacy rule against a Taskless rule as a choice", () => { + expect(route).toContain("do not ask. The legacy ladder in step 4 decides"); + expect(route).not.toContain( + "Present both with trade-offs and let the user\n choose." + ); + }); +}); + +describe("the generic legacy recipe", () => { + const legacy = recipe("create-legacy-rule"); + + it("is topic v3", () => { + expect(legacy).toMatch( + /^# Topic: create-legacy-rule {5}\(CLI v.+ \/ topic v3\)/m + ); + }); + + it("looks for a shipped rule, catch-alls by name, before anything else", () => { + const shipped = legacy.indexOf( + "**Look for a rule the tool already ships.**" + ); + const custom = legacy.indexOf( + "**No shipped rule fits: check the repo's own rules.**" + ); + expect(shipped).toBeGreaterThan(-1); + expect(custom).toBeGreaterThan(shipped); + for (const catchAll of [ + "`no-restricted-syntax`", + "`no-restricted-imports`", + "`banned-api`", + "`noRestrictedImports`", + "`deprecated-modules`", + ]) { + expect(legacy).toContain(catchAll); + } + }); + + it("writes a custom rule only from five of the repo's own", () => { + expect(legacy).toContain( + "**A `count` of 5 or more:** you may write a new rule." + ); + expect(legacy).toContain( + '**Fewer than 5, `"unknown"`, or no `customRules` at all:** do not\n write a custom rule.' + ); + }); + + it("asks before adding a plugin the repo does not have", () => { + expect(legacy).toContain( + "**A plugin the repo does not have is a new dependency.**" + ); + }); + + it("says Taskless does not run the linter", () => { + expect(legacy).toContain("check` does NOT execute external linters"); + }); +}); + +describe("onboarding's legacy limits", () => { + const onboard = recipe("onboard"); + + it("never writes a custom linter rule or adds a dependency", () => { + expect(onboard).toContain( + "During onboarding do not write a custom rule in any linter,\n however many the repo has, and do not add a dependency." + ); + }); +}); diff --git a/packages/cli/test/onboard.test.ts b/packages/cli/test/onboard.test.ts index dd6ad09d..6acdbb35 100644 --- a/packages/cli/test/onboard.test.ts +++ b/packages/cli/test/onboard.test.ts @@ -390,8 +390,8 @@ describe("onboard recipe asks one question and finishes on its own", () => { : stdout.slice(start, start + title.length + next); } - it("is topic v10", () => { - expect(stdout).toMatch(/^# Topic: onboard {5}\(CLI v.+ \/ topic v10\)/m); + it("is topic v11", () => { + expect(stdout).toMatch(/^# Topic: onboard {5}\(CLI v.+ \/ topic v11\)/m); }); // v10 (#554): a worktree holds only commits, and `init` commits nothing, so diff --git a/packages/cli/test/prompts.test.ts b/packages/cli/test/prompts.test.ts index d3352b79..e10c34b6 100644 --- a/packages/cli/test/prompts.test.ts +++ b/packages/cli/test/prompts.test.ts @@ -1,11 +1,12 @@ import { execFile } from "node:child_process"; -import { readFile, readdir } from "node:fs/promises"; +import { mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; import { join, resolve } from "node:path"; import { pathToFileURL } from "node:url"; import { promisify } from "node:util"; import { sprintf } from "sprintf-js"; -import { describe, expect, it } from "vitest"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; import { PROMPTS, @@ -212,6 +213,7 @@ describe("the CLI invocation variable", () => { "DETECT_EVIDENCE", "FIXTURE_SOURCES", "HOST_TOOLS", + "LEGACY_LINTERS", "LOGIN_EVIDENCE", "NAME_DESTINATION", "ONE_COMMAND_PER_CALL", @@ -584,21 +586,96 @@ describe("typed accessor", () => { }); describe("agent command parity", () => { + // The command scans its working directory for the passages that describe + // detected linters, so it runs against an empty one and the export is told + // the same measured answer: no linters. + let emptyDirectory: string; + beforeAll(async () => { + emptyDirectory = await mkdtemp(join(tmpdir(), "taskless-parity-")); + }); + afterAll(async () => { + await rm(emptyDirectory, { recursive: true, force: true }); + }); + it.each([...TOPICS])( "matches `taskless agent %s` byte for byte", async (topic) => { - const { stdout } = await execFileAsync("node", [binPath, "agent", topic]); + const { stdout } = await execFileAsync("node", [ + binPath, + "agent", + topic, + "-d", + emptyDirectory, + ]); // The command trims trailing whitespace before printing; console.log then // adds the single newline that stdout carries. It also asks for the // fetch-time directive, which the export leaves off by default; a // consumer that wants the served text asks for it the same way. expect(stdout.trimEnd()).toBe( - getPrompt(topic, { directive: true }).trimEnd() + getPrompt(topic, { directive: true, linters: [] }).trimEnd() ); } ); }); +describe("route's per-linter passage", () => { + const LINTER_HEADINGS = [ + "**ESLint**", + "**Biome**", + "**Ruff**", + "**pylint**", + "**flake8**", + ]; + + it("describes only the linters it is told were detected", () => { + const route = getPrompt("route", { linters: ["ruff", "prettier"] }); + expect(route).toContain("**Ruff**"); + expect(route).toContain("`banned-api` (TID251)"); + for (const heading of LINTER_HEADINGS.filter((h) => h !== "**Ruff**")) { + expect(route).not.toContain(heading); + } + }); + + it("says so when no detected linter has a legacy recipe", () => { + const route = getPrompt("route", { linters: ["prettier"] }); + for (const heading of LINTER_HEADINGS) { + expect(route).not.toContain(heading); + } + expect(route).toContain("found no linter here with a legacy recipe"); + }); + + it("describes every linter when nothing was scanned", () => { + const route = getPrompt("route"); + for (const heading of LINTER_HEADINGS) { + expect(route).toContain(heading); + } + }); + + it("names the topic rather than a command when mechanics are off", () => { + const route = getPrompt("route", { mechanics: false }); + expect(route).toContain("For any of them, fetch `create-legacy-rule`."); + }); + + it("is rendered from the repository the agent command scans", async () => { + const directory = await mkdtemp(join(tmpdir(), "taskless-route-")); + try { + await writeFile(join(directory, "ruff.toml"), "line-length = 100\n"); + const { stdout } = await execFileAsync("node", [ + binPath, + "agent", + "route", + "-d", + directory, + ]); + expect(stdout).toContain("The linters `detect` found here"); + expect(stdout).toContain("**Ruff**"); + expect(stdout).not.toContain("**ESLint**"); + } finally { + await rm(directory, { recursive: true, force: true }); + } + }); +}); + describe("topic membership", () => { it("classifies every canonical recipe as exported or internal", async () => { const onDisk = await canonicalTopicsOnDisk();