feat(skills): add a planning skill for documentation evals - #259
Draft
czenko wants to merge 5 commits into
Draft
Conversation
Four build-docs evals exist and each was designed from scratch. The most recent cost three false reds and one false green, all of them mistakes an earlier author had already made. The skill plans a documentation eval and stops at the plan. Six phases: enter plan mode and read CONTRIBUTING.md, gather the evidence, pick the one claim the page has to transmit, draft the prompt, design the checks, write the plan, and feed the catalog once the baseline lands. Five references carry the catalogs. sources.md lists every evidence source and marks which need an MCP connector or private access, because an agent that silently skips a gated source produces a thin inventory that reads as complete. check-rules.md is ordered by what a violation costs, and the first three rules are the ones that have sunk a run. flakiness.md is mined from the review feedback on twenty-one PRs, with a citation per entry. It lives under .claude/skills/ rather than skills/, which holds product skills that experiments install into eval sandboxes. Closes DOCS-1228
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Contributor
Author
|
I plan to stress test this SKILL before merging by creating another Docs Eval. |
czenko
commented
Sep 2, 2026
czenko
left a comment
czenko
left a comment
Contributor
Author
There was a problem hiding this comment.
Feedback on my own PR 😁
Applies review feedback. Every reference now states the decision rather than the case that produced it, per Google's timeless documentation guidance. canonical-shape.md drops the genre background, the scorer-structure history, and the reasoning behind not committing solutions. It says to model on build-docs-003 or later, and to never commit solutions. check-rules.md and prompt-rules.md drop their worked examples and state what to do. The motivation guidance now names the constant behind every one of these evals: evidence that an agent can read the page and do the task correctly. sources.md opens with the note that Linear, Slack, Hex, and Supacademy are internal to Supabase, since this repository is open source, and that planning from the public sources alone is a supported outcome.
There was a problem hiding this comment.
🟡 Changes recommended
The new skill is committed under .claude/skills/, but the harness resolves/installable skills from skills/<name>, so it won’t be discoverable via the normal skills mechanism as-is.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a new “docs-eval-planning” agent skill intended to standardize how documentation-page evals are planned in this repo, capturing prior failure modes and conventions so new evals don’t repeat avoidable mistakes.
Changes:
- Introduces a new planning skill (
SKILL.md) describing a phased workflow for designing docs eval prompts and checks. - Adds reference documents covering evidence sources, prompt authoring rules, check design rules, canonical eval shape, and a flakiness catalog.
File summaries
| File | Description |
|---|---|
| .claude/skills/docs-eval-planning/SKILL.md | Main skill entrypoint describing phases and how to produce a planning artifact. |
| .claude/skills/docs-eval-planning/references/sources.md | Catalog of evidence sources to derive checks from (internal + public). |
| .claude/skills/docs-eval-planning/references/prompt-rules.md | Rules for writing prompts that test the page rather than the model. |
| .claude/skills/docs-eval-planning/references/check-rules.md | Ordered checklist of check-design constraints and common traps. |
| .claude/skills/docs-eval-planning/references/flakiness.md | Curated list of previously observed false reds/greens and nondeterminism sources. |
| .claude/skills/docs-eval-planning/references/canonical-shape.md | Conventions for docs-eval folder shape, README content, and orchestration patterns. |
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+1
to
+4
| --- | ||
| name: docs-eval-planning | ||
| description: Plan a documentation eval for supabase/evals, where a docs guide is the subject under test. Use when asked to write, add, or design an eval for a Supabase docs guide, when a ticket asks for a deterministic eval on a page, or when deciding what a guide-under-test eval should check. Produces a plan, not files. Not for debugging a scorer, running an eval, or writing example solutions. | ||
| --- |
Comment on lines
+22
to
+23
| Call `EnterPlanMode` first. Everything in the gathering phase is read-only, and plan mode is what | ||
| enforces that. |
canonical-shape.md read as a thin spec once its background came out. Its six items move into SKILL.md: the naming and id rule, the file layout, the EVAL.ts shape, the README section set, and the guide-read check go into a shape section before the phases; the seed rules and the contract choice join Phase 3, where the seed is drafted; judges join Phase 4; example solutions join Phase 5, where verification is planned. SKILL.md is 1,587 words, inside the range a skill body should hold.
Running the skill on auth/server-side/creating-a-client found five things it does not say. Phase 2 now ends by confirming the harness can measure the claim, before any check is designed. A claim the harness cannot observe is not the claim, and finding that in Phase 4 is a rewrite rather than a design change. The three questions are where the end state lives, whether the scorer can run what has to run, and what the time budget is. Phase 2 also gives a basis for picking. A dense page yields ten evidenced candidates, so pick on consequence and prefer the failure the page itself warns about. sources.md adds the ticket as a source and says to read its rationale as evidence rather than instruction, because a ticket written before the page changed will name a method the page no longer recommends. The diff step now says the page wins on what is acceptable and the check names the class. flakiness.md gains the two entries behind the feasibility step: a framework the eval seeds is not on the host, and a first-of-its-kind toolchain in a scorer is a risk to state rather than a detail. Phase 0 reads flakiness.md, which the intro already asked for.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes DOCS-1228
Problem
Four
build-docs-*evals exist and each was designed from scratch. The most recent cost three false reds and one false green, all of them mistakes an earlier author had already made or could have warned about. The only authoring guidance that exists is a five-step how-to buried inbuild-docs-001/README.md.Solution
.claude/skills/docs-eval-planning/:SKILL.mdplus four references.CONTRIBUTING.md, gather the evidence, pick the one claim the page has to transmit, draft the prompt, design the checks, write the plan, and feed the catalog once the baseline lands.references/sources.mdlists every evidence source and marks which need an MCP connector or private-repo access. The rule is to stop and ask for a missing connector rather than working around it, then record which sources were reachable, because an inventory built from public sources alone reads as complete when it is not.references/check-rules.mdis ordered by what a violation costs. The first three are observability, behaviour over structure, and positive controls a literal cannot satisfy.references/flakiness.mdis mined from review feedback on twenty-one PRs, with a citation per entry. Nine of those PRs were outside the set I started from, including feat: add docs homepage quickstart eval #108, which carries aCHANGES_REQUESTEDreview, and feat: move eval execution from GitHub Actions to Vercel Sandboxes #192, where the CI reproducibility work happened.references/prompt-rules.mdcarries the prompt discipline. The file, frontmatter, and README conventions live inSKILL.mditself.Manual testing
Confirm the skill is discovered and its description reads correctly.
Confirm every relative link resolves and every path the skill names exists.
Typecheck.
Clean. The change is markdown only.
Confirm Biome does not process the skill, since it ships no markdown formatter.
No files were processed in the specified paths.