docs: propose a retrievability lint - #104
Open
coderdan wants to merge 2 commits into
Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
An SEO consultant's read of these docs: "Excellent docs: built to be read by a developer, not retrieved by a model." The mechanical half of that has been worked through — absolute urls in llms.txt, source urls in llms-full.txt, 77 redirects off the pre-V2 tree, the .md rewrite, working links in the markdown views. Those were bugs, and bugs stay fixed. The editorial half does not stay fixed. It regresses one page at a time and nothing notices, because the defect is invisible to a human reviewer by construction: a section headed "How it works" reads fine to someone going through the page top to bottom, and names its subject nowhere once a retriever lifts it out. Defines the four rules concretely enough to check — a heading names its subject, a chunk names an entity, no orphan back-references, frontmatter descriptions present and in range — with today's counts measured against content/docs (624 headings, 26 bare generic, 6 question-shaped, 65 entity-free chunks, 0 back-references, 49 off-spec descriptions). Proposes scripts/validate-retrievability.ts beside the existing validate-* gates in prebuild: rules 1 and 3 as hard failures, 2 and 4 as ratcheting baselines, since 65 and 49 are too many to fail on today and rule 2 has real false positives. States the limit plainly: these are proxies for shape, not for whether a chunk answers anything. The real test is a retrieval eval, which is worth building once the mechanical rules stop finding things. No script yet — this is the proposal to agree on before writing one.
Review question: we have broken-link checks in CI, so why did the root-relative links in the markdown views go unnoticed? Because they are not broken. validate-links.ts resolves against the collections as the app serves them — content/docs at baseUrl "" — so /reference/auth/clients is a correct link to a real page. It also rejects the absolute form outright, since Next's basePath prepends /docs and a /docs prefix would resolve to /docs/docs/…. Both judgements are right for the HTML rendering, which is the only one it models. The markdown view is a second rendering with no basePath and no browser, where the same link resolves against the origin and 404s. A source-level check cannot see that, and should not be bent into trying. Adds rule 5: render what /docs/<path>.mdx returns and assert no root-relative link survives it. Hard failure — an invariant of the route rather than a matter of degree — and it generalises: anything that reshapes content on the way out needs its output asserted, not its input.
coderdan
force-pushed
the
docs/retrievability-lint-proposal
branch
from
September 1, 2026 02:38
7925e1b to
311c4c0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposal only — no script yet. Worth agreeing the rules before writing the thing that enforces them.
Why
The mechanical failures from this session's 404 work were bugs, and bugs stay fixed. The editorial half regresses one page at a time, and nothing notices.
The reason it goes unnoticed is structural. Retrieval matches a query against a chunk, not a page — a section lifted out of the document that gave it context. A section headed
## How it works, on a page titledOverview, in a folder called Concepts, names its subject exactly nowhere. It reads perfectly well to a reviewer going through the page top to bottom, which is why review does not catch it.Measured today
content/docsonmain, 90 hand-written pages, generatedapi-referencetrees excluded:The proposal in one line
scripts/validate-retrievability.tsbesidevalidate-links,validate-content-api,validate-mermaidandvalidate-v2-redirects, wired into the sameprebuild. Rules 1 and 3 as hard failures; rules 2 and 4 as ratcheting baselines inscripts/fixtures/retrievability-baseline.json, failing only on increase.Ratcheting rather than failing outright because 65 and 49 are too many to fix in one pass, and rule 2 has genuine false positives — a section can legitimately inherit its subject from two paragraphs up.
What it will not do
These rules check that a chunk carries its subject, not that it answers anything. A page can pass all four and still be unretrievable. The real test is a retrieval eval — fixed question set, run against the live docs, check what comes back — which is worth building once the mechanical rules stop finding things, not before.
Open questions for review
Related,Next steps,Before you start,In this section) be exactly that set?Related: CIP-3339 (AI-agent readiness — this is its enforcement arm) and CIP-3337 (docs correctness CI — same gate, different axis: that one asks whether the docs are true, this one whether they can be found).