From 2d82dbfa245d0cc4bf48c79a15856ae6c0f44ee4 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 1 Sep 2026 08:51:16 +1000 Subject: [PATCH 1/2] docs: propose a retrievability lint MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An SEO consultant's read of these docs: "Excellent docs: built to be read by a developer, not retrieved by a model." The mechanical half of that has been worked through — absolute urls in llms.txt, source urls in llms-full.txt, 77 redirects off the pre-V2 tree, the .md rewrite, working links in the markdown views. Those were bugs, and bugs stay fixed. The editorial half does not stay fixed. It regresses one page at a time and nothing notices, because the defect is invisible to a human reviewer by construction: a section headed "How it works" reads fine to someone going through the page top to bottom, and names its subject nowhere once a retriever lifts it out. Defines the four rules concretely enough to check — a heading names its subject, a chunk names an entity, no orphan back-references, frontmatter descriptions present and in range — with today's counts measured against content/docs (624 headings, 26 bare generic, 6 question-shaped, 65 entity-free chunks, 0 back-references, 49 off-spec descriptions). Proposes scripts/validate-retrievability.ts beside the existing validate-* gates in prebuild: rules 1 and 3 as hard failures, 2 and 4 as ratcheting baselines, since 65 and 49 are too many to fail on today and rule 2 has real false positives. States the limit plainly: these are proxies for shape, not for whether a chunk answers anything. The real test is a retrieval eval, which is worth building once the mechanical rules stop finding things. No script yet — this is the proposal to agree on before writing one. --- RETRIEVABILITY.md | 147 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 147 insertions(+) create mode 100644 RETRIEVABILITY.md diff --git a/RETRIEVABILITY.md b/RETRIEVABILITY.md new file mode 100644 index 0000000..ae8ab30 --- /dev/null +++ b/RETRIEVABILITY.md @@ -0,0 +1,147 @@ +# Retrievability rules, and a proposal for enforcing them + +An SEO consultant's read of these docs in August 2026: *"Excellent docs: built +to be read by a developer, not retrieved by a model."* + +The mechanical half of that — agents literally could not fetch pages — has been +worked through: absolute URLs in `llms.txt`, source URLs in `llms-full.txt`, 77 +redirects off the pre-V2 URL tree, the `.md` suffix rewrite, working internal +links in the markdown views. Those were bugs, and bugs stay fixed. + +The editorial half does not stay fixed. It regresses one page at a time, and +nothing currently notices. + +This document defines what "retrievable" means here concretely enough to check, +and proposes `scripts/validate-retrievability.ts` to check it. + +## Why it regresses + +Retrieval does not match a query against a page. It matches against a **chunk** — +a section, a few hundred tokens, lifted out of the document that gave it +context. Everything the surrounding page supplied is gone: the H1, the +breadcrumb, the sidebar, the paragraph before. + +So a section headed `## How it works`, on a page titled `Overview`, in a folder +called Concepts, names its subject exactly nowhere. It is a perfectly good +heading for someone reading the page top to bottom, and close to useless for +retrieval. That single pattern is the whole finding, and it is invisible to a +human reviewer precisely because they are reading the page top to bottom. + +## What is mechanically checkable + +There is no off-the-shelf linter for this. But most of what was measured is +shape, and shape is checkable. + +Measured against `content/docs` on `main` (90 hand-written pages; generated +`api-reference` trees excluded): + +| Signal | Today | +| --- | --- | +| Headings (H2–H4) | 624 | +| Bare generic headings (`How it works`, `Examples`, `Overview`, …) | 26 | +| Question-shaped headings | 6 | +| Chunks naming no product entity in their first 600 characters | 65 | +| Orphan back-references (`as shown above`, `the previous section`) | 0 | +| Frontmatter descriptions absent or outside 110–158 characters | 49 | + +## The rules + +### 1. A heading names its own subject + +Flags H2–H4 whose text, lowercased and stripped, is a bare label from a +denylist: `how it works`, `overview`, `examples`, `errors`, `configuration`, +`usage`, `the problem`, `limitations`, and so on. Also flags a heading opening +with a pronoun that has no antecedent inside its own chunk. + +Navigation boilerplate is allowlisted and left alone — `Related`, `Next steps`, +`Before you start`, `In this section`, `Prerequisites`. Those headings are +structural furniture, they are not trying to be retrieved, and renaming them +would make pages read worse for no gain. + +**Enforce as a hard failure.** 26 occurrences today, all of them fixable by +writing a better heading, and the two open PRs that name headings bring that to +zero. + +### 2. A chunk names an entity + +Flags a section whose first ~600 characters mention no product noun — +CipherStash, EQL, ZeroKMS, CTS, keyset, lock context, encrypted column, +Supabase, Prisma, Drizzle, Postgres. + +65 occurrences, and this is the rule with real false positives: a section can +legitimately be pure prose that inherits its subject from two paragraphs up. + +**Enforce as a ratcheting baseline**, not a hard failure. Commit the current +count; fail only when it goes up. The number comes down when someone is editing +that page anyway, and never silently grows. + +### 3. No orphan back-references + +Flags `as shown above`, `as described above`, `the previous section`, `see +above`, `described earlier`. In a chunk, "above" is nothing. + +**Hard failure.** Zero today; keeping zero costs nothing. + +### 4. Frontmatter descriptions are present and useful + +`description` is the one-line summary that goes into `llms.txt` and into search +results, and it is the only text some agents ever see about a page. Flags +missing descriptions and lengths outside 110–158 characters — the range a +previous SEO pass already normalised 52 pages into, which nothing currently +holds. + +49 pages are off-spec today, but many are section index stubs that carry a +short nav description deliberately. **Ratcheting baseline**, with index pages +excluded from the tree exempted outright. + +## How it runs + +Beside the validators that already exist, in the same style, wired into the +same place: + +``` +"validate-retrievability": "tsx scripts/validate-retrievability.ts", +"prebuild": "… && bun run validate-links && bun run validate-redirects && bun run validate-retrievability" +``` + +`scripts/validate-links.ts`, `validate-content-api.ts`, `validate-mermaid.ts` +and `validate-v2-redirects.ts` all follow this shape: read `content/`, print a +one-line verdict on success, print every failure with `file:line` and exit 1. +This one adds a baseline file, `scripts/fixtures/retrievability-baseline.json`, +holding the per-rule counts for the ratcheting rules, and `--update-baseline` +to move it. + +Generated trees are skipped, the same way `validate-content-api.ts` already +skips them. Where a generated page breaks a rule the fix belongs in its +generator, not in a suppression — the same argument that took +`generate-cli-docs.ts` from `## Examples` to `## stash init examples`. + +## What this deliberately does not do + +These rules are proxies. They check that a chunk *carries* its subject, not +that it *answers* anything. A page can pass every rule here and still be +unretrievable because it explains a mechanism nobody would search for. + +The real test is a retrieval eval: a fixed set of questions a developer or +their agent would actually ask, run against the live docs, checking whether the +right page comes back in the top few results. That is a heavier thing to build +and to maintain, and it is worth building once the mechanical rules stop +finding anything. Starting there instead would mean hand-tuning an eval harness +against 26 headings a regex can find today. + +## Rollout + +1. Rules 1 and 3 as hard failures. They are at or near zero once the open + heading PRs land, so they start green and stay green. +2. Rules 2 and 4 recorded as a baseline in the same PR, failing only on + increase. +3. Revisit after a quarter of real content changes: whether the baselines have + actually come down, and whether rule 2's false-positive rate is low enough + to promote it to a hard failure. + +## Related + +- **CIP-3339** — AI-agent readiness. This is the enforcement arm of that + issue's content rules, and of its "negative capability statements" bullet. +- **CIP-3337** — Docs correctness CI. Same gate, different axis: that one asks + whether the docs are *true*, this one asks whether they can be *found*. From 311c4c0950a70bff84b3c1f7500e8ff2df725672 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 1 Sep 2026 08:57:50 +1000 Subject: [PATCH 2/2] docs: add the markdown-view link rule to the retrievability proposal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review question: we have broken-link checks in CI, so why did the root-relative links in the markdown views go unnoticed? Because they are not broken. validate-links.ts resolves against the collections as the app serves them — content/docs at baseUrl "" — so /reference/auth/clients is a correct link to a real page. It also rejects the absolute form outright, since Next's basePath prepends /docs and a /docs prefix would resolve to /docs/docs/…. Both judgements are right for the HTML rendering, which is the only one it models. The markdown view is a second rendering with no basePath and no browser, where the same link resolves against the origin and 404s. A source-level check cannot see that, and should not be bent into trying. Adds rule 5: render what /docs/.mdx returns and assert no root-relative link survives it. Hard failure — an invariant of the route rather than a matter of degree — and it generalises: anything that reshapes content on the way out needs its output asserted, not its input. --- RETRIEVABILITY.md | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/RETRIEVABILITY.md b/RETRIEVABILITY.md index ae8ab30..8aae2b5 100644 --- a/RETRIEVABILITY.md +++ b/RETRIEVABILITY.md @@ -94,6 +94,32 @@ holds. short nav description deliberately. **Ratcheting baseline**, with index pages excluded from the tree exempted outright. +### 5. The markdown views emit no root-relative link + +Renders what `/docs/.mdx` actually returns and asserts no `](/…)` link +survives it. + +This rule exists because `validate-links.ts` cannot catch that class of defect +and should not be changed to try. It resolves links against the collections as +the app serves them (`content/docs` at `baseUrl: ""`), so `/reference/auth/clients` +is a correct link to a real page — and it actively *rejects* the absolute form, +on the grounds that Next's basePath prepends `/docs` and a `/docs` prefix would +resolve to `/docs/docs/…`. Both of those judgements are right for the HTML +rendering. + +There are two renderings. The HTML page has a basePath and a browser to resolve +against. The markdown view is a flat file read from an origin, where that +context is gone and the same link resolves to `cipherstash.com/reference/auth/clients` +— a 404, on every internal link in every page body and in all 3 MB of +`llms-full.txt`. `getLLMText` now absolutises on the way out, so pages keep +authoring the root-relative form the source checker enforces. + +The lesson generalises past this one bug: a source-level check validates one +rendering. Anything that reshapes content on the way out needs its output +asserted, not its input. + +**Hard failure.** It is an invariant of the route, not a matter of degree. + ## How it runs Beside the validators that already exist, in the same style, wired into the @@ -131,8 +157,8 @@ against 26 headings a regex can find today. ## Rollout -1. Rules 1 and 3 as hard failures. They are at or near zero once the open - heading PRs land, so they start green and stay green. +1. Rules 1, 3 and 5 as hard failures. They are at or near zero once the open + heading and markdown-view PRs land, so they start green and stay green. 2. Rules 2 and 4 recorded as a baseline in the same PR, failing only on increase. 3. Revisit after a quarter of real content changes: whether the baselines have