feat(docs): triage script for 404 exports

[coderdan]

What

A triage script for Vercel 404 log exports.

Why

The raw export is unreadable. In the 2026-08-26 → 08-31 window, 4,771 of 5,000 rows were Next.js client segment-cache prefetches — the ~230 failures worth acting on sat under 95% noise, and the top of any naive 404 report is entirely *.segment.rsc.

bun run triage-404s <export.csv>

Dedupes by requestId (the export repeats rows), buckets by shape, drops the prefetch noise, and prints what is left with requester attribution. Against that export:

5000 unique requests (174 duplicate rows dropped)
window: 2026-08-26 16:41:46 → 2026-08-31 05:18:29

bucket             requests
  segment-prefetch   4771  (benign Next.js prefetch — ignore)
  page                147
  source-ref           46
  asset                19
  markdown-suffix      12
  machine-endpoint      5

175 distinct paths worth triaging, 229 requests:

hits  ai search bot browser  path
   7   0      0   1       6  /docs/home
   6   0      0   1       5  /docs/guides/protect-js
   4   0      0   1       3  /docs/how-to/creating-datasets
   ...

That output is what the redirect map in #97 was built from. Not wired into prebuild — it reads a file that only exists once someone pulls an export.

The source-ref bucket (46 requests) is worth its own look: paths like /docs/integrations/prisma/execution/envelope-json.ts:44. Docs prose emits file.ts:NN strings that agents resolve as links. No redirect fixes that; it is a content fix.

Why the segment prefetches 404

Segment data is only generated when cacheComponents is on. Reproduced against production:

$ curl -o/dev/null -w'%{http_code}\n' \
    'https://cipherstash.com/docs/reference/agent-skills.segments/_tree.segment.rsc?_rsc=x'
200

$ curl -o/dev/null -w'%{http_code}\n' -H 'Next-Router-Segment-Prefetch: /_tree' \
    'https://cipherstash.com/docs/reference/agent-skills.segments/_tree.segment.rsc?_rsc=x'
404

The router falls back to a normal prefetch, so nothing is broken for the reader — the cost is ~1,500 wasted requests a day (current 24h Vercel counts: 1,498 × 404 against 144 × 200) and a ruined 404 report.

Next 16.2.6 exposes no flag to turn the client segment cache off — there is no clientSegmentCache key in its config schema. Enabling cacheComponents is the real fix, tracked in CIP-3910. This PR classifies the noise rather than pretending to eliminate it.

Dropped from this PR

An earlier revision added /docs/robots.txt. Removed: robots.txt is only ever read at the origin root, which the marketing app owns and which already allows everything and lists this app's sitemap. A second file on the same host would be inert, and a future edit to it — a Disallow scoped to the docs — would look correct, deploy cleanly, and do nothing.

The one request for /docs/robots.txt in the log window turned out to be my own curl probe while investigating, not an agent.

Checks

• npm run types:check — clean
• biome check — clean
• triage-404s run against the real export, output above

https://claude.ai/code/session_01BsA6C8Y48GDjh6ELL8yVjf

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
public-docs	Ready	Preview	Aug 31, 2026 7:46am