Index the 96 AG-UI documentation pages hosted on docs.copilotkit.ai by jpr5 · Pull Request #164 · CopilotKit/pathfinder

jpr5 · 2026-09-11T07:40:25Z

What is wrong

showcase/shell-docs/src/content/ag-ui/ in CopilotKit/CopilotKit is 96 .mdx files that are live documentation pages — served at https://docs.copilotkit.ai/ag-ui/<slug> by src/app/ag-ui/[[...slug]], and all 96 are in the production sitemap. No configured source claimed a single one of them. The docs source already walked their parent directory (src/content/, widened in #159), but its file_patterns named only the docs/ and reference/ subtrees.

#163's new unclaimed_content check found this on its first real run and deliberately left it unexempted, pending a decision. The decision: index them here — CopilotKit includes AG-UI.

Extend the existing source, not a new one

A type: search tool takes exactly one source, so a separate copilotkit-ag-ui-docs source could not feed search-docs — it would need a third docs tool, sitting between two that already confuse each other (search-docs and search-ag-ui-docs). That is the opposite of "CopilotKit includes AG-UI". The tree also needs no different walk root, no different base_url, no different derivation, and the webhook path_triggers for docs already cover src/content/. One more file_patterns entry is the whole change.

Honest note on overlap: 94 of these 96 files also exist in ag-ui-protocol/ag-ui's docs/ tree, which the ag-ui-docs source already indexes — mostly the same prose, drifted in formatting. They are indexed twice on purpose, because the two copies hand back different links: ag-ui-docs answers with docs.ag-ui.com/..., this one with the docs.copilotkit.ai/ag-ui/... page a CopilotKit reader is actually on. Both tool descriptions now say so, so an agent can route deliberately instead of guessing.

No change to `url_derivation` — and proof that none was needed

The ordered strip_prefix list is first-match-wins, and getting it wrong here would mint plausible URLs that 404 — a search result that lies, which is worse than no result. ag-ui/ is not under content/docs/, so it falls through to the content/ prefix and keeps its directory. Verified against the real route, not by reading the YAML.

Red → green (real enumeration + real URL derivation, fresh checkout `7a36707ab`, no DB, no production)

Harness calls the shipped walkSourceFiles() and deriveUrl() over the real config.

RED (config as on main):

TOTAL enumerated: 866
  content/docs/:      682
  content/reference/: 184
  content/ag-ui/:     0
  [ ABSENT] .../content/ag-ui/concepts/agents.mdx
  [ ABSENT] .../content/ag-ui/concepts/architecture.mdx
  [ ABSENT] .../content/ag-ui/agentic-protocols.mdx
  [ ABSENT] .../content/ag-ui/sdk/js/core/events.mdx

GREEN (this branch):

TOTAL enumerated: 962
  content/docs/:      682
  content/reference/: 184
  content/ag-ui/:     96
  [PRESENT] .../ag-ui/concepts/agents.mdx     -> https://docs.copilotkit.ai/ag-ui/concepts/agents
  [PRESENT] .../ag-ui/concepts/architecture.mdx -> https://docs.copilotkit.ai/ag-ui/concepts/architecture
  [PRESENT] .../ag-ui/agentic-protocols.mdx   -> https://docs.copilotkit.ai/ag-ui/agentic-protocols
  [PRESENT] .../ag-ui/sdk/js/core/events.mdx  -> https://docs.copilotkit.ai/ag-ui/sdk/js/core/events

URL correctness: 96/96

Every one of the 96 derived URLs was checked against the production sitemap (curl -s https://docs.copilotkit.ai/sitemap.xml, 3791 locs):

96 / 96 derived URLs appear in the sitemap. Zero misses in either direction — the sitemap's only extra /ag-ui entry is /ag-ui itself, the overview page with no backing file (its body is rendered inline by OverviewContent), correctly not derived.
Live fetch of all 96 with a genuine 404 positive control (/ag-ui/concepts/agents-definitely-not-a-page → 404): 88 return HTTP 200, 8 return HTTP 500.

The 8 are development/updates, sdk/dart/client/overview, and six sdk/rust/* pages. They are not a derivation error: an invented slug on the same route returns a clean 404, these return 500 after the route matched, they are in the sitemap, and they 500 identically before and after this change (nothing here touches the docs site). That is a pre-existing render fault on docs.copilotkit.ai and worth a separate issue on CopilotKit/CopilotKit. Indexing them is still right — the content is real and the links become good the moment the site is fixed.

No regression: the existing 866 URLs are byte-identical

Full path → URL manifests for the docs/ and reference/ subtrees, before and after, diffed:

red rows: 866   green rows (excluding ag-ui): 866
diff exit=0, 0 lines            cmp: BYTE-IDENTICAL

docs/quickstart.mdx → /quickstart and reference/hooks/useAgent.mdx → /reference/hooks/useAgent both unchanged.

The `unclaimed_content` finding clears

#163's check, run against the updated config over the same checkouts:

	before	after
CopilotKit/CopilotKit	`showcase/shell-docs/src/content/ag-ui .mdx x96`	no unclaimed clusters
ag-ui-protocol/ag-ui	`middlewares .ts x34`	`middlewares .ts x34` (unchanged, unrelated)

The audit that found this goes quiet, and no new cluster appears.

Regression guard

src/__tests__/copilotkit-docs-config.test.ts is new and reads the shipped deploy/copilotkit-docs.yaml, asserting through the real matchesPatterns and deriveUrl which files the docs source claims and what URL each derives. Both of this config's silent failures — the reference tree, and now ag-ui — were invisible to every existing unit test precisely because they lived in the YAML rather than in a fixture. Committed red (4 failing coverage cases), green after the config change.

Deploying this: a reindex is needed, and it should fire by itself

Indexing the pages requires a full walk of the docs source — merging the config does not move content on its own. #160's config_fingerprint should force that automatically, because file_patterns is one of the fields folded into the fingerprint. Measured on the two configs:

2bf63ca62d74ba10ad4698363f3c5f7b   docs source @ origin/main
8274a4799a6d331b97b48a4a74a74cc5   docs source @ this branch

The stored fingerprint no longer matches, so the orchestrator's acquisition branch takes the full-walk path instead of diffing HEAD against itself.

How an operator confirms it landed, after deploy and the next reindex:

/health — the docs source's document count moves from ~866 toward ~962.
The reindex audit reports no unclaimed cluster for CopilotKit/CopilotKit.
search-docs for something only these pages say (e.g. "AG-UI RunAgentInput", "AG-UI Rust SDK subscriber") returns results with docs.copilotkit.ai/ag-ui/... URLs.

If the count does not move, the fingerprint path did not fire and a manual full reindex of the docs source is the fallback.

Local gate

npm run build ✅ · npx tsc --noEmit ✅ · npm test 191 files / 3730 tests passed, 0 failed ✅ · node scripts/check-test-shapes.mjs ✅ · npx prettier --check on all three touched files ✅

The two silent failures this config has had were both invisible to every unit test in the suite, because they lived in the YAML that ships rather than in any synthetic fixture: the API reference tree went unclaimed for months, and the 96 live ag-ui pages still are. This reads deploy/copilotkit-docs.yaml itself and asserts, through the real matchesPatterns and deriveUrl, which files the docs source claims and what URL each one derives. Red as committed: the four ag-ui coverage cases fail.

showcase/shell-docs/src/content/ag-ui/ is 96 .mdx files served live at docs.copilotkit.ai/ag-ui/<slug> by src/app/ag-ui/[[...slug]], and all 96 are in the production sitemap. No configured source claimed one of them: the docs source already walked their parent directory, but its file_patterns named only the docs/ and reference/ subtrees. CopilotKit includes AG-UI, so they are indexed here. They near-duplicate the upstream ag-ui-protocol/ag-ui docs tree that the ag-ui-docs source indexes, which is why this is one more pattern on the existing docs source rather than a separate source: a separate source could not feed search-docs (a search tool takes exactly one source), so it would need a third docs tool sitting between two that already confuse each other. The two tool descriptions now say which site's links each one hands back. No change to url_derivation was needed and none was made. ag-ui/ is not under content/docs/, so it falls through the ordered strip_prefix list to the content/ prefix and keeps its directory: content/ag-ui/concepts/agents.mdx -> /ag-ui/concepts/agents. Measured through the real walkSourceFiles + deriveUrl path against a fresh CopilotKit checkout (7a36707ab): 866 -> 962 files, +96. The manifest of all 866 pre-existing docs/ and reference/ URLs is byte-identical before and after. All 96 derived ag-ui URLs appear in the production sitemap, 96/96; 88 return HTTP 200 live and 8 return HTTP 500 — a pre-existing render fault on the docs site itself, not a derivation error, since an invented slug under the same route returns 404. The unclaimed-content audit now reports zero clusters for this repo, down from the one 96-file .mdx cluster it was built to find.

) ## What is wrong `showcase/shell-docs/src/content/ag-ui/` no longer exists. **CopilotKit#7092, "fix(docs): remove retired AG-UI mirror"**, merged `2026-09-11T22:44:24Z` (`tylerslaton`, merge `3e86b8d55`) — **112 files, 20 additions, 25,059 deletions** — deleted the whole tree plus `src/app/sitemap.ts` and `src/app/llms-mdx/[[...slug]]/route.ts`. Confirmed against the live API: ``` $ gh api repos/CopilotKit/CopilotKit/contents/showcase/shell-docs/src/content --jq '.[].name' docs doctest.json framework-overviews reference snippets ``` #164 (`298b7fa`) had added `showcase/shell-docs/src/content/ag-ui/**/*.mdx` to the `docs` source's `file_patterns`. Its full walk completed `2026-09-11T08:11:58Z` — about **14 hours before** the deletion landed. We indexed a tree that was retired that night. Today the production sitemap still advertises 96 `/ag-ui/` URLs from a stale build; 88 serve 200 and 8 already return 500. As the site rebuilds, those URLs die, and our index keeps handing them out. **No content becomes unsearchable.** `search-ag-ui-docs` already indexes the upstream `ag-ui-protocol/ag-ui` repo (~1,317 chunks) and answers with `docs.ag-ui.com` links, which resolve. Only the link target changes — to the one that will still exist. ## What changed 1. **Removed the `ag-ui` `file_patterns` entry** and the comment block above it that argued for indexing the tree. 2. **Restored both tool descriptions to their pre-#164 text, verbatim.** This is the most important part of the change. `search-docs` claimed to cover "the AG-UI protocol pages hosted on the CopilotKit docs site (docs.copilotkit.ai/ag-ui/...)" — a **false claim**, and it is what an LLM reads to decide which tool to call. It now reads: > Search the CopilotKit product documentation (https://docs.copilotkit.ai) — > guides, concepts, quickstarts, API reference, and how-tos for building with > CopilotKit. Use this for CopilotKit usage and configuration questions. NOT > for AG-UI protocol docs (use search-ag-ui-docs) and NOT for source code > (use search-code). This is a semantic search, so prefer performing multiple > queries with different phrases instead of a single long query, until you > find all the context you need. `search-ag-ui-docs` — which #164 softened into "one of two near-identical copies" — is again described as **the** AG-UI documentation, with no mention of a second copy. 3. **`unclaimed_exempt_paths`: stale comment replaced, no exemption added.** #164's comment said `content/ag-ui/` "is deliberately absent from this list and always will be: it is not exempt, it is CLAIMED". That reasoning is obsolete. The directory is gone, so the `unclaimed_content` audit has nothing to walk there and cannot fire either way; an exemption would only blind the audit to whatever lands at that path next. ### Per-file decision on the rest of #164 `git show 298b7fa --stat` lists three files. | File | Decision | Reasoning | |---|---|---| | `deploy/copilotkit-docs.yaml` | **revert the ag-ui parts** | The subject of this PR. Nothing else #164 touched in it. | | `scripts/test-path-filter.ts` | **KEEP, comment corrected** | #164 changed only comments here. One named "three page subtrees (docs/, reference/, ag-ui/)"; now two. Reverting verbatim would restore a comment that was *already* wrong before #164 — it claimed the source path is `content/docs/`, which #159 had widened to `content/`. | | `src/__tests__/copilotkit-docs-config.test.ts` | **KEEP, assertions updated** | Deploy-config coverage barely existed before this file, and that absence is exactly why both of this config's silent failures — the unindexed reference tree, and now this — stayed invisible. Deleting it would restore the blind spot. Its ag-ui cases are inverted (the tree must **not** be claimed) and a third `describe` block is new: what `search-docs` advertises. Its machine-relay block postdates #164 and is untouched. | ## RED → GREEN Harness calls the **shipped** `walkSourceFiles()` and `deriveUrl()` over the **shipped** `deploy/copilotkit-docs.yaml`. No database, no network, no production. The checkout is CopilotKit **`4b7c0d35fd505e981a03cd7d3cba400234b0eb4b`** — the first parent of the deletion merge, i.e. `main` immediately *before* #7092. That commit is required to demonstrate the before-state at all: at any commit after it the files are simply gone, so a red run there would prove nothing about our config. **RED** (config as on `main`): ``` TOTAL enumerated: 962 content/docs/: 682 content/reference/: 184 content/ag-ui/: 96 [PRESENT] .../content/ag-ui/concepts/agents.mdx -> https://docs.copilotkit.ai/ag-ui/concepts/agents [PRESENT] .../content/ag-ui/concepts/architecture.mdx -> https://docs.copilotkit.ai/ag-ui/concepts/architecture [PRESENT] .../content/ag-ui/agentic-protocols.mdx -> https://docs.copilotkit.ai/ag-ui/agentic-protocols [PRESENT] .../content/ag-ui/sdk/js/core/events.mdx -> https://docs.copilotkit.ai/ag-ui/sdk/js/core/events ``` **GREEN** (this branch, same command, same checkout): ``` TOTAL enumerated: 866 content/docs/: 682 content/reference/: 184 content/ag-ui/: 0 [ ABSENT] .../content/ag-ui/concepts/agents.mdx [ ABSENT] .../content/ag-ui/concepts/architecture.mdx [ ABSENT] .../content/ag-ui/agentic-protocols.mdx [ ABSENT] .../content/ag-ui/sdk/js/core/events.mdx ``` **File-count delta: 962 → 866, exactly −96.** The prose and reference counts do not move. ## Negative check: the existing 866 URLs are byte-identical #159 and #164 both came close to breaking the ordered, first-match-wins `url_derivation.strip_prefix`. Removing a pattern must not disturb it. Full `path → URL` manifests for the `docs/` and `reference/` subtrees, before and after: ``` red rows: 866 green rows: 866 diff exit=0, 0 lines cmp: BYTE-IDENTICAL md5 c88778a79e5b9d4da7e5bf34a0a27cbb (both) docs/quickstart.mdx -> https://docs.copilotkit.ai/quickstart reference/hooks/useAgent.mdx -> https://docs.copilotkit.ai/reference/hooks/useAgent ``` **Mutation-tested, so the empty diff is not vacuous.** Swapping the two `strip_prefix` entries so the shorter `content/` prefix wins first — the exact defect this check exists to catch — produces **1,364 differing rows**, every prose page gaining a spurious `/docs` segment (`/quickstart` → `/docs/quickstart`). The check fires. Restoring the config returns the manifest to byte-identical. The new assertions are mutation-tested the same way: against #164's config, **7 of them fail** (4 ag-ui coverage cases, 3 tool-description cases). ## Local gate `npm run build` ✅ · `npx tsc --noEmit` ✅ · `npm test` **198 files / 3,949 tests passed, 0 failed, 1 skipped** ✅ · `node scripts/check-test-shapes.mjs` ✅ (no new violations) · `npx prettier --check` on all three touched files ✅ No pre-existing failure to control for — the suite is fully green. ## Deploying this: a reindex IS required **Merging this config change does not remove already-indexed chunks.** The ~866 ag-ui chunks sitting in the `docs` source stay there until a full walk of that source rewrites it. The chunks will keep being returned, with dying `docs.copilotkit.ai/ag-ui/...` links, until then. #160's `config_fingerprint` should force that walk automatically, since `file_patterns` is folded into the fingerprint. Measured on the two configs: ``` 8274a4799a6d331b97b48a4a74a74cc5 docs source @ origin/main 2bf63ca62d74ba10ad4698363f3c5f7b docs source @ this branch ``` The branch value is *exactly* the pre-#164 fingerprint, so the stored value no longer matches and the orchestrator takes the full-walk path. **How an operator confirms it, after deploy and the next reindex:** 1. `/health` — the `docs` source's document count falls from ~962 back toward **~866**, i.e. down by roughly the ag-ui contribution (**~866 chunks across 96 files**). 2. `search-docs` for something only those pages said (e.g. "AG-UI RunAgentInput", "AG-UI Rust SDK subscriber") returns **no** `docs.copilotkit.ai/ag-ui/...` URLs; `search-ag-ui-docs` answers the same query with `docs.ag-ui.com` links. 3. The reindex audit reports no new unclaimed cluster for CopilotKit/CopilotKit. If the count does not move, the fingerprint path did not fire and a manual full reindex of the `docs` source is the fallback.

jpr5 added 2 commits September 11, 2026 00:30

jpr5 merged commit 298b7fa into main Sep 11, 2026
7 checks passed

jpr5 deleted the fix/index-ag-ui-docs-tree branch September 11, 2026 08:06

jpr5 mentioned this pull request Sep 13, 2026

Stop indexing the AG-UI documentation mirror that upstream deleted #169

Merged

Navigation Menu

Uh oh!

Index the 96 AG-UI documentation pages hosted on docs.copilotkit.ai - #164

Index the 96 AG-UI documentation pages hosted on docs.copilotkit.ai#164
jpr5 merged 2 commits into
mainfrom
fix/index-ag-ui-docs-tree

jpr5 commented Sep 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

jpr5 commented Sep 11, 2026

What is wrong

Extend the existing source, not a new one

No change to url_derivation — and proof that none was needed

Red → green (real enumeration + real URL derivation, fresh checkout 7a36707ab, no DB, no production)

URL correctness: 96/96

No regression: the existing 866 URLs are byte-identical

The unclaimed_content finding clears

Regression guard

Deploying this: a reindex is needed, and it should fire by itself

Local gate

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

No change to `url_derivation` — and proof that none was needed

Red → green (real enumeration + real URL derivation, fresh checkout `7a36707ab`, no DB, no production)

The `unclaimed_content` finding clears