TINYDOC-3613: Generate the documentation corpus with content, dates and agent entry points.

[kemister85]

Ticket: TINYDOC-3613

Site: Staging branch

Changes:

• lib/antora-extension-page-dates.js, antora-playbook.yml: date every page from its source history, applied to the sitemap lastmod, the page attributes and a page date map.
• scripts/generate-markdown.mjs: seven-field frontmatter on every generated page, no raw HTML links, and a versioned _markdown-manifest.json.
• -scripts/generate-llm-files.js, workflows, package.json, -scripts/README-llm-files.md: full page content in llms-full.txt; new AGENTS.md, sitemap.md and changes.json; titles read from the build instead of production; root LLM files written by explicit path, replacing copy-llms-files.sh.

Note: pull request previews build the site only, so the generated agent files are verified with a staging deploy from this branch.

---

How to test locally:

git fetch origin && git switch TINYDOC-3613-corpus-pipeline
yarn install --frozen-lockfile               # passes on this branch; fails on main today
yarn antora ./antora-playbook.yml            # about 30 s
yarn build:markdown build/site               # about 80 s
yarn generate-llm-files build/site           # about 1 s
npx http-server build/site -p 4000

build/site is what production serves at /docs, so the files are at the server root:

• http://localhost:4000/llms.txt: Generated … on line 3, a Version coverage section near the top, and a For Agents list at the end.
• http://localhost:4000/llms-full.txt: several MB (production today: 57 KB); each page has a ## Title heading, then Source: and Last updated: lines, then its content.
• http://localhost:4000/AGENTS.md: the license key section is first; the token budget example uses real numbers.
• http://localhost:4000/sitemap.md: follows the left navigation, with each page's description and date.
• http://localhost:4000/changes.json: newest first, 500 entries.
• http://localhost:4000/tinymce/latest/basic-setup/index.md: seven frontmatter fields; links written as [text](url), never <a href>.
• http://localhost:4000/sitemap.xml: <lastmod> values differ between pages (production today: one value for every page).

Terminal checks against the same build:

ls -la build/site/llms-full.txt                                      # about 3.9 MB
grep -o '<lastmod>[^<]*' build/site/sitemap.xml | sort -u | wc -l    # hundreds, not 1
find build/site -name 'llms*.txt'                                    # only the root and tinymce/latest/_attachments

Notes for reviewers:

• The playbook builds content from GitHub, so unmerged content changes on tinymce/8 (for example TINYDOC-3617: Backfill missing page description and navigation title attributes. #4361) do not appear in this build.
• The ERROR and WARN lines during yarn antora all come from archived tinymce/5 content and appear on main too.
• The first build clones the content history into .cache/ (ignored), which takes about 2 s; later builds reuse it.
• The build stops with an error if a regression appears: a raw link in the markdown, an llms-full.txt under 2 MB, or an llms file outside the root and latest/_attachments.

---

Pre-checks:

• Branch is correctly prefixed:
• Files removed have been deleted, not just excluded from the build (if applicable).
• Build passes without console errors, warnings, or issues.

---

Review:

• Documentation Team Lead has reviewed.

[kemister85]

Decisions recorded for review

Division of labour between the agent-facing files. Each statement belongs in one file; the license requirement is the only deliberate repetition.

File	Reader	Job
ai-coding-agents.adoc	Human	Per-agent configuration and copy-paste snippets
llms.txt	Agent	Index of the documentation content, version coverage, links to the files below
AGENTS.md	Agent	License requirement, version scheme, frontmatter and citation, manifest schema with a retrieval budget example, one pointer per access path
sitemap.md	Both	The navigation tree with each page's description and last updated date (latest only)
changes.json	Agent	The 500 most recently changed pages across all versions, newest first

The Context7 setup instructions previously in llms.txt are replaced by a pointer in AGENTS.md, since per-agent configuration belongs to ai-coding-agents.adoc.

generate-llm-files-from-url is dropped. It fetched the production sitemap and wrote into the source tree; the build now produces the files, so nothing needs a remote mode.

Where this differs from the brief, deliberately

• last_updated is the newest commit date of the page source or any file it includes, so a page whose content lives in partials moves when a partial changes. 98 of the 434 latest pages are later than git log -1 on their own .adoc for this reason; none are earlier.
• md_url is the page's own markdown URL. On TinyMCE 5, 6 and 7 pages Antora points rel=canonical at the TinyMCE 8 page, so canonical plus index.md would have pointed older markdown at the wrong version. canonical_url still mirrors the HTML.
• The raw-link check ignores fenced code and inline code, where HTML links are legitimate examples, and subtracts links a page shows as literal text.
• The sitemap lastmod is applied inside the Antora build, before publishing, so the rename step in the workflow is unchanged.
• The 404 page is no longer converted to markdown; it was the source of the // key in the old manifest.
• Both LLM files now show license_key: 'gpl' in their self-hosted example; without a key the editor is disabled.
• Removing sanitize-html brings yarn.lock back in step with package.json; yarn install --frozen-lockfile passes on this branch.

Build impact: pull requests against tinymce/8 are unaffected. Previews of pull requests against main add about two seconds for the history lookup. Deploys no longer make 433 requests to production.