🎃 Generate llms.txt and llms-full.txt from the repo docs so AI agents can read them

Part of the October Challenge #997.

Why

AI assistants and coding agents increasingly look for an llms.txt file (the llmstxt.org proposal): a short Markdown index of a project's documentation with links, plus an optional llms-full.txt with the full text in one file. An agent asked "how do I add a SOAP connector to AnythingMCP?" can then read the right page instead of guessing. This repository has about 40 Markdown documents under docs/ (connectors, client integrations, deployment, operations, API reference, the tool definition format) but no llms.txt. The anythingmcp.com/llms.txt linked from README.md:268 belongs to the website and covers the site and its guides, not these docs. AGENTS.md tells coding agents how to work on the code; llms.txt is about using and operating the product.

Self-hosted vs Cloud

Self-hosted: no runtime change. The two files live in the repository and are read from GitHub. They are not copied into the Docker image (the runner stage copies only build output and start.sh, Dockerfile:127-146) and instances do not serve them.
Cloud (DEPLOYMENT_MODE=cloud): no change. Do not put the files into packages/frontend/public/: proxy.ts:42 lets every path containing a dot through without sign-in, so they would be served by every self-hosted instance and by Cloud, which is not the goal.
Who can use it: anyone reading the public repository. Nothing is per user or per organization.

What to build

Generator scripts/generate-llms-txt.mjs: plain Node, no new dependency, in the style of scripts/adapter-count.mjs and scripts/regenerate-catalog.mjs. It writes llms.txt and llms-full.txt at the repository root. With --check it exits 1 when the committed files differ from what it would write, like adapter-count.mjs --check.
llms.txt in the llmstxt.org format: # AnythingMCP, a one-paragraph > summary (reuse the description of the root package.json), then ## sections of - [Title](url): description lines:
- Getting started: README.md, docs/deployment.md, docs/guides.md.
- Connectors: docs/connectors/*.md, docs/tool-definition.md.
- AI clients: docs/integrations/*.md.
- Operations and security: docs/operations/*.md, docs/sso.md, docs/scim-entra-setup.md, docs/license-faq.md.
- Reference: docs/api-reference.md, docs/mcp-resources.md, docs/knowledge-graph.md.
- Contributing: .github/CONTRIBUTING.md, AGENTS.md.
- ## Optional: docs/guides/*.md, docs/ROADMAP.md, docs/ADOPTERS.md.
Titles come from each file's first # heading, descriptions from its first paragraph (first sentence, about 160 characters at most), with an override map in the script for files whose first paragraph is not a good summary. Links are absolute raw URLs on main (https://raw.githubusercontent.com/HelpCode-ai/anythingmcp/main/<path>), which agents can fetch as plain Markdown.
llms-full.txt: the same files in the same order, each introduced by a separator heading with its path and source URL. Rewrite relative links ((connectors/rest.md), (../README.md#…)) to absolute GitHub URLs, and drop images and HTML-only blocks. For README.md:1-35, keep the title and the summary sentence and drop the banner, the star-history <picture> and the badge rows. docs/ is about 300 KB today, which is fine for one file.
Deterministic output: sorted file lists, no timestamps, LF line endings, so --check is stable.
CI: one step in the backend job of .github/workflows/ci.yml, next to "Verify the quoted adapter count and license wording" (:57-58): node scripts/generate-llms-txt.mjs --check. Add scripts/generate-llms-txt.test.mjs to the node --test line that already runs scripts/validate-adapters.test.mjs (:51-52). Changes under .github/ get extra review, so keep the workflow diff to these lines.
Pointers: next to the website line in README.md:268, add one for the repository's own llms.txt, and the same in README.de.md, README.ja.md and README.zh-CN.md (:286). In .github/CONTRIBUTING.md and AGENTS.md, add one line: after changing docs, run node scripts/generate-llms-txt.mjs and commit the result.

Where to look

docs/ and docs/guides.md (the current human-written index).
scripts/adapter-count.mjs: --check pattern and repository-root resolution.
.github/workflows/ci.yml:48-75: existing generated-file checks.
README.md:1-35 (HTML header), :268 (website llms.txt link).
packages/frontend/src/proxy.ts:42: why the files must not go into public/.

Acceptance criteria

node scripts/generate-llms-txt.mjs writes both files; running it twice produces no diff.
Every link in llms.txt points to a file that exists in the repository (checked by the test).
llms-full.txt contains no relative Markdown links and no <picture> or <img> HTML (checked by the test).
--check fails after a doc is edited and passes after regenerating (checked by the test).
node --test scripts/generate-llms-txt.test.mjs passes and the CI step is green.
README lines and contributor notes added.

Out of scope

The website's anythingmcp.com/llms.txt, which is maintained with the website.
content/guides/ (website guides in several languages, synced by scripts/sync-guides-to-website.sh) and the translated READMEs.
Serving the files from the app, or exposing the docs as an MCP resource.

Size

M (1–2 days)

How to claim

Comment "I'd like to work on this" and we'll assign you. Rules in #997.

Strong agree that llms.txt here is about using/operating the product while AGENTS.md is about building it — keeping those audiences separate is the right call. A few specifics:

The biggest failure mode is drift: a hand-written llms.txt rots the moment a connector is added. Generate both files in CI from the same source of truth that already drives the docs (your tool definition format / connector metadata), and fail the build if llms.txt is stale. That makes it a render target rather than a 40th markdown file someone has to remember.
For llms-full.txt across ~40 docs, emit deterministic ordering and a stable per-section anchor (connector name to slug). Agents quote these links back to users, and stable anchors survive reorders.
Split "how to call a connector" (request shape, auth, idempotency notes) from "how to deploy / operate the gateway." An agent wiring a tool at runtime only needs the former; bundling ops content dilutes retrieval.
Mark which connectors are read-only vs have write side effects right in the index. Agents, and the people trusting them, need that before auto-executing, and it is nearly free if it already lives in metadata.

— from experience generating the same index for a large connector catalog (agentfriendly.network · agentuniver.com)

I'd like to work on this.

@Coding-Professional

@Coding-Professional happy to give you this one, but #1009 is still yours with part 2 (tools call and --json) to go, and the challenge is one issue at a time. Part 1 (#1024) is merged now, thanks for that. If you'd rather do llms.txt next and leave part 2 for someone else, just tell me and I'll put #1009 back in the pool and assign this one to you.

@Coding-Professional

@Coding-Professional #1009 is done, so this one is yours if you still want it. Two pointers: generate both files from the docs in the repo with a script (no hand-written index), and add a CI check that fails when they're stale, the same way adapter-count.mjs --check works for the counts. Per the challenge rules, a PR within 7 days, or tell me here if you need longer.

fix(docs): preserve FAQ headings in generated text (HelpCode-ai#1011)

Thanks for assigning this to me and for the pointers. I've opened #1199 with a script that generates both files from the repository docs, plus a CI freshness check that fails when the committed output is stale. The PR includes regression tests and updated contributor instructions, and is ready for review.

fix(docs): render HTML blocks and escape titles safely (HelpCode-ai#1011

fix(docs): preserve code examples during generation (HelpCode-ai#1011)

docs: refresh generated observability reference (HelpCode-ai#1011)

Merge upstream main for HelpCode-ai#1011

fix(docs): resolve llms generation review gaps (HelpCode-ai#1011)

Merge upstream main for HelpCode-ai#1011

fix(docs): reject symlinks in llms generation (HelpCode-ai#1011)

Merge upstream main for HelpCode-ai#1011

docs: refresh generated references after main sync (HelpCode-ai#1011)