Skip to content

feat: generate llms.txt at build time - #1890

Open
jth-nw wants to merge 4 commits into
devfrom
feat/llms-txt
Open

jth-nw wants to merge 4 commits into
devfrom
feat/llms-txt

Conversation

@jth-nw

@jth-nw jth-nw commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Adds an llms-txt Docusaurus plugin (src/plugins/llmsTxt.js) that writes build/llms.txt in postBuild, following the llmstxt.org format.
  • Lists each product's latest version with its description from products.js, plus a Knowledge Base link under Optional.
  • Follows robotsTxt: written on docs.netwrix.com only, and leaves out noIndex and hideFromNavbar products. Uses the full product list, so DOCS_PRODUCT builds don't change it.

Test plan

  • node --test src/plugins/llmsTxt.test.mjs (added to test:scripts)
  • Confirm /llms.txt on the production host after the daily sync, and that Azure serves it as text/plain
  • Confirm /docs/kb/ resolves

🤖 Generated with Claude Code

Add an llms-txt plugin that writes build/llms.txt on docs.netwrix.com,
listing each product's latest version and the Knowledge Base. Unlisted
products (noIndex or hideFromNavbar) are left out.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…iscovery

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
jth-nw and others added 2 commits November 17, 2026 13:44
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@jth-nw
jth-nw deployed to development November 17, 2026 20:03 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown
Contributor

Code Review

One correctness issue. Everything else looks sound.

1. The Knowledge Base link in llms.txt goes to a 404 (src/plugins/llmsTxt.js:43)

`- [Knowledge Base](${origin}/docs/kb/): Troubleshooting and how-to articles across products`,

The kb docs plugin (src/config/products.js:1134-1149) serves docs/kb at routeBasePath: 'docs/kb'. But docs/kb/ has no root page: no index.md or intro.md, no file with slug: /, and nothing in src/pages. Its only contents are per-product folders (1secure/, accessanalyzer/, …) and CLAUDE.md. So https://docs.netwrix.com/docs/kb/ returns a 404.

The broken-link checker won't catch this. The file is written in postBuild, after link validation, as a plain string. Either drop the entry, point it at a route that exists, or add a KB landing page. The test (llmsTxt.test.mjs:50) asserts the current URL, so it needs the same change.

Checked, no issues

  • latestRoute uses the same logic as the plugin generator and navbar (customRoutePath || generateRouteBasePath(...), plus getDefaultVersion), so product links match the real routes, including identitymanager/passwordsecure customRoutePath and current versions. A defaultVersion that doesn't match any version returns null and the product is skipped instead of crashing the build.
  • Product filter: It uses PRODUCTS, so devOnly products are already removed on production builds. noIndex and hideFromNavbar products are left out, which matches robots.txt.
  • Host gating: isIndexableSite means only the docs.netwrix.com (main) build writes the file. Dev, local, and DOCS_PRODUCT builds skip it, so the scoped product list can't produce a partial file.
  • products.js: Only description strings changed (plus the custsomer typo fix). No IDs, paths, or versions changed, so routing, plugins, and sidebars aren't affected. The descriptions are only read by the homepage cards and category data, where longer text is fine.
  • package.json: The new test is appended to the test:scripts chain correctly. Note that I couldn't run the tests in this environment.
  • No security concerns: no secrets, eval, or user-controlled input. The output is built from static config.

This branch was successfully deployed

1 active deployment
development — 044b61ad Deployed Nov 17, 2026 by jth-nw via build-and-deploy #5261
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant