docs: generate version-scoped llms.txt artifacts#2188
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
oss-maintainer
left a comment
There was a problem hiding this comment.
Summary
Documentation update — generates version-scoped llms.txt artifacts for better LLM context management.
Findings
- [Info] Adds 287 lines of documentation across 6 files. Structure looks good.
Verdict
LGTM.
Automated review by github-manager
AgentScopeJavaBot
left a comment
There was a problem hiding this comment.
🤖 AI Review
This PR replaces root-level llms.txt/llms-full.txt with version-scoped outputs (/v1/llms.txt, /v2/llms.txt, etc.) to prevent mixing v1 and v2 documentation in a single LLM index file. The implementation is a new Sphinx build-finished extension (versioned_llms.py) that parses _toc.yml and generates per-version artifacts, removing the third-party sphinx-llms-txt dependency.
The overall architecture is clean and consistent with existing extensions like root_index_notice.py. However, there's one notable gap: the v2 root: page (v2/en/intro) defined in _toc.yml is not parsed by _load_ordered_docs(), which only handles - file: lines. This means the v2 intro page won't appear in v2/llms.txt or v2/llms-full.txt, unlike v1 where the intro is listed as a chapter entry.
Additionally, the text.find("\\n---", 4) in _extract_frontmatter() could match --- within frontmatter values, causing premature truncation. Using python-frontmatter library or a regex-based parser would be more robust.
AgentScopeJavaBot
left a comment
There was a problem hiding this comment.
🤖 AI Review
This PR replaces root-level llms.txt/llms-full.txt with version-scoped outputs (/v1/llms.txt, /v2/llms.txt, etc.) to prevent mixing v1 and v2 documentation in a single LLM index file. The implementation is a new Sphinx build-finished extension (versioned_llms.py) that parses _toc.yml and generates per-version artifacts, removing the third-party sphinx-llms-txt dependency.
The overall architecture is clean and consistent with existing extensions like root_index_notice.py. However, there's one notable gap: the v2 root: page (v2/en/intro) defined in _toc.yml is not parsed by _load_ordered_docs(), which only handles - file: lines. This means the v2 intro page won't appear in v2/llms.txt or v2/llms-full.txt, unlike v1 where the intro is listed as a chapter entry.
Additionally, the text.find("\\n---", 4) in _extract_frontmatter() could match --- within frontmatter values, causing premature truncation. Using python-frontmatter library or a regex-based parser would be more robust.
| docs: list[tuple[str, str]] = [] | ||
| seen: set[str] = set() | ||
| active_index: int | None = None | ||
|
|
There was a problem hiding this comment.
[major] The _toc.yml v2 entry root: v2/en/intro is not parsed by _load_ordered_docs(), which only matches - file: lines. This means the v2 intro page will be missing from v2/llms.txt and v2/llms-full.txt. Consider adding logic to also parse root: lines at the start of the method, or confirm this is intentional and add a comment explaining why.
|
|
||
|
|
||
| def _extract_frontmatter(text: str) -> tuple[dict[str, str], str]: | ||
| if not text.startswith("---\n"): |
There was a problem hiding this comment.
[minor] text.find("\\n---", 4) can match --- substrings within frontmatter values (e.g., a multi-line description containing --- separators), causing premature frontmatter truncation. Consider using a regex like re.search(r'\\n---\\s*\\n', text[4:]) or the python-frontmatter library for more robust parsing.
Summary
Close #2185
This PR replaces the root-level LLM documentation artifacts with version-scoped outputs:
/v1/llms.txt/v1/llms-full.txt/v2/llms.txt/v2/llms-full.txtThe goal is to avoid mixing AgentScope Java 1.x and 2.x documentation in a single
llms-full.txt, so AI coding tools can ingest the correct documentation set for the target version.Changes
build-finishedextension to generate version-scopedllms.txtandllms-full.txtfiles fromdocs/_toc.yml.sphinx_llms_txtintegration to stop generating root-level/llms.txtand/llms-full.txt.v1/enandv2/enpages, excluding blogs and community pages.