Repository navigation
Make robots.txt and the sitemap unable to contradict each other - #87
Merged
Merged
Conversation
The file itself was already right: 200, text/plain, Allow: /, and a Sitemap line on the www host after the canonical fix. Nothing needed changing in what it says. What was missing is anything stopping it from going wrong. The two files are generated in different places, robots.txt.ts and sitemap.xml.ts, and nothing compared them. A Disallow that blocks a URL the sitemap advertises is the one SEO mistake that silently removes pages from search while every other check here still passes: the build succeeds, the links resolve, the pages render. check_links.py gains a seventh rule. Every sitemap URL is tested against the wildcard group's Disallow rules, with * expanded, and the Sitemap line must name the origin the sitemap actually uses. Verified by injecting five faults: a plain prefix Disallow (caught 30 URLs), a wildcard Disallow (4), a Sitemap line pointing at a vercel.app host, and no Sitemap line at all. The fifth is the one it must not catch, a Disallow under a named user-agent rather than the wildcard, which is a deliberate choice about one crawler and not this check's business. The generator now says why nothing is disallowed, because the reasoning is not obvious and the natural instinct is wrong. The pages kept out of search are kept out with a noindex tag, which only works if the crawler is allowed to fetch the page and read it. Disallowing the seven unwritten records would mean the tag is never seen and the URLs could still be indexed from links, with no description. /_astro must stay fetchable or Google cannot render anything. Also checked and found clean: the apex host redirects robots.txt to www, trailing-slash URLs 308 to the canonical form, and the .html duplicates that Vercel serves carry a canonical pointing at the clean URL, so they consolidate rather than compete. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The file itself was already correct: 200,
text/plain,Allow: /, and a Sitemap line on the www host after the canonical fix. Nothing needed changing in what it says. What was missing is anything stopping it from going wrong.Why this needed a check
robots.txtandsitemap.xmlare generated in two different files and nothing compared them. ADisallowthat blocks a URL the sitemap advertises is the one SEO mistake that silently removes pages from search while every other check here still passes: the build succeeds, the links resolve, the pages render, the sitemap validates.The rule
Every sitemap URL is tested against the wildcard group's
Disallowrules, with*expanded, and the Sitemap line must name the origin the sitemap actually uses.Verified by injecting five faults:
Disallow: /compounds/Disallow: /*/bpc-157vercel.apphostDisallowunderUser-agent: GPTBotThe last one is the case it must not flag. A rule for a named crawler is a deliberate decision about that crawler, not a contradiction.
Why nothing is disallowed
The generator now explains this, because the reasoning is not obvious and the natural instinct is the wrong one.
The pages this site keeps out of search are kept out with a
noindexmeta tag, and that only works if a crawler is allowed to fetch the page and read it. Disallowing the seven unwritten records would mean the tag is never seen, and the URLs could still be indexed from links pointing at them, with no description./_astromust stay fetchable or Google cannot render the pages at all.Also checked, all clean
robots.txt.htmlduplicates Vercel servesnoindex, followGates
Content validation (59 records), 36 drafter tests, 17 calculator tests, loop gate 47/47, all seven link rules.
🤖 Generated with Claude Code