Crawlmind

AI-native SEO audits from your terminal or any MCP client. Crawlmind crawls a website, finds technical SEO problems, and goes further than other crawlers: it suggests the internal links you should add, measures page speed per template instead of per page, and shows what changed between two crawls.

What makes it different

1. Internal link suggestions. Local multilingual embeddings find pages that are about the same thing but not linked, and point to the paragraph where a link fits and the words that could become its anchor. They also detect keyword cannibalization.
2. Speed by template. Every crawled page gets TTFB and download timings. Pages are grouped into templates, and Lighthouse (via the PageSpeed Insights API) runs only on one or two representatives of each, so one fix covers thousands of pages.
3. Crawl diffs. Compare two crawls of the same site: new 404s, pages that lost inlinks, TTFB regressions, changed titles and canonicals.

Install

Requires Node.js 22 or newer.

As an MCP server

Claude Code:

claude mcp add crawlmind -- npx -y crawlmind-mcp

Claude Desktop, Cursor and other clients (JSON config):

{
  "mcpServers": {
    "crawlmind": {
      "command": "npx",
      "args": ["-y", "crawlmind-mcp"]
    }
  }
}

crawlmind-mcp is a shortcut for crawlmind mcp; both start the same server over stdio. For internal link suggestions, add the embeddings add-on (see below).

Then ask your AI client:

• "Audit https://example.com and give me the top 10 fixes."
• "Which internal links should I add to my blog?"
• "Which templates on my shop are slow on mobile?"
• "Crawl the site again and tell me what broke since last week."
• "Export an HTML report of the last crawl to ~/Desktop/audit.html."

As a CLI

npx crawlmind --help
# or
npm install -g crawlmind

crawlmind crawl https://example.com          # crawl with live progress, then a summary
crawlmind crawl https://example.com -n 2000  # up to 2,000 pages
crawlmind summary                            # key numbers and top issues of the latest crawl
crawlmind issues --severity critical         # issues with examples
crawlmind issues --check links.broken_internal   # every affected URL of one check
crawlmind page /pricing                      # everything about one URL
crawlmind links /pricing --content-only      # who links here, from the main content
crawlmind top --by underlinked               # important pages that need more links
crawlmind suggest                            # internal links to add (needs the embeddings add-on)
crawlmind suggest --target /pricing          # links that should point to one page
crawlmind cannibalization                    # pages competing for the same topic
crawlmind templates                          # page templates with TTFB per template
crawlmind speed                              # PageSpeed Insights on template representatives
crawlmind diff                               # what changed between the two latest crawls
crawlmind report -o audit.html               # self-contained HTML report (or -f md)
crawlmind sql "SELECT url, ttfb_ms FROM pages ORDER BY ttfb_ms DESC LIMIT 10"
crawlmind crawls                             # past crawls

Every command takes --json for machine-readable output (progress goes to stderr). In CI, crawlmind crawl <url> --fail-on critical exits with code 1 when critical issues are found.

The CLI and the MCP server share one local database, so you can crawl from the terminal and then ask your AI client about the result.

Packages

Package	What it is
crawlmind	Engine, CLI (crawlmind) and MCP server
crawlmind-mcp	Shortcut binary that runs crawlmind mcp

MCP tools

Tool	What it does
start_crawl	Start a background crawl and return its crawl_id
get_crawl_status	Progress of a crawl; wait_seconds waits and streams progress
cancel_crawl	Stop a running crawl, keeping the pages fetched so far
list_crawls	Past and running crawls, filterable by site
get_summary	Key numbers and top issues by severity, with examples and recommendations
get_issues	Issues filtered by severity; with check_id, every affected URL (paginated)
get_page	Everything about one URL: status, redirects, metadata, timings, link metrics
get_links	Inlinks or outlinks of a URL with anchors, rel and placement
get_top_pages	Pages by PageRank, inlinks, depth, TTFB, or underlinked important pages
get_templates	Templates with page counts, TTFB median/p90 and PageSpeed results
measure_templates	Lighthouse (PageSpeed Insights) on 1–2 representatives per template
suggest_links	Internal links to add: source page, paragraph and anchor candidates
find_cannibalization	Page pairs that compete for the same topic
compare_crawls	Diff two crawls: new and fixed errors, lost inlinks, TTFB regressions
export_report	Write a self-contained HTML or Markdown report to a file
query_data	Read-only SQL SELECT on the crawl database, row-limited
ping	Health check

Prompts: seo_audit runs the full audit workflow and ends with a prioritized fix list; internal_linking_plan turns link suggestions into a table of links to add.

Resources: crawlmind://checks/{id} documents each check (why it matters, how to fix), and crawlmind://crawls/{id}/summary gives the summary of a crawl.

Checks

Every issue has the same shape: id, severity (critical, warning, info), title, count, up to five examples and a recommendation.

Area	Checks
Status	4xx and 5xx pages, unreachable pages
Links	broken internal and external links, links to redirects, internal nofollow, generic and empty anchors
Redirects	chains longer than one hop, loops
Sitemap	orphan pages, noindex pages listed, non-200 URLs listed
Canonical	canonical to an error page, to a redirect, to a different page
Content	missing or duplicate titles, meta descriptions and H1, thin content (default < 200 words)
Structure	pages more than 3 clicks deep
Speed	TTFB over 800 ms, HTML over 1 MB, mixed content

Generic anchors ("click here", "read more", "подробнее", "mehr erfahren", …) are listed per language in generic-anchors.ts.

Link graph

Inlinks and outlinks are counted per page and by placement (nav, header, footer, main, other), with links through redirects credited to the final page. Click depth is a breadth-first search from the start URL. Internal PageRank uses damping 0.85, spreads the rank of pages without outlinks evenly, ignores nofollow, and iterates until the total change is below 1e-6; scores are shown 0–100 relative to the strongest page.

Internal link suggestions

Crawlmind embeds every indexable page (title, H1 and the start of the main content) with a local model and looks for pages that are about the same topic as a target but do not link to it yet. For each pair it returns:

• the paragraph of the source page that fits the target best (paragraphs of the main content are embedded separately),
• anchor candidates: phrases from that paragraph that share key words with the target's title and H1, matched by word prefixes so inflected forms count (беговые кроссовки for "Лучшие беговые кроссовки"),
• the similarity and the target's content inlinks and PageRank.

Without a target, the most link-worthy pages come first: rank = similarity × (1 + importance / (1 + content inlinks)), where importance is the larger of the page's relative PageRank and its sitemap priority. Crawlmind never writes the final sentence; your AI client does, from this context. No API keys are involved.

Cannibalization: pairs of indexable pages whose embeddings are at least 0.92 similar and whose title/H1 key words overlap by at least 50%.

The embeddings add-on

The model runs locally with transformers.js. Its ONNX runtime is several hundred megabytes, so it is an optional peer dependency and everything else works without it. To enable link suggestions:

npm install -g crawlmind @huggingface/transformers
# or, without installing:
npx -p crawlmind -p @huggingface/transformers crawlmind suggest

For MCP clients:

claude mcp add crawlmind -- npx -y -p crawlmind-mcp -p @huggingface/transformers crawlmind-mcp

The model is downloaded once (about 120 MB) into ~/.crawlmind/models. Page vectors are stored per crawl; paragraph vectors are computed only for pages that appear in suggestions.

Model choice. Xenova/multilingual-e5-small (8-bit quantized, 384 dimensions) covers about 100 languages, which matters for sites that are not in English, and is strong at retrieval for its size: it is the smallest of the e5 family that clearly beats older multilingual MiniLM models on retrieval benchmarks, while still embedding hundreds of pages per minute on a laptop CPU. Its cosine similarities cluster high, so the default thresholds (0.82 for suggestions, 0.92 for cannibalization) are tuned for it.

Speed by template

Every crawled page gets its time to first byte and download time. Pages are then grouped into templates, and Lighthouse runs only on one or two representatives per template, so a site with thousands of product pages needs a handful of PageSpeed Insights calls instead of thousands:

3,400 pages use template /product/* with mobile LCP 4.2 s (real-user, poor), score 38. Fixing this one template improves all of them.

How pages are grouped:

1. URL pattern. Path segments are compared level by level. Numbers, dates, UUIDs, hex ids and long slugs become *; siblings that differ only in digits (post-1, post-2) become *; below the first level, four or more distinct sibling values become *. First-level sections (/blog, /shop) are never merged by count alone.
2. DOM structure. Each page's skeleton (the set of tag paths below <body>, six levels deep, ignoring text and attributes) is reduced to a 32-value MinHash signature. Within one URL pattern, pages whose estimated Jaccard similarity is at least 0.8 form one template, so /p/* pages with two different layouts become two templates.
3. Representatives are the pages whose TTFB is closest to the template median: typical, not the best or worst case.

PageSpeed Insights runs on demand (measure_templates / crawlmind speed), mobile by default, with a 60 s timeout and retries with backoff. Results (performance score, LCP, CLS, TBT, FCP, page weight and CrUX field data when Google has it) are stored per crawl and reused. Without PSI_API_KEY Crawlmind uses Google's shared keyless quota one request at a time, which is often exhausted; a free API key raises it.

Crawl diffs and reports

Crawls are kept, so any two crawls of a site can be compared (compare_crawls, crawlmind diff): new and fixed 4xx/5xx pages, new and fixed broken links, pages added and removed, the pages that gained or lost the most inlinks, TTFB regressions (at least 200 ms and 50% slower), title and canonical changes, and how each issue count moved. A warning is added when the crawls differ much in size, since "removed" pages may just not have been reached.

Reports (export_report, crawlmind report) contain the summary, every issue with up to ten examples and its fix, under-linked important pages, speed by template, and the strongest and slowest pages. The HTML version is a single file with inline CSS, no scripts, light and dark themes, and works on phones. A report never replaces an existing file unless asked to, and never writes through a symbolic link. MCP clients write only inside ~/.crawlmind/reports unless the server runs with CRAWLMIND_ALLOW_REPORT_PATHS=1, so instructions hidden in a crawled page cannot make the AI client overwrite your files.

Configuration

Variable	Purpose
CRAWLMIND_HOME	Data directory (default ~/.crawlmind)
PSI_API_KEY	PageSpeed Insights API key
CRAWLMIND_ALLOW_PRIVATE_NETWORKS	1 lets MCP clients crawl private and local addresses
CRAWLMIND_LOG_LEVEL	debug, info, warn (default), error or silent; logs go to stderr
CRAWLMIND_ALLOW_REPORT_PATHS	1 lets MCP clients write reports outside ~/.crawlmind/reports

Crawling rules

• Identifies itself as CrawlmindBot/<version> (+https://github.com/Everhard/crawlmind).
• Follows robots.txt, including Crawl-delay (capped at 10 s), unless told not to.
• Defaults: 500 pages (hard cap 10,000), depth 10, 4 parallel requests, 250 ms between requests to the same host, 15 s timeout, 5 MB per page.
• Only text/html is parsed. Sitemaps (from robots.txt and /sitemap.xml, indexes and gzip included) add URLs that links do not reach.
• Refuses private, loopback and link-local addresses, checked on every connection and redirect hop (DNS rebinding included). For intranet or local sites use --allow-private in the CLI. MCP clients may only set allowPrivateNetworks when the server runs with CRAWLMIND_ALLOW_PRIVATE_NETWORKS=1, so a prompt injection on a crawled page cannot turn the AI client into a local network scanner.

Limits

• No JavaScript rendering: pages are analyzed as the server sends them. Sites that build their content in the browser show up nearly empty.
• One site per crawl (optionally with subdomains), up to 10,000 pages.
• Crawlmind only reads. No tool changes the audited site.

Privacy

Everything stays on your machine, in ~/.crawlmind (override with CRAWLMIND_HOME). Crawlmind contacts only the site you crawl, the PageSpeed Insights API when you measure templates, and Hugging Face once to download the embedding model when you first use link suggestions. No telemetry.

Development

npm install
npm run build
npm run lint && npm run typecheck && npm test

Try the MCP server in the official inspector:

npx @modelcontextprotocol/inspector node packages/crawlmind/dist/cli.js mcp

Layout: packages/crawlmind/src/core is the engine (crawler, storage, analysis, speed, embeddings, report, diff) and never imports the MCP SDK or CLI libraries; src/mcp and src/cli are thin layers over it.

See CONTRIBUTING.md.

License

MIT © 2026 Andrew Dorokhov