Skip to content

perf: Warm docs manifests and deduplicate builds - #1242

Open
sentry[bot] wants to merge 3 commits into
mainfrom
seer/perf/warm-docs-manifests
Open

sentry[bot] wants to merge 3 commits into
mainfrom
seer/perf/warm-docs-manifests

Conversation

@sentry

@sentry sentry Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

This PR addresses an N+1 API call issue occurring on docs pages (/_library/$libraryId/$version/docs/$).

Problem:
When navigating to a docs page, the fetchDocsManifest server function was triggering hundreds of individual HTTP calls to GitHub's content API. This happened because buildDocsManifest (which fetchDocsManifest uses) iterates over every markdown file in a library's docs directory and fetches each one individually to extract frontmatter redirect metadata. While a cache (getCachedDocsArtifact) exists, the GitHub webhook handler (src/routes/api/github/webhook.ts) only marked these artifacts as stale, rather than proactively rebuilding them. Consequently, the first user request after a cache invalidation (or on a cold server start) would incur the full N+1 cost.

Solution:

  1. Proactive Cache Warm-up: A new server-only utility, warmDocsArtifacts (src/utils/docs-warm.server.ts), is introduced. After the GitHub webhook handler marks docs artifacts stale, it now calls warmDocsArtifacts in a background task. This rebuilds both the main docs manifest and the docs path manifest immediately, ensuring the cache is warm before subsequent user requests.
  2. In-flight Deduplication: buildDocsManifest now includes in-flight deduplication. If multiple concurrent requests attempt to build the same manifest (e.g., during a cold server start), only one build process will execute, and all other pending requests will await and share its result. This prevents redundant GitHub API calls in high-concurrency scenarios.

These changes ensure that docs manifests are always served from a warm cache, significantly reducing the number of outbound GitHub API calls and improving page load performance.

Fixes TANSTACK-COM-4D2

This PR was automatically generated by Sentry. You can adjust this setting at any time.

@sentry
sentry Bot requested a review from a team September 10, 2026 11:14
@coderabbitai

coderabbitai Bot commented Sep 10, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: cb05527d-3c70-49a5-bbe8-68554cad065e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 10, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
tanstack-com 4ffd44e Commit Preview URL

Branch Preview URL
Oct 20 2026, 12:26 PM

@changeset-bot

changeset-bot Bot commented Oct 20, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 4ffd44e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@tannerlinsley

Copy link
Copy Markdown
Member

The cache warming is worth keeping, but we can use the cache's existing request deduplication. I pushed a follow-up that also tightens validation of cached manifests.

AI review notes:
Head 4ffd44e9d962a396851f9a8716a258ee64c0b3f2 integrates current main and shares a cast-free manifest validator between normal reads and warming. The original warmer accepted { paths: [4], redirects: null }; the corrected validator rejects it, arrays of redirects, and non-string redirect targets.

The extra manifest promise map is removed. Tests against the real artifact cache show eight concurrent cold requests produce one build, eight requests after invalidation produce one rebuild, a fresh read does not rebuild, and a failed cold build can recover on the next request. Builders and stored data in these tests are fixtures; no production webhook or cache was exercised.

Full pnpm test and normal commit hooks passed: 606 unit tests passed with 4 skipped, 2,270 chat tests passed, and 20 desktop tests passed with 1 skipped, plus types and lint. New-head CI and the required server-code team review are still pending. This is pushed, not merged or deployed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant