[Refactor]: source sitemap lastModified from real timestamps

[martyy-code]

⚠️ Security Consideration

Not a security concern. This refactor only adjusts sitemap metadata used by crawlers.

Current State

apps/web/src/app/sitemap.ts declares lastModified: new Date() for:

• The three static routes (/, /docs, /blog).
• Every doc page in pages.map(...).

Only blog posts use a real timestamp — page.data.date from MDX frontmatter.

Problems with the current implementation:

• Every request to /sitemap.xml claims every URL was last modified at the moment the sitemap was generated. Search engines (notably Google) treat this as an unfresh signal and may crawl more aggressively than necessary.
• The home, /docs, and per-doc lastModified values are noise; they do not reflect when the underlying content actually changed.
• The three static routes have the same value, which makes the sitemap harder to inspect for staleness.

Located in: apps/web/src/app/sitemap.ts

Proposed State

After this change:

• Static routes (/, /docs, /blog) — lastModified is the maximum of: (a) the latest MDX file mtime under apps/web/content/, (b) the build timestamp (last-resort fallback).
• Doc pages — lastModified reads the mtime of the corresponding apps/web/content/docs/{slug}.mdx file (via fs.statSync or equivalent), falling back to the directory's most recent mtime for index/parent pages.
• Blog pages — unchanged; still page.data.date.
• changeFrequency values are reviewed:
  • / → weekly if any content changed in the last 7 days, else monthly.
  • /docs and per-doc pages → monthly (matches actual cadence).
  • /blog and per-post pages → keep weekly and monthly.
• Sitemap is built statically at next build time so the values are stable for a deployment.

Expected improvements:

• Crawl-budget signals are honest; Google won't re-fetch unchanged URLs based on stale lastModified.
• The sitemap documents real publication cadence, useful for SEO tooling audits.
• Per-doc lastModified gives downstream tools a meaningful "updated at" signal.

Motivation

This refactoring is needed because:

• The current implementation makes the sitemap dishonest, which is a low-grade SEO footgun (crawl-budget waste, dilutes real change signals).
• Per-doc timestamps are a useful primitive for future features (changelog page, "last updated" badges).

Triggers for this work:

• Technical debt accumulation
• Maintainability concerns

Risks

Potential risks:

• Risk 1: Reading file mtimes during next build requires the FS to be readable from the build context (correct on Vercel; could surprise a custom CI). Mitigation: gate the FS read behind a try/catch and fall back to new Date() if the stat fails.
• Risk 2: lastModified values changing after deploy could trigger a temporary crawl spike. Mitigation: ship in a low-traffic window; values are still bounded between deploys (static at build time).
• Risk 3: changeFrequency is an advisory hint, not a directive; Google ignores it. Not a risk, just confirming we are not over-relying on it.

Migration Plan

Migration approach:

1. Implement lastModified derivation in a small helper file (e.g. apps/web/src/lib/sitemap-mtimes.ts) so the logic is unit-testable.
2. Wire the helper into sitemap.ts.
3. Build and visually verify /sitemap.xml against git log --format=%ad for a sample of files.
4. Roll back by reverting the helper import; behavior returns to today's new Date() everywhere.

Rollback plan: revert the helper and the sitemap.ts edit. No on-disk state changes.

Backward Compatibility

• This refactoring maintains full backward compatibility

Scope

Files/Folders affected:

• apps/web/src/app/sitemap.ts (primary)
• apps/web/src/lib/ (new helper, optional)

Out of scope:

• Changing the OG image or images: array.
• Touching robots.ts or proxy.ts.

Component(s) Affected

• Multiple Components (specifically apps/web/src/app/sitemap.ts and apps/web/src/lib/)

Priority

• p2: Medium - Normal priority

Estimated Effort

• effort: s - Half a day

Test Coverage Requirements

• Need to add new tests for this refactor

(Fumadocs/sitemap does not currently have a test fixture; a small Vitest under apps/web/ for the helper is sufficient.)

Testing Approach

Testing strategy:

• Unit tests: getContentLastModified(slug) returns expected mtime for a known fixture path.
• Integration: build the site and assert /sitemap.xml includes the expected ISO 8601 timestamps for a sample of URLs.

Verification steps:

1. pnpm --filter web test (if tests are added)
2. pnpm --filter web build then grep lastModified .next/server/app/sitemap.xml.body or equivalent
3. Spot-check three URLs (home, one doc, one blog) against stat output.

Related Issues / Pull Requests

• Sibling issue: [Chore]: disallow /llms-full.txt in robots.ts (same audit batch).
• Sibling issue: [BUG]: JSON-LD assemblyVersion / datePublished / operatingSystem are wrong (same audit batch).
• Sibling issue: [Chore]: metadataBase env name + OG image parity + baseUrl canonical (same audit batch).

Relevant Documentation

• docs/internal/tasks/v1.4.x-seo-cleanup.md — full audit and the batch this refactor belongs to.

Pre-Submission Checklist

• I have searched existing issues for related refactoring requests
• Risks and migration plan are documented
• Test coverage approach is defined
• I understand this issue will be labeled according to the project taxonomy
• This is NOT a security vulnerability (see security note above)