[FR] Add /api/health/ready endpoint with dependency health checks (Postgres, Redis, Zoekt)

[Harsh23Kashyap]

Problem

GET /api/health currently returns a hardcoded { "status": "ok" } regardless of whether Sourcebot's actual dependencies (Postgres, Redis, Zoekt) are reachable. Self-hosted operators using Kubernetes, Docker Compose with a healthcheck, an external load balancer, or a monitoring system cannot distinguish a healthy Sourcebot from one whose database connection has dropped, whose Redis is unreachable, or whose Zoekt indexer is dead.

The previous FR for this (#727) was closed because the endpoint already existed, but the original ask was for a "structured response with details on its inner components' health status" — which was never delivered. PR #728 was opened to document the existing endpoint and closed without merging.

Current behavior

$ curl -sS https://sourcebot.example.com/api/health
{"status":"ok"}

This is a 12-line stub in packages/web/src/app/api/(server)/health/route.ts. It runs on every orchestrator probe interval, logs a debug line, and returns the same payload whether or not Postgres, Redis, or Zoekt are reachable. A pod that has lost its database connection will still report "ok" to a Kubernetes readiness probe, and traffic will continue to be routed to a broken instance.

Expected behavior

A new GET /api/health/ready endpoint that:

• Checks Postgres via __unsafePrisma.$queryRaw\SELECT 1``
• Checks Redis via redis.ping()
• Checks Zoekt via a lightweight gRPC call (a List request with q: { and: { children: [] } } and a small opts: { max_wall_time: 1s })
• Runs the three checks in parallel with a per-check timeout (default 2s)
• Returns 200 with {"status":"ok","checks":{...}} if all pass
• Returns 503 with {"status":"degraded","checks":{...}} if any fail; per-check payload includes {status, latencyMs, error?}

The existing GET /api/health stays exactly as it is (liveness probe) — a process-up signal that does not depend on any external service.

The two endpoints give operators the standard Kubernetes liveness/readiness pair out of the box.

Proposed solution

Add packages/web/src/app/api/(server)/health/ready/route.ts:

import { apiHandler } from "@/lib/apiHandler";
import { __unsafePrisma } from "@/prisma";
import { getRedisClient } from "@/lib/redis";
import { createLogger } from "@sourcebot/shared";
import * as grpc from "@grpc/grpc-js";
import * as protoLoader from "@grpc/proto-loader";
import path from "path";
import { env } from "@sourcebot/shared";

const READINESS_TIMEOUT_MS = 2000;
const logger = createLogger("health-ready");

type CheckResult = { status: "ok" | "error"; latencyMs: number; error?: string };
type ReadinessResponse = { status: "ok" | "degraded"; checks: Record<string, CheckResult> };

// Each check returns a CheckResult after at most READINESS_TIMEOUT_MS.
async function checkPostgres(): Promise<CheckResult> { ... }
async function checkRedis(): Promise<CheckResult> { ... }
async function checkZoekt(): Promise<CheckResult> { ... }

export const GET = apiHandler(async () => {
    const [postgres, redis, zoekt] = await Promise.all([
        checkPostgres(), checkRedis(), checkZoekt(),
    ]);
    const checks = { postgres, redis, zoekt };
    const healthy = Object.values(checks).every(c => c.status === "ok");
    const body: ReadinessResponse = {
        status: healthy ? "ok" : "degraded",
        checks,
    };
    return Response.json(body, { status: healthy ? 200 : 503 });
}, { track: false });

Tests: packages/web/src/app/api/(server)/health/ready/route.test.ts covering the healthy / degraded paths with mocked prisma + redis + gRPC client.

Docs: a new docs/docs/api-reference/health.mdx describing both endpoints and the liveness-vs-readiness split, with example probes for Docker Compose and Kubernetes. Add it to the API Reference group in docs/docs.json.

Alternatives considered

• A single /api/health that does both. Rejected: makes Kubernetes-style splitting impossible, and a liveness probe that touches the database can cause cascading restarts when the database is briefly unreachable.
• Prometheus metrics-only (the spirit of FR: Add Prometheus support #119**).** Rejected for this PR: not actionable from an orchestrator probe. Could be a follow-up — checkZoekt and the timing data are already in the right shape to emit as metrics.
• A WebserverService.Health gRPC method. Not available in the vendored Zoekt proto; would require an upstream change. A List with a 1s wall-time cap is the smallest thing that proves the gRPC channel is alive and the server is responsive.

Scope

• One new route file + one new test file + one new docs file + one line in docs/docs.json
• No schema changes
• No new dependencies
• No changes to the existing /api/health liveness endpoint
• No changes to the existing API key, auth, or entitlement systems — this endpoint is unauthenticated and matches the existing /api/health policy (no user data returned)

Acceptance criteria

• GET /api/health/ready returns 200 with {"status":"ok",...} when all three dependencies are reachable
• GET /api/health/ready returns 503 with {"status":"degraded",...} and the failing check's {status:"error", error:...} when any dependency is unreachable
• All three checks are bounded by a 2s timeout; the worst-case request time is ~2s even when one dependency hangs
• All three checks run in parallel (Promise.all) — total request time is max(timeouts), not sum(timeouts)
• GET /api/health (liveness) is unchanged
• Tests pass for both healthy and each-of-three-degraded permutations
• Docs page describes both endpoints with example Docker Compose healthcheck: and Kubernetes livenessProbe/readinessProbe blocks
• No PostHog tracking on the new endpoint (track: false, matching the existing liveness endpoint)
• A CHANGELOG.md entry under [Unreleased] → Added references this PR

Backward compatibility

Fully backwards compatible. The existing /api/health endpoint is unchanged. The new endpoint is purely additive. Operators who do not use it see no change.

Risks

• A misconfigured /api/health/ready could cause Kubernetes to pull all pods out of rotation if the probe interval collides with a database blip. Mitigated by the per-check timeout and the standard recommendation of a 3-failure failureThreshold on the readiness probe.
• The Zoekt gRPC check calls List with a 1s wall-time cap; under load this could add to Zoekt's request rate. The check runs at most once per probe interval per pod; for the default 10s interval on a 3-pod deployment this is ~0.3 RPS, negligible.
• The getRedisClient() lazy singleton means the first readiness probe on a cold pod will create the Redis connection. The 2s timeout is more than enough headroom for the first connect.

Future work (out of scope for this PR)

• Prometheus metrics for each check (sourcebot_readiness_check_duration_seconds{check="postgres"}, etc.) — would build on the same per-check helpers introduced here.
• A /api/health/ready?strict=true mode that treats Zoekt shards-empty as a degraded state (currently List returns success even with zero repos).
• A /api/health/metrics JSON endpoint that returns Prometheus-style text without exposing /metrics directly.
• Cache the readiness result for ~1s to absorb thundering-herd probes from multiple orchestrators.

---

Happy to take this on. Implementation plan, test plan, and docs update all fit in one focused PR.

[Harsh23Kashyap]

Opened PR #1507: #1507

5 commits on Harsh23Kashyap:feat/health-readiness:

• feat(web): extract lazy Zoekt gRPC client into a shared module (preparatory refactor — moves the dynamic gRPC import out of the route so the readiness endpoint can mock it cleanly and avoids the gRPC + @opentelemetry boot cost)
• feat(web): add /api/health/ready endpoint with dependency health checks (the new route)
• test(web): cover /api/health/ready healthy and degraded paths (6 cases, includes the parallel-execution invariant)
• docs(health): document liveness vs readiness split (new docs/api-reference/health.mdx + docs/docs.json entry)
• chore: changelog entry for /api/health/ready

If the design or naming needs to change, happy to iterate. Will not post a daily nudge on the PR.