Skip to content

scheduledauth: decide JWKS warmup fail-open vs fail-closed (currently fail-open on misconfig) #67

Closed

Description

@cristim

Context

Spun out of the adversarial review of PR LeanerCloud/cloud-commitments-cli#1231 (SEC-04, hardened HTTP client for JWKS warmup and fetch). The PR keeps Warmup non-fatal on failure with the rationale "Google's CDN sometimes hiccups and we prefer the validator come up with a stale-on-error fetch on the first real request rather than crashloop the container." Filed for explicit team discussion because the global no-silent-fallbacks rule (feedback_no_silent_fallbacks.md) prefers fail-loud at the earliest checkpoint where the consumer can react.

Current behavior

internal/server/scheduledauth/validator.go (PR LeanerCloud/cloud-commitments-cli#1231):

resp, err := v.httpClient.Do(req)
if err != nil {
    log.Printf("scheduledauth: WARN — JWKS warmup fetch failed for %s: %v "+
        "(validator will retry on first request)", v.jwksURL, err)
    return
}

A misconfigured SCHEDULED_TASK_OIDC_JWKS_URL (typo, blocked egress, expired DNS, scheme mismatch) starts the validator cleanly. Every request to /api/scheduled/* then fails at JWT verification time, returning 401 Unauthorized to the legitimate Cloud Scheduler call. The misconfiguration only surfaces in serving logs, not in the deploy pipeline.

Decision needed

Pick one explicitly and document it in the validator comment:

  • A: keep current behavior (warmup never fatal), make sure the WARN line gets a metric/alert so a CDN-vs-misconfig hiccup is at least observable from the dashboard.
  • B: turn warmup into a fail-closed startup probe and pair with a small retry budget (e.g. 3 attempts, exponential backoff, total cap of 60s) before exiting non-zero. The retry budget absorbs CDN hiccups; persistent failure stops the deploy.

Option B aligns with the no-silent-fallbacks rule and the global "fail-closed at the boundary" pattern; option A is the choice the PR has documented. Whichever the team picks, lock the comment to that decision rather than leaving the behavior implicit.

Closing-issue trailer for the fix PR

Closes #<this issue number>

Activity

  1. cristim commented on Nov 9, 2026

    @cristim
    MemberAuthor

    Claimed by claude-ui-fix-1 (fleet). Premise re-checked on main: Warmup is still best-effort (internal/server/app.go:494, scheduledauth/validator.go:283). Proposed default (A+, awaiting the owner's override): keep warmup non-fatal for serving, expose a scheduled_auth_jwks check in /health that /ready ignores like migrations, and have the post-deploy gate fail the deploy if it stays degraded, so a persistent misconfiguration blocks the rollout while a CDN hiccup does not. Not B: the warmup runs in the shared constructor on every platform, a 60s retry does not fit a Lambda init budget, and a failure would take the whole API down for a path that only serves /api/scheduled/*. Plan gets two cold review passes before any code; no deploy or apply.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions