fix(vault): authenticate auth-api and agents-api with Kubernetes auth

[ExtraToast]

Symptom

Every request through forward-auth to agents.jorisjonkers.dev returned 500:

"detail": "VaultException: Status 403 Forbidden: 2 errors occurred:"
"instance": "/api/v1/auth/verify"
"exception": "org.springframework.vault.VaultException"

jorisjonkers.dev kept working throughout, which is what made the fault look partial rather than systemic.

Root cause

auth-api and agents-api took their Vault session token from a file the vault-agent injector wrote once at pod init:

• annotations vault.hashicorp.com/agent-inject-token + vault.hashicorp.com/agent-pre-populate-only run the agent as an init container only, with no renewing sidecar
• the container command did export SPRING_CLOUD_VAULT_TOKEN="$(cat /vault/secrets/token)" — a one-time read into an env var
• application.yml left spring.cloud.vault.authentication at its TOKEN default

Vault's kubernetes mount issues a 1 h token (token_ttl=3600, token_max_ttl=0), which the Spring Cloud Vault session manager renewed hourly. Renewal is capped by the mount's max_lease_ttl of 2764800 s — exactly 32 days. Past that cap renewal fails; Spring Vault then discards the token and forces a fresh login, but TOKEN authentication has no login mechanism. The token stayed dead for the remaining life of the pod, and every Vault call returned permission denied / invalid token.

The live auth-api pod started 2026-06-21, so the cap was reached on 2026-07-23. lookup-self on its injected token confirmed the token was rejected while the pod itself stayed Ready — the readiness probe does not touch Vault.

The split between hosts follows from which code path signs through Vault transit:

AuthVerificationController.verify
  -> TokenService.createAgentsAssertionToken
    -> VaultTransitJwtEncoder.encode
      -> SpringVaultTransitClient.sign   <-- Vault 403

Only the agents assertion token is minted through transit, so /api/v1/auth/me (app-ui) kept answering 200 while /api/v1/auth/verify (agents forward-auth) 500d.

Change

Both services now authenticate with KUBERNETES against the kubernetes auth roles they already have, so the session manager can re-login indefinitely from the projected service-account token instead of holding one unrenewable token. The injected token file and the SPRING_CLOUD_VAULT_TOKEN export are removed; the vault-agent still renders the secret env file under its own role.

agents-api is configured through the canonical SPRING_CLOUD_VAULT_* property names, since its application.yml lives in the agents-api repository rather than here.

Validation

Both service accounts were confirmed to authenticate against their roles before the change was written:

POST /v1/auth/kubernetes/login  role=auth-api    -> policies ["auth-api","default"],   lease_duration 3600, renewable true
POST /v1/auth/kubernetes/login  role=agents-api  -> policies ["agents-api","default"], lease_duration 3600, renewable true

• ./gradlew :services:auth-api:test :services:auth-api:integrationTest — pass
• cd platform/tests && node --test — pass
• kubectl kustomize platform/cluster/flux/clusters/production — builds

Restarting auth-api restored service immediately, which corroborates the diagnosis; this change stops the failure from returning 32 days after every rollout.

Follow-up, not in this PR

auth-api's rabbit health indicator is DOWN with ACCESS_REFUSED and did not recover on restart with freshly rendered credentials, so the rabbitmq/creds/app-consumer dynamic role is independently broken. Tracked separately.

[ExtraToast]

Post-merge verification on the live cluster.

The new auth-api pod runs with VAULT_AUTHENTICATION=KUBERNETES, VAULT_KUBERNETES_ROLE=auth-api, SPRING_CLOUD_VAULT_TOKEN unset, and /vault/secrets/ containing only auth-api.env — the injected token file is gone. agents-api and agents-api-ws carry the equivalent SPRING_CLOUD_VAULT_* settings.

Transit signing failures went from 4 on the outgoing pod to 0 on the new one, and the previously 500ing endpoints now answer their correct unauthenticated codes:

https://jorisjonkers.dev/                            200
https://agents.jorisjonkers.dev/                     200
https://agents.jorisjonkers.dev/api/v1/workspaces    302
https://agents.jorisjonkers.dev/api/v1/chat-sessions 302
https://auth.jorisjonkers.dev/api/v1/auth/verify     302

The "follow-up" note in the description is now stale: rabbit came back UP on this rollout, and auth-api's aggregate health is UP with db, redis, vault, mail and rabbit all UP. It was still DOWN after a plain restart earlier in the session, so the recovery correlates with the auth change rather than with the restart — plausibly because the dynamic rabbitmq/creds/app-consumer lease could not be maintained while the Vault session was dead. That link is not proven, so the credential path is worth watching rather than considering closed.

Two 403 permission denied entries remain in the startup log for secret/data/application and secret/data/application/prod. Those are Spring Cloud Vault probing its generic config paths, which the auth-api policy has never granted; they are unrelated to this change and harmless.