Repository navigation
Add oap render: a GitOps-ready kustomize overlay for the install - #33
josephschorr wants to merge 1 commit into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
`oap install` brings a cluster up imperatively. `oap render` instead emits a self-contained kustomize overlay into deploy/oap for Argo CD / Flux to reconcile, never mutating the cluster — its only remote call is read-only image-digest resolution, skipped with --no-digest-pin. The overlay is a pure function of its flags, so re-rendering identical inputs is byte-identical and GitOps controllers see no phantom drift. It provisions every internal Secret, with nothing external by default: - A pre-start bootstrap Job (the operator image's `bootstrap-secrets` entrypoint) mints the memory tokens, admind token, passthrough key, NATS identity/TLS/client creds, and the SpiceDB + Postgres tokens — idempotently, before the operator and data plane start. The Job's name carries a hash of its pod template, so an operator-image upgrade yields a new Job object the controller prunes-and-creates rather than an immutable-pod-template apply conflict. - cert-manager (a self-signed Issuer + Certificate + ca-injector annotation on the webhook configuration) provisions the operator's webhook serving cert. All internal-secret minting is consolidated into pkg/platform/bootstrap as the single source; `oap install` delegates to it, and the operator's leader-elected bootstrap runnable is now schema-only. The NATS minting (identity/TLS/creds and the webd publish-surface grant, with its adversarial test) moved verbatim. Escape hatches: --external-spicedb / --external-postgres / --external-webhook-tls for instances you run yourself, and per-component --channelsd/--webd/--authzd- memory-token-secret links. webd's token must resolve to a different Secret than channelsd/authzd (effective-name compare, since the Job mints the defaults regardless of links): webd is browser-facing and its token is read-only, while channelsd/authzd tokens can write the audit log. `oap render --validate` lints the built overlay in-process (every non-optional Secret reference has a creator, AP_CLUSTER_KIND is stamped, first-party images are pinned); `oap check` verifies the memory-token Secrets after a sync. Only --with-postgres is offered among the dependency components — its token is Job-minted, so it works end to end; Graphiti/Neo4j need a user-supplied LLM key the overlay cannot mint, deferred to a follow-up. Guide: site/content/docs/gitops.mdx.
e9c2b80 to
774f28e
Compare
samkim
left a comment
There was a problem hiding this comment.
Summary
oap render writes a kustomize overlay for GitOps (Argo CD / Flux) and adds a pre-start bootstrap Job that mints the internal Secrets, plus a leader-only operator step that applies the SpiceDB schema. The render, Job and mint consolidation are mostly sound, but the GitOps install as shipped cannot come up: no SpiceDB is provisioned, and the new schema step can crash-loop the operator on any restart.
Verdict: needs rework. Two blockers (missing SpiceDB, operator schema write) and two majors need fixing before this is usable for GitOps. The rest can follow as inline fixes.
Issues that matter
- Blocker, operator crash-loop on restart (see inline,
internal/cmd/operator/main.go). On every leader start the operator overwrites the live SpiceDB schema with only the base scaffold, discarding the guardian-composed fragments. SpiceDB rejects this when relationships exist on the removed definitions, the runnable returns the error and the manager stops. This hitsoap installclusters on any operator restart, not only GitOps, and also kills the operator if SpiceDB is not up yet on a cold start. - Blocker, no SpiceDB in the rendered overlay (see inline,
render_overlay.go). The overlay has no SpiceDBCluster instance, nospicebox-spicedb-configConfigMap (the source ofSPICEDB_ENDPOINT) and no database setup.oap installcreates these imperatively. After a GitOps sync the operator, authzd, channelsd and webd have nothing to authorize against, and--validatestill passes. - Major, bootstrap Job cannot reach the API server under the shipped NetworkPolicies (see inline). The Job pod has no
app.kubernetes.io/namelabel, sodefault-denyblocks it. It fails after 6 retries and, because its name is a content hash, a re-sync never recreates it. No Secrets get minted. - Major, default render wires the operator to a Postgres that does not exist (see inline). The Job always mints
spicebox-postgres-token(URI to an in-cluster Postgres), but--with-postgresdefaults to off, so the Service is absent. - Major, operator ClusterRole gains cluster-wide
deployments/patch(see inline). Only the one-shot Job needs it. - Minor:
oap checkdoes not check linked memory-token Secrets; Job name hash misses mint-set changes and failed Jobs; the Job logs nothing about what it minted;--validatehas latent false passes; the memory-token name list is duplicated; stale comment inmint.go. - Possible, needs a check: the cert-manager CA injection leaves
caBundle: ""in the base, which Argo/Flux may keep resetting (see inline).
Questions for the author
- How is SpiceDB supposed to be provisioned on the GitOps path? Is the intent that the overlay ships a SpiceDBCluster, its endpoint ConfigMap and database setup, or that users bring their own (in which case
--external-spicedbshould also cover the endpoint ConfigMap and the docs should say so)? This decides how the SpiceDB-related fixes are shaped, including what Postgres the minted SpiceDBdatastore_urishould point to. - Which ServiceAccount should the bootstrap Job use? Options are a dedicated SA with a namespaced Role (so the operator ClusterRole does not gain
deployments/patch), or dropping the Deployment roll from the Job path. Both are viable and depend on whether the Job should ever re-mint NATS credentials on upgrade. - What ordering mechanism do you want between the Job, the operator and SpiceDB under Argo (PreSync or sync-waves) and Flux (
dependsOn)? Today it is prose only ("eventual convergence"), and the cold-start path is the same one that makes the operator exit in issue 1. - e2e and the bronze/steel threads were deferred. The ship gate in CLAUDE.md requires all three suites; is a run of
mage test:integrationandmage test:e2eplanned before merge, especially for theoap installdelegation and the operator restart path?
Not anchored to a diff line
cmd/oap/internal/installcmd/render.go:157(validateRenderFlags):--with-postgrestogether with--external-postgresis neither refused nor defined, and--external-spicedbdoes not produce the endpoint ConfigMap the consumers read.cmd/oap/internal/installcmd/install.go:2144(operatorMemoryTokenSecrets) andinit.go:1070,1079,1090repeat the names inbootstrap.MemoryTokenSecretNames(inline comment ontoken.go:31).
This review was drafted by an automated pipeline and checked against the code before submission.
| // by this point (construction failure exits above), so SchemaIOFor cannot | ||
| // produce a typed-nil interface. | ||
| if err := mgr.Add(manager.RunnableFunc(func(ctx context.Context) error { | ||
| return bootstrap.RunBootstrap(ctx, spicedb.SchemaIOFor(spiceDBClient), spicedbschema.Schema) |
There was a problem hiding this comment.
[blocker] Leader runnable overwrites the guardian-composed SpiceDB schema with the base scaffold and crash-loops the operator on error
Problem: bootstrap.RunBootstrap (pkg/platform/bootstrap/runnable.go:23-28) calls WriteSchema(spicedbschema.Schema) on every leader acquisition. WriteSchema replaces the live schema wholesale (pkg/authz/guardian/schema/composer.go:424-428: "text must always be scaffold + every fragment"); spicedbschema.Schema is only the scaffold, so the MCPServer/SidecarToolbox/channel-kind fragments and grant_* relations the guardian composed are dropped. If any relationship exists on a removed definition or relation, SpiceDB refuses the write, the RunnableFunc returns the error, controller-runtime stops the manager (mgr.Start at line 3378 returns) and the operator crash-loops on every restart. The same error path kills the operator on a GitOps cold start when SpiceDB is not reachable yet. Main had no startup schema write, so this regresses oap install clusters too.
Fix: Do not write the bare scaffold at startup. Either (a) only write when the live schema is empty or lacks the scaffold (read via SchemaIOAdapter.ReadSchema, pkg/authz/spicedb/client.go:1030, and skip when any definition exists), or (b) trigger the guardian compose (guardianschema.RunAll via the existing reconciler, main.go:2967-2975) instead of writing spicedbschema.Schema directly so the text is scaffold plus every fragment. In either case, do not return an error from the RunnableFunc for a transient SpiceDB failure: retry with backoff inside the runnable and log at INFO with context, per CLAUDE.md "Never silently drop errors". Choose (a) or (b) based on whether the guardian already runs on startup for a cluster with no AgentSessionGrants; check that first.
Test: In pkg/platform/bootstrap/runnable_test.go add cases using a fake SchemaWriter/reader where the live schema already contains a composed fragment: assert it is not replaced, and that a SpiceDB error is retried, not returned. Add an integration test against test/testspicedb that writes a composed schema plus a relationship on a fragment definition, runs the runnable, and asserts the schema and relationship survive. Run go test ./pkg/platform/bootstrap/ and mage test:integration.
Constraints: CLAUDE.md "First-boot-only initialization never belongs in a serving container" (a once-per-cluster seed must be safe on every boot) and "Never silently drop errors".
| // deterministically. Components and additional patches are appended by later | ||
| // render stages; order is fixed so re-renders are byte-identical. | ||
| func renderKustomization(cfg RenderConfig) ([]byte, error) { | ||
| resources := "resources:\n - base/install.yaml\n - base/channelsd.yaml\n - base/nats.yaml\n - " + bootstrapJobFile + "\n" |
There was a problem hiding this comment.
[blocker] Rendered overlay provisions no SpiceDB: no SpiceDBCluster, no spicebox-spicedb-config, no database setup
Problem: renderKustomization lists only base/install.yaml, base/channelsd.yaml, base/nats.yaml, the bootstrap Job, and optional routing/webhook-cert. base/install.yaml contains only the SpiceDBCluster CRD (pkg/platform/manifests/install.yaml:22658), not an instance. oap install creates the instance, the spicebox-spicedb-config ConfigMap (install.go:2794, source of SPICEDB_ENDPOINT, read as optional: true at install.yaml:30551) and the database imperatively. After a GitOps sync of oap render --cluster-kind gke --with-postgres no SpiceDB pods exist, the operator cannot connect, and authzd/channelsd/webd cannot authorize. --validate only checks Secrets, so it passes.
Fix: This depends on a design decision (see question 1 in the review body): either emit the SpiceDBCluster, the endpoint ConfigMap and the database setup as overlay resources (reuse buildSpiceDBClusterDoc and the ConfigMap builder used by oap install in spicedb_cluster.go / install.go:2794, do not copy them), or make the external-SpiceDB path explicit: have --external-spicedb require/emit the endpoint ConfigMap inputs and document it. Whichever is chosen, extend runRenderValidate (render_validate.go) to fail when no SpiceDBCluster and no external endpoint is present. Do not pick a direction without the author's answer.
Test: Add a case in cmd/oap/internal/installcmd/render_validate_test.go that renders a default overlay and asserts --validate fails (or succeeds) according to whether a SpiceDB instance/endpoint is present, and extend the golden in render_golden_test.go. Run go test ./cmd/oap/internal/installcmd/ -run 'Render'.
Constraints: CLAUDE.md "Cluster kinds" and "First-boot-only initialization" (SpiceDB bootstrap files are fatal on a non-empty datastore; do not set datastoreBootstrapFiles for persistent kinds).
| BackoffLimit: &backoff, | ||
| Template: corev1.PodTemplateSpec{ | ||
| Spec: corev1.PodSpec{ | ||
| ServiceAccountName: "spicebox-operator", |
There was a problem hiding this comment.
[major] Bootstrap Job pod has no component label and is blocked by default-deny, so it cannot reach the API server
Problem: buildBootstrapJob sets no pod labels. default-deny selects every pod for Ingress and Egress (pkg/platform/manifests/install.yaml:31375), and the 443/6443 API-server egress rule in egress-external applies only to pods whose app.kubernetes.io/name is channelsd, operator, authzd or webd (install.yaml:31471-31486). On a CNI that enforces NetworkPolicy the Job cannot create Secrets, fails after backoffLimit 6, and its content-hashed name means a re-sync never recreates it. No Secret is ever minted.
Fix: Give the Job's pod template its own label (for example app.kubernetes.io/name: spicebox-bootstrap) and add that value to the egress-external pod selector in the NetworkPolicy source under config/networkpolicy/, then run mage manifests. Do NOT reuse app.kubernetes.io/name: spicebox-operator: the operator Services select on it (config/manager/service.yaml:11,31) and the Job pod would join their endpoints and operator ingress rules.
Test: In render_test.go add a case that builds the overlay (buildOverlay), finds the Job, and asserts its pod template labels match a pod selector in egress-external that allows port 443. Run go test ./cmd/oap/internal/installcmd/ -run 'Render' and go test ./pkg/platform/manifests/ (TestInstallYAMLMatchesKustomize).
Constraints: CLAUDE.md "Manifest regeneration": never edit install.yaml by hand; run mage manifests after editing config/.
| return err | ||
| } | ||
| if !opts.SkipPostgresToken { | ||
| if err := EnsurePostgresToken(ctx, reader, writer); err != nil { |
There was a problem hiding this comment.
[major] Job always mints a Postgres token pointing at an in-cluster Postgres that the default overlay does not ship
Problem: MintAllInternalSecrets mints spicebox-postgres-token (with uri to spicebox-postgres.agentprimitives-system.svc:5432, mint.go:145-150) unless SkipPostgresToken. bootstrapJobArgs (render_bootstrap_job.go:25-35) passes --skip-postgres-token only for --external-postgres, while the Postgres component is added only with --with-postgres, which defaults to false (render.go:72). The operator reads POSTGRES_URI from that Secret as optional: true (config/manager/deployment.yaml:84-89), so oap render --cluster-kind gke gives an operator configured for a Postgres backend whose Service does not exist. --validate counts the token as satisfied (render_validate.go:107-124).
Fix: In bootstrapJobArgs, append --skip-postgres-token unless cfg.WithPostgres is set (also when ExternalPostgres), and mirror that in jobMintedSecretNames (render_validate.go:107) so validation matches. Also make validateRenderFlags (render.go:157) refuse or define --with-postgres together with --external-postgres. Note the SpiceDB datastore_uri (mint.go:65-75) is only minted after the Postgres token, so settle the SpiceDB provisioning question in the review body first; this change must not leave SpiceDB without a datastore URI.
Test: In render_bootstrap_job_test.go (or render_test.go) add a table over {WithPostgres, ExternalPostgres} asserting the Job args and the jobMintedSecretNames set, plus a refused combination. Run go test ./cmd/oap/internal/installcmd/ -run 'Render|Bootstrap'.
| // webhookCAInjectPatch annotates every ValidatingWebhookConfiguration so | ||
| // cert-manager's ca-injector writes the Certificate's CA into each webhook's | ||
| // caBundle. | ||
| func webhookCAInjectPatch() []byte { |
There was a problem hiding this comment.
[minor] CA injection leaves caBundle: "" in the base webhook configurations (needs verification)
Problem: webhookCAInjectPatch only adds the cert-manager.io/inject-ca-from annotation. The base ValidatingWebhookConfigurations spicebox-settings and spicebox-goal-execution still carry literal caBundle: "" (install.yaml:31763-31986). Under Argo CD self-heal or Flux forced server-side apply, the GitOps controller owns that field and the cainjector writes it too, so the two may keep overwriting each other, leaving the app OutOfSync and webhook admission failing TLS in between. Not reproduced here (no Argo/Flux available).
Verify: Run oap render --cluster-kind gke --no-digest-pin --output-dir <tmp> then kustomize build <tmp> and confirm caBundle: "" is present in the output for both VWCs. If yes, fix.
Fix: Add a JSON6902 op: remove patch on /webhooks/<i>/clientConfig/caBundle (one per webhook) to the generated patches/webhook-ca-inject.yaml, driven from the same webhookVWCNames list so a new VWC is covered, and/or document ignoreDifferences for caBundle in site/content/docs/gitops.mdx. Prefer removing the field in the render so there is no tool-specific config for users.
Test: In render_test.go add a case that builds the overlay and asserts no clientConfig.caBundle is set on the webhooks while the inject annotation is present. Run go test ./cmd/oap/internal/installcmd/ -run 'Render'.
| return "", err | ||
| } | ||
| h := sha256.New() | ||
| h.Write(img) |
There was a problem hiding this comment.
[minor] Job name hashes only the image ref and args, so a mint-set change or a failed Job is never re-run
Problem: bootstrapJobName hashes the substituted operator image ref and the entrypoint args. When the operator image ref is unchanged (a mutable tag with --no-digest-pin), an upgrade that adds a NATS grant subject or a new internal Secret keeps the same Job name, so the completed Job never re-runs and the new Secret or grant is never minted. A backoff-exhausted Job also keeps its name and is never recreated by a re-sync. (With default digest pinning a new operator image changes the name, so that path works.)
Fix: Add a deterministic fingerprint of what the Job mints to the hash input: export a function from pkg/platform/bootstrap (for example MintSetFingerprint() over MemoryTokenSecretNames, AdmindTokenSecretName, the NATS secret names and grant subjects in nats.go) and write it after the args at line ~56. Keep the name a pure function of its inputs (no time, no randomness). Document in gitops.mdx that a failed Job needs deleting or a changed input.
Test: In render_test.go add a case asserting the name changes when the mint-set fingerprint changes and is stable for identical inputs. Run go test ./cmd/oap/internal/installcmd/ -run 'Render|JobName'.
Constraints: CLAUDE.md "Server-side apply: keep applied fields idempotent" (the name must stay a pure function of inputs).
| // nonOptionalSecretRefs collects every required (optional!=true) Secret reference | ||
| // from pod templates in docs — env valueFrom.secretKeyRef and volume secrets. | ||
| // A missing optional field defaults to required, matching the kubelet. | ||
| func nonOptionalSecretRefs(docs []*unstructured.Unstructured) []secretRef { |
There was a problem hiding this comment.
[minor] --validate can false-pass: ignores envFrom / projected Secret sources, and mutable-image check passes if any one ref is pinned (latent)
Problem: nonOptionalSecretRefs collects env secretKeyRef and volumes[].secret only, so envFrom.secretRef and volumes[].projected.sources[].secret references are never checked. firstPartyMutableImages (line 254) clears an image if the output contains any <name>:dev@sha256:, even when another reference to the same image is unpinned. Neither is reachable with today's generated bundle (no such refs in the overlay workloads, and substitution rewrites all refs uniformly), so this is a latent false pass for hand-edited overlays or future manifests.
Verify: Build a test overlay that adds an envFrom: secretRef to a Deployment and a second unpinned image ref, and confirm runRenderValidate passes it.
Fix: Extend nonOptionalSecretRefs to also read envFrom[].secretRef (honoring optional) and volumes[].projected.sources[].secret, across containers and initContainers. Change firstPartyMutableImages to scan each image reference occurrence and flag any that lacks @sha256:, instead of a whole-text Contains.
Test: Add negative cases to render_validate_test.go for each (envFrom missing creator, projected missing creator, one pinned plus one unpinned ref). Run go test ./cmd/oap/internal/installcmd/ -run 'Validate'.
| ) | ||
|
|
||
| // MemoryTokenSecretNames is the canonical set of per-component memory-token | ||
| // Secrets the bundle requires. Kept in sync with install.go's |
There was a problem hiding this comment.
[minor] The "canonical" memory-token Secret list is duplicated by hand in install.go and init.go
Problem: The comment says this list is "Kept in sync with install.go's operatorMemoryTokenSecrets". cmd/oap/internal/installcmd/install.go:2144 (operatorMemoryTokenSecrets) and init.go:1070,1079,1090 restate the same three names. Adding a fourth component updates one path and not the other, so oap install and the Job-minted GitOps path diverge.
Fix: Delete operatorMemoryTokenSecrets in install.go and range over bootstrap.MemoryTokenSecretNames in rollOperatorIfPredatesMemoryTokens (install.go:2161); in init.go derive the three ensure*MemoryToken names from the same slice (indexes as in render.go:178-181). Remove the "kept in sync" sentence here.
Test: The existing pkg/platform/manifests/secret_bootstrap_test.go installTimeSecrets should also range over bootstrap.MemoryTokenSecretNames. Run go test ./cmd/oap/internal/installcmd/ ./pkg/platform/manifests/.
| return err | ||
| } | ||
| } | ||
| if _, err := EnsurePassthroughLinkKey(ctx, reader, writer, nil); err != nil { |
There was a problem hiding this comment.
[minor] The bootstrap Job passes a nil logger and logs nothing about what it minted or rotated
Problem: EnsurePassthroughLinkKey(..., nil) here and EnsureNATSClientCreds(ctx, writer, nil) at line 58 are converted to no-op loggers (appsecrets.go:36-38, nats.go:227-229). The Job therefore silently regenerates an unusable passthrough key (appsecrets.go:66) or re-mints NATS credentials and rollout-restarts Deployments (nats.go:281,339), and bootstrapcmd.go logs nothing at all. This breaks the repo's rule that a mutation is visible in logs.
Fix: Add a logf func(string, ...any) (or a logr.Logger) parameter to MintAllInternalSecrets, pass it to both calls, and log one line per Secret created, skipped or repaired. In internal/cmd/operator/bootstrapcmd.go:37 supply a logger (for example from ctrl.Log.WithName("bootstrap-secrets") or fmt.Fprintf to cmd.ErrOrStderr()). Update the existing callers in mint_test.go.
Test: In pkg/platform/bootstrap/mint_test.go assert that the first run logs a line per minted Secret and a second run logs only skips. Run go test ./pkg/platform/bootstrap/.
Constraints: CLAUDE.md "Never silently drop errors" (log at INFO or higher with context).
| // first boot. reader must be a live/uncached reader; writer the controller client. | ||
| // | ||
| // NOTE: the NATS Secrets (identity/tls/creds) are minted by MintNATSSecrets, | ||
| // added in a follow-up commit; this covers the token-shaped ones. |
There was a problem hiding this comment.
[nit] Stale comment: NATS Secrets are minted in this function, not in "a follow-up commit"
Problem: Lines 39-40 say the NATS identity/tls/creds Secrets are minted by MintNATSSecrets, "added in a follow-up commit", but MintAllInternalSecrets mints them at lines 51-59 and no MintNATSSecrets exists.
Fix: Replace the NOTE with a sentence saying the function mints the memory tokens, admind token, passthrough key, NATS identity/TLS/creds, and the optional Postgres/SpiceDB tokens.
Test: None needed; go vet ./pkg/platform/bootstrap/.
What
oap renderemits a GitOps-ready kustomize overlay intodeploy/oapfor Argo CD / Flux — an alternative tooap installmutating a cluster directly. The overlay is a pure function of its flags (byte-identical re-render → no phantom drift) and provisions every internal Secret, with nothing external by default.How
bootstrap-secretsentrypoint) mints the memory/admind/passthrough/NATS/SpiceDB/Postgres secrets idempotently before the operator and data plane start. Its name carries a content hash, so an operator-image upgrade is a new Job object (prune-and-create) instead of an immutable-pod-template apply conflict.pkg/platform/bootstrapas the single source:oap installdelegates to it, the operator's leader-elected bootstrap runnable is now schema-only, and the NATS publish-surface grant moved verbatim with its adversarial test.--external-spicedb/--external-postgres/--external-webhook-tls, and per-component--channelsd/--webd/--authzd-memory-token-secretlinks. webd's token must resolve to a different Secret than channelsd/authzd (effective-name compare) — webd is browser-facing and read-only, the others can write the audit log.oap render --validatelints the built overlay (every non-optional Secret has a creator,AP_CLUSTER_KINDstamped, images pinned);oap checkverifies the memory tokens post-sync. Guide:site/content/docs/gitops.mdx.Scope notes
--with-postgresis the only dependency component offered — its token is Job-minted, so it works end to end. Graphiti/Neo4j need a user-supplied LLM key the overlay cannot mint; their GitOps support is a follow-up (oap installstill handles them).Tests
mage test:unitandmage test:integrationboth green. A fresh whole-branch code review was run; 4 findings were fixed via TDD (including one security-critical webd capability-leak guard bypass) and 2 minors deferred.