From 12b3f7fc8c64744a753b7e21cab7ed4523ab8170 Mon Sep 17 00:00:00 2001 From: Trey Date: Fri, 9 Oct 2026 10:56:02 -0700 Subject: [PATCH 1/6] Fix Connector Gateway configuration gaps Verified against chart v0.22.0. Add the required groupRef, baseline_client_scopes, and control-plane validator to the example; document primaryIdp.subjectClaim, AI Gateway caller auth, name override drift guards, clusterOidcIssuer reachability, the directory NetworkPolicy, the Enterprise Manager KEK Secret format, and global.redis.username. Co-Authored-By: Claude Opus 5.5 --- .../configure-connector-gateway.mdx | 127 ++++++++++++++++-- .../enterprise-platform/deployment.mdx | 31 ++++- 2 files changed, 143 insertions(+), 15 deletions(-) diff --git a/docs/platform/enterprise-platform/configure-connector-gateway.mdx b/docs/platform/enterprise-platform/configure-connector-gateway.mdx index 60371112..6befb809 100644 --- a/docs/platform/enterprise-platform/configure-connector-gateway.mdx +++ b/docs/platform/enterprise-platform/configure-connector-gateway.mdx @@ -39,7 +39,10 @@ embed those names, so adjust them if yours differ. chart doesn't deploy an instance. Configure it once under `global.redis`, as described in [Global Redis/Valkey defaults](./deployment.mdx#global-redisvalkey-defaults), - and the gateway inherits the host, port, and password Secret. + and the gateway inherits the host, port, username, password Secret, and TLS + setting. To connect as a least-privilege ACL user, set `global.redis.username` + and limit that user to keys matching `thv:auth:*` (authorization server + sessions) and `connector-gateway:*` (tool call recording and key checks). ## Prepare the gateway's secrets @@ -98,6 +101,11 @@ kubectl create secret generic enterprise-manager-grpc-ca \ --from-file=ca.crt=./ca.crt ``` +A copied Secret goes stale when cert-manager rotates the CA. For GitOps, keep +the copy in sync automatically instead, for example with a trust-manager +`Bundle` that targets a Secret, or issue the Enterprise Manager's certificate +from a cert-manager `CA` Issuer whose CA you manage and distribute yourself. + ## Enable the Connector Gateway Add the following to the values file you use for the platform. Merge the @@ -119,6 +127,7 @@ global: connector-gateway: vmcpConfig: + groupRef: 'connector-gateway' incomingAuth: type: oidc oidc: @@ -129,8 +138,16 @@ connector-gateway: schema_version: 'v0.1.0' allowed_audiences: - 'https:///gw/mcp' + baseline_client_scopes: + - openid + - offline_access storage: type: redis + controlPlane: + incomingAuth: + oidc: + issuer: 'https://idp.example.com' + audience: '' directory: addr: 'stacklok-enterprise-enterprise-manager.stacklok-system.svc.cluster.local:9443' directoryTLS: @@ -148,8 +165,10 @@ connector-gateway: | `global.stacklok.connectorGateway.enabled` | Deploy the Connector Gateway | | `global.stacklok.connectorGatewayId` | Identify this gateway to the Enterprise Manager and console | | `global.stacklok.authServerIssuer` | Set the gateway's public authorization server URL | +| `vmcpConfig.groupRef` | Name the gateway in its status and logs. Required; any non-empty value works | | `vmcpConfig.incomingAuth` | Require tokens from the gateway's own authorization server on the MCP endpoint | | `enterpriseConfig.authServer` | Configure the embedded authorization server and its Redis storage | +| `enterpriseConfig.controlPlane` | Accept the corporate tokens the console sends to the gateway's REST API | | `enterpriseConfig.directory.addr` | Point the gateway at the Enterprise Manager's TLS gRPC port | | `directoryTLS` | Mount the Enterprise Manager CA bundle and the gateway's ServiceAccount token | | `kek`, `authServerKeys` | Reference the Secrets from [Prepare the gateway's secrets](#prepare-the-gateways-secrets) | @@ -173,6 +192,19 @@ and list the audience in `allowed_audiences` exactly as it appears in `incomingAuth.oidc.audience`. The gateway compares these strings exactly and refuses to start when they disagree. +`vmcpConfig.groupRef` labels the gateway in its status and logs. The gateway +refuses to start while it's empty. It reads its backends from the Enterprise +Manager, so no `MCPGroup` by that name needs to exist. + +`baseline_client_scopes` adds `openid` and `offline_access` to every client that +registers through dynamic client registration. MCP clients such as Claude Code, +Cursor, and VS Code register with `openid` and then request `offline_access` +when they sign in. Without the baseline, the authorization server rejects that +request with `invalid_scope` and the client stays at "needs authentication." +Each baseline scope must also be in the authorization server's supported scopes +(`openid`, `profile`, `email`, and `offline_access` by default), or the gateway +refuses to start. + :::note[Two different issuers] `global.stacklok.primaryIdp.issuer` identifies the corporate identity provider. @@ -187,13 +219,42 @@ loads its identity providers from the Enterprise Manager: your corporate identity provider from `global.stacklok.primaryIdp`, plus the connector identity providers that admins register in the Enterprise Manager. -By default, the Enterprise Manager identifies users by the `sub` claim in your -corporate identity provider's tokens. If your provider identifies users with a -different claim, such as `uid` in Okta, add an -`enterprise-manager.directory.issuers` entry whose `issuer_url` matches -`global.stacklok.primaryIdp.issuer` exactly, and set `binding_claim`: +### Accept console tokens on the REST API + +The console calls the gateway's REST API with the user's corporate identity +provider access token. `enterpriseConfig.controlPlane.incomingAuth.oidc` +configures the validator that accepts those tokens. Without it, the REST API +accepts only tokens from the gateway's own authorization server, and the +console's **Connectors** pages fail with `401`. + +Set `issuer` to `global.stacklok.primaryIdp.issuer`; the gateway refuses to +start if it doesn't match an identity provider it signs users in with. Set +`audience` to the exact `aud` value of the access tokens the console sends. To +find it, decode a console token as shown in +[Configure platform identity](./configure-identity.mdx). Prefer an audience that +only Stacklok components accept over a shared default such as Okta's +`api://default`, which would let tokens minted for unrelated applications reach +the gateway. The token must be a JWT: opaque access tokens, such as those from +Okta's org authorization server, are rejected with `401`. To fetch signing keys +from a URL other than the issuer's discovery document, also set `jwksUrl`. + +### Identify users by a claim other than `sub` + +By default, the gateway and the Enterprise Manager identify users by the `sub` +claim in your corporate identity provider's tokens. If your provider identifies +users with a different claim, such as `uid` in Okta, configure it on both sides. +Set `global.stacklok.primaryIdp.subjectClaim`, which the gateway uses at +sign-in, and add an `enterprise-manager.directory.issuers` entry whose +`issuer_url` matches `global.stacklok.primaryIdp.issuer` exactly, with the same +claim in `binding_claim`: ```yaml title="values.yaml" +global: + stacklok: + primaryIdp: + # issuer and client settings from "Configure identity" stay here. + subjectClaim: 'uid' + enterprise-manager: directory: issuers: @@ -203,7 +264,17 @@ enterprise-manager: binding_claim: 'uid' ``` -The same entry configures SCIM provisioning for that provider. See +The two values must match. When they differ, identity resolution fails and every +user is rejected with `401`. + +The gateway reads the claim from the **ID token**, and it must be a top-level +string claim that also appears on refreshed ID tokens. Okta adds `uid` to access +tokens from a custom authorization server, but not to ID tokens. In Okta, add a +`uid` claim with the value `user.id` and the token type **ID Token**. Choose the +claim before users sign in, because it becomes each user's stable identity in +the directory. + +The same `issuers` entry configures SCIM provisioning for that provider. See [SCIM identity providers](./deployment.mdx#scim-identity-providers). ## Connect the gateway to the Enterprise Manager @@ -242,14 +313,49 @@ this command prints: kubectl get --raw /.well-known/openid-configuration | jq -r .issuer ``` +The Enterprise Manager fetches the issuer's discovery document and signing keys +directly from that URL, not through the Kubernetes API server. On managed +clusters such as Amazon EKS, the issuer is a public URL, so allow egress to it +from the Enterprise Manager pods. If the fetch fails, the Enterprise Manager +rejects every gRPC caller. + Each allowlist covers different calls from the gateway. `saSubjectAllowlist` covers identity resolution and per-user connector configuration. `gatewaySaSubjectAllowlist` covers the gateway's startup registration and its catalog reads. `credentialSaSubjectAllowlist` covers credential resolution, which the gateway needs for the identity provider's client secret. The subject -is `system:serviceaccount::-connector-gateway`. If the -AI Gateway also calls the Enterprise Manager, keep its entries on -`saSubjectAllowlist` alongside this one. +is `system:serviceaccount::-connector-gateway`. The +chart fills in a fourth list, `meteringSaSubjectAllowlist`, from the same +subject for +[tool call recording](#turn-on-tool-call-recording-and-observability). + +### Keep the AI Gateway's access + +Caller authentication covers both the TLS and cleartext gRPC ports, so turning +it on also gates the AI Gateway. Its API key service checks every virtual API +key against the Enterprise Manager. If your platform also runs the AI Gateway +with virtual API keys: + +- Add the API key service's subject to `saSubjectAllowlist`: + `system:serviceaccount::-api-key-service`, + where `` is the name of your `AIGateway` resource. +- Keep `spec.auth.virtualAPIKeys.directory.insecure` on the `AIGateway` at its + default, `false`, so the API key service presents its ServiceAccount token. + +Otherwise every virtual-key request fails with `503` and +`key store temporarily unavailable`. The chart refuses to render while the AI +Gateway is enabled and no subject ending in `-api-key-service` is on the list. + +### Name overrides + +The umbrella chart computes two values from the default gateway name, +`-connector-gateway`: `meteringSaSubjectAllowlist` and +`toolhive-cloud-ui.connectorGatewayUrl`, the in-cluster REST API URL that +enables the console's **Connectors** pages. If you set +`connector-gateway.fullnameOverride` or `nameOverride`, or change the +control-plane Service port, the render fails until you set both by hand. The +error message prints the value to use. Also update the subjects in your three +allowlists to match the new ServiceAccount name. For a development cluster without TLS, set `connector-gateway.enterpriseConfig.directory.insecure: true` in place of @@ -267,6 +373,7 @@ addition to the issuer and client settings in | Key | Purpose | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `clientClaim` | Access-token claim that names the OAuth client. Empty reads `cid` or `client_id`. Allowed values are `cid`, `client_id`, `azp`, and `appid`; set `azp` for Dex, or `azp` or `appid` for Microsoft Entra ID | +| `subjectClaim` | ID-token claim that identifies each user. Defaults to `sub`. Must match the Enterprise Manager's `binding_claim`; see [Identify users by a claim other than `sub`](#identify-users-by-a-claim-other-than-sub) | | `clientClaimAllowUnsafe` | Boolean. Accepts a `clientClaim` outside the allowed values. A weaker claim can match tokens minted for a different client, and the gateway logs a warning at startup | | `allowPrivateIps` | Boolean. Lets discovery and token exchange reach an identity provider on a private or link-local address. HTTPS is still required for non-localhost hosts | | `caBundleRef` | ConfigMap or Secret holding the PEM CA bundle for an identity provider served by a private CA. Set exactly one of `configMapRef` or `secretRef`, each with a `name`, plus an optional `key` (default `ca.crt`) | diff --git a/docs/platform/enterprise-platform/deployment.mdx b/docs/platform/enterprise-platform/deployment.mdx index 138f3f86..4c84b159 100644 --- a/docs/platform/enterprise-platform/deployment.mdx +++ b/docs/platform/enterprise-platform/deployment.mdx @@ -614,13 +614,30 @@ enterprise-manager: app.kubernetes.io/name: connector-gateway ``` +For GitOps, replace `generate: true` with `existingSecret` naming a Secret you +create. The Secret's `kek` key holds a JSON map from version number to a +base64-encoded 32-byte key, and the Enterprise Manager uses the highest version: + +```bash +kubectl create secret generic enterprise-manager-kek \ + --namespace stacklok-system \ + --from-literal=kek="{\"1\":\"$(openssl rand -base64 32)\"}" +``` + TLS is not optional here: the directory refuses a non-TLS peer at call time, so a key-encrypted install served over the cleartext port renders cleanly and then -fails every credential resolution. Point callers at `grpc.tls.port`. If a -service mesh or cluster-wide policy already restricts the gRPC ports, set -`directory.kek.externallyManagedNetworkPolicy=true` to acknowledge that instead -of defining a policy here; there is no equivalent escape for TLS, because a mesh -terminates it in the sidecar and the peer still arrives without it. +fails every credential resolution. Point callers at `grpc.tls.port`. + +With `directory.enabled: true`, the chart also renders a +`-enterprise-manager-directory` NetworkPolicy. It admits Connector +Gateway pods from the release namespace on both gRPC ports, and the AI Gateway's +API key service on the cleartext port. `grpc.networkPolicy` adds a second policy +that restricts both gRPC ports to `allowedClientSelector`. If the directory +policy, a service mesh, or a cluster-wide policy already restricts the gRPC +ports, set `directory.kek.externallyManagedNetworkPolicy: true` to acknowledge +that instead of enabling `grpc.networkPolicy`. There is no equivalent escape for +TLS, because a mesh terminates it in the sidecar and the peer still arrives +without it. #### SCIM identity providers @@ -687,8 +704,12 @@ global: existingSecret: 'redis-auth' # Defaults to redis-password when omitted. existingSecretKey: 'redis-password' + # Optional ACL username. Omit it to authenticate with the password only. + username: '' ``` +The chart refuses to render when `username` is set without `existingSecret`. + The ToolHive operator also accepts `toolhive-operator.upstream.operator.defaultRedis.existingSecret` and `existingSecretKey` for a component-specific credential, which take precedence From 8911fcbd20d8f429fa249b80df9d1974753f0744 Mon Sep 17 00:00:00 2001 From: Trey Date: Fri, 9 Oct 2026 15:38:07 -0700 Subject: [PATCH 2/6] Correct Connector Gateway failure modes and gaps Addresses stacklok/docs-website#1227 review comments: - MEDIUM configure-connector-gateway.mdx (4235091747): discovery failure stops EM startup - MEDIUM configure-connector-gateway.mdx (4235091770): API key guard fires without virtual keys - MEDIUM configure-connector-gateway.mdx (4235091778): cover grpc.insecure for the AI Gateway - MEDIUM configure-connector-gateway.mdx (4235091790): subjectClaim also read from access tokens - MEDIUM configure-connector-gateway.mdx (4235091815): name override guards and httpSaSubjectAllowlist - MEDIUM configure-connector-gateway.mdx (4235091820): give dev cleartext path its own heading - LOW configure-connector-gateway.mdx (4235091824): state minimum chart version - LOW configure-connector-gateway.mdx (4235091846): API key service in release namespace - LOW configure-connector-gateway.mdx (4235091853): main processor allowlist entry - LOW configure-connector-gateway.mdx (4235091861): qualify Redis ACL advice - LOW configure-connector-gateway.mdx (4235091871): explain how to find console audience - LOW configure-connector-gateway.mdx (4235091880): placeholder for control-plane issuer Co-Authored-By: Claude Opus 5.5 --- .../configure-connector-gateway.mdx | 124 +++++++++++++----- 1 file changed, 89 insertions(+), 35 deletions(-) diff --git a/docs/platform/enterprise-platform/configure-connector-gateway.mdx b/docs/platform/enterprise-platform/configure-connector-gateway.mdx index 6befb809..51d00034 100644 --- a/docs/platform/enterprise-platform/configure-connector-gateway.mdx +++ b/docs/platform/enterprise-platform/configure-connector-gateway.mdx @@ -17,6 +17,8 @@ embed those names, so adjust them if yours differ. ## Prerequisites +- **Stacklok Enterprise platform chart v0.22.0 or later.** Earlier charts don't + pass the `global.redis` settings to the gateway. - **A corporate identity provider** at `global.stacklok.primaryIdp`. The gateway resolves every caller to a directory user before it decides what they can see. See [Configure identity](./configure-identity.mdx). @@ -40,9 +42,25 @@ embed those names, so adjust them if yours differ. described in [Global Redis/Valkey defaults](./deployment.mdx#global-redisvalkey-defaults), and the gateway inherits the host, port, username, password Secret, and TLS - setting. To connect as a least-privilege ACL user, set `global.redis.username` - and limit that user to keys matching `thv:auth:*` (authorization server - sessions) and `connector-gateway:*` (tool call recording and key checks). + setting. + +If the Connector Gateway is the only component that uses `global.redis`, you can +connect as a least-privilege ACL user. Set `global.redis.username` to that user, +and apply this reference ACL to your Redis or Valkey instance with your own +password. `global.redis.username` applies to every component that inherits +`global.redis`, so don't restrict the user this way while other components share +it. + +```text title="users.acl" +user default reset off +user cgw reset on > resetchannels ~thv:auth:* ~connector-gateway:* +@read +@write +@connection +@transaction -@admin -@dangerous -@scripting -script -function -fcall -fcall_ro +eval +evalsha -config -acl -flushall -flushdb -swapdb -replicaof -slaveof -cluster -failover -save -bgsave -bgrewriteaof -debug -shutdown -monitor -client|tracking -keys -randomkey -dbsize -scan +``` + +The `thv:auth:*` keys hold authorization server sessions, and the +`connector-gateway:*` keys hold the tool call recording queue, a check for the +gateway's encryption key, and client registration locks. The first line disables +the built-in `default` user, which otherwise grants full access to any +connection that falls back to it. ## Prepare the gateway's secrets @@ -146,7 +164,7 @@ connector-gateway: controlPlane: incomingAuth: oidc: - issuer: 'https://idp.example.com' + issuer: '' # same as global.stacklok.primaryIdp.issuer audience: '' directory: addr: 'stacklok-enterprise-enterprise-manager.stacklok-system.svc.cluster.local:9443' @@ -229,10 +247,14 @@ console's **Connectors** pages fail with `401`. Set `issuer` to `global.stacklok.primaryIdp.issuer`; the gateway refuses to start if it doesn't match an identity provider it signs users in with. Set -`audience` to the exact `aud` value of the access tokens the console sends. To -find it, decode a console token as shown in -[Configure platform identity](./configure-identity.mdx). Prefer an audience that -only Stacklok components accept over a shared default such as Okta's +`audience` to the exact `aud` value of the access tokens the console sends. The +console sends the access token from its own sign-in, so this is the audience +your identity provider assigns to tokens for the console's OAuth client, such as +`cloud-ui` in the Okta example in +[Configure platform identity](./configure-identity.mdx). To confirm it, decode +an access token issued to that client as shown in +[Verify a token](./configure-identity.mdx#verify-a-token). Prefer an audience +that only Stacklok components accept over a shared default such as Okta's `api://default`, which would let tokens minted for unrelated applications reach the gateway. The token must be a JWT: opaque access tokens, such as those from Okta's org authorization server, are rejected with `401`. To fetch signing keys @@ -274,6 +296,11 @@ tokens from a custom authorization server, but not to ID tokens. In Okta, add a claim before users sign in, because it becomes each user's stable identity in the directory. +The console's access tokens must carry the same top-level claim, because the +REST API resolves users from them. Otherwise sign-in works but the console's +**Connectors** pages fail with `401`. Okta's custom authorization servers add +`uid` to access tokens by default. + The same `issuers` entry configures SCIM provisioning for that provider. See [SCIM identity providers](./deployment.mdx#scim-identity-providers). @@ -316,8 +343,9 @@ kubectl get --raw /.well-known/openid-configuration | jq -r .issuer The Enterprise Manager fetches the issuer's discovery document and signing keys directly from that URL, not through the Kubernetes API server. On managed clusters such as Amazon EKS, the issuer is a public URL, so allow egress to it -from the Enterprise Manager pods. If the fetch fails, the Enterprise Manager -rejects every gRPC caller. +from the Enterprise Manager pods. If it can't fetch the discovery document at +startup, the Enterprise Manager fails to start. If it later can't refresh the +signing keys, it rejects every gRPC caller. Each allowlist covers different calls from the gateway. `saSubjectAllowlist` covers identity resolution and per-user connector configuration. @@ -329,6 +357,17 @@ chart fills in a fourth list, `meteringSaSubjectAllowlist`, from the same subject for [tool call recording](#turn-on-tool-call-recording-and-observability). +### Use a cleartext connection for development + +For a development cluster without TLS, set +`connector-gateway.enterpriseConfig.directory.insecure: true` in place of +`directoryTLS`, point `addr` at the cleartext port (`9091`), and set +`enterprise-manager.grpc.insecure: true`. The chart requires both settings +together. This transport carries credentials in cleartext with no caller +authentication, so reserve it for local testing. The Enterprise Manager refuses +to start with `grpc.insecure: true` while `directory.kek` is configured, so this +transport works only on an install without the key-encryption key. + ### Keep the AI Gateway's access Caller authentication covers both the TLS and cleartext gRPC ports, so turning @@ -336,33 +375,48 @@ it on also gates the AI Gateway. Its API key service checks every virtual API key against the Enterprise Manager. If your platform also runs the AI Gateway with virtual API keys: +- Create the `AIGateway` resource in the release namespace. The Enterprise + Manager's directory NetworkPolicy admits the API key service only from that + namespace. - Add the API key service's subject to `saSubjectAllowlist`: - `system:serviceaccount::-api-key-service`, + `system:serviceaccount:stacklok-system:-api-key-service`, where `` is the name of your `AIGateway` resource. - Keep `spec.auth.virtualAPIKeys.directory.insecure` on the `AIGateway` at its default, `false`, so the API key service presents its ServiceAccount token. +- Unset `enterprise-manager.grpc.insecure` if you set it for the AI Gateway. It + skips caller authentication, and the Enterprise Manager refuses to start with + it alongside a directory key-encryption key. Otherwise every virtual-key request fails with `503` and -`key store temporarily unavailable`. The chart refuses to render while the AI -Gateway is enabled and no subject ending in `-api-key-service` is on the list. +`key store temporarily unavailable`. -### Name overrides +The chart refuses to render while the AI Gateway is enabled and no subject +ending in `-api-key-service` is on the list, even if you don't use virtual API +keys. In that case, set +`enterprise-manager.grpc.callerAuth.acknowledgeNoApiKeyServiceCaller: true` +instead of adding a subject. -The umbrella chart computes two values from the default gateway name, -`-connector-gateway`: `meteringSaSubjectAllowlist` and -`toolhive-cloud-ui.connectorGatewayUrl`, the in-cluster REST API URL that -enables the console's **Connectors** pages. If you set -`connector-gateway.fullnameOverride` or `nameOverride`, or change the -control-plane Service port, the render fails until you set both by hand. The -error message prints the value to use. Also update the subjects in your three -allowlists to match the new ServiceAccount name. +If you set `global.stacklok.directoryGrpcEndpoint` to add group labels to AI +Gateway telemetry, also add +`system:serviceaccount:stacklok-system:-main-processor` to +`saSubjectAllowlist`. Without it, the AI Gateway records telemetry without group +labels and logs no error. -For a development cluster without TLS, set -`connector-gateway.enterpriseConfig.directory.insecure: true` in place of -`directoryTLS`, point `addr` at the cleartext port (`9091`), and set -`enterprise-manager.grpc.insecure: true`. The chart requires both settings -together. This transport carries credentials in cleartext with no caller -authentication, so reserve it for local testing. +### Override the gateway's name + +The umbrella chart computes these `enterprise-manager.grpc.callerAuth` and +console values from the default gateway name, +`-connector-gateway`. If you change that name, set each one by +hand: + +| Value | Render fails when | +| -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| `meteringSaSubjectAllowlist` | The gateway's ServiceAccount name changes: `fullnameOverride`, `nameOverride`, `serviceAccount.name`, or `serviceAccount.create: false` | +| `toolhive-cloud-ui.connectorGatewayUrl`, the in-cluster REST API URL that enables the console's **Connectors** pages | The control-plane Service name or port changes, or `controlPlane.enabled` is `false` | +| `httpSaSubjectAllowlist`, set only when `global.stacklok.enterpriseManager.url` is set | Never. The Enterprise Manager refuses the gateway's HTTP calls, and nothing warns you at render time | + +When a render fails, the error message prints the value to use. Also update the +subjects in your three allowlists to match the new ServiceAccount name. ## Corporate identity provider settings @@ -370,13 +424,13 @@ The gateway reads these optional keys from `global.stacklok.primaryIdp`, in addition to the issuer and client settings in [Configure platform identity](./configure-identity.mdx): -| Key | Purpose | -| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `clientClaim` | Access-token claim that names the OAuth client. Empty reads `cid` or `client_id`. Allowed values are `cid`, `client_id`, `azp`, and `appid`; set `azp` for Dex, or `azp` or `appid` for Microsoft Entra ID | -| `subjectClaim` | ID-token claim that identifies each user. Defaults to `sub`. Must match the Enterprise Manager's `binding_claim`; see [Identify users by a claim other than `sub`](#identify-users-by-a-claim-other-than-sub) | -| `clientClaimAllowUnsafe` | Boolean. Accepts a `clientClaim` outside the allowed values. A weaker claim can match tokens minted for a different client, and the gateway logs a warning at startup | -| `allowPrivateIps` | Boolean. Lets discovery and token exchange reach an identity provider on a private or link-local address. HTTPS is still required for non-localhost hosts | -| `caBundleRef` | ConfigMap or Secret holding the PEM CA bundle for an identity provider served by a private CA. Set exactly one of `configMapRef` or `secretRef`, each with a `name`, plus an optional `key` (default `ca.crt`) | +| Key | Purpose | +| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `clientClaim` | Access-token claim that names the OAuth client. Empty reads `cid` or `client_id`. Allowed values are `cid`, `client_id`, `azp`, and `appid`; set `azp` for Dex, or `azp` or `appid` for Microsoft Entra ID | +| `subjectClaim` | Claim that identifies each user, read from ID tokens at sign-in and from the console's access tokens. Defaults to `sub`. Must match the Enterprise Manager's `binding_claim`; see [Identify users by a claim other than `sub`](#identify-users-by-a-claim-other-than-sub) | +| `clientClaimAllowUnsafe` | Boolean. Accepts a `clientClaim` outside the allowed values. A weaker claim can match tokens minted for a different client, and the gateway logs a warning at startup | +| `allowPrivateIps` | Boolean. Lets discovery and token exchange reach an identity provider on a private or link-local address. HTTPS is still required for non-localhost hosts | +| `caBundleRef` | ConfigMap or Secret holding the PEM CA bundle for an identity provider served by a private CA. Set exactly one of `configMapRef` or `secretRef`, each with a `name`, plus an optional `key` (default `ca.crt`) | Set the boolean keys as unquoted `true` or `false`. The chart refuses a quoted string, and `--set-string` produces one. From 50edbf485ca9b781867bd5a237c5e5a6ef09bf56 Mon Sep 17 00:00:00 2001 From: Trey Date: Fri, 9 Oct 2026 15:38:09 -0700 Subject: [PATCH 3/6] Fix directory NetworkPolicy and KEK guidance Addresses stacklok/docs-website#1227 review comments: - MEDIUM deployment.mdx (4235091799): NetworkPolicies are additive; match the example - MEDIUM deployment.mdx (4235091808): warn against replacing an existing KEK Co-Authored-By: Claude Opus 5.5 --- .../enterprise-platform/deployment.mdx | 41 +++++++++++++++---- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/docs/platform/enterprise-platform/deployment.mdx b/docs/platform/enterprise-platform/deployment.mdx index 4c84b159..55a27743 100644 --- a/docs/platform/enterprise-platform/deployment.mdx +++ b/docs/platform/enterprise-platform/deployment.mdx @@ -624,20 +624,43 @@ kubectl create secret generic enterprise-manager-kek \ --from-literal=kek="{\"1\":\"$(openssl rand -base64 32)\"}" ``` +```yaml title="values.yaml" +enterprise-manager: + directory: + kek: + enabled: true + existingSecret: 'enterprise-manager-kek' +``` + +Make this choice before the Enterprise Manager stores any credentials. A new key +makes every credential encrypted with the old one permanently unreadable. To +move an install that already ran with `generate: true`, copy the `kek` value +from the chart-created `-enterprise-manager-directory-kek` Secret +into your own Secret instead of generating a new key. To rotate the key, add a +higher-numbered entry to the map and keep the earlier ones. + TLS is not optional here: the directory refuses a non-TLS peer at call time, so a key-encrypted install served over the cleartext port renders cleanly and then fails every credential resolution. Point callers at `grpc.tls.port`. With `directory.enabled: true`, the chart also renders a -`-enterprise-manager-directory` NetworkPolicy. It admits Connector -Gateway pods from the release namespace on both gRPC ports, and the AI Gateway's -API key service on the cleartext port. `grpc.networkPolicy` adds a second policy -that restricts both gRPC ports to `allowedClientSelector`. If the directory -policy, a service mesh, or a cluster-wide policy already restricts the gRPC -ports, set `directory.kek.externallyManagedNetworkPolicy: true` to acknowledge -that instead of enabling `grpc.networkPolicy`. There is no equivalent escape for -TLS, because a mesh terminates it in the sidecar and the peer still arrives -without it. +`-enterprise-manager-directory` NetworkPolicy that limits which +pods reach the Enterprise Manager's gRPC ports. It admits these pods, all from +the release namespace: + +- Connector Gateway pods, on both gRPC ports +- The AI Gateway's API key service and main processor, on the cleartext port +- The AI Gateway main processor and operator, on the webhook port, when + `webhookTLS.enabled` is `true` + +NetworkPolicies are additive, so the `grpc.networkPolicy` policy in the example +can only admit more pods. It admits pods in the release namespace that match +`allowedClientSelector`, on both gRPC ports. Keep the selector to the Connector +Gateway's labels as shown. The chart requires either `grpc.networkPolicy` or, +when a service mesh or cluster-wide policy already restricts the gRPC ports, +`directory.kek.externallyManagedNetworkPolicy: true` to acknowledge it. There is +no equivalent escape for TLS, because a mesh terminates it in the sidecar and +the peer still arrives without it. #### SCIM identity providers From a26a719022bce03b1079932f2f6d2520619ad84d Mon Sep 17 00:00:00 2001 From: Trey Date: Tue, 13 Oct 2026 07:44:17 -0700 Subject: [PATCH 4/6] Separate setup, advanced config, and troubleshooting Addresses stacklok/docs-website#1227 review comments: - MEDIUM review body (5498084449): move symptoms to Troubleshooting, group optional config under Advanced configuration, move groupRef, link from AI Gateway guide - MEDIUM review body (5498084449): nameOverride changes the pod label; update allowedClientSelector - MEDIUM configure-connector-gateway.mdx (4254095820): move Redis ACL recipe to Advanced configuration - MEDIUM configure-connector-gateway.mdx (4254095821): shorten baseline_client_scopes text - MEDIUM configure-connector-gateway.mdx (4254095827): move issuer failure diagnosis to Troubleshooting - MEDIUM configure-connector-gateway.mdx (4254095830): make the name override table task-oriented Co-Authored-By: Claude Opus 5.5 --- .../configure-ai-gateway.mdx | 5 + .../configure-connector-gateway.mdx | 242 +++++++++++------- 2 files changed, 157 insertions(+), 90 deletions(-) diff --git a/docs/platform/enterprise-platform/configure-ai-gateway.mdx b/docs/platform/enterprise-platform/configure-ai-gateway.mdx index 84b9c743..41e17fbf 100644 --- a/docs/platform/enterprise-platform/configure-ai-gateway.mdx +++ b/docs/platform/enterprise-platform/configure-ai-gateway.mdx @@ -21,6 +21,11 @@ resources in the order described below. See It is optional and off by default. See [PCI/PII controls](../../ai-gateway/pci-pii-controls.mdx). +If the Enterprise Manager authenticates gRPC callers, as it must when the +Connector Gateway is enabled, also allow the AI Gateway's ServiceAccounts as +described in +[Keep the AI Gateway's access](./configure-connector-gateway.mdx#keep-the-ai-gateways-access). + ## Enable it Set the install toggle in your platform values and upgrade: diff --git a/docs/platform/enterprise-platform/configure-connector-gateway.mdx b/docs/platform/enterprise-platform/configure-connector-gateway.mdx index 51d00034..126c6cce 100644 --- a/docs/platform/enterprise-platform/configure-connector-gateway.mdx +++ b/docs/platform/enterprise-platform/configure-connector-gateway.mdx @@ -42,25 +42,8 @@ embed those names, so adjust them if yours differ. described in [Global Redis/Valkey defaults](./deployment.mdx#global-redisvalkey-defaults), and the gateway inherits the host, port, username, password Secret, and TLS - setting. - -If the Connector Gateway is the only component that uses `global.redis`, you can -connect as a least-privilege ACL user. Set `global.redis.username` to that user, -and apply this reference ACL to your Redis or Valkey instance with your own -password. `global.redis.username` applies to every component that inherits -`global.redis`, so don't restrict the user this way while other components share -it. - -```text title="users.acl" -user default reset off -user cgw reset on > resetchannels ~thv:auth:* ~connector-gateway:* +@read +@write +@connection +@transaction -@admin -@dangerous -@scripting -script -function -fcall -fcall_ro +eval +evalsha -config -acl -flushall -flushdb -swapdb -replicaof -slaveof -cluster -failover -save -bgsave -bgrewriteaof -debug -shutdown -monitor -client|tracking -keys -randomkey -dbsize -scan -``` - -The `thv:auth:*` keys hold authorization server sessions, and the -`connector-gateway:*` keys hold the tool call recording queue, a check for the -gateway's encryption key, and client registration locks. The first line disables -the built-in `default` user, which otherwise grants full access to any -connection that falls back to it. + setting. To connect as a least-privilege ACL user, see + [Restrict the Redis user](#restrict-the-redis-user). ## Prepare the gateway's secrets @@ -191,6 +174,14 @@ connector-gateway: | `directoryTLS` | Mount the Enterprise Manager CA bundle and the gateway's ServiceAccount token | | `kek`, `authServerKeys` | Reference the Secrets from [Prepare the gateway's secrets](#prepare-the-gateways-secrets) | +Set `vmcpConfig.groupRef` to any non-empty name; the gateway refuses to start +while it's empty. The gateway reads its backends from the Enterprise Manager, so +no `MCPGroup` by that name needs to exist. + +Set `baseline_client_scopes` to `openid` and `offline_access` so MCP clients +that register through dynamic client registration, such as Claude Code, Cursor, +and VS Code, can obtain refresh tokens. + ### Choose the gateway ID and issuer Choose a stable `connectorGatewayId` such as `prod-eu` or `platform-staging`, @@ -210,28 +201,6 @@ and list the audience in `allowed_audiences` exactly as it appears in `incomingAuth.oidc.audience`. The gateway compares these strings exactly and refuses to start when they disagree. -`vmcpConfig.groupRef` labels the gateway in its status and logs. The gateway -refuses to start while it's empty. It reads its backends from the Enterprise -Manager, so no `MCPGroup` by that name needs to exist. - -`baseline_client_scopes` adds `openid` and `offline_access` to every client that -registers through dynamic client registration. MCP clients such as Claude Code, -Cursor, and VS Code register with `openid` and then request `offline_access` -when they sign in. Without the baseline, the authorization server rejects that -request with `invalid_scope` and the client stays at "needs authentication." -Each baseline scope must also be in the authorization server's supported scopes -(`openid`, `profile`, `email`, and `offline_access` by default), or the gateway -refuses to start. - -:::note[Two different issuers] - -`global.stacklok.primaryIdp.issuer` identifies the corporate identity provider. -`authServerIssuer` identifies the Connector Gateway's authorization server. -Configure both values. See -[Configure platform identity](./configure-identity.mdx). - -::: - Omit `upstreams` from `enterpriseConfig.authServer`. The authorization server loads its identity providers from the Enterprise Manager: your corporate identity provider from `global.stacklok.primaryIdp`, plus the connector identity @@ -241,9 +210,7 @@ providers that admins register in the Enterprise Manager. The console calls the gateway's REST API with the user's corporate identity provider access token. `enterpriseConfig.controlPlane.incomingAuth.oidc` -configures the validator that accepts those tokens. Without it, the REST API -accepts only tokens from the gateway's own authorization server, and the -console's **Connectors** pages fail with `401`. +configures the validator that accepts those tokens. Set `issuer` to `global.stacklok.primaryIdp.issuer`; the gateway refuses to start if it doesn't match an identity provider it signs users in with. Set @@ -256,9 +223,9 @@ an access token issued to that client as shown in [Verify a token](./configure-identity.mdx#verify-a-token). Prefer an audience that only Stacklok components accept over a shared default such as Okta's `api://default`, which would let tokens minted for unrelated applications reach -the gateway. The token must be a JWT: opaque access tokens, such as those from -Okta's org authorization server, are rejected with `401`. To fetch signing keys -from a URL other than the issuer's discovery document, also set `jwksUrl`. +the gateway. The token must be a JWT, so use an authorization server that issues +JWT access tokens rather than Okta's org authorization server. To fetch signing +keys from a URL other than the issuer's discovery document, also set `jwksUrl`. ### Identify users by a claim other than `sub` @@ -286,9 +253,6 @@ enterprise-manager: binding_claim: 'uid' ``` -The two values must match. When they differ, identity resolution fails and every -user is rejected with `401`. - The gateway reads the claim from the **ID token**, and it must be a top-level string claim that also appears on refreshed ID tokens. Okta adds `uid` to access tokens from a custom authorization server, but not to ID tokens. In Okta, add a @@ -297,9 +261,8 @@ claim before users sign in, because it becomes each user's stable identity in the directory. The console's access tokens must carry the same top-level claim, because the -REST API resolves users from them. Otherwise sign-in works but the console's -**Connectors** pages fail with `401`. Okta's custom authorization servers add -`uid` to access tokens by default. +REST API resolves users from them. Okta's custom authorization servers add `uid` +to access tokens by default. The same `issuers` entry configures SCIM provisioning for that provider. See [SCIM identity providers](./deployment.mdx#scim-identity-providers). @@ -343,9 +306,7 @@ kubectl get --raw /.well-known/openid-configuration | jq -r .issuer The Enterprise Manager fetches the issuer's discovery document and signing keys directly from that URL, not through the Kubernetes API server. On managed clusters such as Amazon EKS, the issuer is a public URL, so allow egress to it -from the Enterprise Manager pods. If it can't fetch the discovery document at -startup, the Enterprise Manager fails to start. If it later can't refresh the -signing keys, it rejects every gRPC caller. +from the Enterprise Manager pods. Each allowlist covers different calls from the gateway. `saSubjectAllowlist` covers identity resolution and per-user connector configuration. @@ -355,18 +316,11 @@ which the gateway needs for the identity provider's client secret. The subject is `system:serviceaccount::-connector-gateway`. The chart fills in a fourth list, `meteringSaSubjectAllowlist`, from the same subject for -[tool call recording](#turn-on-tool-call-recording-and-observability). - -### Use a cleartext connection for development - -For a development cluster without TLS, set -`connector-gateway.enterpriseConfig.directory.insecure: true` in place of -`directoryTLS`, point `addr` at the cleartext port (`9091`), and set -`enterprise-manager.grpc.insecure: true`. The chart requires both settings -together. This transport carries credentials in cleartext with no caller -authentication, so reserve it for local testing. The Enterprise Manager refuses -to start with `grpc.insecure: true` while `directory.kek` is configured, so this -transport works only on an install without the key-encryption key. +[tool call recording](#turn-on-tool-call-recording-and-observability). If you +change the gateway's name, update these values as described in +[Override the gateway's name](#override-the-gateways-name). For a development +cluster without TLS, see +[Use a cleartext connection for development](#use-a-cleartext-connection-for-development). ### Keep the AI Gateway's access @@ -387,9 +341,6 @@ with virtual API keys: skips caller authentication, and the Enterprise Manager refuses to start with it alongside a directory key-encryption key. -Otherwise every virtual-key request fails with `503` and -`key store temporarily unavailable`. - The chart refuses to render while the AI Gateway is enabled and no subject ending in `-api-key-service` is on the list, even if you don't use virtual API keys. In that case, set @@ -399,24 +350,7 @@ instead of adding a subject. If you set `global.stacklok.directoryGrpcEndpoint` to add group labels to AI Gateway telemetry, also add `system:serviceaccount:stacklok-system:-main-processor` to -`saSubjectAllowlist`. Without it, the AI Gateway records telemetry without group -labels and logs no error. - -### Override the gateway's name - -The umbrella chart computes these `enterprise-manager.grpc.callerAuth` and -console values from the default gateway name, -`-connector-gateway`. If you change that name, set each one by -hand: - -| Value | Render fails when | -| -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -| `meteringSaSubjectAllowlist` | The gateway's ServiceAccount name changes: `fullnameOverride`, `nameOverride`, `serviceAccount.name`, or `serviceAccount.create: false` | -| `toolhive-cloud-ui.connectorGatewayUrl`, the in-cluster REST API URL that enables the console's **Connectors** pages | The control-plane Service name or port changes, or `controlPlane.enabled` is `false` | -| `httpSaSubjectAllowlist`, set only when `global.stacklok.enterpriseManager.url` is set | Never. The Enterprise Manager refuses the gateway's HTTP calls, and nothing warns you at render time | - -When a render fails, the error message prints the value to use. Also update the -subjects in your three allowlists to match the new ServiceAccount name. +`saSubjectAllowlist`. ## Corporate identity provider settings @@ -622,6 +556,68 @@ and 3. Direct users to the deployment-specific instructions in the console. See [Roll out gateway clients](./roll-out-gateway-clients.mdx). +## Advanced configuration + +These settings apply only to some installs. Configure them alongside the values +above. + +### Restrict the Redis user + +If the Connector Gateway is the only component that uses `global.redis`, you can +connect as a least-privilege ACL user. Set `global.redis.username` to that user, +and apply this reference ACL to your Redis or Valkey instance with your own +password. `global.redis.username` applies to every component that inherits +`global.redis`, so don't restrict the user this way while other components share +it. + +```text title="users.acl" +user default reset off +user cgw reset on > resetchannels ~thv:auth:* ~connector-gateway:* +@read +@write +@connection +@transaction -@admin -@dangerous -@scripting -script -function -fcall -fcall_ro +eval +evalsha -config -acl -flushall -flushdb -swapdb -replicaof -slaveof -cluster -failover -save -bgsave -bgrewriteaof -debug -shutdown -monitor -client|tracking -keys -randomkey -dbsize -scan +``` + +The `thv:auth:*` keys hold authorization server sessions, and the +`connector-gateway:*` keys hold the tool call recording queue, a check for the +gateway's encryption key, and client registration locks. The first line disables +the built-in `default` user, which otherwise grants full access to any +connection that falls back to it. + +### Use a cleartext connection for development + +For a development cluster without TLS, set +`connector-gateway.enterpriseConfig.directory.insecure: true` in place of +`directoryTLS`, point `addr` at the cleartext port (`9091`), and set +`enterprise-manager.grpc.insecure: true`. The chart requires both settings +together. This transport carries credentials in cleartext with no caller +authentication, so reserve it for local testing. The Enterprise Manager refuses +to start with `grpc.insecure: true` while `directory.kek` is configured, so this +transport works only on an install without the key-encryption key. + +### Override the gateway's name + +The umbrella chart computes several Enterprise Manager and console values from +the gateway's default name, `-connector-gateway`. If you set +`connector-gateway.fullnameOverride`, `nameOverride`, `serviceAccount.name`, or +`serviceAccount.create: false`, or change the control-plane Service, update +these values to match: + +| Setting | Replacement value | +| ------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------- | +| `enterprise-manager.grpc.callerAuth.saSubjectAllowlist`, `gatewaySaSubjectAllowlist`, `credentialSaSubjectAllowlist`, and `meteringSaSubjectAllowlist` | `system:serviceaccount::` | +| `enterprise-manager.grpc.callerAuth.httpSaSubjectAllowlist`, only when `global.stacklok.enterpriseManager.url` is set | The same subject, alongside any AI Gateway entry already on the list | +| `toolhive-cloud-ui.connectorGatewayUrl`, which enables the console's **Connectors** pages | `http://.:` | +| `enterprise-manager.grpc.networkPolicy.allowedClientSelector`, only with `nameOverride` | `app.kubernetes.io/name: ` | + +To read the names, render the chart with `helm template` and find the gateway's +ServiceAccount and its `-control-plane` Service. With `fullnameOverride`, the +ServiceAccount and Services take the override as their name; with +`nameOverride`, they become `-`. + +`nameOverride` also changes the pods' `app.kubernetes.io/name` label. The +Enterprise Manager's directory NetworkPolicy admits only pods labeled +`connector-gateway`, so renamed pods reach the gRPC ports only through +`grpc.networkPolicy`. If you don't enable that policy, use `fullnameOverride`, +which keeps the label. + ## Next steps - [Connector Gateway quickstart](../../connector-gateway/quickstart.mdx) to @@ -658,6 +654,72 @@ kubectl logs deployment/stacklok-enterprise-connector-gateway -n stacklok-system restart the gateway; it reads the bundle only at startup. - **Connection timeouts.** Check that the Enterprise Manager's gRPC NetworkPolicy admits the gateway pods, and that the gateway can reach the - Redis or Valkey host. + Redis or Valkey host. After a `nameOverride`, see + [Override the gateway's name](#override-the-gateways-name). + + + +
+The Enterprise Manager fails to start or rejects every gRPC caller + +The Enterprise Manager fetches the `clusterOidcIssuer` discovery document at +startup. If it can't, it exits with `cluster OIDC bootstrap failed`. If it +starts but later can't refresh the issuer's signing keys, it rejects every gRPC +caller, including the gateway. In both cases, check that `clusterOidcIssuer` +matches your cluster's issuer and that the Enterprise Manager pods can reach it. + +
+ +
+The console's Connectors pages return 401 + +The gateway's REST API rejected the console's access token. Check these causes: + +- `enterpriseConfig.controlPlane.incomingAuth.oidc` is unset. Without it, the + REST API accepts only tokens from the gateway's own authorization server. +- `audience` doesn't match the console token's `aud` claim, or the token is + opaque rather than a JWT. See + [Accept console tokens on the REST API](#accept-console-tokens-on-the-rest-api). +- With a `subjectClaim` other than `sub`, the console's access token lacks that + claim, or `subjectClaim` and the Enterprise Manager's `binding_claim` differ. + When they differ, users can't be resolved and sign-in through MCP clients + fails too. See + [Identify users by a claim other than `sub`](#identify-users-by-a-claim-other-than-sub). + +
+ +
+MCP clients stay at "needs authentication" + +If the authorization server rejects the client's sign-in with `invalid_scope`, +the client registered with `openid` and then requested `offline_access`. Set +`enterpriseConfig.authServer.baseline_client_scopes` to `openid` and +`offline_access` as shown in +[Enable the Connector Gateway](#enable-the-connector-gateway). + +
+ +
+AI Gateway requests fail or lose group labels + +- **Every virtual-key request fails with `503` and + `key store temporarily unavailable`.** The Enterprise Manager refused the API + key service. Check its subject on `saSubjectAllowlist`, its namespace, and its + `directory.insecure` setting, as described in + [Keep the AI Gateway's access](#keep-the-ai-gateways-access). +- **AI Gateway telemetry has no group labels, and no error is logged.** The main + processor's subject is missing from `saSubjectAllowlist`. + +
+ +
+The render fails or the gateway is refused after a name override + +The umbrella chart checks `meteringSaSubjectAllowlist` and +`toolhive-cloud-ui.connectorGatewayUrl` against the gateway's actual names and +fails the render when they differ. The error message prints the value to set. No +check covers `httpSaSubjectAllowlist`, so a stale entry there shows up only as +refused HTTP calls from the gateway to the Enterprise Manager. See +[Override the gateway's name](#override-the-gateways-name).
From f152cacf040cad5c310f20a345a963e9515e98dc Mon Sep 17 00:00:00 2001 From: Trey Date: Tue, 13 Oct 2026 07:44:50 -0700 Subject: [PATCH 5/6] Split new and existing KEK paths; add restart Addresses stacklok/docs-website#1227 review comments: - MEDIUM deployment.mdx (4254095833): place the credential-loss warning before the commands and separate new from existing installs - MEDIUM review body (5498084449): restart the Enterprise Manager after KEK rotation Co-Authored-By: Claude Opus 5.5 --- .../enterprise-platform/deployment.mdx | 44 ++++++++++++++----- 1 file changed, 32 insertions(+), 12 deletions(-) diff --git a/docs/platform/enterprise-platform/deployment.mdx b/docs/platform/enterprise-platform/deployment.mdx index 55a27743..7c2b30f3 100644 --- a/docs/platform/enterprise-platform/deployment.mdx +++ b/docs/platform/enterprise-platform/deployment.mdx @@ -616,13 +616,31 @@ enterprise-manager: For GitOps, replace `generate: true` with `existingSecret` naming a Secret you create. The Secret's `kek` key holds a JSON map from version number to a -base64-encoded 32-byte key, and the Enterprise Manager uses the highest version: +base64-encoded 32-byte key, and the Enterprise Manager uses the highest version. -```bash -kubectl create secret generic enterprise-manager-kek \ - --namespace stacklok-system \ - --from-literal=kek="{\"1\":\"$(openssl rand -base64 32)\"}" -``` +Replacing the key makes every credential encrypted with it permanently +unreadable, so create the Secret the way that matches your install: + +- **New install, with no credentials stored yet.** Generate a key: + + ```bash + kubectl create secret generic enterprise-manager-kek \ + --namespace stacklok-system \ + --from-literal=kek="{\"1\":\"$(openssl rand -base64 32)\"}" + ``` + +- **Existing install that ran with `generate: true`.** Copy the current key from + the chart-created Secret instead of generating a new one: + + ```bash + kubectl get secret stacklok-enterprise-enterprise-manager-directory-kek \ + --namespace stacklok-system -o jsonpath='{.data.kek}' | base64 -d > kek.json + kubectl create secret generic enterprise-manager-kek \ + --namespace stacklok-system \ + --from-file=kek=./kek.json + ``` + +Then reference the Secret in your values: ```yaml title="values.yaml" enterprise-manager: @@ -632,12 +650,14 @@ enterprise-manager: existingSecret: 'enterprise-manager-kek' ``` -Make this choice before the Enterprise Manager stores any credentials. A new key -makes every credential encrypted with the old one permanently unreadable. To -move an install that already ran with `generate: true`, copy the `kek` value -from the chart-created `-enterprise-manager-directory-kek` Secret -into your own Secret instead of generating a new key. To rotate the key, add a -higher-numbered entry to the map and keep the earlier ones. +To rotate the key, add a higher-numbered entry to the map and keep the earlier +ones. The Enterprise Manager reads the Secret only at startup, so restart it to +start using the new version: + +```bash +kubectl rollout restart deployment/stacklok-enterprise-enterprise-manager \ + --namespace stacklok-system +``` TLS is not optional here: the directory refuses a non-TLS peer at call time, so a key-encrypted install served over the cleartext port renders cleanly and then From e8800f906fdd4aad9cd750cba39cc9b987615dd8 Mon Sep 17 00:00:00 2001 From: Trey Date: Tue, 13 Oct 2026 07:44:57 -0700 Subject: [PATCH 6/6] Fix allowedClientSelector example nesting The chart wraps grpc.networkPolicy.allowedClientSelector in matchLabels itself, so the example's nested matchLabels key rendered an invalid label selector. Addresses stacklok/docs-website#1227 review comments: - LOW review body (5498084449): allowedClientSelector takes the label map directly Co-Authored-By: Claude Opus 5.5 --- docs/platform/enterprise-platform/deployment.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/platform/enterprise-platform/deployment.mdx b/docs/platform/enterprise-platform/deployment.mdx index 7c2b30f3..95e1d363 100644 --- a/docs/platform/enterprise-platform/deployment.mdx +++ b/docs/platform/enterprise-platform/deployment.mdx @@ -609,9 +609,9 @@ enterprise-manager: existingSecret: 'enterprise-manager-grpc-tls' networkPolicy: enabled: true + # The chart wraps this label map in matchLabels itself. allowedClientSelector: - matchLabels: - app.kubernetes.io/name: connector-gateway + app.kubernetes.io/name: connector-gateway ``` For GitOps, replace `generate: true` with `existingSecret` naming a Secret you