diff --git a/docs/platform/enterprise-platform/configure-ai-gateway.mdx b/docs/platform/enterprise-platform/configure-ai-gateway.mdx index 84b9c743..41e17fbf 100644 --- a/docs/platform/enterprise-platform/configure-ai-gateway.mdx +++ b/docs/platform/enterprise-platform/configure-ai-gateway.mdx @@ -21,6 +21,11 @@ resources in the order described below. See It is optional and off by default. See [PCI/PII controls](../../ai-gateway/pci-pii-controls.mdx). +If the Enterprise Manager authenticates gRPC callers, as it must when the +Connector Gateway is enabled, also allow the AI Gateway's ServiceAccounts as +described in +[Keep the AI Gateway's access](./configure-connector-gateway.mdx#keep-the-ai-gateways-access). + ## Enable it Set the install toggle in your platform values and upgrade: diff --git a/docs/platform/enterprise-platform/configure-connector-gateway.mdx b/docs/platform/enterprise-platform/configure-connector-gateway.mdx index 60371112..126c6cce 100644 --- a/docs/platform/enterprise-platform/configure-connector-gateway.mdx +++ b/docs/platform/enterprise-platform/configure-connector-gateway.mdx @@ -17,6 +17,8 @@ embed those names, so adjust them if yours differ. ## Prerequisites +- **Stacklok Enterprise platform chart v0.22.0 or later.** Earlier charts don't + pass the `global.redis` settings to the gateway. - **A corporate identity provider** at `global.stacklok.primaryIdp`. The gateway resolves every caller to a directory user before it decides what they can see. See [Configure identity](./configure-identity.mdx). @@ -39,7 +41,9 @@ embed those names, so adjust them if yours differ. chart doesn't deploy an instance. Configure it once under `global.redis`, as described in [Global Redis/Valkey defaults](./deployment.mdx#global-redisvalkey-defaults), - and the gateway inherits the host, port, and password Secret. + and the gateway inherits the host, port, username, password Secret, and TLS + setting. To connect as a least-privilege ACL user, see + [Restrict the Redis user](#restrict-the-redis-user). ## Prepare the gateway's secrets @@ -98,6 +102,11 @@ kubectl create secret generic enterprise-manager-grpc-ca \ --from-file=ca.crt=./ca.crt ``` +A copied Secret goes stale when cert-manager rotates the CA. For GitOps, keep +the copy in sync automatically instead, for example with a trust-manager +`Bundle` that targets a Secret, or issue the Enterprise Manager's certificate +from a cert-manager `CA` Issuer whose CA you manage and distribute yourself. + ## Enable the Connector Gateway Add the following to the values file you use for the platform. Merge the @@ -119,6 +128,7 @@ global: connector-gateway: vmcpConfig: + groupRef: 'connector-gateway' incomingAuth: type: oidc oidc: @@ -129,8 +139,16 @@ connector-gateway: schema_version: 'v0.1.0' allowed_audiences: - 'https:///gw/mcp' + baseline_client_scopes: + - openid + - offline_access storage: type: redis + controlPlane: + incomingAuth: + oidc: + issuer: '' # same as global.stacklok.primaryIdp.issuer + audience: '' directory: addr: 'stacklok-enterprise-enterprise-manager.stacklok-system.svc.cluster.local:9443' directoryTLS: @@ -148,12 +166,22 @@ connector-gateway: | `global.stacklok.connectorGateway.enabled` | Deploy the Connector Gateway | | `global.stacklok.connectorGatewayId` | Identify this gateway to the Enterprise Manager and console | | `global.stacklok.authServerIssuer` | Set the gateway's public authorization server URL | +| `vmcpConfig.groupRef` | Name the gateway in its status and logs. Required; any non-empty value works | | `vmcpConfig.incomingAuth` | Require tokens from the gateway's own authorization server on the MCP endpoint | | `enterpriseConfig.authServer` | Configure the embedded authorization server and its Redis storage | +| `enterpriseConfig.controlPlane` | Accept the corporate tokens the console sends to the gateway's REST API | | `enterpriseConfig.directory.addr` | Point the gateway at the Enterprise Manager's TLS gRPC port | | `directoryTLS` | Mount the Enterprise Manager CA bundle and the gateway's ServiceAccount token | | `kek`, `authServerKeys` | Reference the Secrets from [Prepare the gateway's secrets](#prepare-the-gateways-secrets) | +Set `vmcpConfig.groupRef` to any non-empty name; the gateway refuses to start +while it's empty. The gateway reads its backends from the Enterprise Manager, so +no `MCPGroup` by that name needs to exist. + +Set `baseline_client_scopes` to `openid` and `offline_access` so MCP clients +that register through dynamic client registration, such as Claude Code, Cursor, +and VS Code, can obtain refresh tokens. + ### Choose the gateway ID and issuer Choose a stable `connectorGatewayId` such as `prod-eu` or `platform-staging`, @@ -173,27 +201,49 @@ and list the audience in `allowed_audiences` exactly as it appears in `incomingAuth.oidc.audience`. The gateway compares these strings exactly and refuses to start when they disagree. -:::note[Two different issuers] - -`global.stacklok.primaryIdp.issuer` identifies the corporate identity provider. -`authServerIssuer` identifies the Connector Gateway's authorization server. -Configure both values. See -[Configure platform identity](./configure-identity.mdx). - -::: - Omit `upstreams` from `enterpriseConfig.authServer`. The authorization server loads its identity providers from the Enterprise Manager: your corporate identity provider from `global.stacklok.primaryIdp`, plus the connector identity providers that admins register in the Enterprise Manager. -By default, the Enterprise Manager identifies users by the `sub` claim in your -corporate identity provider's tokens. If your provider identifies users with a -different claim, such as `uid` in Okta, add an -`enterprise-manager.directory.issuers` entry whose `issuer_url` matches -`global.stacklok.primaryIdp.issuer` exactly, and set `binding_claim`: +### Accept console tokens on the REST API + +The console calls the gateway's REST API with the user's corporate identity +provider access token. `enterpriseConfig.controlPlane.incomingAuth.oidc` +configures the validator that accepts those tokens. + +Set `issuer` to `global.stacklok.primaryIdp.issuer`; the gateway refuses to +start if it doesn't match an identity provider it signs users in with. Set +`audience` to the exact `aud` value of the access tokens the console sends. The +console sends the access token from its own sign-in, so this is the audience +your identity provider assigns to tokens for the console's OAuth client, such as +`cloud-ui` in the Okta example in +[Configure platform identity](./configure-identity.mdx). To confirm it, decode +an access token issued to that client as shown in +[Verify a token](./configure-identity.mdx#verify-a-token). Prefer an audience +that only Stacklok components accept over a shared default such as Okta's +`api://default`, which would let tokens minted for unrelated applications reach +the gateway. The token must be a JWT, so use an authorization server that issues +JWT access tokens rather than Okta's org authorization server. To fetch signing +keys from a URL other than the issuer's discovery document, also set `jwksUrl`. + +### Identify users by a claim other than `sub` + +By default, the gateway and the Enterprise Manager identify users by the `sub` +claim in your corporate identity provider's tokens. If your provider identifies +users with a different claim, such as `uid` in Okta, configure it on both sides. +Set `global.stacklok.primaryIdp.subjectClaim`, which the gateway uses at +sign-in, and add an `enterprise-manager.directory.issuers` entry whose +`issuer_url` matches `global.stacklok.primaryIdp.issuer` exactly, with the same +claim in `binding_claim`: ```yaml title="values.yaml" +global: + stacklok: + primaryIdp: + # issuer and client settings from "Configure identity" stay here. + subjectClaim: 'uid' + enterprise-manager: directory: issuers: @@ -203,7 +253,18 @@ enterprise-manager: binding_claim: 'uid' ``` -The same entry configures SCIM provisioning for that provider. See +The gateway reads the claim from the **ID token**, and it must be a top-level +string claim that also appears on refreshed ID tokens. Okta adds `uid` to access +tokens from a custom authorization server, but not to ID tokens. In Okta, add a +`uid` claim with the value `user.id` and the token type **ID Token**. Choose the +claim before users sign in, because it becomes each user's stable identity in +the directory. + +The console's access tokens must carry the same top-level claim, because the +REST API resolves users from them. Okta's custom authorization servers add `uid` +to access tokens by default. + +The same `issuers` entry configures SCIM provisioning for that provider. See [SCIM identity providers](./deployment.mdx#scim-identity-providers). ## Connect the gateway to the Enterprise Manager @@ -242,21 +303,54 @@ this command prints: kubectl get --raw /.well-known/openid-configuration | jq -r .issuer ``` +The Enterprise Manager fetches the issuer's discovery document and signing keys +directly from that URL, not through the Kubernetes API server. On managed +clusters such as Amazon EKS, the issuer is a public URL, so allow egress to it +from the Enterprise Manager pods. + Each allowlist covers different calls from the gateway. `saSubjectAllowlist` covers identity resolution and per-user connector configuration. `gatewaySaSubjectAllowlist` covers the gateway's startup registration and its catalog reads. `credentialSaSubjectAllowlist` covers credential resolution, which the gateway needs for the identity provider's client secret. The subject -is `system:serviceaccount::-connector-gateway`. If the -AI Gateway also calls the Enterprise Manager, keep its entries on -`saSubjectAllowlist` alongside this one. - -For a development cluster without TLS, set -`connector-gateway.enterpriseConfig.directory.insecure: true` in place of -`directoryTLS`, point `addr` at the cleartext port (`9091`), and set -`enterprise-manager.grpc.insecure: true`. The chart requires both settings -together. This transport carries credentials in cleartext with no caller -authentication, so reserve it for local testing. +is `system:serviceaccount::-connector-gateway`. The +chart fills in a fourth list, `meteringSaSubjectAllowlist`, from the same +subject for +[tool call recording](#turn-on-tool-call-recording-and-observability). If you +change the gateway's name, update these values as described in +[Override the gateway's name](#override-the-gateways-name). For a development +cluster without TLS, see +[Use a cleartext connection for development](#use-a-cleartext-connection-for-development). + +### Keep the AI Gateway's access + +Caller authentication covers both the TLS and cleartext gRPC ports, so turning +it on also gates the AI Gateway. Its API key service checks every virtual API +key against the Enterprise Manager. If your platform also runs the AI Gateway +with virtual API keys: + +- Create the `AIGateway` resource in the release namespace. The Enterprise + Manager's directory NetworkPolicy admits the API key service only from that + namespace. +- Add the API key service's subject to `saSubjectAllowlist`: + `system:serviceaccount:stacklok-system:-api-key-service`, + where `` is the name of your `AIGateway` resource. +- Keep `spec.auth.virtualAPIKeys.directory.insecure` on the `AIGateway` at its + default, `false`, so the API key service presents its ServiceAccount token. +- Unset `enterprise-manager.grpc.insecure` if you set it for the AI Gateway. It + skips caller authentication, and the Enterprise Manager refuses to start with + it alongside a directory key-encryption key. + +The chart refuses to render while the AI Gateway is enabled and no subject +ending in `-api-key-service` is on the list, even if you don't use virtual API +keys. In that case, set +`enterprise-manager.grpc.callerAuth.acknowledgeNoApiKeyServiceCaller: true` +instead of adding a subject. + +If you set `global.stacklok.directoryGrpcEndpoint` to add group labels to AI +Gateway telemetry, also add +`system:serviceaccount:stacklok-system:-main-processor` to +`saSubjectAllowlist`. ## Corporate identity provider settings @@ -264,12 +358,13 @@ The gateway reads these optional keys from `global.stacklok.primaryIdp`, in addition to the issuer and client settings in [Configure platform identity](./configure-identity.mdx): -| Key | Purpose | -| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `clientClaim` | Access-token claim that names the OAuth client. Empty reads `cid` or `client_id`. Allowed values are `cid`, `client_id`, `azp`, and `appid`; set `azp` for Dex, or `azp` or `appid` for Microsoft Entra ID | -| `clientClaimAllowUnsafe` | Boolean. Accepts a `clientClaim` outside the allowed values. A weaker claim can match tokens minted for a different client, and the gateway logs a warning at startup | -| `allowPrivateIps` | Boolean. Lets discovery and token exchange reach an identity provider on a private or link-local address. HTTPS is still required for non-localhost hosts | -| `caBundleRef` | ConfigMap or Secret holding the PEM CA bundle for an identity provider served by a private CA. Set exactly one of `configMapRef` or `secretRef`, each with a `name`, plus an optional `key` (default `ca.crt`) | +| Key | Purpose | +| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `clientClaim` | Access-token claim that names the OAuth client. Empty reads `cid` or `client_id`. Allowed values are `cid`, `client_id`, `azp`, and `appid`; set `azp` for Dex, or `azp` or `appid` for Microsoft Entra ID | +| `subjectClaim` | Claim that identifies each user, read from ID tokens at sign-in and from the console's access tokens. Defaults to `sub`. Must match the Enterprise Manager's `binding_claim`; see [Identify users by a claim other than `sub`](#identify-users-by-a-claim-other-than-sub) | +| `clientClaimAllowUnsafe` | Boolean. Accepts a `clientClaim` outside the allowed values. A weaker claim can match tokens minted for a different client, and the gateway logs a warning at startup | +| `allowPrivateIps` | Boolean. Lets discovery and token exchange reach an identity provider on a private or link-local address. HTTPS is still required for non-localhost hosts | +| `caBundleRef` | ConfigMap or Secret holding the PEM CA bundle for an identity provider served by a private CA. Set exactly one of `configMapRef` or `secretRef`, each with a `name`, plus an optional `key` (default `ca.crt`) | Set the boolean keys as unquoted `true` or `false`. The chart refuses a quoted string, and `--set-string` produces one. @@ -461,6 +556,68 @@ and 3. Direct users to the deployment-specific instructions in the console. See [Roll out gateway clients](./roll-out-gateway-clients.mdx). +## Advanced configuration + +These settings apply only to some installs. Configure them alongside the values +above. + +### Restrict the Redis user + +If the Connector Gateway is the only component that uses `global.redis`, you can +connect as a least-privilege ACL user. Set `global.redis.username` to that user, +and apply this reference ACL to your Redis or Valkey instance with your own +password. `global.redis.username` applies to every component that inherits +`global.redis`, so don't restrict the user this way while other components share +it. + +```text title="users.acl" +user default reset off +user cgw reset on > resetchannels ~thv:auth:* ~connector-gateway:* +@read +@write +@connection +@transaction -@admin -@dangerous -@scripting -script -function -fcall -fcall_ro +eval +evalsha -config -acl -flushall -flushdb -swapdb -replicaof -slaveof -cluster -failover -save -bgsave -bgrewriteaof -debug -shutdown -monitor -client|tracking -keys -randomkey -dbsize -scan +``` + +The `thv:auth:*` keys hold authorization server sessions, and the +`connector-gateway:*` keys hold the tool call recording queue, a check for the +gateway's encryption key, and client registration locks. The first line disables +the built-in `default` user, which otherwise grants full access to any +connection that falls back to it. + +### Use a cleartext connection for development + +For a development cluster without TLS, set +`connector-gateway.enterpriseConfig.directory.insecure: true` in place of +`directoryTLS`, point `addr` at the cleartext port (`9091`), and set +`enterprise-manager.grpc.insecure: true`. The chart requires both settings +together. This transport carries credentials in cleartext with no caller +authentication, so reserve it for local testing. The Enterprise Manager refuses +to start with `grpc.insecure: true` while `directory.kek` is configured, so this +transport works only on an install without the key-encryption key. + +### Override the gateway's name + +The umbrella chart computes several Enterprise Manager and console values from +the gateway's default name, `-connector-gateway`. If you set +`connector-gateway.fullnameOverride`, `nameOverride`, `serviceAccount.name`, or +`serviceAccount.create: false`, or change the control-plane Service, update +these values to match: + +| Setting | Replacement value | +| ------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------- | +| `enterprise-manager.grpc.callerAuth.saSubjectAllowlist`, `gatewaySaSubjectAllowlist`, `credentialSaSubjectAllowlist`, and `meteringSaSubjectAllowlist` | `system:serviceaccount::` | +| `enterprise-manager.grpc.callerAuth.httpSaSubjectAllowlist`, only when `global.stacklok.enterpriseManager.url` is set | The same subject, alongside any AI Gateway entry already on the list | +| `toolhive-cloud-ui.connectorGatewayUrl`, which enables the console's **Connectors** pages | `http://.:` | +| `enterprise-manager.grpc.networkPolicy.allowedClientSelector`, only with `nameOverride` | `app.kubernetes.io/name: ` | + +To read the names, render the chart with `helm template` and find the gateway's +ServiceAccount and its `-control-plane` Service. With `fullnameOverride`, the +ServiceAccount and Services take the override as their name; with +`nameOverride`, they become `-`. + +`nameOverride` also changes the pods' `app.kubernetes.io/name` label. The +Enterprise Manager's directory NetworkPolicy admits only pods labeled +`connector-gateway`, so renamed pods reach the gRPC ports only through +`grpc.networkPolicy`. If you don't enable that policy, use `fullnameOverride`, +which keeps the label. + ## Next steps - [Connector Gateway quickstart](../../connector-gateway/quickstart.mdx) to @@ -497,6 +654,72 @@ kubectl logs deployment/stacklok-enterprise-connector-gateway -n stacklok-system restart the gateway; it reads the bundle only at startup. - **Connection timeouts.** Check that the Enterprise Manager's gRPC NetworkPolicy admits the gateway pods, and that the gateway can reach the - Redis or Valkey host. + Redis or Valkey host. After a `nameOverride`, see + [Override the gateway's name](#override-the-gateways-name). + + + +
+The Enterprise Manager fails to start or rejects every gRPC caller + +The Enterprise Manager fetches the `clusterOidcIssuer` discovery document at +startup. If it can't, it exits with `cluster OIDC bootstrap failed`. If it +starts but later can't refresh the issuer's signing keys, it rejects every gRPC +caller, including the gateway. In both cases, check that `clusterOidcIssuer` +matches your cluster's issuer and that the Enterprise Manager pods can reach it. + +
+ +
+The console's Connectors pages return 401 + +The gateway's REST API rejected the console's access token. Check these causes: + +- `enterpriseConfig.controlPlane.incomingAuth.oidc` is unset. Without it, the + REST API accepts only tokens from the gateway's own authorization server. +- `audience` doesn't match the console token's `aud` claim, or the token is + opaque rather than a JWT. See + [Accept console tokens on the REST API](#accept-console-tokens-on-the-rest-api). +- With a `subjectClaim` other than `sub`, the console's access token lacks that + claim, or `subjectClaim` and the Enterprise Manager's `binding_claim` differ. + When they differ, users can't be resolved and sign-in through MCP clients + fails too. See + [Identify users by a claim other than `sub`](#identify-users-by-a-claim-other-than-sub). + +
+ +
+MCP clients stay at "needs authentication" + +If the authorization server rejects the client's sign-in with `invalid_scope`, +the client registered with `openid` and then requested `offline_access`. Set +`enterpriseConfig.authServer.baseline_client_scopes` to `openid` and +`offline_access` as shown in +[Enable the Connector Gateway](#enable-the-connector-gateway). + +
+ +
+AI Gateway requests fail or lose group labels + +- **Every virtual-key request fails with `503` and + `key store temporarily unavailable`.** The Enterprise Manager refused the API + key service. Check its subject on `saSubjectAllowlist`, its namespace, and its + `directory.insecure` setting, as described in + [Keep the AI Gateway's access](#keep-the-ai-gateways-access). +- **AI Gateway telemetry has no group labels, and no error is logged.** The main + processor's subject is missing from `saSubjectAllowlist`. + +
+ +
+The render fails or the gateway is refused after a name override + +The umbrella chart checks `meteringSaSubjectAllowlist` and +`toolhive-cloud-ui.connectorGatewayUrl` against the gateway's actual names and +fails the render when they differ. The error message prints the value to set. No +check covers `httpSaSubjectAllowlist`, so a stale entry there shows up only as +refused HTTP calls from the gateway to the Enterprise Manager. See +[Override the gateway's name](#override-the-gateways-name).
diff --git a/docs/platform/enterprise-platform/deployment.mdx b/docs/platform/enterprise-platform/deployment.mdx index 138f3f86..95e1d363 100644 --- a/docs/platform/enterprise-platform/deployment.mdx +++ b/docs/platform/enterprise-platform/deployment.mdx @@ -609,18 +609,78 @@ enterprise-manager: existingSecret: 'enterprise-manager-grpc-tls' networkPolicy: enabled: true + # The chart wraps this label map in matchLabels itself. allowedClientSelector: - matchLabels: - app.kubernetes.io/name: connector-gateway + app.kubernetes.io/name: connector-gateway +``` + +For GitOps, replace `generate: true` with `existingSecret` naming a Secret you +create. The Secret's `kek` key holds a JSON map from version number to a +base64-encoded 32-byte key, and the Enterprise Manager uses the highest version. + +Replacing the key makes every credential encrypted with it permanently +unreadable, so create the Secret the way that matches your install: + +- **New install, with no credentials stored yet.** Generate a key: + + ```bash + kubectl create secret generic enterprise-manager-kek \ + --namespace stacklok-system \ + --from-literal=kek="{\"1\":\"$(openssl rand -base64 32)\"}" + ``` + +- **Existing install that ran with `generate: true`.** Copy the current key from + the chart-created Secret instead of generating a new one: + + ```bash + kubectl get secret stacklok-enterprise-enterprise-manager-directory-kek \ + --namespace stacklok-system -o jsonpath='{.data.kek}' | base64 -d > kek.json + kubectl create secret generic enterprise-manager-kek \ + --namespace stacklok-system \ + --from-file=kek=./kek.json + ``` + +Then reference the Secret in your values: + +```yaml title="values.yaml" +enterprise-manager: + directory: + kek: + enabled: true + existingSecret: 'enterprise-manager-kek' +``` + +To rotate the key, add a higher-numbered entry to the map and keep the earlier +ones. The Enterprise Manager reads the Secret only at startup, so restart it to +start using the new version: + +```bash +kubectl rollout restart deployment/stacklok-enterprise-enterprise-manager \ + --namespace stacklok-system ``` TLS is not optional here: the directory refuses a non-TLS peer at call time, so a key-encrypted install served over the cleartext port renders cleanly and then -fails every credential resolution. Point callers at `grpc.tls.port`. If a -service mesh or cluster-wide policy already restricts the gRPC ports, set -`directory.kek.externallyManagedNetworkPolicy=true` to acknowledge that instead -of defining a policy here; there is no equivalent escape for TLS, because a mesh -terminates it in the sidecar and the peer still arrives without it. +fails every credential resolution. Point callers at `grpc.tls.port`. + +With `directory.enabled: true`, the chart also renders a +`-enterprise-manager-directory` NetworkPolicy that limits which +pods reach the Enterprise Manager's gRPC ports. It admits these pods, all from +the release namespace: + +- Connector Gateway pods, on both gRPC ports +- The AI Gateway's API key service and main processor, on the cleartext port +- The AI Gateway main processor and operator, on the webhook port, when + `webhookTLS.enabled` is `true` + +NetworkPolicies are additive, so the `grpc.networkPolicy` policy in the example +can only admit more pods. It admits pods in the release namespace that match +`allowedClientSelector`, on both gRPC ports. Keep the selector to the Connector +Gateway's labels as shown. The chart requires either `grpc.networkPolicy` or, +when a service mesh or cluster-wide policy already restricts the gRPC ports, +`directory.kek.externallyManagedNetworkPolicy: true` to acknowledge it. There is +no equivalent escape for TLS, because a mesh terminates it in the sidecar and +the peer still arrives without it. #### SCIM identity providers @@ -687,8 +747,12 @@ global: existingSecret: 'redis-auth' # Defaults to redis-password when omitted. existingSecretKey: 'redis-password' + # Optional ACL username. Omit it to authenticate with the password only. + username: '' ``` +The chart refuses to render when `username` is set without `existingSecret`. + The ToolHive operator also accepts `toolhive-operator.upstream.operator.defaultRedis.existingSecret` and `existingSecretKey` for a component-specific credential, which take precedence