diff --git a/app/authzed/concepts/rate-limiting/page.mdx b/app/authzed/concepts/rate-limiting/page.mdx index 823d8de9..cb96501e 100644 --- a/app/authzed/concepts/rate-limiting/page.mdx +++ b/app/authzed/concepts/rate-limiting/page.mdx @@ -298,6 +298,7 @@ The `unit` field supports: - `"day"` You can also specify custom durations using Go duration syntax (e.g., `"30s"`, `"15m"`, `"2h"`, `"90s"`). +Custom durations require SpiceDB Enterprise v1.57.0 or later; earlier versions reject them with `invalid unit`. ### Self-Hosted Configuration @@ -305,19 +306,27 @@ The following sections apply only to self-hosted SpiceDB Enterprise deployments. #### Basic Setup -For self-hosted SpiceDB Enterprise deployments, use the following command-line flag: +The rate limit extender is disabled by default. +It depends on the [Restricted API Access](/authzed/concepts/restricted-api-access) extender (`authzed-fgam`) for authentication, so enable both and point them at the same FGAM configuration file: -| Flag | Description | Default | -| --------------------- | --------------------------------------------------- | ------- | -| `--rate-limit-config` | Path to YAML file containing rate limit definitions | | +| Flag | Description | Default | +| ------------------------------ | ----------------------------------------------------------------------------------------- | ------- | +| `--extender-enabled` | Comma-separated list of extenders to enable; must include `authzed-fgam` and `rate-limit` | | +| `--extender-rate-limit-config` | Path to YAML file containing rate limit definitions | | -```bash /dev/null/self-hosted-basic.sh#L1-3 +```bash /dev/null/self-hosted-basic.sh#L1-5 spicedb serve \ - --rate-limit-config=/path/to/config.yaml \ + --extender-enabled=authzed-fgam,rate-limit \ + --extender-authzed-fgam-endpoint=file:///path/to/fgam.yaml \ + --extender-rate-limit-config=/path/to/fgam.yaml \ ... ``` -The YAML file follows the same format as shown in the configuration examples above. +The YAML file follows the same format as shown in the configuration examples above, with rate limits under the top-level `rate_limits` key. +Enabling `rate-limit` without `authzed-fgam` fails at startup with `referenced dependency does not exist on chain: authzed-fgam`. +Once `authzed-fgam` is enabled, API requests must authenticate with tokens defined in the FGAM file; the preshared key is no longer accepted. + +Matching or bucketing by `service_account` (including `serviceAccount` in CEL expressions) requires the corresponding tokens and service accounts to be defined in the same file. Matching by `role` also requires the role to be assigned to the service account there. #### Distributed Rate Limiting @@ -326,36 +335,46 @@ The YAML file follows the same format as shown in the configuration examples abo Enterprise deployments. AuthZed Dedicated handles this automatically. -For self-hosted deployments, you can enable distributed coordination across replicas using gossip for accurate global rate limits. +Gossip coordination across replicas is enabled by default, for accurate global rate limits. +Peers are discovered automatically through Kubernetes EndpointSlices, so gossip only works when SpiceDB runs in Kubernetes. + +Set `--extender-rate-limit-gossip-replicas` to the actual number of SpiceDB replicas. +When gossip is unavailable (or disabled), each replica falls back to safe mode and enforces its share of the limit (`limit / replicas`) with a reduced burst. -##### Enabling Gossip +Outside Kubernetes, replicas cannot discover each other, so disable gossip with `--extender-rate-limit-gossip-enabled=false` and rely on safe mode. + +##### Configuring Gossip ```bash /dev/null/gossip-flags.sh#L1-10 spicedb serve \ - --rate-limit-config=/path/to/config.yaml \ - --rate-limit-gossip-enabled=true \ - --rate-limit-gossip-listen-addr=:6000 \ - --rate-limit-gossip-target-service=spicedb \ - --rate-limit-gossip-port-name=gossip \ - --rate-limit-gossip-replicas=3 \ - --rate-limit-gossip-use-dispatch-tls=true \ + --extender-enabled=authzed-fgam,rate-limit \ + --extender-authzed-fgam-endpoint=file:///path/to/fgam.yaml \ + --extender-rate-limit-config=/path/to/fgam.yaml \ + --extender-rate-limit-gossip-listen-addr=:6000 \ + --extender-rate-limit-gossip-target-service=spicedb \ + --extender-rate-limit-gossip-port-name=gossip \ + --extender-rate-limit-gossip-replicas=3 \ + --extender-rate-limit-gossip-use-dispatch-tls=true \ ... ``` +If no TLS certificates are configured, gossip uses TLS without certificate verification. +For mutual TLS between replicas, set `--extender-rate-limit-gossip-tls-cert`, `--extender-rate-limit-gossip-tls-key`, and `--extender-rate-limit-gossip-tls-ca`, or use `--extender-rate-limit-gossip-use-dispatch-tls`. + #### Gossip Configuration Flags -| Flag | Default | Description | -| -------------------------------------- | --------- | ------------------------------------------- | -| `--rate-limit-gossip-enabled` | `false` | Enable distributed rate limiting via gossip | -| `--rate-limit-gossip-listen-addr` | `:6000` | Address for gossip connections | -| `--rate-limit-gossip-target-service` | `spicedb` | Kubernetes service name for peer discovery | -| `--rate-limit-gossip-port-name` | `""` | Port name to use for peer addresses | -| `--rate-limit-gossip-replicas` | `1` | Number of replicas for rate division | -| `--rate-limit-gossip-use-dispatch-tls` | `false` | Use dispatch TLS certificates for gossip | -| `--rate-limit-gossip-tls-cert` | `""` | TLS certificate for gossip | -| `--rate-limit-gossip-tls-key` | `""` | TLS key for gossip | -| `--rate-limit-gossip-tls-ca` | `""` | TLS CA for mutual TLS | -| `--rate-limit-gossip-tls-server-name` | `""` | Server name for TLS verification | +| Flag | Default | Description | +| ----------------------------------------------- | -------- | --------------------------------------------------------------------------------------------------- | +| `--extender-rate-limit-gossip-enabled` | `true` | Enable distributed rate limiting via gossip | +| `--extender-rate-limit-gossip-listen-addr` | `:6000` | Address for gossip connections | +| `--extender-rate-limit-gossip-target-service` | `""` | Kubernetes service name for peer discovery (defaults to the `authzed.com/cluster` Kubernetes label) | +| `--extender-rate-limit-gossip-port-name` | `gossip` | Port name to use for peer addresses | +| `--extender-rate-limit-gossip-replicas` | `1` | Number of replicas for rate division | +| `--extender-rate-limit-gossip-use-dispatch-tls` | `false` | Use dispatch TLS certificates for gossip | +| `--extender-rate-limit-gossip-tls-cert` | `""` | TLS certificate for gossip | +| `--extender-rate-limit-gossip-tls-key` | `""` | TLS key for gossip | +| `--extender-rate-limit-gossip-tls-ca` | `""` | TLS CA for mutual TLS | +| `--extender-rate-limit-gossip-tls-server-name` | `""` | Server name for TLS verification (defaults to `{service}.{namespace}.svc.cluster.local`) | #### Monitoring @@ -379,7 +398,7 @@ When a rate limit is exceeded, the API returns: - **Response Trailers**: - `x-ratelimit-id`: The rate limit ID that was exceeded - `x-ratelimit-key`: The bucket key -- `retry-after`: Seconds until the client can retry +- `retry-after`: Seconds until the client can retry (omitted when zero) Example error handling in Go: @@ -410,7 +429,8 @@ if err != nil { #### Rate Limits Not Applied -- Verify the configuration file is being loaded with `--rate-limit-config` +- Verify both extenders are enabled with `--extender-enabled=authzed-fgam,rate-limit` and the configuration file is set with `--extender-rate-limit-config` +- On startup, look for `rate limit configuration loaded successfully` in the logs; `rate limit extender disabled: no config path provided` means the config flag is missing - Check logs for configuration parsing errors - Ensure match criteria are correctly specified (arrays for endpoints, service accounts, etc.) @@ -423,7 +443,7 @@ if err != nil { #### Rate Limits Too Restrictive in Safe Mode -- Increase `--rate-limit-gossip-replicas` if it doesn't match actual deployment +- Ensure `--extender-rate-limit-gossip-replicas` matches the actual number of replicas (a value that's too high makes limits stricter) - Fix gossip connectivity to enable coordinated mode - Consider adjusting base rate limits to account for safe mode operation