From 4ad917015741e9d08b3815de72072499c9828d58 Mon Sep 17 00:00:00 2001 From: Derek Tu Date: Wed, 22 Jul 2026 10:15:24 -0700 Subject: [PATCH 1/3] Add AI Gateway providers and models doc page New public-facing reference listing supported LLM providers (OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, Google Vertex AI, Google Gemini), model routing, API compatibility, and provider-agnostic capabilities. Content verified against the enterprise stacklok-llm-gateway codebase. Co-Authored-By: Claude Opus 4.8 --- .../supported-providers-and-models.mdx | 164 ++++++++++++++++++ 1 file changed, 164 insertions(+) create mode 100644 docs/ai-gateway/supported-providers-and-models.mdx diff --git a/docs/ai-gateway/supported-providers-and-models.mdx b/docs/ai-gateway/supported-providers-and-models.mdx new file mode 100644 index 00000000..f69ea214 --- /dev/null +++ b/docs/ai-gateway/supported-providers-and-models.mdx @@ -0,0 +1,164 @@ +--- +title: Supported providers and models +sidebar_label: Providers and models +description: + LLM providers, model APIs, and capabilities supported by the Stacklok AI + Gateway, and how to route models to provider backends. +--- + +The Stacklok AI Gateway is a self-hosted proxy that sits between your +applications and the large language model (LLM) providers they call. It gives +platform teams a single, policy-enforced entry point for AI traffic, with +guardrails, budgets, authentication, and observability, while letting developers +keep using the OpenAI and Anthropic API formats they already know. + +This page lists the providers and model APIs the gateway supports and explains +how models are routed to provider backends. + +## Supported providers + +Providers are configured declaratively. You register each provider backend once, +with its endpoint and credentials, and the gateway handles routing, policy, and +metering for every request. + +| Provider | API schema | Authentication | +| -------------------------- | -------------------------- | ----------------------- | +| OpenAI | `OpenAI` | API key | +| Anthropic | `Anthropic` | API key | +| AWS Bedrock | `AWSBedrock` | AWS IAM (role + region) | +| Azure OpenAI | `AzureOpenAI` | API key | +| Google Vertex AI | `GCPVertexAI` | Google service account | +| Google Gemini (AI Studio) | `GeminiAIStudio` | API key | +| Google Gemini (Gemini API) | `GoogleGenerativeLanguage` | API key | + +Google Gemini can be reached two ways: through Google AI Studio's +OpenAI-compatible endpoint (`GeminiAIStudio`), or through the native Gemini API +at `generativelanguage.googleapis.com` (`GoogleGenerativeLanguage`). Both +authenticate with an API key. Google Vertex AI (`GCPVertexAI`) connects to a +regional Vertex endpoint and authenticates with a Google service account. + +You can also point the OpenAI schema at any OpenAI-compatible endpoint +(self-hosted or third-party) by setting a custom hostname and API key. This is +useful for gateways, inference servers, and open-model providers that expose the +OpenAI API. + +:::info[Need a provider that isn't listed?] + +The gateway uses a pluggable provider model, and Stacklok adds backends based on +customer demand. If you need a provider that isn't listed here, contact your +Stacklok representative. + +::: + +## Supported models + +The gateway does not maintain a fixed list of models. Any model your configured +provider exposes is available through the gateway. You map model names to +provider backends in configuration, and new models work as soon as the provider +offers them, with no gateway upgrade required. + +You route models to providers by declaring routes. For example: + +```yaml +routes: + - name: gpt-4o + match: + model: 'gpt-4o' + backendRefs: + - provider: openai + - name: claude + match: + model: 'claude-sonnet-4-5' + backendRefs: + - provider: anthropic + - name: gemini + match: + model: 'gemini-2.5-pro' + backendRefs: + - provider: gemini + - name: default + backendRefs: + - provider: openai +``` + +Requests are matched on the requested model name and forwarded to the +corresponding provider. Any request whose model does not match a specific route +falls through to the default route. + +For teams that want to restrict which models a given group or application may +use, the gateway supports per-principal model allow-lists through its policy +layer. This is optional. Without a policy, routing is unrestricted. + +## API compatibility + +Applications call the gateway using standard provider APIs. No client SDK +changes are required beyond pointing the base URL at the gateway. + +| API | Endpoint | Streaming | +| ----------------------- | ------------------------ | :-------: | +| OpenAI Chat Completions | `/v1/chat/completions` | Yes | +| Anthropic Messages | `/anthropic/v1/messages` | Yes | + +Server-sent event (SSE) streaming is fully supported for both APIs. Requests to +Google and Azure providers are accepted in these same formats and translated to +the provider's native API by the gateway. + +## Capabilities across every provider + +The following capabilities are provider-agnostic. They apply to all traffic +routed through the gateway, regardless of which provider serves the request: + +- **Sensitive-data guardrails**: detect and block or redact personally + identifiable information (PII) and payment card (PCI) data in prompts, with + built-in detectors for identifiers such as credit cards, national IDs, emails, + and phone numbers, plus custom patterns and optional machine-learning + detection. +- **Token budgets and rate limiting**: enforce per-team, per-application, or + per-model token budgets. +- **Failover and resilience**: weighted load balancing, priority-based failover + between providers, retries, circuit breaking, and health checks. +- **Authentication**: OpenID Connect (OIDC) authentication with role-based + authorization, plus virtual API keys for issuing scoped, revocable credentials + to teams. +- **Observability**: OpenTelemetry traces, access logs, token-usage metrics, and + prebuilt Grafana dashboards. +- **Interaction journaling**: optional capture of prompts and responses for + audit and compliance. + +## How to add a provider + +1. Store the provider credential as a Kubernetes Secret. +2. Add the provider to your gateway configuration with its schema, endpoint, and + credential reference. +3. Add at least one route that references the provider. +4. Apply the configuration. The gateway reconciles the backend automatically. + +```yaml +providers: + - name: openai + schema: OpenAI + endpoint: + hostname: api.openai.com + port: 443 + credentials: + type: APIKey + secretRef: + name: openai-key + key: apiKey +routes: + - name: default + backendRefs: + - provider: openai +``` + +Credentials can be rotated in place by updating the Secret. No configuration +change or restart is required. + +## Deployment requirements + +The AI Gateway is Kubernetes-native and deployed with a Helm-installed operator. +It is built on Envoy AI Gateway and Envoy Gateway, which the operator installs +and manages for you. + +For full installation and day-2 operations guidance, see the operator +documentation. From 4cc78d20c1b256674931d691bb9ce85df7d5b94f Mon Sep 17 00:00:00 2001 From: Derek Tu Date: Wed, 22 Jul 2026 13:31:29 -0700 Subject: [PATCH 2/3] Add supported clients section to AI Gateway page Documents which AI coding tools connect and how: token-helper tools (Claude Code, Gemini CLI, Codex CLI) authenticate directly, static-key tools (Cursor, VS Code, Cline, etc.) connect via ToolHive's thv llm localhost proxy. Verified against the enterprise connecting-clients runbook. Co-Authored-By: Claude Opus 4.8 --- .../supported-providers-and-models.mdx | 40 +++++++++++++++++-- 1 file changed, 37 insertions(+), 3 deletions(-) diff --git a/docs/ai-gateway/supported-providers-and-models.mdx b/docs/ai-gateway/supported-providers-and-models.mdx index f69ea214..9e8f485a 100644 --- a/docs/ai-gateway/supported-providers-and-models.mdx +++ b/docs/ai-gateway/supported-providers-and-models.mdx @@ -103,6 +103,41 @@ Server-sent event (SSE) streaming is fully supported for both APIs. Requests to Google and Azure providers are accepted in these same formats and translated to the provider's native API by the gateway. +## Supported clients + +Because the gateway speaks the standard OpenAI and Anthropic APIs, any tool that +can target a custom base URL works with it. The gateway authenticates callers +with OpenID Connect (OIDC) tokens, and AI coding tools fall into two groups +based on how they handle authentication: + +- **Token-helper tools** run a command on each request and use its output as the + bearer token, so they authenticate against the gateway directly. +- **Static-key tools** accept only a fixed API key and base URL and cannot + perform an OIDC login on their own. They connect through a small localhost + proxy that injects the token for them. + +| Client | Auth style | How it connects | +| ----------- | ------------ | --------------------------- | +| Claude Code | Token helper | Directly (`apiKeyHelper`) | +| Gemini CLI | Token helper | Directly | +| Codex CLI | Token helper | Directly (`auth.command`) | +| avante.nvim | Token helper | Directly (`cmd:` prefix) | +| Cursor | Static key | Through the localhost proxy | +| VS Code | Static key | Through the localhost proxy | +| Roo Code | Static key | Through the localhost proxy | +| Cline | Static key | Through the localhost proxy | +| Continue | Static key | Through the localhost proxy | +| Aider | Static key | Through the localhost proxy | +| Zed | Static key | Through the localhost proxy | +| Xcode | Static key | Through the localhost proxy | + +[ToolHive](https://github.com/stacklok/toolhive) bridges both paths through its +`thv llm` command: it supplies fresh tokens to token-helper tools and runs the +localhost proxy that injects credentials for static-key tools. A single +`thv llm setup` can auto-configure detected tools. The list above is +representative, not exhaustive: any tool that fits one of the two auth styles +can connect the same way. + ## Capabilities across every provider The following capabilities are provider-agnostic. They apply to all traffic @@ -117,9 +152,8 @@ routed through the gateway, regardless of which provider serves the request: per-model token budgets. - **Failover and resilience**: weighted load balancing, priority-based failover between providers, retries, circuit breaking, and health checks. -- **Authentication**: OpenID Connect (OIDC) authentication with role-based - authorization, plus virtual API keys for issuing scoped, revocable credentials - to teams. +- **Authentication**: OIDC authentication with role-based authorization, plus + virtual API keys for issuing scoped, revocable credentials to teams. - **Observability**: OpenTelemetry traces, access logs, token-usage metrics, and prebuilt Grafana dashboards. - **Interaction journaling**: optional capture of prompts and responses for From 5c455ab644664f1c9fb5fbf8088af169f906da48 Mon Sep 17 00:00:00 2001 From: Derek Tu Date: Wed, 22 Jul 2026 13:53:37 -0700 Subject: [PATCH 3/3] Clarify clients and providers are independent Note that the gateway routes by model and translates API formats, so any client can reach any configured provider, and add same-API-family guidance for single-route failover. Co-Authored-By: Claude Opus 4.8 --- .../ai-gateway/supported-providers-and-models.mdx | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/ai-gateway/supported-providers-and-models.mdx b/docs/ai-gateway/supported-providers-and-models.mdx index 9e8f485a..2cad9478 100644 --- a/docs/ai-gateway/supported-providers-and-models.mdx +++ b/docs/ai-gateway/supported-providers-and-models.mdx @@ -138,6 +138,21 @@ localhost proxy that injects credentials for static-key tools. A single representative, not exhaustive: any tool that fits one of the two auth styles can connect the same way. +### Clients and providers are independent + +Clients are not tied to a specific provider. The gateway routes each request to +a provider by model name and translates between API formats, so any client can +reach any provider you have configured. For example, a tool that speaks the +OpenAI API can call an Anthropic, Bedrock, or Gemini model, and the gateway +handles the translation. The authentication style a client uses (token helper or +static key) is likewise independent of which provider serves the request. + +One point to keep in mind when configuring automatic failover: within a single +route, keep the backends in the same API family (for example, OpenAI with an +OpenAI-compatible provider, or Anthropic with an Anthropic-compatible provider). +To spread traffic across providers that speak different APIs, put them on +separate routes. + ## Capabilities across every provider The following capabilities are provider-agnostic. They apply to all traffic