Add multi-protocol Portkey gateway provider - #54
Merged
Merged
Conversation
A first-class portkey provider routing each model over its natural wire protocol against one gateway, in two modes derived from the required base URL (the URL determines what the stored key is, so it can never be defaulted): - Hosted (exact https://api.portkey.ai origin only): key sent as x-portkey-api-key; paginated Model Catalog discovery of @slug/model ids via a new fetchFresh seam on createCachedModelFetcher. Conservatively stamps the Claude family only until the hosted Phase 0 probes run (TODO(phase0-gate) markers); other families are excluded from discovery rather than shipped unverified. - Self-hosted OSS (any other URL): the key is the single upstream's key, no discovery (GET /v1/models is broken on the OSS gateway, re-verified 2026-08-08); models declared via models.custom with optional per-model protocol; x-portkey-provider defaults to anthropic with user-supplied routing headers winning. resolvePortkeyConnection is the single owner of URL classification, /v1 normalization, secret-header sanitization (x-portkey-api-key / x-portkey-virtual-key stripped case-insensitively from customHeaders), and the chat/discovery header split (routing headers chat-only). The protocol-dispatching client mirrors the litellm dispatcher (extraction deferred until a third gateway provider) and rejects per-request baseUrl overrides that don't normalize to the connection's gateway, closing a credential-redirection hazard. Also: stripCatalogSlug in ai-config as the shared @slug/ convention owner; classifyPortkeyModel decision object + capability helpers with an ID-only Claude-or-conservative rule (TODO(phase1)); clarifyBlankRequestError in the shared AI-SDK seam so blank APICallError/RetryError messages (e.g. Portkey's {"html-message":...} 401 and {"status":"failure"} envelopes) surface status + body excerpt for every provider; PORTKEY_API_KEY / PORTKEY_BASE_URL env mappings. Backed by 2026-08-08 OSS gateway probes (1.15.2): tools/streaming/ images pass for anthropic, openai, and gemini upstreams; /v1/responses stateless reasoning continuity passes for openai; Bedrock translation preserves cache_control and thinking signatures.
… guard Review findings on the initial portkey provider: - Discovery headers now pass through the shared additive-header filter (additiveHeaderRecord). The provider-owned fetchFresh fetch bypasses the cached fetcher's merge, so SDK-managed headers (Authorization, x-api-key, ...) placed in customHeaders were reaching hosted discovery even though the chat delegates strip them. Mixed-case regression fixtures added. - The pagination "no-progress guard" only detected an empty page. A server that ignores offset and repeats a non-empty page would duplicate models and keep requesting until the model bound. The guard now tracks seen model ids and stops when a page adds none, with an explicit page-count bound (100) and truncation at the 10k-model bound for oversized pages. Both regression tests verified failing on the unfixed code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a first-class
portkeyprovider so organizations routing LLM traffic through a Portkey gateway can use it with each model speaking its natural wire protocol, mirroring the landed LiteLLM provider. Two deployment shapes are supported: hosted Portkey (https://api.portkey.ai/v1, Portkey API key, Model Catalog@provider/modelids with automatic discovery) and a self-hosted OSS gateway (the stored key is one upstream's key, models are user-declared). Claude models route over the Anthropic-shaped/v1/messagespassthrough so explicit prompt-cache breakpoints and thinking signatures survive; other families route over the OpenAI-shaped endpoints. The provider is conservative where verification is still pending: hosted discovery currently stamps Claude-family models only, because hosted Portkey behavior could not be probed without an account.Portkey can also back
providers.custom.<name>entries withtype: "portkey". The custom provider id owns model-fetch/cache identity and model stamping, while chat resolves through the factory registered under theportkeyclient kind, so live type changes follow the current catalog entry. Current user surfaces support a keyless, noncanonical self-hosted gateway or credential-injecting front proxy: models use bare upstream ids declared inmodels.custom, and the ordinary self-hosted path performs no HTTP discovery. A securely stored key for a custom id can reach the shared hosted fetcher, but the canonical hosted origin always requires a non-empty key and there is no custom-id credential UI or environment mapping yet.Changes
portkeyadded to the provider/client-kind vocabulary, built-in schema/catalog builder, and the base-onlyproviders.customdiscriminated union (withproviders.schema.jsonregenerated);stripCatalogSlugexported as the single owner of the@slug/id convention;model-capabilities/portkey-helpers.tswithclassifyPortkeyModel(one per-entry decision object — family, capability model id, supported/excluded with reason, protocol — consumed by both discovery filtering and capability stamping) andgetPortkeyModelCapabilities(Anthropic/OpenAI helper delegation, conservative fallback); canonical hosted base-URL constant in the base-URL policy module;PORTKEY_BASE_URLconnection env mapping.providers/portkey-provider.tsbuilt around a singleresolvePortkeyConnectionseam that both the client factory and model fetchers consume — required base URL, hosted-vs-OSS classification (exact canonical HTTPS origin only; lookalike hosts classify as OSS,http://api.portkey.aiis a local error before any request), mode-dependent key validation (hosted requires a non-empty key; OSS permits an empty key for an injecting proxy),/v1normalization, and case-insensitive provider-local filtering ofx-portkey-api-key/x-portkey-virtual-keyfromcustomHeaders. The chat client is a protocol-dispatching wrapper over oneAnthropicClientand oneOpenAIClientdelegate (mirroring the LiteLLM dispatcher), with a same-gateway check that rejects per-requestbaseUrloverrides pointing at a different host before any request is made. Model discovery uses thefetchFreshseam oncreateCachedModelFetcherso Portkey owns its hosted pagination loop (with an atomic no-progress guard), the OSS no-fetch short-circuit, and the chat-vs-discovery header split (routing headers never sent toGET /v1/models).registerCustomPortkeyProviderregisters a custom-id fetcher with independent cache/model identity and the shared kind-keyed client factory. Also fixes blank AI-SDK error messages (clarifyBlankRequestErrorinai-sdk-helpers.ts) surfaced by Portkey's error shapes, benefiting all AI-SDK clients.PORTKEY_API_KEYenv mapping for the built-in provider, plus the customportkeyauth descriptor (apikey, optional at catalog-synthesis time so a base-URL-only OSS/front-proxy connection can run).Design decisions
PORTKEY_BASE_URLand the configure form.resolvePortkeyConnectiondecides whether that credential is valid for the resolved deployment mode: hosted rejects empty/whitespace keys locally, while OSS accepts an empty key for a credential-injecting proxy. Discovery and chat consume the same rule.clientKind, avoiding a stale custom-id factory after a liveproviders.jsontype change.x-portkey-api-keyon both delegates) does not parameterize cleanly; the convention stays documented inproviderGuide.md.Verification
Probed against a local OSS gateway (1.15.2): streaming
/v1/messagespassthrough,/v1/chat/completionswith tools across OpenAI/Anthropic/Gemini upstreams, stateless encrypted-reasoning continuity over/v1/responses, and Bedrock passthrough with cache and thinking intact. Custom-provider tests cover schema/catalog preservation, custom-id model stamping and Portkey capability inference, operation without the built-in registrar, keyless OSS no-fetch behavior, and local rejection of keyless hosted discovery/chat. ai-lib check-types (including the shape guards), lint, and all package test suites pass.Not included
Known limitations
Content-Encoding: gzip. A fixed gateway release resolves both upstream.