Add litellm provider (built-in and custom client kind) - #51
Open
wch wants to merge 6 commits into
Open
Conversation
Adds a first-class LiteLLM provider that targets the proxy's unified
/v1/messages endpoint (Anthropic-shaped) so explicit prompt caching and
thinking-block round-trips survive for Claude models behind the proxy.
- New provider id "litellm" in PROVIDER_IDS / BUILTIN_PROVIDER_IDS /
CLIENT_KIND_VALUES (built-in only; not a custom client kind)
- Model discovery via GET /v1/model/info with tolerant parsing (null
metadata for local upstreams, mode filtering that keeps unknown modes)
- Capability helper delegates Claude-family ids to the Anthropic table;
non-Claude upstreams get conservative defaults and no thinking levels
(LiteLLM's translation drops reasoning continuity for them)
- Chat reuses AnthropicClient pointed at {baseUrl}/v1
- Env mappings: LITELLM_API_KEY (secret), LITELLM_BASE_URL (connection)
The catalog pipeline forwards the raw connection baseUrl (e.g. http://localhost:4000) as a per-model routing override (params.baseUrl), which wins over the constructor URL inside AnthropicClient — so chat requests hit /messages instead of /v1/messages and 404. Wrap the client so every path into the proxy is normalized through litellmV1BaseUrl(). Found during in-app e2e verification; repro test fails without the fix.
Allow providers.custom entries with type "litellm" so organizations can register their own (often custom-branded) LiteLLM gateways under their own names, alongside or instead of the built-in litellm provider. - ai-credentials/ai-config: add "litellm" to the supported custom client kinds (apikey auth, key optional) and the providers.json schema's custom discriminated union (base connection fields only) - bridge: split litellm registration into shared fetcher/factory builders and export registerCustomLitellmProvider(registry, providerId, logger). The model fetcher is keyed by the custom id (independent per-gateway cache, id stamped into discovered models); the client factory is registered under the kind key "litellm" only, so chat routing goes through the clientKind fallback and cannot go stale when an entry's type changes on live reload - drop hard-coded kind counts from schema comments and memory-bank prose
…soning via /v1/responses, rest via /v1/chat/completions - classifyLitellmModel in ai-config: single source of truth for alias family (underlying id trusted; alias fallback only without metadata; OpenAI needs a provider signal) - getLitellmModelCapabilities gains an OpenAI branch (thinking effort levels gated in by the verified stateless encrypted-reasoning round-trip on LiteLLM 1.95.0 /v1/responses) - fetcher stamps per-alias protocol; client factory returns a dispatching client with AnthropicClient + OpenAIClient delegates, undefined protocol keeps the Anthropic-shaped route, customHeaders flow to both delegates
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
litellmprovider so LiteLLM gateways work with full Anthropic-protocol features (explicit prompt-cache breakpoints, thinking-block preservation) instead of the degraded openai-compatible path. The provider targets LiteLLM's unified/v1/messagesendpoint by reusingAnthropicClientwith a normalized base URL, and discovers models from/v1/model/info. Consumed by posit-dev/assistant#1876's implementation PR.Changes
litellmprovider ID and registrar (registerLitellmProvider); client factory wrappingAnthropicClientthat normalizes both the constructor base URL and the per-requestparams.baseUrlrouting override throughlitellmV1BaseUrl()(the raw override previously won and chat 404'd); cached/v1/model/infofetcher tolerant of null-metadata entries (Ollama-backed aliases) with empty fallback models; a second registrarregisterCustomLitellmProvider(registry, providerId, logger)for custom gateway entries — per-id model fetchers with the custom id stamped into discovered models and independent caches, while the client factory is registered idempotently under the kind key"litellm"so chat resolution goes through theclientKindfallback and cannot go stale when an entry's type changes on live catalog reload.litellmin the built-in provider vocabulary, schema block (baseUrl,customHeaders,enabled, …), and supported custom client kinds; capability helper that delegates Claude-family models (including Bedrock inference-profile-prefixed IDs) togetAnthropicModelCapabilitiesand falls back to conservative defaults with thinking disabled for other upstreams;LITELLM_BASE_URLconnection-env mapping; regeneratedproviders.schema.json.litellmcustom-kind auth descriptor (apikey, key optional) andLITELLM_API_KEYenv mapping.