feat(pi): OSS mlflow provider + broaden OSS cohort (reasoning, limits)#217
Open
dgokeeffe wants to merge 4 commits into
Open
feat(pi): OSS mlflow provider + broaden OSS cohort (reasoning, limits)#217dgokeeffe wants to merge 4 commits into
dgokeeffe wants to merge 4 commits into
Conversation
…h reasoning/limits Builds on the merged OpenCode OSS support (kimi/glm) to close the remaining gaps: - Pi had NO OSS provider. Add databricks-mlflow (api openai-completions -> /ai-gateway/mlflow/v1) so Pi can select OSS chat models, with a local SSE-repair proxy (_mlflow_proxy) for models like inkling whose gateway stream omits finish_reason (Pi's parser rejects that; OpenCode's @ai-sdk/openai tolerates it, so it needs no proxy). - Broaden _OSS_MODEL_FAMILIES from kimi/glm to the full chat-completions cohort (inkling, llama, qwen, gpt-oss, gemma), excluding non-chat services (embeddings/rerankers) that share a family substring (e.g. qwen3-embedding). - Correct + extend _MODEL_TOKEN_LIMITS from probed gateway caps: glm is 1M context / 65536 output (was 200k/25k); per-family output ceilings for the cohort. Add model_is_reasoning() from probed openai_reasoning capability; Pi sets reasoning:true so it renders streamed reasoning_content as thinking. Both agents read the shared model_token_limits/model_is_reasoning tables. Verified live: glm-5-2 answers through Pi (6*7 -> 42).
Pi databricks-mlflow OSS provider + broadened OSS cohort with reasoning/limits.
gpt-oss-* contains 'gpt-' so it matched the codex bucket, but it's a chat-completions-only OSS model (served via /ai-gateway/mlflow/v1), not an openai-responses model. Broadening _OSS_MODEL_FAMILIES to include gpt-oss put it in BOTH codex_models and oss_models, so it was offered under Pi's databricks-openai provider too — where the openai-responses route 400s for it. Exclude gpt-oss from the codex bucket so it's OSS-only.
Proxy (_mlflow_proxy.py): - Close the connection at stream end (self.close_connection = True) so HTTP/1.1 clients detect the message boundary instead of hanging after [DONE] (the proxy sends no Content-Length/chunking). [critical] - Only repair text/event-stream responses; relay non-streaming and error bodies verbatim with their real Content-Type instead of forcing SSE. [high] - Catch URLError/OSError around urlopen and return a clean 502 instead of a dead socket on connection-level failures. [high] - Restrict forwarded paths to /ai-gateway/mlflow/ (404 otherwise) so the localhost relay can't be used as an authenticated arbitrary-path workspace client. [security] - Move terminator-repair writes inside the OSError guard so a client disconnect doesn't traceback on the daemon thread. [medium] - Parse SSE data lines with or without the optional space after the colon. [low] Discovery (databricks.py): - OSS chat models matching a broad family (qwen, llama-) but lacking a specific _MODEL_TOKEN_LIMITS entry now fall back to a conservative floor (128k/8192) so they're never offered uncapped (which 400s). Non-OSS and embedding ids still return None. Verified live: glm-5-2 streams content (42) + finish_reason + [DONE] through the proxy; non-stream/error/404/502 paths covered by tests. 898 passed.
Author
|
@rohita5l @AarushiShah-db — would appreciate a review when you have a moment. This builds directly on your OSS work (#178 Kimi, #186 GLM): adds a Pi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Pi support for OSS (mlflow chat-completions-only) foundation models and broadens the OSS cohort across both agents, building on the OpenCode OSS support already merged (#178 Kimi, #186 GLM).
Supersedes #216 (which was based on a pre-#178/#186 main and conflicted with the merged OSS work). Closes #204.
Gaps this closes on current
mainmaindiscovers OSS models via UC model-services and buckets them by name-substring allowlist_OSS_MODEL_FAMILIES = ("kimi-", "glm-"), wired into OpenCode only. That leaves:mainpins200k context / 25k output; the gateway actually accepts1M / 65536(probed 2026-07-16).Changes
databricks-mlflowprovider (api: openai-completions→/ai-gateway/mlflow/v1,compat {supportsStore, supportsStrictMode}=false). Selector/default-fallback/stale-cleanup wired;default_modelfalls back to OSS last._mlflow_proxy.py— a stdlib SSE-repair proxy started in Pi'slaunch(). Some OSS models (inkling) omit the spec-requiredfinish_reasonon the streaming terminal chunk; Pi'sopenai-completionsparser rejects that. The proxy injects it on any stream end (clean or an abnormal/429 mid-stream drop) and re-derives its origin from the workspace each launch (never a stale persisted port). No-op for well-behaved models. OpenCode needs no proxy — verified@ai-sdk/openaitolerates the missingfinish_reason._OSS_MODEL_FAMILIESto the full cohort (inkling, llama, qwen, gpt-oss, gemma + existing kimi/glm), with_is_oss_chat_modelexcluding non-chat services (embeddings/rerankers) that share a family substring — e.g.qwen3-embeddingis dropped though it matchesqwen._MODEL_TOKEN_LIMITSfrom probed gateway caps (glm 1M/65536, kimi/inkling 65536, gpt-oss/qwen35 25000, qwen3-next 10000, llama/gemma 8192). Addmodel_is_reasoning()from the probedopenai_reasoningcapability. Both agents read these shared tables; Pi setsreasoning/contextWindow/maxTokens, OpenCode setslimit.{context,output}.uv.locknet diff is the oneucodeversion line.Verification
ruff+tyclean.42through Pi via the new provider.Notes for reviewers
main's model-services + per-family-table architecture rather than a capability-API approach (model-services doesn't expose api_types/capabilities). Reasoning/limits are therefore hardcoded per-family, consistent with the existing_MODEL_TOKEN_LIMITS._MODEL_TOKEN_LIMITSoutput caps are probed (dated 2026-07-16); they're the thing most likely to drift if the gateway changes. Unlisted/new models fail safe (no override → client default).6×7 → 42; inkling confirmed by the author). Note the selector uses the full UC id, e.g.databricks-mlflow/system.ai.inkling— a staledatabricks-<name>selector from an earlier build won't resolve (Pi warns and falls back). The gatewayfinish_reasonomission is tracked in AI Gateway MLflow chat-completions omits finish_reason on natural stop (breaks inkling in Pi) #215.delta.contentafter theirreasoning_contentdeltas, and the proxy passes it through verbatim (verified in the emitted bytes at low/medium/highreasoning_effort). If a reasoning model appears to print nothing in Pi's non-interactive--printmode, the content is present in the stream — this is a Pi client rendering behavior, not a gap in what ucode produces.Code-review fixes (post-initial-review)
Addressed a review pass on
_mlflow_proxy.py: close the connection at stream end so HTTP/1.1 clients detect the boundary (was a potential hang); only repairtext/event-streamresponses and relay non-streaming/error bodies verbatim; return a clean502on connection-level upstream failures; restrict forwarded paths to/ai-gateway/mlflow/(SSRF guard); tolerate client-disconnect in the terminator-repair path; parsedata:lines with or without the optional space. Discovery: OSS chat models matching a broad family but lacking a specific limits entry now fall back to a conservative128k/8192cap so they're never offered uncapped (which would 400). Full suite: 898 passed.