Skip to content

fix: declare long-context Claude and GPT metadata#223

Open
dgokeeffe wants to merge 5 commits into
databricks:mainfrom
dgokeeffe:fix/1m-context-claude-gpt-pi
Open

fix: declare long-context Claude and GPT metadata#223
dgokeeffe wants to merge 5 commits into
databricks:mainfrom
dgokeeffe:fix/1m-context-claude-gpt-pi

Conversation

@dgokeeffe

Copy link
Copy Markdown

Summary

Fixes long-context metadata for Claude and GPT gateway models, with GPT handling scoped to Pi.

  • Claude Code: add [1m] to Sonnet 4.5+ and Opus 4.6+ aliases.
  • Pi: emit explicit Claude contextWindow, maxTokens, reasoning, image, and adaptive-thinking metadata instead of accepting Pi's custom-model defaults.
  • Pi: emit model-specific GPT limits; GPT-5.6 Sol/Terra/Luna use the opt-in 1,050,000-token window rather than treating every GPT-5 model as 1M.
  • Handle dotted, system.ai.*, databricks-*, and provider-qualified model IDs.

Closes #222.

Root cause and reproducible proof

Pi 0.80.6 documents these defaults for custom models when fields are omitted:

contextWindow: 128000
maxTokens: 4096

ucode previously emitted Claude models as only {"id": ...}, so Pi displayed and compacted Opus 4.8 as a 128K model even though the gateway model is long-context.

The installed Pi catalog independently declares Claude Opus 4.8 as:

contextWindow: 1000000
maxTokens: 128000

Pi's model override docs also explicitly show GPT-5.6 Sol can opt into:

{"gpt-5.6-sol": {"contextWindow": 1050000}}

After this patch, rendering a ucode Pi overlay produces:

{
  "id": "databricks-claude-opus-4-8",
  "reasoning": true,
  "input": ["text", "image"],
  "contextWindow": 1000000,
  "maxTokens": 128000,
  "compat": {"forceAdaptiveThinking": true}
}

and:

{
  "id": "databricks-gpt-5-6-sol",
  "contextWindow": 1050000,
  "maxTokens": 128000,
  "reasoning": true,
  "input": ["text", "image"]
}

Claude Code 2.1.214 bundle inspection also confirms that [1m] selects context-1m-2025-08-07 independently of CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS. Therefore beta suppression was not the root cause of Pi's 128K display; the omitted Pi custom-model metadata was.

Validation

uv run pytest tests/test_agent_pi.py tests/test_databricks.py tests/test_agent_claude.py tests/test_lint.py -q
272 passed in 5.53s

uv run ruff check .
All checks passed!

A full local run reached 913 passed, 36 skipped; its only behavioral failure was the managed-environment Claude binary timing out before reaching the test capture server. The other failure was formatting, subsequently fixed and covered by the passing tests/test_lint.py run above.

Stacking

This PR is stacked on dgokeeffe:feat/pi-oss-provider-v2 / #217. Retarget to databricks/ucode:main after #217 merges.

…h reasoning/limits

Builds on the merged OpenCode OSS support (kimi/glm) to close the remaining gaps:

- Pi had NO OSS provider. Add databricks-mlflow (api openai-completions ->
  /ai-gateway/mlflow/v1) so Pi can select OSS chat models, with a local
  SSE-repair proxy (_mlflow_proxy) for models like inkling whose gateway stream
  omits finish_reason (Pi's parser rejects that; OpenCode's @ai-sdk/openai
  tolerates it, so it needs no proxy).
- Broaden _OSS_MODEL_FAMILIES from kimi/glm to the full chat-completions cohort
  (inkling, llama, qwen, gpt-oss, gemma), excluding non-chat services
  (embeddings/rerankers) that share a family substring (e.g. qwen3-embedding).
- Correct + extend _MODEL_TOKEN_LIMITS from probed gateway caps: glm is
  1M context / 65536 output (was 200k/25k); per-family output ceilings for the
  cohort. Add model_is_reasoning() from probed openai_reasoning capability;
  Pi sets reasoning:true so it renders streamed reasoning_content as thinking.

Both agents read the shared model_token_limits/model_is_reasoning tables.
Verified live: glm-5-2 answers through Pi (6*7 -> 42).
Pi databricks-mlflow OSS provider + broadened OSS cohort with reasoning/limits.
gpt-oss-* contains 'gpt-' so it matched the codex bucket, but it's a
chat-completions-only OSS model (served via /ai-gateway/mlflow/v1), not an
openai-responses model. Broadening _OSS_MODEL_FAMILIES to include gpt-oss put
it in BOTH codex_models and oss_models, so it was offered under Pi's
databricks-openai provider too — where the openai-responses route 400s for it.
Exclude gpt-oss from the codex bucket so it's OSS-only.
Proxy (_mlflow_proxy.py):
- Close the connection at stream end (self.close_connection = True) so HTTP/1.1
  clients detect the message boundary instead of hanging after [DONE] (the
  proxy sends no Content-Length/chunking). [critical]
- Only repair text/event-stream responses; relay non-streaming and error bodies
  verbatim with their real Content-Type instead of forcing SSE. [high]
- Catch URLError/OSError around urlopen and return a clean 502 instead of a dead
  socket on connection-level failures. [high]
- Restrict forwarded paths to /ai-gateway/mlflow/ (404 otherwise) so the
  localhost relay can't be used as an authenticated arbitrary-path workspace
  client. [security]
- Move terminator-repair writes inside the OSError guard so a client
  disconnect doesn't traceback on the daemon thread. [medium]
- Parse SSE data lines with or without the optional space after the colon. [low]

Discovery (databricks.py):
- OSS chat models matching a broad family (qwen, llama-) but lacking a specific
  _MODEL_TOKEN_LIMITS entry now fall back to a conservative floor
  (128k/8192) so they're never offered uncapped (which 400s). Non-OSS and
  embedding ids still return None.

Verified live: glm-5-2 streams content (42) + finish_reason + [DONE] through the
proxy; non-stream/error/404/502 paths covered by tests. 898 passed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1M-token context not applied to Sonnet 4.5 and GPT-5 (pi)

1 participant