Skip to content

fix(runtime): don't count image bytes as raw chars in the context-budget verdict - #4291

Open
liuxiaocs7 wants to merge 1 commit into
apache:mainfrom
liuxiaocs7:fix/image-context-budget-overcount
Open

fix(runtime): don't count image bytes as raw chars in the context-budget verdict#4291
liuxiaocs7 wants to merge 1 commit into
apache:mainfrom
liuxiaocs7:fix/image-context-budget-overcount

Conversation

@liuxiaocs7

@liuxiaocs7 liuxiaocs7 commented Aug 30, 2026

Copy link
Copy Markdown
Member

Summary

A first-turn message carrying a single user-uploaded image can fail immediately with context_budget_exhausted / no_safe_completed_span — the runtime record shows Steps (0), ~263 ms, no provider round-trip. The turn is killed before anything is sent.

Root cause: the mid-turn final-request capacity verdict sized the outgoing payload with JSON.stringify(messages).length. That serializes an inline image at the size of its transported bytes — a Uint8Array/Buffer expands to a digit map ({"0":..}, or {"type":"Buffer","data":[...]} once JSON.stringify applies Buffer.toJSON), and a base64 image is counted at full length. Divided by charsPerToken a single ~150 KB screenshot reads as hundreds of thousands of "tokens", exceeding the window on its own. It surfaces most on a model with no known context window: resolveContextBudgetCapacity returns policy_fallback (48,384 tokens), and policy_fallback is exactly the case where the verdict runs on step 0, before the first request. Being the first turn, the current user message is the pinned head anchor and nothing is foldable, so the verdict resolves to no_safe_completed_span. The reported Desktop path reads a session-file upload as a Node Buffer (packages/storage/src/artifact-attachments.ts), so the estimate must survive Buffer.toJSON.

The fix walks the ModelMessage content and discounts only genuine images to a bounded per-image estimate (~1.6K tokens, the vision per-image ceiling) scaled by the policy's charsPerToken. Image classification mirrors the AI SDK's own normalization (convert-to-language-model-prompt), so the estimate matches what the provider is billed: an image part, an image top-level mediaType (bare image or image/*, including a data: URL's own type), or inline bytes whose signature is a known image (the SDK overrides the declared mediaType from the byte signature). Each image is charged once regardless of how it is carried (inline bytes, a data: URL, or a remote reference); only inline bytes are stripped from the serialized size, and a remote http(s) URL stays verbatim. Everything else is left to serialize verbatim:

  • Non-image files (PDF, text, audio, …) stay fully measured — the budget must not silently under-count content the provider really bills.
  • Look-alike content (e.g. an arbitrary { type: 'data' } tool-call input) is untouched.

This removes the over-count without reintroducing the opposite image-counts-as-zero under-count fixed in #3372. All four capacity call sites share the one function, so they stay consistent.

Fixes #4290

Verification

  • Estimator unit tests (mid-turn-image-payload-chars.test.ts, 13/13): the Buffer / Uint8Array / base64 / data: URL / remote-URL-string / bare-image-type / generic-MIME-with-magic-bytes / PDF / tool-call-input shapes, charsPerToken scaling, and no change for text-only messages.
  • End-to-end backend test (mid-turn-capacity-backend.test.ts): an unknown-window model, a first turn whose only message is a 200 KB Buffer image upload — the provider now receives the request (with the image materialized) instead of a pre-send context_budget_exhausted. The fixture's attachment reader was made to return a Node Buffer, matching production.
  • No regressions in the adjacent suites, run locally: context-budget-mid-turn-policy (8), mid-turn-capacity-backend (67), overflow-reactive-recovery (43), ai-sdk-backend (217), history-compaction (22).
  • tsc -p packages/runtime/tsconfig.json clean; biome check clean on the changed files.

Review focus

The policy_fallback fail-open (letting the provider's real token count decide for a first, unshrinkable user message) is intentionally not in this PR — it would change the verdict for large first-turn text, files, and tool schemas, and belongs in its own issue/PR with reactive-recovery coverage.

AI use

Select exactly one:

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: Claude Code (Anthropic) performed the root-cause diagnosis, wrote the runtime fix in ai-sdk-compaction.ts, and authored the regression tests. The commit carries a Generated-by trailer.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

@github-actions github-actions Bot added the effort/M Under 500 readable lines label Aug 30, 2026
@liuxiaocs7
liuxiaocs7 force-pushed the fix/image-context-budget-overcount branch 4 times, most recently from d89637b to 416e2d3 Compare August 31, 2026 03:18
@github-actions github-actions Bot added effort/L Under 1000 readable lines and removed effort/M Under 500 readable lines labels Aug 31, 2026
…get verdict

The mid-turn final-request capacity verdict sized the outgoing payload with
`JSON.stringify(messages).length`, which serializes an inline image at the
size of its transported bytes: a Uint8Array/Buffer expands to a digit map
(`{"0":..}` or `{"type":"Buffer","data":[...]}` once toJSON runs) and a base64
string is counted at full length. Divided by charsPerToken that reads as
hundreds of thousands of tokens, so a single attached image could trip
`context_budget_exhausted` before any request was sent -- most visibly on an
unknown-window model whose `policy_fallback` capacity runs the verdict on
step 0 (Steps 0, ~263ms, no provider round-trip). The reported Desktop upload
path reads a session-file as a Node Buffer.

Walk the message content and discount only genuine IMAGES to a bounded
per-image estimate (~1.6K tokens, the vision per-image ceiling) scaled by the
policy's charsPerToken. Image classification mirrors the AI SDK's own
normalization: an `image` part, an `image` top-level mediaType (bare `image`
or `image/*`, incl. a data: URL's own type), or inline bytes whose signature
is a known image (which the SDK uses to override the declared type). Each
image is charged once regardless of how it is carried (inline bytes, a data:
URL, or a remote reference); only inline bytes are stripped, and a remote URL
stays verbatim. Non-image files (PDF, text, audio) and look-alike content
(e.g. a `{type:'data'}` tool-call input) stay fully measured, so the budget
never silently under-counts them, and the image=0 under-count of apache#3372 is
avoided.

Covered by estimator unit tests (Buffer / Uint8Array / base64 / data: URL /
remote URL string / bare `image` type / generic-MIME + magic bytes / PDF /
tool-call input / charsPerToken) and an end-to-end backend test: an
unknown-window first turn with a 200 KB Buffer image now reaches the provider
instead of a pre-send context_budget_exhausted.

Fixes apache#4290

Generated-by: Claude Code
@liuxiaocs7
liuxiaocs7 force-pushed the fix/image-context-budget-overcount branch from 416e2d3 to efbfcf2 Compare August 31, 2026 06:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/L Under 1000 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(runtime): a single attached image trips context_budget_exhausted on the first turn (image bytes counted as raw serialized chars)

1 participant