fix(runtime): don't count image bytes as raw chars in the context-budget verdict - #4291
Open
liuxiaocs7 wants to merge 1 commit into
Open
fix(runtime): don't count image bytes as raw chars in the context-budget verdict#4291liuxiaocs7 wants to merge 1 commit into
liuxiaocs7 wants to merge 1 commit into
Conversation
liuxiaocs7
force-pushed
the
fix/image-context-budget-overcount
branch
4 times, most recently
from
August 31, 2026 03:18
d89637b to
416e2d3
Compare
…get verdict
The mid-turn final-request capacity verdict sized the outgoing payload with
`JSON.stringify(messages).length`, which serializes an inline image at the
size of its transported bytes: a Uint8Array/Buffer expands to a digit map
(`{"0":..}` or `{"type":"Buffer","data":[...]}` once toJSON runs) and a base64
string is counted at full length. Divided by charsPerToken that reads as
hundreds of thousands of tokens, so a single attached image could trip
`context_budget_exhausted` before any request was sent -- most visibly on an
unknown-window model whose `policy_fallback` capacity runs the verdict on
step 0 (Steps 0, ~263ms, no provider round-trip). The reported Desktop upload
path reads a session-file as a Node Buffer.
Walk the message content and discount only genuine IMAGES to a bounded
per-image estimate (~1.6K tokens, the vision per-image ceiling) scaled by the
policy's charsPerToken. Image classification mirrors the AI SDK's own
normalization: an `image` part, an `image` top-level mediaType (bare `image`
or `image/*`, incl. a data: URL's own type), or inline bytes whose signature
is a known image (which the SDK uses to override the declared type). Each
image is charged once regardless of how it is carried (inline bytes, a data:
URL, or a remote reference); only inline bytes are stripped, and a remote URL
stays verbatim. Non-image files (PDF, text, audio) and look-alike content
(e.g. a `{type:'data'}` tool-call input) stay fully measured, so the budget
never silently under-counts them, and the image=0 under-count of apache#3372 is
avoided.
Covered by estimator unit tests (Buffer / Uint8Array / base64 / data: URL /
remote URL string / bare `image` type / generic-MIME + magic bytes / PDF /
tool-call input / charsPerToken) and an end-to-end backend test: an
unknown-window first turn with a 200 KB Buffer image now reaches the provider
instead of a pre-send context_budget_exhausted.
Fixes apache#4290
Generated-by: Claude Code
liuxiaocs7
force-pushed
the
fix/image-context-budget-overcount
branch
from
August 31, 2026 06:17
416e2d3 to
efbfcf2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A first-turn message carrying a single user-uploaded image can fail immediately with
context_budget_exhausted/no_safe_completed_span— the runtime record showsSteps (0), ~263 ms, no provider round-trip. The turn is killed before anything is sent.Root cause: the mid-turn final-request capacity verdict sized the outgoing payload with
JSON.stringify(messages).length. That serializes an inline image at the size of its transported bytes — aUint8Array/Bufferexpands to a digit map ({"0":..}, or{"type":"Buffer","data":[...]}onceJSON.stringifyappliesBuffer.toJSON), and a base64 image is counted at full length. Divided bycharsPerTokena single ~150 KB screenshot reads as hundreds of thousands of "tokens", exceeding the window on its own. It surfaces most on a model with no known context window:resolveContextBudgetCapacityreturnspolicy_fallback(48,384 tokens), andpolicy_fallbackis exactly the case where the verdict runs on step 0, before the first request. Being the first turn, the current user message is the pinned head anchor and nothing is foldable, so the verdict resolves tono_safe_completed_span. The reported Desktop path reads a session-file upload as a Node Buffer (packages/storage/src/artifact-attachments.ts), so the estimate must surviveBuffer.toJSON.The fix walks the
ModelMessagecontent and discounts only genuine images to a bounded per-image estimate (~1.6K tokens, the vision per-image ceiling) scaled by the policy'scharsPerToken. Image classification mirrors the AI SDK's own normalization (convert-to-language-model-prompt), so the estimate matches what the provider is billed: animagepart, animagetop-level mediaType (bareimageorimage/*, including adata:URL's own type), or inline bytes whose signature is a known image (the SDK overrides the declared mediaType from the byte signature). Each image is charged once regardless of how it is carried (inline bytes, adata:URL, or a remote reference); only inline bytes are stripped from the serialized size, and a remotehttp(s)URL stays verbatim. Everything else is left to serialize verbatim:{ type: 'data' }tool-call input) is untouched.This removes the over-count without reintroducing the opposite image-counts-as-zero under-count fixed in #3372. All four capacity call sites share the one function, so they stay consistent.
Fixes #4290
Verification
mid-turn-image-payload-chars.test.ts, 13/13): the Buffer / Uint8Array / base64 /data:URL / remote-URL-string / bare-image-type / generic-MIME-with-magic-bytes / PDF / tool-call-input shapes,charsPerTokenscaling, and no change for text-only messages.mid-turn-capacity-backend.test.ts): an unknown-window model, a first turn whose only message is a 200 KB Buffer image upload — the provider now receives the request (with the image materialized) instead of a pre-sendcontext_budget_exhausted. The fixture's attachment reader was made to return a NodeBuffer, matching production.context-budget-mid-turn-policy(8),mid-turn-capacity-backend(67),overflow-reactive-recovery(43),ai-sdk-backend(217),history-compaction(22).tsc -p packages/runtime/tsconfig.jsonclean;biome checkclean on the changed files.Review focus
The
policy_fallbackfail-open (letting the provider's real token count decide for a first, unshrinkable user message) is intentionally not in this PR — it would change the verdict for large first-turn text, files, and tool schemas, and belongs in its own issue/PR with reactive-recovery coverage.AI use
Select exactly one:
Tool(s) and scope: Claude Code (Anthropic) performed the root-cause diagnosis, wrote the runtime fix in
ai-sdk-compaction.ts, and authored the regression tests. The commit carries aGenerated-bytrailer.Checklist
Does this PR entail a change in behavior?