Skip to content

Support stored screenshot references in computer-use tasks - #15587

Draft
danielpeng2 wants to merge 1 commit into
masterfrom
daniel/stored-screenshot-refs
Draft

Support stored screenshot references in computer-use tasks#15587
danielpeng2 wants to merge 1 commit into
masterfrom
daniel/stored-screenshot-refs

Conversation

@danielpeng2

@danielpeng2 danielpeng2 commented Aug 27, 2026

Copy link
Copy Markdown
Member

Description

Client half of computer-use screenshot offload: the server now uploads each screenshot to object storage at ingestion and sends an UpdateTaskMessage swapping the inline bytes in the task message for a StoredScreenshotRef, so screenshots stop traveling base64-inline in every request. This PR advertises the supports_stored_screenshots capability (behind a new StoredScreenshots dogfood-enabled feature flag — the server only swaps for clients that opt in), applies those updates, and keeps screenshot UX working when only a ref exists.

The substantive change is in UpdateTaskMessage handling, which previously assumed updates target a message from the current response stream. The swap targets messages from earlier exchanges, so upsert_message now resolves the owning exchange by message ID (via added_message_ids), tolerates the current stream having added no exchange for the task, and applies the rendered-output upsert to the exchange that actually owns the message — falling back to current-stream behavior only for genuinely new messages. Same-stream flows are unchanged; this is the area most worth review scrutiny.

Elsewhere: UseComputerResult::Success now carries an optional stored ref alongside the action result, ref-only screenshots restore as "no inline image" with the ref retained (the proto task source is what echoes back to the server, so refs must survive round-trips), and the lightbox fetches ref-only screenshots on demand — signed URL from GET /api/v1/agent/conversations/{cid}/screenshots/{uid}, bytes into the AssetCache, error toast on failure. Live sessions keep rendering locally captured bytes and never fetch.

Draft caveats: depends on warpdotdev/warp-proto-apis#367Cargo.toml carries a temporary [patch] pointing the proto crate at a local checkout (plus the resulting Cargo.lock delta); both revert when the proto change is published and the git rev is bumped. Server half: warpdotdev/warp-server#16148. Master has not been merged in yet; that plus full presubmit are still owed before this leaves draft.

Linked Issue

  • No linked issue — this tracks the screenshot-offload work planned in the linked Warp conversation below.
  • No screenshots included — the change has no visible UI beyond the existing lightbox, which is unchanged when bytes are present.

Testing

Added 4 unit tests in task_tests.rs: same-stream update regression, cross-exchange stored-ref swap on a UseComputer result (including shared-session input replacement), an update arriving when the current stream added no exchange, and the new-message ExchangeNotFound error case. Ran: cargo nextest run -p warp -E 'test(test_upsert_message)' (4/4 pass), cargo check -p ai -p warp_features, cargo check -p warp --lib, ./script/format, and cargo clippy -p warp -p ai --lib --tests -- -D warnings — all pass. Full presubmit not run yet (draft).

  • Not manually tested yet — end-to-end manual testing needs the server flag on staging or a local server with the staging bucket applied; planned before marking ready.

Agent Mode

  • Warp Agent Mode - This PR was created via Warp's AI Agent Mode

CHANGELOG-NONE


Warp conversation · Plan: Offload computer-use screenshots to GCS, keep references in the MAA task

The server can now offload computer-use screenshots to object storage,
swapping inline bytes in task messages for StoredScreenshotRefs via
UpdateTaskMessage. Advertise the supports_stored_screenshots capability
(behind the StoredScreenshots dogfood flag), apply UpdateTaskMessage to
messages from earlier exchanges, preserve stored refs through
conversion/restore round-trips, and fetch ref-only screenshots via a
signed URL for the lightbox in restored conversations.

Co-Authored-By: Warp <agent@warp.dev>
@cla-bot cla-bot Bot added the cla-signed label Aug 27, 2026

Copy link
Copy Markdown
Member Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant