Support stored screenshot references in computer-use tasks - #15587
Draft
danielpeng2 wants to merge 1 commit into
Draft
Support stored screenshot references in computer-use tasks#15587danielpeng2 wants to merge 1 commit into
danielpeng2 wants to merge 1 commit into
Conversation
The server can now offload computer-use screenshots to object storage, swapping inline bytes in task messages for StoredScreenshotRefs via UpdateTaskMessage. Advertise the supports_stored_screenshots capability (behind the StoredScreenshots dogfood flag), apply UpdateTaskMessage to messages from earlier exchanges, preserve stored refs through conversion/restore round-trips, and fetch ref-only screenshots via a signed URL for the lightbox in restored conversations. Co-Authored-By: Warp <agent@warp.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Description
Client half of computer-use screenshot offload: the server now uploads each screenshot to object storage at ingestion and sends an
UpdateTaskMessageswapping the inline bytes in the task message for aStoredScreenshotRef, so screenshots stop traveling base64-inline in every request. This PR advertises thesupports_stored_screenshotscapability (behind a newStoredScreenshotsdogfood-enabled feature flag — the server only swaps for clients that opt in), applies those updates, and keeps screenshot UX working when only a ref exists.The substantive change is in
UpdateTaskMessagehandling, which previously assumed updates target a message from the current response stream. The swap targets messages from earlier exchanges, soupsert_messagenow resolves the owning exchange by message ID (viaadded_message_ids), tolerates the current stream having added no exchange for the task, and applies the rendered-output upsert to the exchange that actually owns the message — falling back to current-stream behavior only for genuinely new messages. Same-stream flows are unchanged; this is the area most worth review scrutiny.Elsewhere:
UseComputerResult::Successnow carries an optional stored ref alongside the action result, ref-only screenshots restore as "no inline image" with the ref retained (the proto task source is what echoes back to the server, so refs must survive round-trips), and the lightbox fetches ref-only screenshots on demand — signed URL fromGET /api/v1/agent/conversations/{cid}/screenshots/{uid}, bytes into the AssetCache, error toast on failure. Live sessions keep rendering locally captured bytes and never fetch.Draft caveats: depends on warpdotdev/warp-proto-apis#367 —
Cargo.tomlcarries a temporary[patch]pointing the proto crate at a local checkout (plus the resultingCargo.lockdelta); both revert when the proto change is published and the git rev is bumped. Server half: warpdotdev/warp-server#16148. Master has not been merged in yet; that plus full presubmit are still owed before this leaves draft.Linked Issue
Testing
Added 4 unit tests in
task_tests.rs: same-stream update regression, cross-exchange stored-ref swap on aUseComputerresult (including shared-session input replacement), an update arriving when the current stream added no exchange, and the new-messageExchangeNotFounderror case. Ran:cargo nextest run -p warp -E 'test(test_upsert_message)'(4/4 pass),cargo check -p ai -p warp_features,cargo check -p warp --lib,./script/format, andcargo clippy -p warp -p ai --lib --tests -- -D warnings— all pass. Full presubmit not run yet (draft).Agent Mode
CHANGELOG-NONE
Warp conversation · Plan: Offload computer-use screenshots to GCS, keep references in the MAA task