Repository navigation
feat(computers): an annotated recording draws each input over the frames - #826
Open
earakely-scale wants to merge 12 commits into
Open
earakely-scale wants to merge 12 commits into
earakely-scale wants to merge 12 commits into
Conversation
computer-recording/v1 start takes annotate, an object of options (none yet, so {} draws everything). The driver then
draws each computer_* input on the frames captured while it shows, in the colour of the role that sent it: a ring
where a click landed, a trail for a drag, an arrow for a scroll, and a chip for keys, typed text and each driver tool
that acts. The card lists annotate under the recording params, the manifest records it, start_computer_recording
passes it through and refuses a driver that lists none, and the conformance probe checks a click is drawn where it
landed from the frame its effect shows on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd never stop an input A mark shows on the first frame after its input however late it comes, so a low fps or a slow capture can't miss it; making a mark is best effort, reusing the display a mapped input read; a recording of a native recorder's finished segments names no annotate, its marks kept only so long; the probe keeps each frame's findings, not the frames. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ents, and every chip shows The probe finds the annotate check not applicable to a recording of a native recorder's finished segments, which aren't drawn on; chips past the most a frame shows wait for a later frame instead of counting as drawn; the drawing's sizes and timings are named in one Style. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rdings-show Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…by its own annotations, and the probe's settings are named Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…re its frames can't show it Frames polled from captures that leave the pointer out, as cua-driver's, now show it where the agent's last input left it, along a drag's path as it moves, after a fading line from where it jumped; a frame that draws the pointer itself gets no second one. A click's dot is hollow, so the pointer's tip shows through, and the probe reads the clicked cell's colour up and left of the click, clear of a pointer there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p by its exact served name
The hook fills in {environment_name} from the mounted card, as the SDK does when it mounts the tools, so one
tool's name can't match another's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… and pointer Where the frames can't show the pointer, each role, and each pointer one acts through, gets a pointer where its last input left it: the one that moved last at full strength, the others fainter, each named once there are several. Inputs carry the pointer they act through, the computer's own until agents can drive more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ation_settings turning it off A recording draws its inputs unless its step's annotation_settings says is_enabled: false. A computer whose driver can't, as one composed before agent-env drew inputs, is recorded plain with a warning rather than failing, and the run's recording entry says whether it was annotated. The scripted desktop recording, which compares its frames with the screenshots, turns it off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… their operations, pointer moves their own type The driver's annotate options are one Pydantic model, AnnotateOptions, which the card lists and the step's AnnotationSettings extends with is_enabled, so a later option is added once. A mark's kind is its operation's name, one vocabulary for the inputs drawn at a point, and a pointer move is a PointerMove with where it was and its path, not a Mark. A test drives every input tool on a page fake, moved to the shared fakes, through an annotated recording and checks the marks each makes, so a tool whose arguments change, or a new one, shows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on, and falsy annotation settings are refused Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rdings-show Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A computer recording can now show what the agent did, drawn over what it saw.
computer-recording/v1starttakesannotate, an object of options, and the computer's driver draws each input into the frames as it records, in the colour of the role that sent it:key Ctrl+S), typed text (type "Q4 165 97"), and each call to the driver's own or forwarded tools that its annotations don't mark read-only, by name and main argument (cua-driver'sclick,hotkey cmd+S).annotatehas no options yet, so{}draws everything; options can be added later without changing its shape.This PR:
computer/annotate.py).ComputerRuntime.actmarks eachcomputer_*input as it's delivered, with its caller's role from the gateway's_metaorAgentEnv-Role.ComputerDriverEnvironment.on_tool_callmarks the driver's own and forwarded tools, which act withoutact.Video._add_framedraws the marks a frame shows before encoding it. Points are fractions of the display, so they land right on a HiDPI capture and aftermax_edgescaling.agent,user,agent p2) once there are several. Inputs carry the pointer they act through, the computer's own until agents can drive more (FD-6390). A recording whose frames draw the real pointer, as the reference driver's x11grab does, gets no second one. A click's dot is hollow, so the pointer's tip shows through.Style.annotate: {options: []}under the recording params; the manifest records theannotatea recording drew with, and leaves it out for a recording that took a native recorder's finished segments, which aren't drawn on.start_computer_recordingannotates by default. Itsannotation_settingsobject turns it off with{"is_enabled": false}, and will hold more options later; a step without it serializes as before. It sends the driverannotate(the settings butis_enabled) only when enabled. A computer whose driver can't draw inputs, as one composed before this, is recorded plain with a warning rather than failing, and the run's recording entry saysannotated. The scripted desktop recording test, which compares its frames with the agent's screenshots, turns annotations off and checks it wasn't annotated; the example's agent test checks it was._annotation): an annotated recording draws a click where it landed, from no later than the first frame that shows the click's effect, and its first frame, before the click, draws none. All its times are the computer's own. It doesn't apply to a recording of a native recorder's finished segments, which aren't drawn on. The probe's fake fixture now repaints a clicked cell, as the real one does.annotateoptions are one Pydantic model,AnnotateOptions, which the card lists and the step'sAnnotationSettingsextends withis_enabled, so a later option is added once. A mark's kind is its operation's name, and a pointer move is aPointerMovewith where it was and its path.FD-6314. The design was settled on the issue: the driver draws, no event track yet, typed text drawn as typed.
Not here, each a follow-up on FD-6314:
For the docs: Computer Environments › Example Tasks can show an annotated recording, and say that
start_computer_recordingannotates by default,annotation_settings: {"is_enabled": false}turning it off.Proof
Tests (
packages/agentenv-protocol/tests/test_computer_recording_annotate.py):userdraws in the second colour. With drawing patched out, the drawing tests fail.computer_screenshot, don't.{environment_name}is told read-only by its own annotations under its served name.annotateout of the manifest, and the probe finds its check not applicable to them.startrefuses anannotatethat isn't an object or names an option; the card listsannotate; each input's mark and label; a mark lands at its point on a frame of any size.The probe (
test_computer_conformance.py): the fake driver passes the new check atfull; with drawing patched out it fails withnone of its 11 frames draws the click's mark.The step (
tst/unit/task_step/computer_recording_test.py): a recording draws its inputs unlessis_enabledis false; a card withoutannotaterecords plain with a warning, even when its failures are fatal; the settings round-trip and take only a booleanis_enabled.Suites:
packages/agentenv-protocol/testsandtst/unit/task_stepgave 5,881 passed, 4 skipped (andpackages/agentenv-protocol/testswith the step's tests 2,030 passed at the final commit). Ruff is clean.An independent review found no ordering or coordinate bug. Its findings are fixed above: missed marks at a low fps or slow capture, an input refused when its mark couldn't be made, native-segment recordings claiming
annotate, and the probe holding every frame.Live, on an agent-environments-admin VM (amd64, local Docker), the shipped
desktopexample with annotations on its recording step (then"annotate": {}, now the default):agent-env run ./desk --task probe, then with--set envs/desktop.driver=cua-driver): both passed, the newX an annotated recording draws a click where it landed, from the frame its effect shows onamong their checks. Each computer's compose also probed it atfullbefore marking its build ready.key Ctrl+Shift+N,type "Reports[Enter]",key Ctrl+Landtype "/home/user/Downloads/[Enter]", each where and when the agent acted.On every OS agent-env ships a computer for, at 458a322, each with the probe at
full(run_probe, one result per check) and then a recording around a scripted drive of the computer's tools (click,type "annotated 123", Escape, scroll, drag, a pointer move), with annotations on by default:chromium(browser)annotated: truemacos-26vm_bundle/arm64)known_failures.macosannotated: trueomarchy-4.5disk/amd64)annotated: truewindows-11disk/amd64)annotated: trueThe desktops' gate-3 probes, which include the new check, marked each build ready. In every video the click's ring lands where the drive clicked, the
type "annotated 123"andkey Escchips show, and the scroll's arrow and the drag's trail are drawn. macOS, Omarchy and Windows, whose cua-driver frames leave the pointer out, show the agent's drawn pointer following each input, with a fading line where it jumped to the drag's start. Chromium's page driver shows its own cursor and gets none drawn.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge.Summary
Adds input marks and role-coloured pointers to computer recordings, with annotations enabled by default in the recording task step.
earakely-scaleexplicitly deferred masking typed secrets. Typed text remains visible in annotated recordings; tasks can turn annotations off.Diagram
%%{init: {'theme': 'neutral'}}%% flowchart LR A[Computer input] --> B[Role-coloured marks] C[Acting driver tool] --> B B --> D[Draw over captured frames] E[Captured frames] --> D D --> F[Encode video segments] G[Finished native segments] --> H[Keep unchanged] F --> I[Recording manifest] H --> IReviews (15) · Last reviewed commit: "Merge origin/main into edgararakelyan/fd..." · Reviewed by Greptile