Skip to content

feat(computers): an annotated recording draws each input over the frames - #826

Open
earakely-scale wants to merge 12 commits into
mainfrom
edgararakelyan/fd-6314-annotated-computer-recordings-show
Open

earakely-scale wants to merge 12 commits into
mainfrom
edgararakelyan/fd-6314-annotated-computer-recordings-show

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

What

A computer recording can now show what the agent did, drawn over what it saw. computer-recording/v1 start takes annotate, an object of options, and the computer's driver draws each input into the frames as it records, in the colour of the role that sent it:

  • a ring where a click landed: a square for the right button, a ring per click of a double or triple;
  • a trail for a drag, an arrow for a scroll, a cross for a pointer move;
  • a chip for a key chord (key Ctrl+S), typed text (type "Q4 165 97"), and each call to the driver's own or forwarded tools that its annotations don't mark read-only, by name and main argument (cua-driver's click, hotkey cmd+S).

annotate has no options yet, so {} draws everything; options can be added later without changing its shape.

This PR:

  • Draws in the driver SDK, on the video's own clock (computer/annotate.py).
    • ComputerRuntime.act marks each computer_* input as it's delivered, with its caller's role from the gateway's _meta or AgentEnv-Role.
    • ComputerDriverEnvironment.on_tool_call marks the driver's own and forwarded tools, which act without act.
    • Video._add_frame draws the marks a frame shows before encoding it. Points are fractions of the display, so they land right on a HiDPI capture and after max_edge scaling.
  • Draws the pointers where the frames can't show them. A recording polled from captures that leave the pointer out, as cua-driver's, shows a pointer for each role and pointer that has acted, where its last input left it, along a drag's path as it moves, after a fading line from where it jumped. The one that moved last is drawn at full strength and the others fainter, each named (agent, user, agent p2) once there are several. Inputs carry the pointer they act through, the computer's own until agents can drive more (FD-6390). A recording whose frames draw the real pointer, as the reference driver's x11grab does, gets no second one. A click's dot is hollow, so the pointer's tip shows through.
  • Shows every mark at least once. A mark shows on each frame while it lasts, and on the first one after its input however late, as at a low fps or after a slow capture. A frame shows at most the newest four chips; older ones wait for a later frame. The drawing's sizes and timings are named in one Style.
  • Never stops an input. Making a mark is best effort: a display read that fails leaves the input unmarked, not refused. A mapped click reuses the display it already read, and chips read nothing.
  • Says so on the card and in the manifest. The card lists annotate: {options: []} under the recording params; the manifest records the annotate a recording drew with, and leaves it out for a recording that took a native recorder's finished segments, which aren't drawn on.
  • start_computer_recording annotates by default. Its annotation_settings object turns it off with {"is_enabled": false}, and will hold more options later; a step without it serializes as before. It sends the driver annotate (the settings but is_enabled) only when enabled. A computer whose driver can't draw inputs, as one composed before this, is recorded plain with a warning rather than failing, and the run's recording entry says annotated. The scripted desktop recording test, which compares its frames with the agent's screenshots, turns annotations off and checks it wasn't annotated; the example's agent test checks it was.
  • The conformance probe checks it (_annotation): an annotated recording draws a click where it landed, from no later than the first frame that shows the click's effect, and its first frame, before the click, draws none. All its times are the computer's own. It doesn't apply to a recording of a native recorder's finished segments, which aren't drawn on. The probe's fake fixture now repaints a clicked cell, as the real one does.
  • One place for each thing that will grow. The driver's annotate options are one Pydantic model, AnnotateOptions, which the card lists and the step's AnnotationSettings extends with is_enabled, so a later option is added once. A mark's kind is its operation's name, and a pointer move is a PointerMove with where it was and its path.
  • Docs: the protocol README's "Recording the display" and AGENTS.md's recording steps.

FD-6314. The design was settled on the issue: the driver draws, no event track yet, typed text drawn as typed.

Not here, each a follow-up on FD-6314:

  • masking typed secrets (typed text is drawn as typed, so don't annotate a recording of a run that types a real password);
  • an event track beside the segments, with links from marks to trajectory steps;
  • points for driver tools that can report where an element click landed;
  • browser recordings drawn from their action log;
  • one agent driving several pointers (FD-6390, on FD-4430's leases), which the drawing already keys by pointer.

For the docs: Computer Environments › Example Tasks can show an annotated recording, and say that start_computer_recording annotates by default, annotation_settings: {"is_enabled": false} turning it off.

Proof

  • Tests (packages/agentenv-protocol/tests/test_computer_recording_annotate.py):

    • A recording of a click on the fake driver draws the agent's mark around the click on the frames after it and none before; a plain recording draws nothing; a call with role user draws in the second colour. With drawing patched out, the drawing tests fail.
    • A forwarded tool that acts shows as a chip; one whose annotations say it only reads, and the SDK's own computer_screenshot, don't.
    • A driver's own tool named with {environment_name} is told read-only by its own annotations under its served name.
    • A mark no frame caught while it lasted shows fully on the next frame, once; marks no frame draws aren't kept past 30 s.
    • A scroll is delivered when the display read for its mark fails.
    • A native recorder's finished segments leave annotate out of the manifest, and the probe finds its check not applicable to them.
    • Chips past the most a frame shows wait for a later frame rather than go unseen.
    • After a click's mark is over, a driver whose frames leave the pointer out still shows it there in the agent's colour, and one whose frames draw it gets none; the pointer follows the last input before each frame.
    • The agent, a user simulator and a second pointer of the agent's each get a pointer, the last to move unfaded, each named.
    • Every input tool, driven on a page fake (moved to the shared fakes) through an annotated recording, makes the marks its arguments call for, with no marking warning; the list must name every input tool the driver serves, so a new one or a changed one shows.
    • start refuses an annotate that isn't an object or names an option; the card lists annotate; each input's mark and label; a mark lands at its point on a frame of any size.
  • The probe (test_computer_conformance.py): the fake driver passes the new check at full; with drawing patched out it fails with none of its 11 frames draws the click's mark.

  • The step (tst/unit/task_step/computer_recording_test.py): a recording draws its inputs unless is_enabled is false; a card without annotate records plain with a warning, even when its failures are fatal; the settings round-trip and take only a boolean is_enabled.

  • Suites: packages/agentenv-protocol/tests and tst/unit/task_step gave 5,881 passed, 4 skipped (and packages/agentenv-protocol/tests with the step's tests 2,030 passed at the final commit). Ruff is clean.

  • An independent review found no ordering or coordinate bug. Its findings are fixed above: missed marks at a low fps or slow capture, an input refused when its mark couldn't be made, native-segment recordings claiming annotate, and the probe holding every frame.

  • Live, on an agent-environments-admin VM (amd64, local Docker), the shipped desktop example with annotations on its recording step (then "annotate": {}, now the default):

    • The probe on the reference driver and on cua-driver (agent-env run ./desk --task probe, then with --set envs/desktop.driver=cua-driver): both passed, the new X an annotated recording draws a click where it landed, from the frame its effect shows on among their checks. Each computer's compose also probed it at full before marking its build ready.
    • The agent task on each driver (Claude Code makes a folder in Documents and moves a CSV into it): both passed, 47 s and 42 s. Their recordings show a ring at each click and chips such as key Ctrl+Shift+N, type "Reports[Enter]", key Ctrl+L and type "/home/user/Downloads/[Enter]", each where and when the agent acted.
    • The cua-driver recording shows the agent's pointer in its colour at each place it clicked, staying there between inputs, with a fading line from where it jumped. The reference driver's shows its real cursor.
    • This last ran at 04992c8, the pointer's commit; earlier runs, before the review's and Greptile's fixes, passed the same way.
  • On every OS agent-env ships a computer for, at 458a322, each with the probe at full (run_probe, one result per check) and then a recording around a scripted drive of the computer's tools (click, type "annotated 123", Escape, scroll, drag, a pointer move), with annotations on by default:

    Computer Driver Where Its build's gate-3 probe Probe task Annotate check Recording
    chromium (browser) page amd64 VM, local Docker not logged every check passed passed annotated: true
    macos-26 cua-driver EC2 Mac, Lume (vm_bundle/arm64) ready, 4 min 12 s 48 of 49 passed; the one failure, P6 typed text as key events, is in cua-driver's known_failures.macos passed annotated: true
    omarchy-4.5 cua-driver KVM host (disk/amd64) ready, 9 min 11 s every check passed passed annotated: true
    windows-11 cua-driver KVM host (disk/amd64) ready, 32 min 36 s every check passed passed annotated: true

    The desktops' gate-3 probes, which include the new check, marked each build ready. In every video the click's ring lands where the drive clicked, the type "annotated 123" and key Esc chips show, and the scroll's arrow and the drag's trail are drawn. macOS, Omarchy and Windows, whose cua-driver frames leave the pointer out, show the agent's drawn pointer following each input, with a fading line where it jumped to the drag's start. Chromium's page driver shows its own cursor and gets none drawn.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge.

Summary

Adds input marks and role-coloured pointers to computer recordings, with annotations enabled by default in the recording task step.

  • Computer recordings draw input marks over captured frames.
  • Recording steps annotate supported videos by default and track the result.

earakely-scale explicitly deferred masking typed secrets. Typed text remains visible in annotated recordings; tasks can turn annotations off.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[Computer input] --> B[Role-coloured marks]
    C[Acting driver tool] --> B
    B --> D[Draw over captured frames]
    E[Captured frames] --> D
    D --> F[Encode video segments]
    G[Finished native segments] --> H[Keep unchanged]
    F --> I[Recording manifest]
    H --> I
Loading

Reviews (15) · Last reviewed commit: "Merge origin/main into edgararakelyan/fd..." · Reviewed by Greptile

earakely-scale and others added 2 commits November 19, 2026 11:00
computer-recording/v1 start takes annotate, an object of options (none yet, so {} draws everything). The driver then
draws each computer_* input on the frames captured while it shows, in the colour of the role that sent it: a ring
where a click landed, a trail for a drag, an arrow for a scroll, and a chip for keys, typed text and each driver tool
that acts. The card lists annotate under the recording params, the manifest records it, start_computer_recording
passes it through and refuses a driver that lists none, and the conformance probe checks a click is drawn where it
landed from the frame its effect shows on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd never stop an input

A mark shows on the first frame after its input however late it comes, so a low fps or a slow capture can't miss
it; making a mark is best effort, reusing the display a mapped input read; a recording of a native recorder's
finished segments names no annotate, its marks kept only so long; the probe keeps each frame's findings, not the
frames.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 19, 2026 19:26
earakely-scale and others added 2 commits November 19, 2026 11:33
…ents, and every chip shows

The probe finds the annotate check not applicable to a recording of a native recorder's finished segments, which
aren't drawn on; chips past the most a frame shows wait for a later frame instead of counting as drawn; the drawing's
sizes and timings are named in one Style.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rdings-show

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…by its own annotations, and the probe's settings are named

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale and others added 4 commits November 19, 2026 14:54
…re its frames can't show it

Frames polled from captures that leave the pointer out, as cua-driver's, now show it where the agent's last input
left it, along a drag's path as it moves, after a fading line from where it jumped; a frame that draws the pointer
itself gets no second one. A click's dot is hollow, so the pointer's tip shows through, and the probe reads the
clicked cell's colour up and left of the click, clear of a pointer there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p by its exact served name

The hook fills in {environment_name} from the mounted card, as the SDK does when it mounts the tools, so one
tool's name can't match another's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… and pointer

Where the frames can't show the pointer, each role, and each pointer one acts through, gets a pointer where its last
input left it: the one that moved last at full strength, the others fainter, each named once there are several. Inputs
carry the pointer they act through, the computer's own until agents can drive more.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ation_settings turning it off

A recording draws its inputs unless its step's annotation_settings says is_enabled: false. A computer whose driver
can't, as one composed before agent-env drew inputs, is recorded plain with a warning rather than failing, and the
run's recording entry says whether it was annotated. The scripted desktop recording, which compares its frames with
the screenshots, turns it off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… their operations, pointer moves their own type

The driver's annotate options are one Pydantic model, AnnotateOptions, which the card lists and the step's
AnnotationSettings extends with is_enabled, so a later option is added once. A mark's kind is its operation's name,
one vocabulary for the inputs drawn at a point, and a pointer move is a PointerMove with where it was and its path,
not a Mark. A test drives every input tool on a page fake, moved to the shared fakes, through an annotated recording
and checks the marks each makes, so a tool whose arguments change, or a new one, shows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale and others added 2 commits November 19, 2026 16:53
…on, and falsy annotation settings are refused

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rdings-show

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant