Skip to content

fix(computer): the probe's recording-limit check holds on a computer already recording - #782

Merged
earakely-scale merged 4 commits into
mainfrom
edgararakelyan/fd-6254-conformance-probe-the-recording-limit
Nov 18, 2026
Merged

earakely-scale merged 4 commits into
mainfrom
edgararakelyan/fd-6254-conformance-probe-the-recording-limit

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

What

The full probe's check X a recording past max_concurrent is refused no longer fails on a computer that's already recording, and it still fails every driver it failed before.

limited() started max_concurrent recordings, then expected the next start to be refused. A recording the probe didn't start, as when a task records the screen while it probes, counts against the limit too (Recordings.start). So the loop's own second start was refused with limit_exceeded, and the check failed though the limit held.

  • computer-recording/v1 has a list op (GET), which returns {recordings}, each still recording as {recording_id, started_at}. It's served by the SDK's shared Recordings (listed()), so every SDK driver has it. Like every framework-only route, it takes the env's control token.
  • The limit check counts around each start. When the card advertises list, the probe lists the open recordings before each start. Below max_concurrent the start must succeed; at it, the start must be refused limit_exceeded. Other recordings may start or end meanwhile, so an unexpected outcome is checked against a listing right after:
    • A refusal fails the check only while fewer than max_concurrent are still recording.
    • A start past the limit fails it only once more than max_concurrent are recording.
    • Otherwise the probe goes on, up to LIMIT_STARTS (3) starts per slot, then reports that the computer's recordings kept changing.
  • A driver that doesn't advertise list is checked exactly as before: the probe starts the whole max_concurrent. On a computer already recording, that driver still fails the check, as today.

So nothing the check caught before passes now:

  • A driver that refuses short of its declared limit still fails, idle or recording.
  • A driver that doesn't keep its limit still fails.

The probe still ends only the recordings it started, so a task's own stays running.

The protocol README documents list and the probe's counting.

FD-6254, under FD-6253.

Not here:

  • A computer already at max_concurrent: the probe's first recording check can't start one there, so it and the repeated-request_id check fail. That's as before, and outside FD-6254.
  • playwright MCP's recorder (env/envs/playwright_mcp/recorder.py) serves the extension's start and end but no list. It isn't a computer driver the probe checks.

Proof

  • Tests (test_computer_conformance_recording.py, test_computer_recording.py):
    • A computer recording already passes all three recording checks, and only the task's recording is still listed afterwards. It fails on main.
    • A driver that refuses short of its card's limit (card 3, driver 2) fails the check, idle and recording. This guards against the regression: run against this PR's first version, which took any refusal as the limit holding, the test fails, because that version let this driver pass.
    • A driver whose card doesn't advertise list passes idle and fails while recording, as on main.
    • A recording that ends, or one that starts, between the probe's listing and its start doesn't fail the check. Both cases fail against the version Greptile flagged, which counted once: one reads recording 3 of at most 2 started, the other an early limit_exceeded.
    • list names each recording still recording, as its start answered it, and none once ended. Without the control token it's refused 401.
    • The card's recording methods are now start, end, end_all and list.
    • The existing test of a driver that doesn't keep its limit (card 1, driver 2) still fails as it should.
  • Suites: the protocol package's tests plus tst/unit/env and tst/unit/task_step gave 7,770 passed, 4 skipped, on the version before the race fix. The protocol package alone, on the final version: 1,663 passed, 1 skipped. Two failed in the larger run:
    • test_disk_recipe.py::test_a_windows_install_boots_its_iso_first_..., which fails on main on this Mac too.
    • process_scope_socket_test.py::test_an_armed_time_trigger_keeps_firing_after_a_client_leaves. It passed 3 of 3 runs on its own, and this PR touches no gateway code.
    • Ruff is clean.
  • Live, on an agent-environments-admin VM (amd64, local Docker): the shipped desktop example with the issue's task (deploy, start_computer_recording, the probe as env_outcome_verifier, end_computer_recording).
    • An agent-env without this change (built from v0.9.1559; nothing has changed the probe or the recorder since): scored below 1 (probe: 0).
    • This branch, final version: passed (probe: 1). The composed driver served list: without it the probe starts the whole limit, which fails on a computer recording already.
    • The shipped probe task, which doesn't record: passed (probe: 1).

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no blocking issues were found.

Summary

The probe counts recordings already running when it checks the recording limit, without ending recordings owned by the task.

  • The recording-limit probe counts recordings already in progress.

The latest changes preserve behavior. The earlier finding about ended recordings is addressed. No new issues were found.

Reviews (8) · Last reviewed commit: "refactor(computer): plainer limit-check ..." · Reviewed by Greptile

…already recording

The full probe's limit check started max_concurrent recordings, catching nothing, and expected only the next to be
refused. On a computer recording already, as a task's that records while it probes, the loop's own second start was
refused (limit_exceeded) and the check failed, though the limit held. The probe can't count recordings it didn't start,
so a limit_exceeded refusal at any of its starts now holds the limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 18, 2026 18:40
…so an early refusal still fails

A refusal at any start passed the limit check, so a driver that refused short of its declared max_concurrent passed even
on an idle computer, which the check caught before. computer-recording/v1 now has a list op, served by the SDK's
recordings, naming those still recording. The probe starts only what's left of the limit beside the recordings a driver
lists, and from a driver that doesn't list them starts the whole limit, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale and others added 2 commits November 18, 2026 11:13
…sted around it

The limit check counted the other recordings once, so one that ended before the probe's last start made that start
succeed and read as a violation, and one that started meanwhile made an earlier start refused. Each start is now judged
by a listing before it, and an unexpected outcome by a listing after it: a refusal fails only while fewer than
max_concurrent are recording, and a start past the limit only once more than max_concurrent are. The path for a driver
that doesn't list its recordings is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tries, and one card helper in its tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit 42b0e40 into main Nov 18, 2026
37 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/fd-6254-conformance-probe-the-recording-limit branch November 18, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant