Skip to content

test(computers): a by-hand check of a computer's clock on a remote sandbox provider, which passes on Modal (FD-5487) - #736

Merged
earakely-scale merged 9 commits into
mainfrom
edgararakelyan/fd-5487-hb1-follow-up-the-computers-clock-on
Nov 18, 2026
Merged

earakely-scale merged 9 commits into
mainfrom
edgararakelyan/fd-5487-hb1-follow-up-the-computers-clock-on

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

What

FD-5487 (part of FD-5420): the computer's universe clock on remote sandbox providers, starting with Modal.

The local clock test's steps, run on Modal for the desktop and the browser computer, all hold with no change to the clock path. This PR adds them as a by-hand check, tst/manual/remote_computer_clock_check.py; it changes nothing else.

A multi env's computer that can't follow the clock, such as a desktop booted from its disk form, is handled from its card like any server, with no special case:

  • Tolerated (the default): it's skipped and stays on real time.
  • Strict (tolerate_missing_sync_time = false): the arm refuses it as not advertising clock/v1.

Results on Modal

The multi env's gateway runs on modal_vm, and its computer on the provider in each row. Every reading is taken inside the computer: the gateway's universe, read where the computer reaches the gateway as its follower does, and the session's time, each between two readings of real time. Each skew is recorded with how far its reading may be off: half its brackets, at the clock's rate.

Computer Provider Skew over 10 steps Rate 10 / frozen Follower killed, back in Reset
desktop modal_vm (container in a VM) at most 0.12 s (readings ±0.30 s) 10.0 / 0.0 2.1 s done, the fresh computer on the current version
desktop modal (container sandbox) at most 0.08 s, but 0.81 s once frozen after rate 10 (below) 10.0 / 0.0 1.3 s refused: "can't be reset in place", as before
browser computer modal_vm at most 0.11 s 10.0 / 0.0 2.4 s done; the page's Date.now() within 0.4 s, before and after
browser computer modal at most 0.12 s 10.0 / 0.0 1.4 s refused, as above; the page within 0.6 s
desktop, disk form modal_vm — — — arming skips it when tolerated (the default); a strict arm refuses it, "desktop does not advertise urn:agentenv:clock/v1"

What each row covers:

  • The steps: armed half a year back, then a day on, a week on, a month back, rate 10, frozen, running again, 90 s with no change, the follower killed, and a reset.
  • What depends on the provider, all working:
    • the root channel core writes the follower's config through: docker exec -u 0 on modal_vm, and the sandbox's own exec on modal;
    • the gateway's address as the computer's sandbox reaches it;
    • the root start;
    • the 25 s watch through Modal's tunnels: after 90 s with no change (more than three holds), the next write's wait for the follower was as quick as any other.
  • The link to the gateway varies run to run. A reading's bracket was 0.01 to 0.6 s, and a clock write that waits for followers took 0.23 to 0.87 s.

How

tst/manual/remote_computer_clock_check.py is a by-hand check, run with pytest -s, against multi envs written to stores the provider reaches. Its docstring says what it needs and how to write them. Its three tests:

  • the desktop over modal_vm,modal (two of the rows above);
  • the browser computer over the same (two more);
  • the disk form, on providers that boot disks (the last row).

No behavior changes.

Found along the way, not changed here

  • A rate change reaches a computer late by its follower's delivery delay. The follower applies a rate change when the change arrives, so the session runs at the old rate until then, and stays that far off until the next jump.
    • On modal, freezing after rate 10 left the desktop 0.81 s ahead: about 80 ms of delay at rate 10.
    • At rate 1 the offset is just the delay itself. On a slow link, with readings of 0.6 s as in one run, a change away from rate 10 could leave more than a second.
    • The local provider has the same behavior; its delay is only smaller.
  • A modal_vm gateway VM died in 3 of 9 deploys. Each exited 137 about 95 to 140 s after it was created, just as it finished building its images from their contexts; the next exec said "Sandbox is shutting down".
    • Ruled out: seven VMs created the same way (three ports, a 2400 s timeout) built the same four images and all survived 90 s after. The lowest available memory was 7.1 GB, dmesg had no OOM, and the disk is a real block device, not RAM.
    • Modal gives no reason, and the gateway deploy doesn't retry a lost VM. It isn't specific to the clock.
  • Registering computers to remote stores from an arm64 machine. bundle register builds every image on the registering machine for amd64 and pushes it. I wrote the bases as build contexts instead, which the Modal builder VM builds (VmBuilder._base).
  • ECR region. EcrImageStore still can't take a region through config without keys.
  • The gateway image isn't published for main yet. The last two releases stopped before publishing. Main's CI is green again, so the next release should publish it. Until then, a run on remote stores needs its infra envs written as build contexts.
  • Not run on other providers. Only Modal credentials were available.

Verification

  • On Modal: the table above, from the final version of the check (each row passing on it; a modal_vm desktop run lost its gateway VM once and passed on its rerun). Gate 3 also probed each computer ready on modal_vm when it was composed there: the desktop's container and disk, and the browser computer.
  • Unit tier: unchanged; the PR touches no code it covers.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new actionable issues were found.

Summary

Adds an opt-in manual check of computer clocks on remote sandbox providers, without changing production behavior.

  • Operators can check remote desktop and browser clocks against their gateway.
  • Operators can check how clock arming treats disk-booted desktops.

No new actionable issues found. The three earlier findings are addressed. Remote checks were not run during this review.

Reviews (15) · Last reviewed commit: "revert(task_step): a multi env's compute..." · Reviewed by Greptile

earakely-scale and others added 2 commits November 17, 2026 00:18
… can't follow it

A computer whose card offers no clock/v1, one booted from a disk or a VM bundle or on a base without agent-env's
clock, was skipped as a server without a clock: arming succeeded and its session stayed on real time. sync_env_clock
now refuses it, naming the computer and what to deploy instead. Its apps, which run on the computer's clock, are still
skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndbox provider

It takes the local clock test's steps, plus a minute and a half idle and the disk form's refusal, on modal_vm and
modal, for the desktop and a browser computer, reading the gateway's universe and the session's time inside the
computer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 17, 2026 08:19
earakely-scale and others added 2 commits November 17, 2026 00:23
…ake narrowly, instead of using it

Ten readings each slower than the bracket allows used to leave the last one to be measured, so a slow request could
fail a healthy clock. The reading now says whether one was narrow enough, and the check fails naming the widths when
none was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ty of its own reading

At a rate of 10, each 0.1 s of the computer's round trip to the gateway is a second of universe, more than the
skew allowed; a slow link failed a healthy clock. Each reading keeps the narrowest of its ten tries and reports how
far it may be off, the skew passes when it's within the limit give or take that, and the rate is read from the
session's own readings alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale and others added 2 commits November 17, 2026 08:22
…tarts

The waits, timeouts, attempt counts, tolerances and the arming offset its helpers held inline are named constants
beside the others, and the reading script inside the computer takes its attempts and gateway timeout as arguments.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uter too

A computer that offers no clock/v1 is skipped when the step tolerates servers without sync_time, as it is by default,
and stays on real time; a strict arm refuses it, naming it and what to deploy instead, rather than refusing it
whatever the step says.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale earakely-scale changed the title feat(task_step): arming the clock refuses a computer that can't follow it, and the clock is checked on Modal (FD-5487) feat(task_step): a strict clock arm refuses a computer that can't follow it, and the clock is checked on Modal (FD-5487) Nov 17, 2026
… computer, and the clock check reads as core does

- BackingServer.is_computer names the rule sync_env_clock used inline (a server on a computer's own sandbox whose
  card offers computer-exec/v1), beside the code that lists those servers, with a test of its own.
- sync_env_clock's tests share one helper for a multi env with a computer, and the module's imports are at its top.
- The remote clock check expects a reset to be refused where core refuses it, on a sandbox that isn't a VM, rather
  than by the refusal's wording, and its docstring says how to write the envs it needs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…an in-place reset as the reset's outcome

On a sandbox that isn't a VM, any RuntimeError from the reset passed as the refusal, though the multi env wraps every
child's failure in one. The check now also requires core's refusal for that provider, which names it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… as any server does

With tolerate_missing_sync_time covering computers, the computer-specific branch only reworded a strict arm's refusal.
A computer whose card offers no clock/v1 is skipped when tolerated and refused, as not advertising clock/v1, when not,
from its card like any server; sync_env_clock, legacy_protocol and their tests are as on main again. The remote check's
disk case expects that refusal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale earakely-scale changed the title feat(task_step): a strict clock arm refuses a computer that can't follow it, and the clock is checked on Modal (FD-5487) test(computers): a by-hand check of a computer's clock on a remote sandbox provider, which passes on Modal (FD-5487) Nov 18, 2026
@earakely-scale
earakely-scale merged commit a57b8d3 into main Nov 18, 2026
37 of 38 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/fd-5487-hb1-follow-up-the-computers-clock-on branch November 18, 2026 05:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant