[factory] Build the open-source sandbox provider factory and evidence-gated catalog

[khaliqgant]

Objective

Build the open-source provider factory for AgentWorkforce Sandbox: one minimal, stable interface for selecting and using supported sandbox providers, plus a repeatable adapter-production path that can expand the long tail safely.

This work unit must be claimed and orchestrated by the cloud Factory. Do not manually spawn it on a laptop or mini. Remote implementation execution may be placed by the cloud Factory relay backend on a named fleet node.

Product boundary

Open source:

• provider interface and registry/factory
• provider manifests, explicit capability flags, and environment-variable contracts
• manual provider selection and basic fallback
• adapter generator/scaffold and conformance tests
• benchmark/provenance schema and public verified/unknown status
• Relay, Relayfile, and RelayAuth integration contracts that preserve portability

Paid control plane, out of scope here:

• continuously learned routing and placement
• proactive migration and mid-stream transfer policy
• multi-tenant governance, billing, audit, and SLA enforcement

Requirements

1. Preserve the current AdapterPort/provider contract and existing Daytona, E2B, Microsandbox, and Agent37 behavior. Integrate Box and Freestyle only from their canonical landed work; do not duplicate or overwrite in-flight adapter work.
2. Provide a simple provider-agnostic construction surface and CLI vocabulary inspired by the clarity of Box: create/new, connect or ssh/scp where supported, exec/prompt, fork/snapshot, stop/suspend, list, and move as a distinct orchestration operation.
3. Model unsupported capabilities explicitly. Never silently emulate a provider semantic or claim parity that has not been proven.
4. Keep vendor SDKs behind provider internals. Reconcile configured capabilities with runtime support at construction time. Keep provisioning policy outside provider adapters.
5. Add a provider manifest/catalog schema with pinned source provenance, credential-variable names only, lifecycle semantics, regions/images/snapshots, pricing source metadata, and verification timestamps. Never persist secret values.
6. Add a generator/scaffold that produces the adapter skeleton, manifest, contract-test fixture, benchmark stub, cleanup ledger integration, and documentation checklist for a new provider.
7. Add an evidence gate: a provider cannot be advertised as verified until conformance tests and safe live benchmark artifacts prove create, ready, execute, lifecycle/cleanup, and must-fire/must-not-fire teardown behavior. Unknown stays UNKNOWN.
8. Reuse lessons from Amika provider architecture and its Freestyle hazards: SDK/runtime type drift, soft-deleted list rows, stop/start races, suspend/start workarounds, non-streaming execution, and opaque snapshot identifiers.
9. Document the one-path contribution workflow for adding a provider and the OSS versus hosted boundary.
10. Add focused tests and run the repository CI appropriate to changed files.

Acceptance evidence

• GitHub issue claim by the deployed cloud Factory
• stable Factory run/work-unit correlation
• implementation agent name and named remote fleet node
• factory/<issue>-... branch and PR opened by the configured GitHub App path
• tests and conformance results attached
• no merge; terminal state is human review
• no credentials in files, argv, logs, reports, or issue/PR text

Coordination

Current Box and Freestyle provider lanes are already active. Treat their work as dependencies and avoid overlapping edits. Read the existing benchmark artifacts and the cloud Factory history before changing code. Never git stash.

Competitive correction: ComputeSDK integration gate

Before implementing a new provider framework or adapter factory, audit the current MIT-licensed computesdk/computesdk provider contract, packages, conformance utilities, benchmark schemas, and add-provider scaffold. It currently exposes an open multi-provider benchmark harness plus unified TypeScript sandbox API and documents roughly two dozen providers.

Do not duplicate that ecosystem by default. Produce an explicit adopt / bridge / extend / replace decision backed by contract and dependency analysis:

• Prefer a compatibility bridge or upstream contribution when ComputeSDK already provides the commodity create/list/get/destroy/exec/filesystem adapter behavior.
• Own AgentWorkforce-specific semantics above it: durable agent-session checkpoints, Relay identity and messaging continuity, Relayfile state, RelayHistory trajectory, RelayAuth reauthorization, lifecycle normalization, evidence-gated cleanup, time-to-resumed-execution measurement, and policy-driven routing/failover.
• Keep an AgentWorkforce-native adapter only where ComputeSDK cannot express a required capability or fails the safety/conformance bar; document the precise gap.
• Public benchmarks are evidence feeding the router, not the primary product. Avoid building a competing generic provider leaderboard.

Acceptance now includes a written dependency/interop decision before code and a proof that the chosen boundary does not fork commodity provider logic unnecessarily.

State substrate correction: Mesa and Archil

Treat portable workspace storage as a pluggable substrate, not a proprietary prerequisite. Before implementing content-addressed blobs, virtual filesystems, checkpoint storage, or merge/version engines, audit Mesa and Archil and record a build / partner / adapter decision.

• Mesa already provides a cross-sandbox versioned POSIX filesystem, checkpoints, forks, merges, and concurrent-agent workflows.
• Archil already provides strongly consistent versioned disks, branches/checkpoints, data-source mounts, serverless execution, and persistent sandboxes. Its published ComputeSDK integration demonstrates a unified filesystem across multiple providers.
• Define PortableAgentSession around an opaque workspace backend reference plus AgentWorkforce-owned continuity state: Relay identity/address, RelayHistory cursor and execution trajectory, RelayAuth capability rebind, process/service manifest, routing state, policy, and audit.
• Support Relayfile as a native backend and make Mesa, Archil, S3/Git, or other state substrates adapter candidates. Do not hard-wire AgentWorkforce continuity to a single storage vendor.
• Treat Archil as both a possible storage backend and a compute provider. Preserve explicit capability differences.
• The owned moat is cross-provider agent-session continuity, coordination, authorization, placement, recovery, and measurable resumed useful work—not generic filesystem persistence.

Acceptance now also includes a state-backend interface decision and a benchmark plan comparing Relayfile, Mesa, and Archil across at least two compute providers: mount-to-first-file, checkpoint/fork latency, cross-provider remount, time-to-resumed-useful-execution, correctness/RPO, concurrent-writer behavior, secret reauthorization, and total storage/egress/compute cost. Vendor performance claims remain unverified until reproduced.

[khaliqgant]

Agent37 long-running substrate proof — 2026-08-23 authentication gate (agent37-longrun-proof-0823)

Verdict: BLOCKED BEFORE LIVE MUTATION. I could not authenticate to Agent37, so I stopped as instructed. No Agent37 instance, Relay workspace, Relayfile delegation, or RelayHistory session was created in this run; teardown was therefore not needed and no billed resource exists from this run.

Environment report:

• cwd: /Users/khaliqgant/Projects/AgentWorkforce/sandbox
• host: Khaliqs-MBP.home
• Node: v22.22.2
• disk: 72 GiB free of 460 GiB; 84% used
• GitHub: gh authenticated as khaliqgant; active scopes admin:public_key, delete_repo, gist, read:org, repo, workflow
• checkout: main, 14 commits behind origin/main; I read origin/main without pulling the shared worktree
• Agent37 auth checks: AGENT37_API_KEY absent from the process environment; ~/.agentworkforce/provider-creds/AGENT37_API_KEY does not exist; 1Password CLI is installed but op whoami returns "account is not signed in" (two accounts are configured). I did not read, copy, rotate, or print any credential.

How the previous harness was invoked:
sed -n '/^AGENT37_API_KEY=/p' /Users/khaliqgant/Projects/AgentWorkforce/.worktrees/sandbox-provider-comparison-0819/bench/env.tpl | op run --env-file=/dev/stdin -- /Users/khaliqgant/Projects/AgentWorkforce/.worktrees/sandbox-agent37-0819/node_modules/.bin/tsx .agent37-proof-0820/proof.mjs

That path keeps the secret in 1Password/env transport rather than argv or an artifact, but it cannot run today while the CLI is signed out. Required unblock: sign one configured 1Password account into the desktop-integrated CLI, then resume this lane.

Existing proof evidence rechecked:

• report: .agent37-proof-0820/report.json
• prior run a37-20260820124257-7d3cee
• bare_create_ms = 12671
• provider_ready_ms, agent_relay_ready_ms, bootstrap_gap_ms, history_first_cloud_receipt_ms, targeted_spawn_ready_ms, repo_mount_read_ms, history_drain_before_destroy_ms, destroy_ms, verified_gone_ms = UNKNOWN
• failure: the live provider exec returned shell exit 1 because launch(workdir="/opt/agent37-proof") caused the adapter to prepend "cd /opt/agent37-proof" before the later command that would create that directory.
• /opt/agent37-proof was meant to be the remote proof root for app/, state/, repo/ (Relayfile mount), trajectories/, and src/ (RelayHistory build); it was not a disposable step. The correct repair is to launch in an existing directory such as /root, create the proof root, upload/install there, and only then execute with that cwd. A later reference branch reportedly fixed the root to /root, but it has not been re-run live.
• prior failure cleanup did destroy instance 5qsy67njou and getById verified it gone, but those two durations were not timed; both metrics remain UNKNOWN.

Actual API/substrate finding, limited to what live evidence supports:

• Agent37 is NOT conversation/SSE-only. The prior live run created an instance and executed a shell command through the provider; the returned remote shell diagnostic ("sh: 1: cd: can't cd...") is an actual exec response, not a documentation inference.
• origin/main's adapter maps exec to POST /v1/instances/{id}/exec and files to the instance-plane PUT/GET /v1/files/content. The exec half is live-observed; the dedicated file plane and full mount lifecycle were never reached, so file-surface usability remains UNKNOWN rather than claimed.
• Async exec is not native in the current adapter; it is buffered synchronous exec with a provider-side 280s cap.

Idle/cost finding:

• This run cannot settle provider-auto-sleep or controller-stop/resume without auth. Existing observed auto_sleep:false is not evidence of sleep. Therefore provider-auto-sleep = UNKNOWN and controller-stop/resume = UNKNOWN.
• The $0.0248/day idle-sleep figure remains unsubstantiated. The honest current planning figure remains $0.1565/day until a live sleep/wake or stop/start billing-and-readiness proof closes the gate.

Measurement method for all nine requested metrics in this run: no clock was started because authentication failed before any provider-side operation; each remains UNKNOWN(auth_unavailable), not zero and not success.

[khaliqgant]

Agent37 long-running substrate proof resumed — 2026-08-23 (agent37-longrun-proof-0823b)

I have read the predecessor report and Khaliq’s correction. I will run the existing proof through op run without op whoami; an interactive auth prompt is expected and is not a blocker. The harness fix is limited to ordering: launch in an existing cwd, create /opt/agent37-proof, then use it. I will preserve /opt/agent37-proof, treat timeouts as UNKNOWN, test the file surface and idle mechanism explicitly, and unconditionally destroy only the Agent37 instance this run creates, followed by a verified-gone assertion. No provider-side mutation has occurred yet.

[khaliqgant]

Agent37 proof progress — first corrected op run attempt reached the 1Password operation but timed out awaiting desktop authorization. Exact stderr: [ERROR] 2026/08/23 13:52:43 error initializing client: authorization timeout. The harness emitted nothing and never started, so this attempt created no Agent37 resource and required no provider teardown. I am retrying with a fresh authorization window; I am not using op whoami.

[khaliqgant]

Agent37 long-running substrate proof — corrected retry result (agent37-longrun-proof-0823b)

Verdict: BLOCKED BY op run ITSELF BEFORE HARNESS EXECUTION OR LIVE MUTATION.

I followed the correction exactly: no op whoami; each attempt piped the existing Agent37 entry from the existing env.tpl into op run and invoked the original harness path. Three fresh authorization windows ended inside 1Password:

• [ERROR] 2026/08/23 13:52:43 error initializing client: authorization timeout
• [ERROR] 2026/08/23 13:54:17 error initializing client: authorization timeout
• [ERROR] 2026/08/23 13:55:33 error initializing client: authorization timeout

Each process emitted no harness stdout or stderr before that error. The local report and ledger still end at the prior 2026-08-20 run a37-20260820124257-7d3cee; no new run id, Relay workspace, Relayfile delegation, RelayHistory session, or Agent37 instance was created. No provider teardown was possible or needed, and no billed resource exists from these retries.

Nine requested metrics for this retry:

• provider_ready: UNKNOWN(op-run-authorization-timeout-before-create)
• agent_relay_ready: UNKNOWN(op-run-authorization-timeout-before-create)
• bootstrap_gap: UNKNOWN(op-run-authorization-timeout-before-create)
• history_first_cloud_receipt: UNKNOWN(op-run-authorization-timeout-before-create)
• targeted_spawn_ready: UNKNOWN(op-run-authorization-timeout-before-create)
• repo_mount_read: UNKNOWN(op-run-authorization-timeout-before-create)
• history_drain_before_destroy: UNKNOWN(op-run-authorization-timeout-before-create)
• destroy: UNKNOWN(no-instance-created)
• verified_gone: UNKNOWN(no-instance-created)

Measurement method: no provider clock was started because the harness never received its environment and never executed. UNKNOWN is deliberate; an authorization timeout is not readiness or success.

The untracked proof harness is prepared for the next authorized invocation:

• launch cwd is /root; /opt/agent37-proof is then created and retained as the intended app/state/repo/trajectories/src root
• provider readiness requires a successful remote node --version
• Agent Relay readiness requires roster status online plus spawn:proof
• provider file surface requires an exact PUT/GET byte round trip; Relayfile mount requires an exact remote sentinel read
• targeted spawn requires both the worker ready file and mount-read file containing the run id, plus terminal placement confirmation and online roster assertion
• RelayHistory first and final receipts require matching marker, session id, and event id from cloud pair-check
• controller idle requires stop status stopped, resume status running, and a successful post-resume exec
• all failure paths recover only the unique run-labeled instance, destroy it unconditionally, poll GET-by-id to null, and time both destroy and verified-gone

File surface remains UNKNOWN until that live PUT/GET and mount assertion executes. Provider auto-sleep remains UNKNOWN; observed auto_sleep:false from prior evidence is not a sleep mechanism. Controller stop/resume also remains UNKNOWN until live assertion. The defensible cost remains $0.1565/day; no $0.0248 figure is claimed.

Required unblock: approve the 1Password desktop authorization prompt while the next op run window is open, then rerun the already-corrected harness.

[khaliqgant]

Agent37 proof progress — authentication cleared on the retry launched at 2026-08-23T13:01:12.508Z (15:01:12 Oslo). A live box was created as run a37-20260823130128-4f4b6a, with bare create 12226 ms, then the first readiness exec failed because this template has no /root: remote exit 1: sh: 1: cd: can\x27t cd to /root. Failure cleanup destroyed the unique run instance in 5102 ms and GET-by-id verified it gone in 295 ms; cleanup failures were empty. Provider metadata observed auto_sleep:false. No box leaked. I am changing only the bootstrap cwd to guaranteed-existing /, preserving /opt/agent37-proof as the created proof root, then rerunning.

[khaliqgant]

Agent37 proof progress — second authenticated live run a37-20260823130245-1e2eee proved / is a usable bootstrap cwd and a real node --version exec succeeded. Timings: bare create 12877 ms; provider ready 1397 ms. The next assertion failed because the template user cannot create the required proof root: mkdir: cannot create directory /opt/agent37-proof: Permission denied (for each child). Unconditional cleanup again succeeded: destroy 4369 ms, verified gone 346 ms, no cleanup failures or leak. Provider metadata again reported auto_sleep:false. Next narrow attempt preserves /opt/agent37-proof and uses non-interactive passwordless sudo to create and chown that root if this image exposes it; otherwise the intended /opt root is live-proven unattainable for this template.

[khaliqgant]

Agent37 proof progress — third authenticated live run a37-20260823130417-47184f tested the only narrow way to preserve /opt/agent37-proof: non-interactive sudo create+chown. The image exposes /usr/bin/sudo, but live stderr is sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the nosuid option set or an NFS file system without root privileges?. Thus /opt/agent37-proof is live-proven unattainable for this template user; this is no longer an ordering issue. Timings: create 13045 ms, provider ready 1280 ms, destroy 4501 ms, verified gone 311 ms; cleanup failures empty, no leak. I will preserve this negative result and run the identical proof tree from writable /tmp/agent37-proof, with worker/build paths parameterized away from nonexistent /root, so the requested lifecycle metrics can still be measured without falsely claiming /opt support.

[khaliqgant]

Live Agent37 proof progress — run a37-20260823130615-f745a3 is active. It created the box, passed a real provider exec, and passed an exact dedicated file-plane round trip: PUT then GET returned the same 46 bytes. This is live evidence that Agent37 exposes usable provider files, not only exec. The Relayfile mount assertion and remaining lifecycle stages are still running. Teardown remains guarded and unconditional.

[khaliqgant]

Agent37 proof progress — fourth authenticated run a37-20260823130615-f745a3 passed create, real provider exec, exact provider PUT/GET, Agent Relay install/version, Relayfile binary SHA check, and the pinned ai-hist build. It then failed before mount at the Relayfile sentinel API: HTTP 400 from /v1/workspaces/rw_7ccfea89/fs/bulk. Metrics captured: create 12294 ms, provider ready 1301 ms, destroy 3697 ms, verified gone 291 ms. Cleanup failures empty; Agent37 box verified gone. File PUT/GET remains proven true; Relayfile mount is UNKNOWN, not success. I am checking the landed Relayfile bulk-write schema before the next billed run.

[khaliqgant]

Agent37 proof progress — fifth run a37-20260823131059-3c6ac7 revalidated provider PUT/GET and reached the corrected Relayfile bulk request. The server returned the precise remaining contract error: HTTP 400 ... {"code":"bad_request","message":"missing X-Correlation-Id header","correlationId":""}. Timings: create 12050 ms, provider ready 1389 ms, destroy 3236 ms, verified gone 341 ms; all provider and logical cleanup failures empty. I am adding the required correlation header and moving this sentinel write before the expensive build so any further data-plane rejection fails fast.

[khaliqgant]

Live progress — run a37-20260823131549-35674b has passed the corrected scoped Relayfile sentinel write with the required correlation header. Provider PUT/GET also passed again. The run is proceeding through pinned installation/build toward the actual mount-read assertion.

[khaliqgant]

Agent37 proof progress — sixth run a37-20260823131549-35674b crossed the major substrate gates. Metrics: bare create 12843 ms; provider ready 1393 ms; Agent Relay ready 190973 ms; bootstrap gap 178130 ms; first cloud history receipt 2477 ms. Exact provider PUT/GET passed; Relayfile mount read matched the sentinel; remote secret scan passed. Targeted placement then failed with Action "spawn:proof" not found even though the node roster had asserted status online plus capability spawn:proof. Therefore targeted_spawn_ready, worker repo_mount_read, and final history drain remain UNKNOWN for this run. Cleanup passed: destroy 7016 ms, verified gone 302 ms, Relayfile sentinel deleted, cleanup failures empty. I am checking the exact installed 11.8.0 placement API action vocabulary before rerunning.

[khaliqgant]

Agent37 auth-window update — the proof attempt launched at 2026-08-23T13:21:50.906Z (15:21:50.906 Oslo) never started the harness. Exact op run stderr: [ERROR] 2026/08/23 15:22:51 error initializing client: authorization timeout. No run id or resource was created. Opening one immediate retry window now; no silent retry loop.

[khaliqgant]

Agent37 auth-window update — the single permitted retry launched at 2026-08-23T13:23:20.664Z (15:23:20.664 Oslo) also never started the harness. Exact stderr: [ERROR] 2026/08/23 15:24:21 error initializing client: authorization timeout. No run id or resource was created. I am not opening another authorization window without fresh instruction. The latest completed live evidence remains run a37-20260823131549-35674b: provider ready 1393 ms, Agent Relay ready 190973 ms, bootstrap gap 178130 ms, first cloud history receipt 2477 ms, Agent37 PUT/GET true, Relayfile mount true, destroy 7016 ms, verified gone 302 ms; targeted spawn/final drain/idle remain UNKNOWN pending the corrected proof:run rerun.