fix(sandboxes): retry transient background-job status timeouts

[samsja]

Summary

• retry transient background-job status reads after the VM runtime or SDK HTTP layer times out
• keep background-command launch exactly once; only the idempotent status read is retried
• apply the same bounded behavior to sync and async clients
• retry only the known VM-runtime timeout response; unrelated runtime errors still fail immediately

Motivation

Production RL workloads intermittently received structured per-job errors while polling commands that were already running:

Background job batch status failed: <sandbox>/<job>: Timed out reading background job status from the VM runtime

Surfacing that transient status-read failure immediately aborts the rollout even though relaunching the command would be unsafe and the original command may complete normally. This change retries the read instead, allowing four consecutive status attempts with 1s/2s/4s backoff. A successful poll resets that consecutive-failure budget, and all polling remains bounded by the caller's existing command deadline.

This does not retry sandbox creation, command launch, rate limits, egress failures, or persistent runtime errors.

Live A/B result

We ran two simultaneous, launch-order-reversed waves against the active service using the production prime-sandboxes==0.2.39 client. Each arm used four VMs and ran 800 commands total over the same service windows.

Variant	Successful commands	Success rate	Terminal status failures	Raw VM status-timeout items	Recovered	Command launches
Stock	796/800	99.50%	4	4	0	800
Retry patch	800/800	100%	0	5	5	800

The headline 99.50% to 100% success-rate difference might just be noise: the Wilson 95% intervals overlap (stock 98.72–99.81%, patched 99.52–100%) and a two-sided Fisher exact test gives p=0.125. It is still interesting because the patched arm actually saw slightly more raw backend timeout items and recovered all five, while preserving exactly one launch per command. That instrumented recovery path is stronger evidence for the behavior than the aggregate score alone.

The timeout items were correlated status-batch events: stock fanout was 2 + 2, while patched fanout was 3 + 2. Median/p95 command latency was 1.357s/11.044s for stock and 1.309s/10.645s for patched. The sample is too small to attribute the latency difference to this patch.

The live A/B used the exact production 0.2.39 implementation. This PR ports the same behavior onto current main / prime-sandboxes 0.2.41 and narrows the classifier to the exact structured timeout message.

Validation

• uv run pytest packages/prime-sandboxes/tests/test_batch_status.py -q — 43 passed
• uv run pre-commit run --all-files — passed
• broader sandbox package run — 263 passed, 4 skipped; the remaining 10 failures and 22 errors were live integration tests without PRIME_API_KEY

Tests cover sync and async recovery, bounded exhaustion, exactly-once command launch, and immediate failure for unrelated RUNTIME_ERROR responses.