Skip to content

perf(agents): store claude-code's workspace archive uncompressed in its gzip stream - #694

Open
andrewpark-scaleai wants to merge 5 commits into
mainfrom
andrew-park/rel-snapshot-archive-format
Open

andrewpark-scaleai wants to merge 5 commits into
mainfrom
andrew-park/rel-snapshot-archive-format

Conversation

@andrewpark-scaleai

@andrewpark-scaleai andrewpark-scaleai commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Why

The goal is that every failure of an agent run leaves enough in the logs to debug it, and that code heals what it can. The claude-code agent's snapshot save builds its workspace archive with tarfile w:gz at the default level 9. Deflate runs on one core, so it limits the whole save, and it gains nothing on large incompressible media in the workspace. A workspace of a few GB of video spends minutes in deflate before the upload starts, and the caller's wait runs out.

What changed

  • _write_workspace_archive opens the tar with compresslevel=0. The archive is still a gzip stream, but its blocks are stored, not deflated.
  • Restore is unchanged. It reads r:gz, which opens both stored and deflated gzip streams.

Design decisions

  • gzip level 0 over a plain tar. A plain tar is about as fast (480 to 550 MB/s against 390 to 440 MB/s in the same run). But an agent image built before this change restores with r:gz and would reject a plain tar. Level 0 keeps the container, so a snapshot saved by a new agent loads into an old one, and the reverse. No magic-byte sniffing is needed.
  • No zstd or pigz. The image runs Python 3.11 with no zstd in the standard library, and install.sh installs the agent into images this repo does not control. A codec the sandbox might lack would make saves fail.
  • Cost. Text-heavy workspaces store larger. They are small next to media, so the cost is mostly object-store bytes.

Scope

  • Covers the claude-code agent, whose save hit the limit.
  • Leaves the codex, openhands, pi and pydantic-ai agents for follow-ups. They also gzip at level 9. Codex caps the compressed archive size, so dropping compression there changes which workspaces it accepts and needs its own decision.

Blast Radius

Every claude-code snapshot save and load. Archives grow to about the workspace's size. Archives saved earlier still load (tested).

Verification

  • .venv/bin/python -m pytest tst/unit/agents/claude_code -q: 283 passed. Without the source commit, test_the_workspace_archive_stores_bytes_as_they_are_in_a_gzip_stream_older_loaders_read fails: 1.16 MB of repeated text archives to a few KB. The restore test passes for archives at both levels.
  • Archive build, measured by calling the agent's own _build_workspace_archive on a synthetic workspace of 300 MB random bytes plus 22 MB of text in 400 files (337 MB in all), 5 runs each on an 8 vCPU devbox shared with other jobs: old 25.2 to 38.1 s (8.8 to 13.4 MB/s), new 0.78 to 1.46 s (231 to 433 MB/s). The old build was CPU-bound on one core (CPU time 25 to 29 s, equal to wall time on quiet runs). The new build reads files from the page cache, so it measures the CPU cost, not a cold disk.
  • After deploy: the agent's Snapshot saved: ... bytes=N line shows N close to the workspace's size, and saves of large media workspaces finish in seconds of archive time instead of minutes.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new blocking issues were found.

Summary

Claude Code snapshot saves now skip compression by default while keeping the gzip format that older agents can read.

  • Claude Code builds workspace snapshots quickly and compresses them when needed.

No new actionable issues found.

Reviews (5) · Last reviewed commit: "refactor(agents): name the gzip levels c..." · Reviewed by Greptile

@andrewpark-scaleai
andrewpark-scaleai marked this pull request as ready for review November 16, 2026 16:33
@andrewpark-scaleai
andrewpark-scaleai requested a review from a team as a code owner November 16, 2026 16:34
@andrewpark-scaleai
andrewpark-scaleai force-pushed the andrew-park/rel-snapshot-archive-format branch from 7f2a179 to 10945c0 Compare November 18, 2026 18:17
@andrewpark-scaleai
andrewpark-scaleai force-pushed the andrew-park/rel-snapshot-archive-format branch from 10945c0 to 71a0c4e Compare November 18, 2026 18:25
Ubuntu added 5 commits November 18, 2026 18:31
…ts gzip stream

Deflate on one core limited the snapshot save, and it gains nothing on
media. Level 0 keeps the gzip container, so every loader still reads it.
…its upload grant

An uncompressed archive of a workspace near the grant's max_bytes could exceed
it and fail the save with transfer_too_large, where the compressed archive fit.
The save rebuilds the archive compressed in that case.
…as no room for it uncompressed

A level-0 write that hit ENOSPC was taken for a workspace that changed mid-read and retried with staged files,
which needs more disk, so a workspace that fit compressed failed to save. ENOSPC now falls back to the compressed
build.
@andrewpark-scaleai
andrewpark-scaleai force-pushed the andrew-park/rel-snapshot-archive-format branch from 5c72497 to bd0719e Compare November 18, 2026 18:32

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant