Repository navigation
Read Codex rollouts compressed to .jsonl.zst; release 0.12.0 - #26
Conversation
Codex's experimental `local_thread_store_compression` rewrites rollouts idle for a week as `rollout-*.jsonl.zst` and drops the `.jsonl`. Those threads vanished from discovery and every reader that opened the file. Vendor the decompression half of zstd 1.5.7 as a `CZstd` target (macOS ships no zstd library), wrap it in `ZstdStreamDecoder`, and route `JSONLLineScanner` / `JSONLHeadTail` through a new `SessionLogByteReader` that picks plain or compressed by suffix, so every adapter reads both. `CodexSessionAdapter` discovers both spellings and derives the filename id from the stem without either suffix. `CodexRolloutFile` collects the naming rules, including the canonical (`.jsonl`) path hosts should key caches on. Tests build raw-block zstd frames without a compressor and also decode a fixture produced by the zstd CLI. The Rust lane is unchanged and still sees only plain rollouts; noted in the changelog. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 400335aad8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Review on #26: a compressed rollout's tail pass decoded the whole transcript for 30 lines, `lineCountIfSmall` buffered an unbounded decompressed expansion of a small archive, and the tail ring shifted `count` elements per line. Both passes now stream through `JSONLLineScanner.scanLines(in:while:)` and stop at `JSONLHeadTail.compressedReadBudget` (32 MiB decoded), reporting `[]` / `nil` so callers take their existing fallbacks; the tail keeps a fixed ring indexed modulo `count`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ed65667bc6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
1 participant
Why
Codex's experimental
[features] local_thread_store_compression = truerewrites every rollout idle for a week asrollout-<stamp>-<uuid>.jsonl.zst(same JSONL, zstd-framed) and removes the.jsonl. Discovery matched onlypathExtension == "jsonl", so those threads disappeared from the session list, the search index and every host that reads through this package.What
CZstd— the decompression half of Zstandard 1.5.7 (lib/common+lib/decompress), vendored unmodified as a C target; no compressor, no legacy formats, no assembly. macOS ships no zstd library. BSD-3;THIRD_PARTY_NOTICES.mdandREADME.mdupdated.ZstdStreamDecoder— streaming decoder over a file withread(upToCount:),skip(toOffset:);.truncated/.corrupt(name)errors; memory bounded by zstd's recommended buffer sizes.SessionLogByteReader— plain or compressed by suffix, withseek(toOffset:)in decompressed offsets.JSONLLineScannerandJSONLHeadTailnow read through it, so every adapter inherits support.tailLineson a compressed log is one streaming pass keeping the last N lines.CodexSessionAdapterdiscovers.jsonland.jsonl.zst; filename id comes from the stem without either suffix.CodexRolloutFile— naming helpers incl.canonicalPath(.jsonl.zst→.jsonl) for hosts that key caches / ledgers on the rollout path.0.12.0, changelog entry.Tests
swift test: 404 tests, 0 failures. New coverage: raw-block zstd frames built without a compressor (multi-block, empty, truncated, garbage, forward/backward seek), a CLI-compressed fixture decoded byte-for-byte, head/tail/count on compressed logs, compressed Codex discovery + metadata + transcript + deletion plan, filename-mismatch refusal.Not in this PR
The Rust lane still reads only plain rollouts (needs its own zstd decoder);
contracts/unchanged. The live tailer ignores.jsonl.zston purpose: a compressed rollout is never live.🤖 Generated with Claude Code