Skip to content

Add a download command for saving channel files locally#26

Open
closestfriend wants to merge 1 commit into
aredotna:mainfrom
closestfriend:feat/download-command
Open

Add a download command for saving channel files locally#26
closestfriend wants to merge 1 commit into
aredotna:mainfrom
closestfriend:feat/download-command

Conversation

@closestfriend

Copy link
Copy Markdown

The CLI can push files into Are.na (upload, import) but there's no way to pull them back out — you can list a channel's contents, but not save the actual images and attachments to disk. This adds arena download <slug> [dir] to close that gap.

Works on any channel you can access — your own private channels (via arena login) and any public channel — which makes it a building block toward #22 (backup functionality).

What it does

arena download worldmaking --dir ./refs
arena download my-private-channel --type Image --size large
  • Paginates /v3/channels/{id}/contents and resolves each block to a download target: Images (with --size original|large|medium|small|square), Attachments, and Text (opt-in via --include-text).
  • Streams assets concurrently (--concurrency, default 4) with retry/backoff on transient errors.
  • Always writes a manifest.json so Link/Embed/Text metadata (titles, source URLs) is preserved even though those have no hosted file.
  • Skips files that already exist unless --overwrite; --type filters to one block type.
  • Supports --json (NDJSON progress stream) for scripting, consistent with import.

One thing worth flagging

Are.na's CloudFront CDN returns 202 Accepted with an empty body when a request has no User-Agent, which silently produces 0-byte files. The command sends an explicit User-Agent to avoid that.

Design

Logic lives in src/lib/download.ts behind an injectable adapter, mirroring import.ts. Unit tests (src/lib/download.test.ts) cover pagination, per-type target resolution, --size, filename fallbacks, failures + exit codes, transient retries, skip-existing, and --type filtering.

Tested

  • Public channel (anonymous): 126-block channel → 58 images downloaded, 0 empty files, one genuinely-dead link surfaced as a failure rather than a silent 0-byte file.
  • Private channel (arena login): authenticated listing → all images downloaded, links preserved in the manifest.
  • npm run check green (typecheck + 86 tests + build).

Scope

This is per-channel download, not the full automated whole-account backup #22 describes — but it's the core primitive that makes that possible.

Adds `arena download <slug> [dir]` to pull a channel's images and
attachments to disk — the inverse of the existing upload/import commands.
Works with private channels, since the listing goes through the
authenticated API client.

- Paginates /v3/channels/{id}/contents, resolving each block to a target:
  Images (with --size), Attachments, and Text (opt-in via --include-text).
- Streams assets concurrently (--concurrency) with retry/backoff, and
  sends a User-Agent so Are.na's CDN doesn't return empty bodies.
- Always writes a manifest.json so Link/Embed/Text metadata is preserved.
- Skips existing files unless --overwrite; supports --type filtering.

Logic lives in src/lib/download.ts behind an injectable adapter (mirroring
import.ts), with unit tests covering pagination, target resolution,
failures, retries, skipping, and type filters.
Comment thread src/lib/download.ts

@dzucconi dzucconi Jun 17, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this file has a literal NUL byte in a regex, which makes Github classify it as a binary file. Makes it hard to review. Could escape it as a range like [\x00-\x1F]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants