From 1c029061b5893059af35facc677106e28fae888f Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Fri, 11 Sep 2026 21:14:34 +0000 Subject: [PATCH 1/2] 0.17.0: a channel relayed to another nixamp, losslessly compressed (PRD 0001) One nixamp can now carry another's channel with fewer bytes on the wire and every byte restored, through a negotiated envelope on a path of its own. Nothing about ordinary playback changes, and all of it is off until a channel's policy says otherwise. The envelope (application/vnd.nixamp.stream, docs/stream-compression.md): a 16-byte stream header naming the boundary and generation, then 48-byte frames each carrying the mode, sequence, lengths and SHA-256 of the bytes they stand for, ending in a marker with the whole generation's total and digest. Every field is checked against negotiated limits before a byte is allocated; a stream that stops without its marker is reported cut off. Compression is Zstandard from node:zlib (Bun 1.4 and Node 24 both ship it), off the event loop on the runtime's pool behind a bounded queue with deadlines. One encoder per channel however many receivers; a late joiner is sent the channel's opening bytes for itself and then the shared blocks. A block is flushed when full or after 100 ms, never waiting for a packet boundary; it is sent compressed only when that saves 3% and 512 bytes, stored otherwise, and `auto` stops trying after eight stored blocks in a row until a cooldown. A compressor that falls behind ends its receivers; a receiver that stops draining is cut off alone. Ordinary listeners now also have a ceiling on unsent bytes, where before a stalled socket's buffer grew until the channel ended. Per channel: GET/PATCH /api/channels/:id/compression (conditional on the version you saw), POST .../compression/analyses for a bounded, deduped, cancellable analysis job, GET/POST/DELETE .../relay to serve, bring in, or stop a relay. Server-wide: GET /api/compression and PATCH {enabled} as the kill switch. A library file gets a representation at /api/media/:n/relay, built once, published by rename, checked against the file on every lookup, under a byte budget; the original keeps answering ranges and the representation refuses them. `nixamp compression analyze|status|set|off|on|pull|fetch` speak the same routes; `analyze FILE` runs here with no server, reporting the container as told by the bytes, the transport-stream layout and null share, the bitrate the PCR implies, and every codec's complete wire size with a checked round trip. HLS can be packaged as fragmented MP4, opt-in: the same boxes copied into .m4s files with an init segment named per packager run so an old init cannot be paired with new media, the key on the EXT-X-MAP line, and the playlist's real segment lengths reported rather than the target. The experimental ts-zstd transform groups 188-byte packet headers apart from payloads in front of Zstandard, reversibly, incomplete tails and malformed adaptation fields included; it is offered only when a policy asks and a receiver can undo it, and chosen only when smaller. Not here yet, and answered honestly when asked for: a live source at the source boundary (SOURCE_BOUNDARY_UNAVAILABLE), PWA and desktop controls, MCP tools, a lower-bitrate quality profile, birnpack, and the real-feed, client-matrix and soak measurements that gate any change of default. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01MxNif5tsYq4LczgG7aE8Jp --- README.md | 33 ++ desktop/package.json | 2 +- docs/stream-compression.md | 396 +++++++++++++ package.json | 2 +- prd/0000-template.md | 50 ++ ...on-and-efficient-hls-delivery-to-nixamp.md | 320 +++++++++++ prd/README.md | 14 + src/channels.ts | 102 +++- src/compression/analyze.ts | 270 +++++++++ src/compression/blocks.ts | 93 ++++ src/compression/cli.ts | 340 ++++++++++++ src/compression/codec.ts | 202 +++++++ src/compression/envelope.ts | 237 ++++++++ src/compression/jobs.ts | 157 ++++++ src/compression/metrics.ts | 109 ++++ src/compression/policy.ts | 171 ++++++ src/compression/receiver.ts | 107 ++++ src/compression/relay.ts | 462 +++++++++++++++ src/compression/routes.ts | 294 ++++++++++ src/compression/service.ts | 525 ++++++++++++++++++ src/compression/static.ts | 264 +++++++++ src/compression/store.ts | 126 +++++ src/compression/ts-transform.ts | 148 +++++ src/hls.ts | 102 +++- src/main.ts | 6 + src/server.ts | 86 ++- test/compression-analyze.test.ts | 105 ++++ test/compression-core.test.ts | 295 ++++++++++ test/compression-hls-fmp4.test.ts | 73 +++ test/compression-relay.test.ts | 220 ++++++++ test/compression-routes.test.ts | 256 +++++++++ test/compression-service.test.ts | 213 +++++++ web/package.json | 2 +- 33 files changed, 5761 insertions(+), 21 deletions(-) create mode 100644 docs/stream-compression.md create mode 100644 prd/0000-template.md create mode 100644 prd/0001-add-lossless-stream-compression-and-efficient-hls-delivery-to-nixamp.md create mode 100644 prd/README.md create mode 100644 src/compression/analyze.ts create mode 100644 src/compression/blocks.ts create mode 100644 src/compression/cli.ts create mode 100644 src/compression/codec.ts create mode 100644 src/compression/envelope.ts create mode 100644 src/compression/jobs.ts create mode 100644 src/compression/metrics.ts create mode 100644 src/compression/policy.ts create mode 100644 src/compression/receiver.ts create mode 100644 src/compression/relay.ts create mode 100644 src/compression/routes.ts create mode 100644 src/compression/service.ts create mode 100644 src/compression/static.ts create mode 100644 src/compression/store.ts create mode 100644 src/compression/ts-transform.ts create mode 100644 test/compression-analyze.test.ts create mode 100644 test/compression-core.test.ts create mode 100644 test/compression-hls-fmp4.test.ts create mode 100644 test/compression-relay.test.ts create mode 100644 test/compression-routes.test.ts create mode 100644 test/compression-service.test.ts diff --git a/README.md b/README.md index 1a12114..7b20264 100644 --- a/README.md +++ b/README.md @@ -223,6 +223,39 @@ is the whole point. Publishing is administering the server, so it needs the control link or the owner's account. Listening only needs the share link, like any other audio. +### Relaying a channel to another nixamp, compressed + +A channel can be carried from one nixamp to another with fewer bytes on the +wire and every byte restored at the far end. It is off until you turn it on, +per channel, and nothing about ordinary playback changes when you do. + +``` +nixamp compression analyze --channel cnn what a codec would make of it +nixamp compression set --channel cnn --mode auto compress when it pays, store when it does not +nixamp compression status --channel cnn what it is doing, in bytes +nixamp compression off the whole server, at once +``` + +On the receiving nixamp: + +``` +nixamp compression pull --channel cnn --from https://host:4321/api/channels/cnn/relay --from-key KEY +``` + +and `cnn` is a channel there, heard at `/api/channels/cnn` like any other. +The relay is `GET /api/channels//relay` as `application/vnd.nixamp.stream`, +a framed stream of Zstandard blocks each carrying the length and SHA-256 of +what it stands for, ending in a marker; a block that would not shrink is +sent as it is, and the metrics say so rather than claiming a saving. A +library file gets the same treatment at `/api/media//relay`, built once +and kept. `nixamp compression analyze FILE` measures a file here with no +server at all. The wire format, the policy, the limits and the switch are +in [docs/stream-compression.md](docs/stream-compression.md). + +HLS can be packaged as fragmented MP4 instead of MPEG-TS +(`--hls fmp4` on `compression set`, or server-wide): the same boxes the +channel already carries, copied into files, never re-encoded. + ## Streaming into it A nixamp can be the thing you broadcast *to*, not just from. diff --git a/desktop/package.json b/desktop/package.json index 19b13d0..bbe7f8a 100644 --- a/desktop/package.json +++ b/desktop/package.json @@ -1,7 +1,7 @@ { "name": "@nixamp/desktop", "productName": "nixamp", - "version": "0.15.1", + "version": "0.17.0", "private": true, "description": "nixamp as a desktop app: the PWA in a window, with the CLI bundled in.", "author": "Profullstack, Inc. ", diff --git a/docs/stream-compression.md b/docs/stream-compression.md new file mode 100644 index 0000000..60d89f4 --- /dev/null +++ b/docs/stream-compression.md @@ -0,0 +1,396 @@ +# Stream compression: the relay envelope and how nixamp uses it + +This is the wire specification for `application/vnd.nixamp.stream`, the +representation one nixamp sends another when it relays a channel or a +library file losslessly compressed. It also says how the server decides +what to compress, what the operator can set, and how to turn it all off. + +Nothing here changes ordinary playback. A browser, a phone, a TV or a CLI +player asks for `/api/channels/` or `/api/media/` and gets what it +always got. The envelope is a different media type on a different path, and +only a client that asks for it by name receives it. + +## Three settings that are never the same thing + +| Setting | What it changes | What it promises | +| --- | --- | --- | +| `losslessCompression` | How bytes travel between two nixamps | Decoding restores every byte at the named boundary | +| `hlsPackaging` | The container HLS segments are wrapped in | The media is copied, not re-encoded | +| `qualityProfile` | Whether media is re-encoded | Only `source` ships; anything else is a separate, explicit setting | + +Changing one never changes another. The API refuses a `qualityProfile` other +than `source` rather than quietly re-encoding. + +## The boundary + +Every envelope names where its bytes were captured. + +- `channel`: the bytes the channel pipeline emits, after ffmpeg. For a video + channel that is fragmented MP4; for audio, MP3. This is what a listener on + `/api/channels/` receives, and it is what every relay today carries. +- `source`: the bytes as they arrived, before ffmpeg. A library file's + representation (`/api/media//relay`) is at this boundary. A live channel + cannot be relayed at it yet: ffmpeg reads the source itself and the original + bytes never pass through the server. Asking for it answers + `SOURCE_BOUNDARY_UNAVAILABLE`, not a remux labelled as the original. + +## Wire layout + +All integers are big-endian. Lengths are unsigned. + +### Stream header, 16 bytes, once + +| Offset | Size | Field | +| --- | --- | --- | +| 0 | 4 | Magic `NXS1` (`4e 58 53 31`) | +| 4 | 1 | Version, `01` | +| 5 | 1 | Flags, `00` (none defined) | +| 6 | 1 | Boundary: `00` source, `01` channel | +| 7 | 1 | Reserved, `00` | +| 8 | 4 | Generation | +| 12 | 4 | `maxFrameBytes`: the largest decoded size any frame may claim | + +The generation changes whenever the source starts over. Frames from two +generations are never on one connection: a new generation is a new +connection with a new stream header. + +A receiver refuses a header whose `maxFrameBytes` is larger than its own +ceiling (`BAD_LIMIT`), before it reads a frame. + +### Frame header, 48 bytes, per frame + +| Offset | Size | Field | +| --- | --- | --- | +| 0 | 1 | Type: `01` data, `02` end | +| 1 | 1 | Mode: `00` stored, `01` zstd, `02` gzip, `03` ts-zstd | +| 2 | 2 | Reserved, `0000` | +| 4 | 4 | Sequence number, from 0, consecutive | +| 8 | 4 | Original length | +| 12 | 4 | Encoded length: how many payload bytes follow | +| 16 | 32 | SHA-256 | + +For a data frame the SHA-256 is of the original bytes, and the payload +follows the header. Every data frame is independently decodable: no +dictionary, no window carried between frames. + +For the end frame there is no payload; the two length fields together are +the total original bytes of the generation (high word at offset 8, low word +at offset 12), and the SHA-256 is of every original byte in order. A stream +that stops without an end frame was cut off, and a receiver reports +`TRUNCATED`. It never reports a cut-off file as complete. + +### Modes + +- `stored`: the payload is the original bytes. Encoded length equals original length. +- `zstd`: one Zstandard frame. +- `gzip`: one gzip member. Benchmarked; not selected by any policy. +- `ts-zstd`: the transport-stream transform (below), then Zstandard. + +### Validation, in order, before any allocation + +A receiver checks, for every frame header: + +1. type is data or end (`BAD_FRAME_TYPE`) +2. mode is known (`BAD_MODE`) and was negotiated (`UNSUPPORTED_MODE`) +3. sequence number is the one expected (`BAD_SEQUENCE`) +4. original length is at most `maxFrameBytes` (`FRAME_TOO_LARGE`) +5. encoded length is at most original length + 1024 (`EXPANSION_BUDGET`) +6. a stored frame's lengths are equal (`LENGTH_MISMATCH`) + +and after decoding: + +7. decoded length equals original length (`LENGTH_MISMATCH`), with the + decoder capped at that length so a payload cannot grow past it + (`FRAME_TOO_LARGE`) +8. SHA-256 of the decoded bytes equals the header's (`CHECKSUM_MISMATCH`) + +and for the end frame: the total and the stream digest match what was +received; anything after it is `AFTER_END`. A frame that fails is never +forwarded, and the connection is dropped. + +The SHA-256 is an integrity check on the media bytes. It is not +authentication: that comes from the TLS connection and the key. + +### Test vectors + +Stream header, version 1, channel boundary, generation 7, 256 KiB frames: + +``` +4e585331 01 00 01 00 00000007 00040000 +``` + +Data frame 3, stored, original `hello`: + +``` +01 00 0000 00000003 00000005 00000005 +2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824 +68656c6c6f +``` + +End frame after one frame of five bytes, the same digest (the stream was +just `hello`): + +``` +02 00 0000 00000001 00000000 00000005 +2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824 +``` + +`test/compression-core.test.ts` checks the first two against the encoder; +`test/compression-relay.test.ts` runs a whole stream through the decoder. + +## Negotiation over HTTP + +A receiver asks for a channel with: + +``` +GET /api/channels//relay +Accept: application/vnd.nixamp.stream +X-Nixamp-Stream-Codecs: zstd, ts-zstd +X-Nixamp-Key: +``` + +The server answers a stream with: + +``` +200 +Content-Type: application/vnd.nixamp.stream; version=1 +Content-Encoding: identity +X-Nixamp-Stream-Codecs: stored,zstd +X-Nixamp-Kind: video +X-Nixamp-Generation: 1757548800 +``` + +`X-Nixamp-Stream-Codecs` on the response lists the modes the stream may use: +the receiver's list intersected with the policy. `ts-zstd` is only offered +when the channel's policy has `tsAware` on. The receiver never sees a mode +it did not offer. + +Or it answers JSON and one of: + +| Status | `code` | Meaning | +| --- | --- | --- | +| 404 | `NO_SUCH_CHANNEL` | nothing is playing there | +| 406 | (none) | no `Accept` for the envelope: a player followed the link; `playback` says where to go | +| 406 | `RECEIVER_UNSUPPORTED` | the receiver offered nothing the server would compress with | +| 409 | `COMPRESSION_OFF` | the channel's policy, or the server's switch, is off | +| 409 | `SOURCE_BOUNDARY_UNAVAILABLE` | the policy asks for the source boundary, which this server cannot provide | +| 409 | `VARIANT_IN_USE` | another receiver holds the channel's one compressor under a different codec set | +| 416 | (none) | a `Range` header: a relay is never a range | +| 503 | `CHANNEL_GONE` | the channel ended before the relay could start | + +Every refusal carries `playback`, the ordinary URL, which is unaffected. + +The envelope is not gzipped again by the server, and `Content-Encoding: +identity` asks proxies not to. Do not apply a second layer. + +### A library file + +``` +GET /api/media//relay +Accept: application/vnd.nixamp.stream +``` + +answers `202 {"building": true}` with `Retry-After` while the +representation is being made, `200` with `Content-Length`, +`X-Nixamp-Sha256` (of the original file) and `X-Nixamp-Original-Length` once +it is, `409 COMPRESSION_OFF` when the `static` policy is off, `416` for a +`Range`, `503` when the last build failed (the reason is in the body). + +The representation is built once, whole, written to a temporary file and +renamed into place, so a request never reads half of one. It is keyed by +the file's path and the policy variant, checked against the file's size and +mtime on every lookup and against its own digest at the end of the build. +A file that changed while it was being read is not published. The cache has +a byte budget; the least recently served representation goes first. The +original file is never touched. + +### Bringing a channel in + +``` +POST /api/channels//relay (control key) +{ "from": "https://host:4321/api/channels//relay", "key": "", "name": "CNN, relayed" } +``` + +starts a receiver on this server that dials the address, decodes the +envelope and feeds a channel here named ``, which listeners hear at +`/api/channels/` exactly as if it were decoded here. It dials again +after a clean end (the upstream started over: listeners here are ended and +rejoin, as they would for a redial) and after a broken one (reported in +`status.incoming.error`), and gives up after five dials without a byte. +`DELETE /api/channels/` stops it. The address is not shown to listeners. + +## What gets compressed + +A channel's bytes go into blocks of at most `maxBlockBytes`. A block is +flushed when it is full or when its first byte has been held `maxHoldMs`, +whichever comes first; it never waits for a packet boundary. Each block is +compressed once, on the runtime's thread pool, and the result is written to +every receiver: one compressor per channel however many receivers, and no +compressor at all when nobody is receiving. + +A compressed block is sent only if it saves at least `minSavingsBytes` **and** +`minSavingsPercent` against the block stored. Both frames carry the same +header, so the comparison of complete representations is the comparison of +payloads. Otherwise the block is sent stored. A stored block costs its bytes +plus 48; the envelope's whole cost on data that does not compress is 16 bytes +plus 48 per block plus 48 at the end, and the metrics say so rather than +claiming a saving. + +Under `auto`, eight ineligible blocks in a row stop the compressor trying +for `resampleAfterMs`, after which it tries again. Under `zstd` every block +is tried. Under `off` there is no relay: the endpoint answers +`COMPRESSION_OFF`. + +A compressor that falls more than `maxChannelQueueBytes` behind ends its +receivers rather than growing or stalling the channel. A receiver that has +more than `maxListenerQueueBytes` unsent is cut off on its own; the others +carry on. Neither touches the channel or any ordinary listener. A codec that +refuses or times out stores the block and counts a failure. + +## The transport-stream transform (`ts-zstd`, experimental) + +For a block that is a run of aligned 188-byte packets, the transform writes +every packet's header (the 4 bytes, plus the adaptation field when there is +one) into one region and every payload into another, with any bytes before +the first aligned packet and after the last kept as they are. The headers, +grouped, are a regular sequence that Zstandard squeezes well; interleaved +they are lost among payload bytes that do not compress. It is a reversible +rearrangement in front of an ordinary compressor, and nothing more is +claimed for it. `tsJoin(tsSplit(x))` is `x` for every input, including +malformed adaptation lengths (the packet is kept whole), null packets (kept +whole, never regenerated), and streams that lose sync mid-block. + +192- and 204-byte layouts are detected and left to plain Zstandard. The +transform is only used when its complete output is smaller than plain +Zstandard's on the same block, and only when the policy has `tsAware` on +and the receiver offered `ts-zstd`. On a synthetic padded stream it wins; +on a real corpus that has not been measured yet, and `auto` does not +select it until it has. + +## Policy + +Per channel, at `/api/channels//compression`: + +```json +{ + "losslessCompression": { + "mode": "off", + "boundary": "channel", + "zstdLevel": 1, + "minSavingsPercent": 3, + "minSavingsBytes": 512, + "maxBlockBytes": 262144, + "maxHoldMs": 100, + "resampleAfterMs": 60000, + "maxChannelQueueBytes": 8388608, + "maxListenerQueueBytes": 1048576, + "tsAware": false + }, + "hlsPackaging": "mpegts", + "qualityProfile": "source", + "version": 0 +} +``` + +`GET` returns the configured policy, the effective one (after the server's +switch and this server's boundary support) with a plain-language reason +when they differ, the server's settings, and the metrics of the current +generation. `PATCH` takes any subset; unknown fields are refused, not +ignored; ranges are checked; `If-Match: ""` (or `version` in the +body) makes the change conditional on the version you read. Every accepted +change bumps the version. A change that alters what a running relay would +produce ends that relay; its receivers see a cut-off, report it, and dial +again under the new policy. + +The pseudo-channel `static` holds the policy for library file +representations. + +Server-wide, at `/api/compression`: `GET` for every channel's status, the +pool, the jobs and the cache; `PATCH {"enabled": false}` is the kill switch. +It ends every running relay and refuses new ones; nothing else changes, and +`{"enabled": true}` puts every channel back on its own policy. +`{"hlsPackaging": "fmp4"}` sets the packaging for channels that have no +policy of their own. + +Policies are kept in `compression.json` beside `channels.json`, keyed by +port, and written whole then renamed. + +## Analysis + +`POST /api/channels//compression/analyses {"seconds": 30}` starts a +job that listens to the channel for up to that long or 25 MiB, whichever +comes first, and runs every codec over the sample. One job runs at a time; +a second request for the same channel and window is handed the running +job; finished jobs are kept ten minutes. `GET +/api/compression/analyses/` reports progress and then the result; +`DELETE` cancels. All three need the controls. + +The result names the boundary, the sample size and duration, the container +as told by the bytes, the transport-stream report (packet size, packets, +null share, PIDs, the bitrate the PCR clock implies) when it is one, and a +row per codec: complete wire bytes with every header counted, stored and +compressed block counts, encode and decode time, and whether every block +restored exactly. The recommendation is what `auto` would do under the +channel's thresholds, and it says "already efficiently compressed" when +nothing beat stored. The runtime and library versions are in `tools`. + +`nixamp compression analyze FILE` runs the same on a file, here, with no +server, at the source boundary. + +## Metrics + +Per channel and generation: `inputBytes` (from the channel), +`representationBytes` (payloads, once), `wireBytes` (payloads plus headers, +summed over every receiver), block counts by mode, the active mode, whether +`auto` is bypassing and until when, block latency (first byte in to encoded +block ready) as last, p50, p95, max over the last 512 blocks, queue depth, +receivers, receivers dropped, codec failures, and the last fallback reason +in plain words: "already efficiently compressed", "receiver does not support +this format", "processing budget exceeded", "original source bytes are +unavailable", "slow listener", "compression is off". A saving is +`inputBytes - representationBytes`; nothing is reported as saved that was +not. + +## HLS packaging + +`hlsPackaging: "fmp4"` wraps a channel's HLS segments as fragmented MP4 +(`seg00012.m4s`) with an initialisation segment named per packager run +(`init-<8 hex>.mp4`, in `#EXT-X-MAP`), instead of MPEG-TS (`seg00012.ts`). +The media is copied either way; ffmpeg cuts at keyframes, so a segment can +run longer than the two-second target when the source's keyframes are +further apart, and the status reports the longest segment the playlist +actually offers rather than the target. The key travels on the +`#EXT-X-MAP` URI as it does on every segment line, and the init segment is +authorised like any segment. An init from a previous run is a different +name and a 404, so old initialisation cannot be paired with new media. + +MPEG-TS remains the default and the fallback. The supported-client matrix +for preferring fMP4 has not been run; do not change the default until it +has. + +## Turning it off + +- One channel: `nixamp compression set --channel --mode off`, or + `PATCH` its policy. Its relays end; its receivers report a disconnect + and, on redial, are answered `COMPRESSION_OFF` and stop. +- The server: `nixamp compression off`, or `PATCH /api/compression + {"enabled": false}`. Every relay ends, none starts, every policy is kept. +- A receiver: `DELETE /api/channels/` on the receiving server. + +None of it touches source media, ordinary listeners, or the original files +behind static representations. The cache directory (`relay-cache` under the +state directory) can be deleted at any time; representations are rebuilt on +demand. + +## What is not here yet + +- A live source relayed at the `source` boundary: needs a tee in front of + ffmpeg for direct HTTP(S) transport-stream sources. +- The PWA and desktop controls, and MCP tools: the API is the contract they + will call. +- A lower-bitrate quality profile: a separate setting, explicitly labelled, + extending the existing capped encoder. +- Real-feed benchmarks on target hardware, the client matrix for fMP4, and + the 24-hour soak: the gates for changing any default. Everything here is + off until they pass. +- birnpack: an offline benchmark adapter only, once it has a streaming form. diff --git a/package.json b/package.json index d994e71..c4f4e54 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "nixamp", - "version": "0.15.1", + "version": "0.17.0", "description": "It really whips the terminal's ass. A Winamp-shaped audio player for your terminal.", "license": "MIT", "type": "module", diff --git a/prd/0000-template.md b/prd/0000-template.md new file mode 100644 index 0000000..85a4b19 --- /dev/null +++ b/prd/0000-template.md @@ -0,0 +1,50 @@ +--- +openprd: "0.2" +id: "0000" +title: "Short imperative title — start with a verb if possible" +status: Draft +authors: + - you@example.com +created: 2026-01-01 +updated: 2026-01-01 +repo: +discussion: +implementation: +tags: +supersedes: +superseded-by: +--- + +## Problem + +The user/business problem, and why it matters now. Cite the ask, the incident, +or the constraint — not aesthetics. + +## Goals + +What success looks like, as outcomes (not features). + +## Non-Goals + +Explicitly out of scope, to bound the work. + +## Users + +Who this is for; personas or segments. + +## Requirements + +- R1 [P0] First required capability. +- R2 [P1] Next capability. + +## UX Notes + +Flows, states, and constraints that shape the experience. + +## Success Metrics + +How the goals will be measured. + +## Risks & Open Questions + +- Known risk or decision still owed. diff --git a/prd/0001-add-lossless-stream-compression-and-efficient-hls-delivery-to-nixamp.md b/prd/0001-add-lossless-stream-compression-and-efficient-hls-delivery-to-nixamp.md new file mode 100644 index 0000000..6a4b69e --- /dev/null +++ b/prd/0001-add-lossless-stream-compression-and-efficient-hls-delivery-to-nixamp.md @@ -0,0 +1,320 @@ +--- +openprd: "0.3" +id: "0001" +title: Add lossless stream compression and efficient HLS delivery to NixAmp +status: Draft +authors: + - anthony@profullstack.com +owner: anthony@profullstack.com +repo: profullstack/nixamp +created: "2026-09-11" +updated: "2026-09-11" +discussion: "https://www.reddit.com/r/compression/comments/1wdc46u/comment/p97uoz0/" +implementation: +tags: [nixamp, streaming, compression, lossless, mpegts, hls, performance] +supersedes: +superseded-by: +--- + +## Problem + +NixAmp receives MPEG transport streams (`.ts`) from static files and live sources. Operators want to reduce transferred and stored bytes without degrading the picture or sound, increasing buffering, or breaking existing players. The requested capability is an additional lossless compression layer over binary stream data, with a format-aware algorithm investigated where ordinary compression leaves useful redundancy behind. + +A `.ts` container is not necessarily uncompressed media. Container redundancy and already-encoded audio/video must be measured separately. Three operations must remain distinct: + +| Operation | Preservation contract | Intended outcome | +| --- | --- | --- | +| Lossless transport compression | Decompression reproduces every input byte, including padding and metadata. | Fewer bytes on a controlled connection or in a stored representation. | +| Repackaging / remuxing | Selected encoded media is copied; the container and possibly framing change. | More efficient, compatible delivery without media re-encoding. | +| Lower-bitrate encoding | The media is re-encoded; output is not byte-identical. | Smaller media with explicitly accepted quality and compute tradeoffs. | + +The source review for this proposal used NixAmp commit `38c2641354a456919ef1377062202061a1de7891`, not a verified production deployment. At that revision, `src/audio.ts` contains codec-aware fragmented-MP4 output and a capped encoding path; `src/channels.ts` shares channel output with listeners; and `src/hls.ts` copies channel media into MPEG-TS HLS segments. These are integration points, not instructions to replace the server. [S3–S5] + +The Reddit discussion led to birnpack, an experimental byte-prediction compressor. Its current file-oriented implementation and published non-video results do not establish suitability for endless live streams. It is a benchmark candidate, not a production dependency. [S6] + +The earlier research bundle is exploratory evidence only. Its synthetic results do not predict savings on production sources. The implementation must reproduce relevant experiments and benchmark authorized real feeds before enabling compression by default. + +## Goals + +Reduce total delivered bytes where measurable redundancy exists, while preserving exact bytes in lossless mode and leaving incompressible data on an efficient passthrough path. + +Support static files and indefinite live streams with bounded memory, bounded processing queues, independent recovery points, and shared work across viewers. Preserve NixAmp's existing authentication, channel lifecycle, playback, and compatibility behavior. + +Add efficient HLS packaging independently of binary compression. Expose an optional, explicitly labeled lower-bitrate mode by extending the existing encoder rather than conflating transcoding with lossless compression. + +Provide one capability through the existing server, CLI, API, MCP, PWA, and desktop surfaces. Publish reproducible measurements and an experimental TS-aware transform without claiming unproven savings or algorithmic novelty. + +## Non-Goals + +This proposal does not promise that every binary input becomes smaller, invent a new audio/video codec, replace FFmpeg, bypass content protection, or introduce DRM circumvention. + +It does not send a proprietary compressed payload to an unchanged native HLS player, require a browser extension, or turn NixAmp into a separate compression SaaS. A custom browser decoder, trained shared dictionaries, codec-level entropy recoding, and cross-user content deduplication are outside v1. + +Changing a container is not a byte-exact archival operation. Discarding null packets, audio tracks, subtitles, program information, or timing data is not allowed in exact-byte mode. HTTP response compression after ingestion does not reduce traffic on the upstream provider-to-NixAmp link. + +## Users + +**Server operators** configure policies, inspect savings and resource use, and roll back without interrupting unrelated channels. + +**Broadcasters and relay operators** move authorized static or live sources between NixAmp instances, including constrained uplinks, without changing original media bytes. + +**Viewers** use the existing mobile-first player, PWA, desktop app, or an external compatible player without being asked to understand compression algorithms. + +**Developers and agents** manage the same policies and diagnostics through the CLI, API, and MCP, with the same permissions and structured errors. + +## Requirements + +Priorities: P0 is required for the initial production-capable feature; P1 follows the compatibility and resource gates; P2 is experimental research. Configuration and routes below are proposed contracts, not claims that these commands already exist. + +### Preservation and architecture + +- R1 [P0] Distinguish `losslessCompression`, `hlsPackaging`, and `qualityProfile` in configuration, telemetry, APIs, and UI; changing lossless compression must never silently enable re-encoding. +- R2 [P0] Define exactness relative to a named byte boundary: `source` means bytes captured before FFmpeg, while `channel` means bytes emitted by the existing channel pipeline; record the boundary in every analysis and negotiated relay session. +- R3 [P0] Integrate a shared compression stage into the existing streaming architecture, with at most one active compression result per source generation, boundary, and policy variant, rather than one compressor per viewer. +- R4 [P0] Keep existing media endpoints and compatibility behavior unchanged by default; introduce compressed delivery only through an explicitly negotiated NixAmp-controlled relay representation. + +Proposed flow: + +```text +Static .ts or live source + ├─ bounded source tee, when supported and explicitly enabled + │ → exact-source relay compressor + │ → authenticated NixAmp relay receiver + │ → original source bytes → receiver's existing media pipeline + │ + └─ existing FFmpeg / channel pipeline + ├─ ordinary MP4 / MP3 delivery + ├─ existing TS HLS or new fMP4 HLS variant + └─ optional exact-channel relay compression + → authenticated NixAmp relay receiver + → original channel-output bytes → ordinary delivery +``` + +A source tee must be a bounded streaming operation, not an additional full download. Where FFmpeg owns source acquisition and the original bytes are unavailable, exact-source mode must return `SOURCE_BOUNDARY_UNAVAILABLE` or use an explicitly selected channel boundary. It must not label a remuxed or transcoded output as original source bytes. Support authorized local files and direct HTTP(S) TS sources first; do not silently proxy unsupported protocols through a new ingestion path. + +### Diagnostics and adaptive compression + +- R5 [P0] Add a bounded analyzer that reports container and codec information, TS packet layout, null-packet share, observed bitrate, and complete-wire-size compression results; inspect bytes rather than trusting the filename alone. +- R6 [P0] Benchmark identity, gzip, and Zstandard on identical samples and block boundaries, reporting encoder/decoder time, memory, framing cost, exact round-trip verification, source provenance, and tool versions. +- R7 [P0] Implement `off`, `auto`, and `zstd` lossless policies with a maintained, pinned Zstandard implementation; use stored frames whenever compression does not beat the complete stored representation by the required margin. +- R8 [P0] Run adaptive selection without holding playback for a full diagnostic sample; evaluate bounded blocks, bypass low-value compression, and periodically resample after cooldown or a source-generation change. +- R9 [P0] Run compression and decompression outside the server's JavaScript event loop with bounded worker concurrency and explicit cancellation, timeout, and resource limits. + +Initial tunable defaults, subject to measured release gates: + +```json +{ + "losslessCompression": { + "mode": "off", + "boundary": "channel", + "zstdLevel": 1, + "minSavingsPercent": 3, + "minSavingsBytes": 512, + "maxBlockBytes": 262144, + "maxHoldMs": 100, + "resampleAfterMs": 60000, + "maxChannelQueueBytes": 8388608, + "maxListenerQueueBytes": 1048576 + }, + "hlsPackaging": "mpegts", + "qualityProfile": "source" +} +``` + +`off` preserves existing delivery. `auto` uses measured eligibility; `zstd` chooses the codec but still stores incompressible blocks. A compressed relay can carry stored blocks, so bypass does not require a protocol change mid-session. Eligibility compares all representation bytes, not compressed payload alone. Report any framing expansion against the original unwrapped byte stream honestly. + +The 100 ms hold limit starts when the first byte enters a block. Flush at the byte limit or time limit, whichever comes first. Do not wait indefinitely for TS alignment or a large block; carry unsupported or incomplete data as stored bytes. Sample limits default to 30 seconds or 25 MiB, whichever occurs first. Reuse bounded copies of an existing ingest where possible; a new source connection requires an explicit authorized diagnostic request. + +### Live relay, recovery, and security + +- R10 [P0] Specify and implement a versioned binary relay envelope with explicit algorithm negotiation, source generation, sequence number, original length, encoded length, original-byte integrity check, and an unambiguous clean-end marker. +- R11 [P0] Make each compressed block independently decodable without an unbounded shared history; reconnects must start a new generation or resume from a validated checkpoint rather than mix bytes from different source generations. +- R12 [P0] Bound all queues and apply transport-aware backpressure; disconnect and resynchronize slow live listeners without blocking healthy listeners or dropping arbitrary bytes inside a continued media stream. +- R13 [P0] Negotiate compression only between authenticated endpoints advertising a compatible decoder; retain ordinary playback URLs and gracefully fall back before a compressed session begins when support is absent. +- R14 [P0] Validate versions, modes, lengths, sequence continuity, checksums, decoder window limits, and expansion budgets before forwarding decoded bytes; reject corruption and unsupported frames rather than silently emitting damaged media. +- R15 [P0] Preserve existing source-access controls and validate newly introduced fetches against SSRF, redirect, credential-forwarding, path-traversal, and tenant-isolation threats; never place credentials in compression dictionaries or metric labels. + +Publish the envelope's byte layout, endianness, negotiation examples, and cross-language test vectors under `docs/stream-compression.md` before the first transport implementation merges. Use a dedicated custom media type, provisionally `application/vnd.nixamp.stream`, rather than pretending the envelope is an ordinary `.ts` response or a standard HTTP Zstandard body. Do not apply a second HTTP compression layer to the envelope. + +Each block declares its actual mode, including `stored`. The stored-frame integrity path is identical to the compressed path. SHA-256 of original bytes is the initial integrity contract; transport authentication still comes from the authenticated TLS connection, not that hash. Reject length fields above negotiated limits before allocating. Initial decoded-frame limit is 256 KiB; compressed data, frame headers, and decoder windows have separately enforced limits. + +The custom transport's independent blocks are not necessarily media random-access points. A joining viewer still needs the appropriate initialization data and codec keyframe. Recovery must therefore reuse existing channel initialization/backlog logic or restart the receiving demuxer and obtain valid media initialization. An incomplete static transfer must never be reported as a complete file; an unexpected live EOF must be reported as a disconnect. + +For live overflow, close that listener and reconnect at a valid media boundary. For finite file transfer, pause or resume with a verified byte/checkpoint offset. Do not pause an indefinitely producing shared upstream merely to accommodate one slow consumer. Fix ignored `write()` backpressure signals in touched channel/packager paths as part of this work. + +### Static files and HLS delivery + +- R16 [P0] Support precomputed, immutable compressed representations for authorized static files, with atomic publication, source-identity validation, bounded retention, corruption detection, and the original file retained. +- R17 [P0] Preserve byte-range and seek semantics by keeping the original representation available; never apply original-file offsets to compressed bytes or serve a whole-file compressed cache entry as an arbitrary original range. +- R18 [P0] Add opt-in fMP4 HLS packaging alongside MPEG-TS HLS without re-encoding selected compatible media; correctly serve initialization segments, playlists, media segments, and discontinuities. +- R19 [P0] Extend HLS routing, MIME types, authorization, cleanup, generation-specific filenames, and cache behavior for `.m4s` media and initialization `.mp4` files, including authentication of every `EXT-X-MAP` URI. +- R20 [P0] Validate segment independence against actual source keyframes and timestamps; do not advertise independent segments or guaranteed two-second cuts solely because an HLS flag or duration target was set. +- R21 [P1] Permit fMP4 HLS as the preferred packaging only after the supported-client matrix passes, retaining the TS variant for clients or source codecs requiring it. + +The static cache key must include content identity, representation boundary, algorithm/format version, and settings. When byte identity cannot be established cheaply, do not reuse a questionable entry. Include source modification checks and a verified identity before final publication. Keep media on existing storage volumes; do not place media blobs in PostgreSQL. + +Initial static seeking uses the uncompressed endpoint or standard HLS representation. A seekable compressed archive/index is not required for v1. The custom relay representation may explicitly reject Range requests; it must not silently return the wrong range. Preserve authorization on all cached variants; generation-specific resource names must prevent an old initialization segment from being combined with new media. + +At the reviewed revision, HLS uses a two-second target and six listed segments. Retain current defaults unless tests justify changes. Copying media does not create missing keyframes. Re-encoding to change GOP structure requires the separate quality policy. [S5] + +### Existing encoder and product surfaces + +- R22 [P1] Extend the existing bitrate-capped encoding path into an explicitly selected lower-bitrate quality option, sharing each rendition across viewers and keeping `source` quality as the default policy. +- R23 [P0] Expose authorized policy reads, conditional updates, diagnostic jobs, cancellation, and structured compression metrics through the existing server API without creating a separate service or identity system. +- R24 [P0] Expose CLI analysis, status, and policy commands using those same handlers, with JSON output on stdout, progress on stderr, nonzero error exits, and no credentials in arguments where safer existing mechanisms are available. +- R25 [P1] Add mobile-first PWA and desktop controls for lossless savings, packaging, and lower-bitrate mode as distinct settings, with visible effective policy and fallback reasons. +- R26 [P1] Expose matching MCP tools with the same schemas, owner/control authorization, job limits, and audit events; read-only tokens must not change policies or start expensive unbounded work. +- R27 [P0] Export per-boundary input/output bytes, complete representation bytes, active codec, stored-block share, processing latency, queue depth, worker health, cache behavior, and fallback reason without claiming hypothetical savings as realized savings. + +Proposed API surface, aligned with the existing `/api/channels` namespace: + +| Method and route | Purpose | +| --- | --- | +| `GET /api/channels/:id/compression` | Configured policy, effective policy, boundary, and measured status. | +| `PATCH /api/channels/:id/compression` | Validated policy change with an expected policy version / `If-Match`. | +| `POST /api/channels/:id/compression/analyses` | Start a bounded, deduplicated analysis; return a job ID. | +| `GET /api/compression/analyses/:jobId` | Authorized job status and reproducible result. | +| `DELETE /api/compression/analyses/:jobId` | Cancel a diagnostic job. | +| `GET /api/channels/:id/relay` | Explicitly negotiated compressed channel relay; not a public replacement playback URL. | + +Exact-source relay acquisition must be bound to an existing authorized source record, not an arbitrary URL appended to a public endpoint. Static diagnostics and configured source relays reuse that source-access layer. Diagnostic job identifiers and results are scoped to the requesting owner or authorized server role. + +Proposed CLI examples: + +```bash +nixamp compression analyze ./sample.ts --seconds 30 --format json +nixamp compression analyze --channel main --seconds 30 --format json +nixamp compression status --channel main --format json +nixamp compression set --channel main --mode auto --boundary channel +nixamp compression set --channel main --mode off +``` + +Proposed MCP tools: `nixamp_compression_analyze`, `nixamp_compression_status`, and `nixamp_compression_set`. Do not present these interfaces as installed until implementation and tests exist. + +Policy updates are audited and idempotent where applicable. Apply a compatible compression-policy change at a complete frame boundary; changing boundary, envelope version, media initialization, or rendition requires a documented reconnect/discontinuity. Do not silently mutate an active stream's contract. + +### Format-aware research and rollout + +- R28 [P1] Implement a feature-flagged `ts-zstd` experimental transform that groups original TS headers/adaptation data and payload bytes reversibly, preserving full null packets, ordering, timestamps, continuity counters, and all exceptions. +- R29 [P1] Select TS-aware mode only for validated packet layouts and when its complete framed output beats the baseline; preserve unsupported, malformed, scrambled, or unrecognized layouts through byte-exact stored or ordinary Zstandard modes. +- R30 [P2] Benchmark birnpack in an isolated, offline adapter with pinned source, timeouts, memory limits, and exact round trips; it cannot enter the live path without a streaming design and the same release gates as other codecs. +- R31 [P0] Add deterministic fixtures, property-based tests, cross-decoder vectors, corruption tests, player integration tests, and soak/load tests for both finite files and live feeds before enabling production traffic. +- R32 [P0] Ship staged feature flags and a per-channel/global kill switch, with metrics-only observation first, opt-in canaries second, and defaults changed only after documented compatibility and performance acceptance. +- R33 [P0] Commit the OpenPRD, protocol documentation, tests, benchmark harness, fixtures with redistribution rights, and operator rollback instructions to the NixAmp repository using its existing open-source license and contribution workflow. + +The TS-aware experiment initially targets verified 188-byte TS packets. Detect 192/204-byte variants but leave them untouched unless separately implemented and tested. Never reconstruct a supposedly standard null packet instead of preserving its original bytes. Preserve incomplete tails and any unsynchronized regions; no source byte may disappear during parsing or regrouping. Choose among stored, ordinary Zstandard, and transformed Zstandard using the entire output size. + +Treat this as a reversible preprocessing technique plus an established compressor. Claims of algorithmic novelty require a separate prior-art review. Promotion requires a reproducible improvement on a held-out representative TS corpus, not just on padded synthetic fixtures; if it does not beat the baseline under the latency budget, retain it as research-only. + +Suggested integration locations, adjusted to actual repository conventions: + +| Location | Change | +| --- | --- | +| `src/audio.ts` | Preserve existing codec and bitrate logic; extend only the explicit quality path. | +| `src/channels.ts` | Shared output taps, bounded queues, generation handling, and slow-listener recovery. | +| `src/hls.ts` | TS/fMP4 variants, init/media lifecycle, valid keyframe/discontinuity behavior. | +| `src/server.ts` | Protected routes, capability negotiation, metrics, and resource serving. | +| New `src/compression/` modules | Codec adapters, framing, worker management, policy, analysis, and optional TS transform. | +| Existing web/desktop/CLI/MCP adapters | Thin clients over shared service contracts; no duplicated compression logic. | +| `test/`, `scripts/`, `docs/` | Round-trip fixtures, benchmark/soak harness, wire specification, and operational guide. | + +Implementation sequence: establish baseline and bounded queues; implement negotiated identity/Zstandard relay and static caching; add optional fMP4 HLS; canary against the client matrix; expose UI/MCP controls and the existing capped quality path; evaluate TS-aware and birnpack research independently. No algorithm experiment blocks useful baseline delivery. + +## UX Notes + +Ordinary viewers see the existing player. They do not select compressors or install a decoder. Relay compression is terminated by a compatible NixAmp receiver, which serves normal media to downstream clients. + +The operator's channel settings separate **Lossless bandwidth savings**, **HLS packaging**, and **Video/audio quality**. Show configured versus effective settings, measurement window, representation boundary, and a plain-language reason when bypassed: “Already efficiently compressed,” “Receiver does not support this format,” “Processing budget exceeded,” or “Original source bytes are unavailable.” + +“Analyze” starts a bounded job with progress and cancellation. Results distinguish source-to-server, server-to-relay, and server-to-viewer links. A source-boundary saving must not be presented as a viewer-delivery saving. Remuxed output must not carry a “byte-identical original” badge. + +Lower-bitrate mode displays “Re-encodes media; picture or sound may change.” Packaging changes display “Changes the container; does not re-encode media in this stage.” Lossless mode displays “Restores the exact bytes at the selected boundary.” + +Use existing mobile-first panels, keyboard/accessibility conventions, and desktop components. Do not redesign NixAmp's player as part of this feature. Rollback is a clearly labeled operator action; viewers either remain on an unaffected ordinary endpoint or receive a bounded reconnect through the established player recovery flow. + +## Tech Stack + +Extend `profullstack/nixamp`; do not introduce a greenfield service or migrate its architecture. + +Use the existing TypeScript/ESM server and CLI, its supported Node.js runtime and Bun-compatible workflow, FFmpeg/ffprobe, existing test runner, and repository package manager/lockfile. The reviewed README specifies Node 24 or newer for the npm distribution; verify the working tree and deployed runtime before choosing bindings. [S7] + +Use a maintained native Zstandard binding or a bounded persistent native worker compatible with the supported platforms. Pin versions, verify licensing, and test deterministic decode compatibility. Avoid synchronous codec calls on the request/event-loop thread and avoid spawning a process for every block. Use the current worker/process conventions where suitable. + +Retain self-hosted PostgreSQL for account-backed configuration, audit records, and durable analysis summaries where persistence is needed. Do not introduce Turso, SQLite, Redis, or another datastore merely for this feature. Standalone local NixAmp instances retain their existing local configuration/state model and must not require PostgreSQL just to play or relay a stream. + +Use existing writable storage volumes for HLS and static caches, with quotas and cleanup. Object storage, Cloudflare R2, and a paid transcoding vendor are not required. Retain the existing deployment paths, including self-hosted Linux/Docker and supported Railway deployments. Native decoder and worker availability must be checked in the actual deployment image. + +Reuse the PWA/desktop UI stack and existing shared components. New controls may use the existing shadcn-style components where available; this feature does not require a framework migration. Native compression runs server-side in v1, not on viewer phones. + +## Monetization + +This is an infrastructure-efficiency capability inside NixAmp, not a new separately billed product. Add no compression fee, subscription requirement, payment provider, or new billing flow. Existing self-hosting, access controls, and monetization remain unchanged. + +Measure operational benefit as actual network/storage reduction against incremental CPU, memory, and storage activity. Compression is not automatically profitable when it saves bytes. Report these measurements without assigning invented infrastructure prices; operators may supply their own unit costs for an optional estimate. + +Optional future paid transcoding capacity or hosting tiers require a separate product decision. This PRD does not authorize them. + +## Success Metrics + +These are proposed release gates, not claims of achieved production performance. Record hardware, runtime, versions, workload, source identities, and the measurement interval for every result. + +| Gate | Acceptance criterion | +| --- | --- | +| Exactness | Every accepted lossless fixture and transferred static file restores byte-for-byte; SHA-256 matches at the declared boundary. Every live frame validates before downstream delivery. | +| Adaptive savings | Compressed blocks satisfy both the configured 3% and 512-byte net-saving thresholds versus the complete stored frame; all other blocks are stored. Also report aggregate bytes versus unwrapped original traffic, including envelope overhead. | +| Incompressible data | No misleading positive savings; additional transmitted bytes are limited to the documented envelope/end-marker overhead when all blocks are stored. | +| Live delay | On approved target hardware and admitted concurrency, p95 added delay from first byte entering a block to decoded block availability is at most 150 ms; separately report p99, startup, and media latency. | +| Processing headroom | Encoder and decoder sustain at least twice the tested aggregate peak input rate on the admitted workload; otherwise lower concurrency or bypass compression. | +| Memory and queues | Configured channel/listener/global limits hold; a 24-hour soak shows no sustained memory or queue growth after warm-up. | +| Viewer compatibility | Supported Chrome/Chromium, Firefox, Safari/iOS, desktop, and representative external HLS/TV clients pass their ordinary media path or an explicit TS fallback. | +| HLS integrity | Authenticated init and media requests work; keyframe joins, discontinuities, reconnects, live cleanup, and static seeking pass. No init/media generation mixing. | +| Isolation | A slow viewer, bad source, crashing codec worker, or cancelled diagnostic job cannot stall healthy unrelated channels. | +| Work sharing | Adding viewers to the same channel/representation does not create additional encoder/compressor instances; increased network-write work is measured separately. | +| Recovery | Decoder failures and source restarts cause an explicit bounded reconnect or error, never silent corruption or false completion. | +| Rollback | Global/per-channel disable stops new compressed sessions and returns active relays through a documented safe restart path without changing source media or deleting originals. | + +The test corpus must include authorized real static and live samples, low- and high-motion video, audio-only streams, padded and unpadded TS, already-efficient fMP4, random bytes, tiny inputs, empty finite files, multiple programs, multiple audio tracks, subtitles/metadata, damaged packets, timestamp discontinuities, variable bitrate, and 188/192/204-byte packet layouts. Unsupported layouts must prove exact passthrough, not silently fail detection. + +Test disconnections mid-header and mid-payload, corrupt lengths/checksums, decompression bombs, stale caches, source mutation during preprocessing, worker death, unavailable FFmpeg/codecs, unauthorized init requests, redirects, slow clients, and concurrent policy updates. Test at the operator-declared channel/viewer scale; do not infer server capacity from a single-stream benchmark. + +For TS-aware promotion, publish both its incremental gain over ordinary Zstandard and its incremental CPU/latency cost on held-out sources. No mandatory improvement percentage is promised before measurement. If the transform fails to outperform the baseline usefully, `auto` must not select it. + +## Risks & Open Questions + +**Limited compressibility.** Some sources will have little redundancy after existing encoding and repackaging. That is a valid bypass outcome, not a reason to force CPU-intensive compression or quietly lower quality. + +**Source-boundary availability.** Direct FFmpeg acquisition may hide original TS bytes. The first implementation must document which source adapters support an exact pre-FFmpeg tee and reject unsupported requests explicitly. Refactoring ingestion must preserve reconnect, pacing, source credentials, and cancellation behavior. + +**Live behavior and compatibility.** Independent compression blocks do not create video keyframes. HLS segment targets are not guarantees under stream copy. Native player behavior must be checked using real clients and supported codecs, especially initialization and authenticated fMP4 delivery. + +**Codec resource and security exposure.** Native decoder bugs, hostile frames, unbounded windows, and poorly handled backpressure can turn modest bandwidth savings into an outage. Enforce admission limits, isolate workers, pin dependencies, and retain kill switches. Compression is not encryption; use authenticated TLS and avoid cross-tenant compression state. + +**Cost and metrics.** Input-byte counts, channel-produced bytes, cache storage, and bytes delivered to multiple viewers are different measurements. Report each separately. Defaults cannot be justified by synthetic padding savings or by assuming the development machine matches production. + +**Experimental algorithm maturity.** TS-aware preprocessing and birnpack remain gated until reproducible benchmarks, cross-platform decode tests, licensing review, and resource tests pass. Do not market either as a new universal binary compressor. + +**Repository numbering.** This standalone proposal uses `0001`, matching the reviewed root without a `prd/` collection. Before committing, inspect the current collection, assign the next available contiguous ID if necessary, update the filename/front matter together, regenerate the index, and run the OpenPRD validator. No repository changes or number reservations are made by this document. + +Remaining operator inputs are target hardware/concurrency, the supported viewer/TV matrix, and representative authorized feeds. These do not block instrumentation or opt-in implementation; they block changing production defaults. Keep all new behavior disabled until its specific gates pass. + +### Sources and implementation references + +The sources below establish the format and reviewed implementation context, not benchmark guarantees. Verify the working tree before making changes. + +- [S1] OpenPRD 0.3 source specification: . Reviewed file blob: `74d5d0b1df4ccc8e9990acc470ad07ecda1a93f6`. +- [S2] OpenPRD front-matter schema: . Reviewed file blob: `51a44594b6f6a2d2a6b5d5e61b5d61e0c924ae0e`. +- [S3] NixAmp codec selection and capped encoding: . +- [S4] NixAmp channel lifecycle and listener fan-out: . +- [S5] NixAmp HLS packager and URL authorization helper: . +- [S6] birnpack research candidate: . Review the README and `src/welle_fast.c` at a pinned commit before running experiments. +- [S7] NixAmp runtime and deployment guidance: . +- [S8] HLS specification and fMP4 requirements: . +- [S9] FFmpeg HLS muxer reference: . +- [S10] Zstandard reference implementation and format documentation: . + +Repository completion checks: + +```bash +logicsrc prd index --write +logicsrc prd validate --strict +``` diff --git a/prd/README.md b/prd/README.md new file mode 100644 index 0000000..feee7fd --- /dev/null +++ b/prd/README.md @@ -0,0 +1,14 @@ +# LogicSRC PRDs + +Numbered [OpenPRD](https://github.com/profullstack/logicsrc/blob/master/docs/openprd.md) product requirements documents for this repo. One file +per PRD at `prd/-.md`, four-digit ids, no gaps. `0000-template.md` is the copy-paste +starting point. + +Status lives in each file's front-matter and is the source of truth: +`Draft → Review → Accepted → Final`, or `Rejected` / `Withdrawn` / `Superseded`. + + + +| ID | Title | Status | Tags | +| --- | --- | --- | --- | +| [0001](./0001-add-lossless-stream-compression-and-efficient-hls-delivery-to-nixamp.md) | Add lossless stream compression and efficient HLS delivery to NixAmp | Draft | nixamp, streaming, compression, lossless, mpegts, hls, performance | diff --git a/src/channels.ts b/src/channels.ts index bdffe12..461e0e5 100644 --- a/src/channels.ts +++ b/src/channels.ts @@ -25,6 +25,12 @@ import { Fragments, isOpening } from "./fragments.ts"; export interface Listener { write(chunk: Buffer): boolean; end(): void; + /** + * How many bytes it has accepted and not yet sent, when it can say. A + * response's writableLength. Without it a listener that stopped reading + * is a buffer that grows until the channel ends. + */ + pending?(): number; } export interface ChannelInfo { @@ -33,8 +39,8 @@ export interface ChannelInfo { name: string; /** The container it is sending, e.g. webm from a browser, flv over RTMP. */ format: string; - /** How it arrived. */ - via: "http" | "rtmp" | "pull"; + /** How it arrived. `relay` is another nixamp's channel, decoded from its envelope. */ + via: "http" | "rtmp" | "pull" | "relay"; startedAt: number; bytes: number; listeners: number; @@ -100,6 +106,14 @@ const TAIL = 2000; */ export const BACKLOG_VIDEO = 4 * 1024 * 1024; export const BACKLOG_AUDIO = 64 * 1024; +/** + * How far behind a listener may fall before it is let go. Sixteen + * megabytes is half a minute of 720p television that a socket has accepted + * and not delivered: nobody is watching that, and every byte of it was + * sitting in this process. Before this a stalled listener's buffer grew + * until the channel ended, however long that was. + */ +export const LISTENER_QUEUE = 16 * 1024 * 1024; /** The four-letter name in a box header, or "" for something too short. */ function boxType(box: Buffer): string { @@ -145,6 +159,8 @@ export interface ChannelOptions { onEnd?: (info: ChannelInfo) => void; /** How long an on-demand channel outlives its last viewer. Tests shorten it. */ idleMs?: number; + /** Unsent bytes a listener may hold before it is dropped. Tests shrink it. */ + maxListenerQueueBytes?: number; } /** @@ -515,11 +531,22 @@ export class Channel { this.send(chunk); } - /** Write to everyone, and drop anybody whose socket has gone. */ + /** + * Write to everyone, and drop anybody whose socket has gone -- or has + * stopped taking anything. A write that returns false is ordinary: the + * socket is a little behind and will catch up. One that returns false + * with a queue past the limit is a listener that is not reading, and + * ending it is the only thing that stops its queue growing. + */ private send(chunk: Buffer): void { + const cap = this.options.maxListenerQueueBytes ?? LISTENER_QUEUE; for (const listener of this.listeners) { try { - listener.write(chunk); + const drained = listener.write(chunk); + if (!drained && (listener.pending?.() ?? 0) > cap) { + this.listeners.delete(listener); + listener.end(); + } } catch { // One listener's broken socket is not the channel's problem. this.listeners.delete(listener); @@ -528,6 +555,45 @@ export class Channel { this.info.listeners = this.listeners.size; } + /** + * What a new listener is written before the live bytes: the opening + * boxes when there are any, then the recent backlog. The same rule as + * `listen`, handed out so a relay can compress it for one receiver. + */ + opening(): Buffer[] { + const out: Buffer[] = []; + if (this.fragments?.ready) out.push(this.fragments.header); + if (!this.fragments || this.fragments.ready) out.push(...this.recent); + return out; + } + + /** + * Bytes decoded from another nixamp's relay: the channel's own output as + * it was there, so they go out here exactly as ffmpeg's would, whole + * boxes at a time with the backlog kept. + */ + receive(chunk: Buffer): void { + if (this.closing) return; + this.info.bytes += chunk.byteLength; + this.emit(chunk); + } + + /** Ready a channel that will be fed by `receive`: pictures need their boxes tracked. */ + prepare(): void { + if (this.info.kind === "video") this.fragments = new Fragments(); + } + + /** + * The feed behind `receive` started over: a new generation upstream, with + * new opening boxes. Everybody listening is ended, as they are when our + * own ffmpeg is dialled again, and a newcomer gets the new beginning. + */ + rollover(): void { + if (this.closing) return; + this.info.redials = (this.info.redials ?? 0) + 1; + this.startOver(); + } + listen(listener: Listener): () => void { // What the stream is, before any of what it is currently saying. Without // this a listener who arrives after the first second gets fragments that @@ -794,6 +860,34 @@ export class Channels { return channel; } + /** + * A channel carried in from another nixamp's relay. Like `attach`, no + * ffmpeg of our own; unlike it, the kind is known up front, so a picture + * gets its fragment tracking and a newcomer gets the opening boxes. + */ + relayIn(id: string, name: string, kind: "audio" | "video", source: string): Channel | null { + if (this.open.has(id)) return null; + const channel = new Channel( + { id, name: name || id, format: kind === "video" ? "mp4" : "mp3", via: "relay", startedAt: Date.now(), bytes: 0, listeners: 0, kind, source, live: true }, + this.options, + (gone) => this.open.delete(gone), + ); + channel.prepare(); + this.open.set(id, channel); + this.options.onStart?.(channel.info); + return channel; + } + + /** What a new listener would be written first, for a relay's preface. */ + opening(id: string): Buffer[] { + return this.open.get(id)?.opening() ?? []; + } + + /** The kind of a channel, for a relay to say what it is carrying. */ + kindOf(id: string): "audio" | "video" | undefined { + return this.open.get(id)?.info.kind; + } + stop(id: string): boolean { const channel = this.open.get(id); if (!channel) return false; diff --git a/src/compression/analyze.ts b/src/compression/analyze.ts new file mode 100644 index 0000000..9d14878 --- /dev/null +++ b/src/compression/analyze.ts @@ -0,0 +1,270 @@ +/** + * What is in a sample, and what each codec makes of it. + * + * Bytes are inspected, not filenames: a .ts that is really an MP4 is told + * apart by its sync bytes, or their absence. The transport-stream report + * counts what can be counted without decoding anything -- packets, null + * packets, PIDs, the bitrate the PCR clock implies. The benchmark then runs + * every codec over the same blocks and reports the complete wire size, + * headers included, with a round trip checked on every block. Nothing here + * is a promise about production sources; it is a measurement of this + * sample on this machine, and says so. + */ +import { createHash } from "node:crypto"; +import { decode, encode, toolVersions } from "./codec.ts"; +import { type Boundary, FRAME_HEADER_BYTES, type Mode, STREAM_HEADER_BYTES } from "./envelope.ts"; +import { DEFAULT_LOSSLESS, eligible, type LosslessPolicy } from "./policy.ts"; +import { SYNC, TS_PACKET, tsLayout } from "./ts-transform.ts"; + +/** Sample limits: bytes, and seconds of a live channel. */ +export const SAMPLE_MAX_BYTES = 25 * 1024 * 1024; +export const SAMPLE_MAX_SECONDS = 30; + +export type Container = "mpegts" | "fmp4" | "mp4" | "mp3" | "webm" | "unknown"; + +/** Which container the first bytes say they are. */ +export function sniffContainer(bytes: Uint8Array): Container { + if (bytes.length >= 12) { + const box = Buffer.from(bytes.subarray(4, 8)).toString("latin1"); + if (box === "ftyp" || box === "styp") { + const brand = Buffer.from(bytes.subarray(8, 12)).toString("latin1"); + return brand === "iso5" || brand === "iso6" || brand === "dash" || brand === "cmfc" ? "fmp4" : "mp4"; + } + if (box === "moof" || box === "moov" || box === "sidx") return "fmp4"; + } + if (tsLayout(bytes) !== null) return "mpegts"; + if (bytes.length >= 4 && bytes[0] === 0x1a && bytes[1] === 0x45 && bytes[2] === 0xdf && bytes[3] === 0xa3) return "webm"; + if (bytes.length >= 3 && bytes[0] === 0x49 && bytes[1] === 0x44 && bytes[2] === 0x33) return "mp3"; + if (bytes.length >= 2 && bytes[0] === 0xff && ((bytes[1] as number) & 0xe0) === 0xe0) return "mp3"; + return "unknown"; +} + +export interface TsReport { + packetSize: 188 | 192 | 204; + offset: number; + packets: number; + nullPackets: number; + /** Of all packets. Padding a compressor removes for free, and that a copy must keep. */ + nullShare: number; + scrambled: number; + transportErrors: number; + adaptationOnly: number; + /** Bytes not inside an aligned packet: a ragged head or tail. Kept, always. */ + unsynced: number; + pids: { pid: number; packets: number }[]; + /** What the PCR clock says the whole stream runs at, when there is a PCR to read. */ + pcrBitrateKbps?: number; +} + +/** PCR base and extension, as a count of 27 MHz ticks, from an adaptation field that has one. */ +function pcrOf(packet: Uint8Array, at: number): number | null { + const afc = ((packet[at + 3] as number) >> 4) & 0x3; + if ((afc & 0b10) === 0) return null; + const length = packet[at + 4] as number; + if (length < 7) return null; + const flags = packet[at + 5] as number; + if ((flags & 0x10) === 0) return null; + const b = packet.subarray(at + 6, at + 12); + const base = + ((b[0] as number) * 2 ** 25) + ((b[1] as number) << 17) + ((b[2] as number) << 9) + ((b[3] as number) << 1) + ((b[4] as number) >> 7); + const ext = (((b[4] as number) & 0x1) << 8) | (b[5] as number); + return base * 300 + ext; +} + +/** The transport-stream report, or null for something that is not one. */ +export function analyzeTs(bytes: Uint8Array): TsReport | null { + const layout = tsLayout(bytes); + if (layout === null) return null; + const { packetSize, offset } = layout; + const skip = packetSize === 192 ? 4 : 0; + const pids = new Map(); + let packets = 0; + let nullPackets = 0; + let scrambled = 0; + let transportErrors = 0; + let adaptationOnly = 0; + let firstPcr: { pid: number; value: number; at: number } | null = null; + let lastPcr: { value: number; at: number } | null = null; + let cursor = offset; + while (cursor + packetSize <= bytes.length && bytes[cursor + skip] === SYNC) { + const at = cursor + skip; + const b1 = bytes[at + 1] as number; + const b2 = bytes[at + 2] as number; + const b3 = bytes[at + 3] as number; + const pid = ((b1 & 0x1f) << 8) | b2; + packets += 1; + pids.set(pid, (pids.get(pid) ?? 0) + 1); + if (pid === 0x1fff) nullPackets += 1; + if (b1 & 0x80) transportErrors += 1; + if (b3 & 0xc0) scrambled += 1; + if (((b3 >> 4) & 0x3) === 0b10) adaptationOnly += 1; + const pcr = pcrOf(bytes, at); + if (pcr !== null) { + if (firstPcr === null) firstPcr = { pid, value: pcr, at: cursor }; + else if (firstPcr.pid === pid && pcr > firstPcr.value) lastPcr = { value: pcr, at: cursor }; + } + cursor += packetSize; + } + const report: TsReport = { + packetSize, + offset, + packets, + nullPackets, + nullShare: packets === 0 ? 0 : nullPackets / packets, + scrambled, + transportErrors, + adaptationOnly, + unsynced: offset + (bytes.length - cursor), + pids: [...pids.entries()] + .sort((a, b) => b[1] - a[1]) + .slice(0, 12) + .map(([pid, count]) => ({ pid, packets: count })), + }; + if (firstPcr && lastPcr) { + const seconds = (lastPcr.value - firstPcr.value) / 27_000_000; + if (seconds > 0.5) report.pcrBitrateKbps = Math.round(((lastPcr.at - firstPcr.at) * 8) / seconds / 1000); + } + return report; +} + +export interface BenchRow { + mode: Mode; + level: number; + blocks: number; + inputBytes: number; + /** Payload bytes after the eligibility rule: stored where compression did not pay. */ + payloadBytes: number; + /** The complete representation: stream header, every frame header, every payload. */ + wireBytes: number; + storedBlocks: number; + compressedBlocks: number; + /** Against the unwrapped original, headers included. Negative means it grew. */ + savingsPercent: number; + encodeMs: number; + decodeMs: number; + roundTrip: boolean; + /** Set when a mode could not be applied at all, e.g. ts-zstd on an MP4. */ + note?: string; +} + +export interface BenchOptions { + blockBytes?: number; + zstdLevels?: number[]; + policy?: Pick; + /** Try the transport-stream transform too. */ + tsAware?: boolean; + signal?: AbortSignal; +} + +/** + * Every codec over the same blocks. The identity row is the honest floor: + * what the envelope costs when nothing is saved. + */ +export async function benchmark(bytes: Buffer, options: BenchOptions = {}): Promise { + const blockBytes = options.blockBytes ?? DEFAULT_LOSSLESS.maxBlockBytes; + const policy = options.policy ?? DEFAULT_LOSSLESS; + const plan: { mode: Mode; level: number }[] = [{ mode: "stored", level: 0 }, { mode: "gzip", level: 6 }]; + for (const level of options.zstdLevels ?? [1, 3]) plan.push({ mode: "zstd", level }); + if (options.tsAware) plan.push({ mode: "ts-zstd", level: 1 }); + const rows: BenchRow[] = []; + for (const { mode, level } of plan) { + if (options.signal?.aborted) break; + const row: BenchRow = { + mode, level, blocks: 0, inputBytes: bytes.length, payloadBytes: 0, wireBytes: STREAM_HEADER_BYTES + FRAME_HEADER_BYTES, + storedBlocks: 0, compressedBlocks: 0, savingsPercent: 0, encodeMs: 0, decodeMs: 0, roundTrip: true, + }; + try { + for (let at = 0; at < bytes.length; at += blockBytes) { + if (options.signal?.aborted) throw new Error("cancelled"); + const block = bytes.subarray(at, Math.min(bytes.length, at + blockBytes)); + row.blocks += 1; + const t0 = performance.now(); + const encoded = mode === "stored" ? block : await encode(mode, block, level); + row.encodeMs += performance.now() - t0; + const keep = mode !== "stored" && eligible(block.length, encoded.length, policy); + const payload = keep ? encoded : block; + if (keep) row.compressedBlocks += 1; + else row.storedBlocks += 1; + row.payloadBytes += payload.length; + row.wireBytes += FRAME_HEADER_BYTES + payload.length; + const t1 = performance.now(); + const back = keep ? await decode(mode, payload, block.length) : payload; + row.decodeMs += performance.now() - t1; + if (!back.equals(block)) row.roundTrip = false; + } + } catch (error) { + row.note = (error as Error).message; + row.roundTrip = false; + } + row.savingsPercent = bytes.length === 0 ? 0 : ((bytes.length - row.wireBytes) * 100) / bytes.length; + row.encodeMs = Math.round(row.encodeMs * 100) / 100; + row.decodeMs = Math.round(row.decodeMs * 100) / 100; + row.savingsPercent = Math.round(row.savingsPercent * 100) / 100; + rows.push(row); + } + return rows; +} + +export interface Analysis { + /** Which bytes these are: the source as it arrived, or the channel's output. */ + boundary: Boundary; + source: { kind: "file" | "channel" | "bytes"; name: string }; + sampleBytes: number; + sampleMs?: number; + truncated: boolean; + sha256: string; + container: Container; + ts: TsReport | null; + /** From ffprobe, when the caller had one to ask. */ + codecs?: { video: string; audio: string; container: string; duration?: number }; + /** What the sample's own length and duration imply, when a duration is known. */ + observedKbps?: number; + bench: BenchRow[]; + /** The rows' verdict under the policy thresholds: which mode `auto` would take. */ + recommendation: { mode: Mode; level: number; reason: string }; + tools: ReturnType & { ffprobe?: string }; + at: string; +} + +export interface AnalyzeOptions extends BenchOptions { + boundary: Boundary; + source: Analysis["source"]; + sampleMs?: number; + truncated?: boolean; + codecs?: Analysis["codecs"]; + ffprobeVersion?: string; +} + +export async function analyzeSample(bytes: Buffer, options: AnalyzeOptions): Promise { + const ts = analyzeTs(bytes); + const bench = await benchmark(bytes, { ...options, tsAware: options.tsAware ?? ts?.packetSize === TS_PACKET }); + const stored = bench.find((row) => row.mode === "stored"); + const best = bench + .filter((row) => row.roundTrip && row.mode !== "stored") + .sort((a, b) => a.wireBytes - b.wireBytes)[0]; + const policy = options.policy ?? DEFAULT_LOSSLESS; + let recommendation: Analysis["recommendation"]; + if (!best || !stored || !eligible(stored.wireBytes, best.wireBytes, policy)) { + recommendation = { mode: "stored", level: 0, reason: "already efficiently compressed: no codec beat stored by the configured margin" }; + } else { + recommendation = { mode: best.mode, level: best.level, reason: `${best.mode} level ${best.level} saves ${best.savingsPercent}% of the complete wire size on this sample` }; + } + const analysis: Analysis = { + boundary: options.boundary, + source: options.source, + sampleBytes: bytes.length, + truncated: options.truncated ?? false, + sha256: createHash("sha256").update(bytes).digest("hex"), + container: sniffContainer(bytes), + ts, + bench, + recommendation, + tools: { ...toolVersions(), ...(options.ffprobeVersion ? { ffprobe: options.ffprobeVersion } : {}) }, + at: new Date().toISOString(), + }; + if (options.sampleMs !== undefined) analysis.sampleMs = options.sampleMs; + if (options.codecs) analysis.codecs = options.codecs; + const seconds = options.sampleMs !== undefined ? options.sampleMs / 1000 : options.codecs?.duration; + if (seconds && seconds > 0 && !options.truncated) analysis.observedKbps = Math.round((bytes.length * 8) / seconds / 1000); + return analysis; +} diff --git a/src/compression/blocks.ts b/src/compression/blocks.ts new file mode 100644 index 0000000..94b1cb9 --- /dev/null +++ b/src/compression/blocks.ts @@ -0,0 +1,93 @@ +/** + * Bytes into blocks, without ever waiting for the stream's convenience. + * + * A block is flushed when it is full or when its first byte has been held + * for `maxHoldMs`, whichever is sooner. It never waits for a packet + * boundary that has not come, and a chunk bigger than a block is sliced + * rather than refused. Alignment, when asked for, only decides where a full + * block is cut; a timed flush sends whatever is there, and the transform + * that wanted the alignment copes with a ragged edge. + */ +export interface BlockerOptions { + maxBlockBytes: number; + maxHoldMs: number; + /** Cut full blocks at a multiple of this many bytes. 0 for wherever. */ + align?: number; + onBlock: (block: Buffer) => void; + /** Injected by tests. */ + setTimer?: (fn: () => void, ms: number) => { clear(): void }; +} + +const realTimer = (fn: () => void, ms: number): { clear(): void } => { + const handle = setTimeout(fn, ms); + handle.unref?.(); + return { clear: () => clearTimeout(handle) }; +}; + +export class Blocker { + private pieces: Buffer[] = []; + private held = 0; + private timer: { clear(): void } | null = null; + private ended = false; + + constructor(private readonly options: BlockerOptions) {} + + get pendingBytes(): number { + return this.held; + } + + push(chunk: Buffer): void { + if (this.ended || chunk.length === 0) return; + const { maxBlockBytes, align = 0 } = this.options; + let offset = 0; + while (offset < chunk.length) { + const room = maxBlockBytes - this.held; + const take = Math.min(room, chunk.length - offset); + this.pieces.push(chunk.subarray(offset, offset + take)); + this.held += take; + offset += take; + if (this.held >= maxBlockBytes) { + // Cut at the alignment, carrying the remainder into the next block. + // Only a full block is cut this way: a whole block with no boundary + // in it is sent as it is, or nothing would ever leave. + const cut = align > 1 && this.held - (this.held % align) > 0 ? this.held - (this.held % align) : this.held; + this.flushBytes(cut); + } else if (this.timer === null) { + // The clock starts when the first byte enters an empty block. + this.timer = (this.options.setTimer ?? realTimer)(() => { + this.timer = null; + this.flush(); + }, this.options.maxHoldMs); + } + } + } + + /** Send whatever is held, now. */ + flush(): void { + this.flushBytes(this.held); + } + + end(): void { + this.flush(); + this.ended = true; + } + + private flushBytes(count: number): void { + if (this.timer) this.timer.clear(); + this.timer = null; + if (count <= 0 || this.held === 0) return; + const whole = this.pieces.length === 1 ? (this.pieces[0] as Buffer) : Buffer.concat(this.pieces, this.held); + const block = whole.subarray(0, count); + const rest = whole.subarray(count); + this.pieces = rest.length > 0 ? [rest] : []; + this.held = rest.length; + this.options.onBlock(block); + // A remainder is a new block whose first byte arrived just now. + if (this.held > 0 && this.timer === null && !this.ended) { + this.timer = (this.options.setTimer ?? realTimer)(() => { + this.timer = null; + this.flush(); + }, this.options.maxHoldMs); + } + } +} diff --git a/src/compression/cli.ts b/src/compression/cli.ts new file mode 100644 index 0000000..b78d9ba --- /dev/null +++ b/src/compression/cli.ts @@ -0,0 +1,340 @@ +/** + * `nixamp compression` -- measure it, see it, set it, bring a relay in. + * + * nixamp compression analyze ./sample.ts [--seconds 30] [--format json] + * nixamp compression analyze --channel main [--seconds 30] + * nixamp compression status [--channel main] + * nixamp compression set --channel main --mode auto [--level 1] [--ts-aware on] + * nixamp compression set --channel main --hls fmp4 + * nixamp compression off | on + * nixamp compression pull --channel cnn --from https://host:4321/api/channels/cnn/relay --from-key KEY + * nixamp compression fetch URL --out FILE [--key KEY] + * + * A file is analysed here, with no server. Everything else talks to a + * running server over the same routes a browser would: the local daemon + * by default, or --url and --key for another machine, exactly as + * `nixamp admin` finds its target. JSON goes to stdout and only JSON; + * progress and complaints go to stderr; a failure is a nonzero exit. + */ +import { createWriteStream } from "node:fs"; +import { resolveTarget } from "../admin.ts"; +import { detectTools } from "../audio.ts"; +import { KEY_HEADER } from "../share.ts"; +import type { Analysis } from "./analyze.ts"; +import { RelayError } from "./envelope.ts"; +import type { Job } from "./jobs.ts"; +import { receiveRelay, RelayRefused } from "./receiver.ts"; +import { analyzeFile, type ChannelStatus } from "./service.ts"; + +const USAGE = `nixamp compression — lossless relay compression: measure it, see it, set it. + + nixamp compression analyze FILE [--seconds N] [--format json|text] + nixamp compression analyze --channel ID [--seconds N] + nixamp compression status [--channel ID] + nixamp compression set --channel ID [--mode off|auto|zstd] [--level 1-19] + [--boundary channel|source] [--ts-aware on|off] [--hls mpegts|fmp4] + nixamp compression off | on the whole server's switch + nixamp compression pull --channel ID --from URL [--from-key KEY] [--name NAME] + nixamp compression fetch URL --out FILE [--key KEY] + + --url U --key K a server other than the local daemon, as for \`nixamp admin\` + --format json JSON on stdout (the default when stdout is not a terminal) +`; + +interface Flags { + positional: string[]; + named: Map; +} + +function parse(argv: string[]): Flags { + const positional: string[] = []; + const named = new Map(); + for (let i = 0; i < argv.length; i += 1) { + const arg = argv[i] as string; + if (arg.startsWith("--")) { + const eq = arg.indexOf("="); + if (eq !== -1) named.set(arg.slice(2, eq), arg.slice(eq + 1)); + else if (i + 1 < argv.length && !(argv[i + 1] as string).startsWith("--")) named.set(arg.slice(2), argv[(i += 1)] as string); + else named.set(arg.slice(2), "true"); + } else { + positional.push(arg); + } + } + return { positional, named }; +} + +function wantsJson(flags: Flags): boolean { + const format = flags.named.get("format"); + if (format === "json") return true; + if (format === "text") return false; + return !process.stdout.isTTY; +} + +/** Talk to the server the way a browser does, key in the header. */ +async function call(flags: Flags, method: string, path: string, body?: unknown): Promise<{ status: number; body: Record }> { + const target = resolveTarget([ + ...(flags.named.has("url") ? ["--url", flags.named.get("url") as string] : []), + ...(flags.named.has("key") ? ["--key", flags.named.get("key") as string] : []), + ]); + const headers: Record = { accept: "application/json" }; + if (target.key) headers[KEY_HEADER] = target.key; + if (body !== undefined) headers["content-type"] = "application/json"; + const response = await fetch(`${target.url}${path}`, { method, headers, ...(body !== undefined ? { body: JSON.stringify(body) } : {}) }); + let parsedBody: Record = {}; + try { + parsedBody = (await response.json()) as Record; + } catch { + parsedBody = { error: `${response.status} ${response.statusText}` }; + } + return { status: response.status, body: parsedBody }; +} + +function kb(bytes: number): string { + if (bytes < 1024) return `${bytes} B`; + if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KiB`; + return `${(bytes / 1024 / 1024).toFixed(2)} MiB`; +} + +function describeAnalysis(a: Analysis): string { + const lines = [ + ` ${a.source.kind} ${a.source.name}`, + ` boundary ${a.boundary}; sample ${kb(a.sampleBytes)}${a.truncated ? " (truncated)" : ""}${a.sampleMs ? ` over ${(a.sampleMs / 1000).toFixed(1)}s` : ""}; container ${a.container}`, + ]; + if (a.codecs) lines.push(` ffprobe: ${a.codecs.container} video=${a.codecs.video || "-"} audio=${a.codecs.audio || "-"}${a.codecs.duration ? ` ${a.codecs.duration.toFixed(0)}s` : ""}`); + if (a.observedKbps) lines.push(` observed ${a.observedKbps} kbps`); + if (a.ts) { + lines.push(` transport stream: ${a.ts.packetSize}-byte packets, ${a.ts.packets} packets, ${(a.ts.nullShare * 100).toFixed(1)}% null, ${a.ts.pids.length} PIDs${a.ts.pcrBitrateKbps ? `, PCR says ${a.ts.pcrBitrateKbps} kbps` : ""}${a.ts.scrambled ? `, ${a.ts.scrambled} scrambled` : ""}`); + } + lines.push(" mode level wire bytes saving enc ms dec ms round trip"); + for (const row of a.bench) { + lines.push( + ` ${row.mode.padEnd(9)} ${String(row.level).padStart(5)} ${String(row.wireBytes).padStart(10)} ${(row.savingsPercent >= 0 ? "+" : "") + row.savingsPercent.toFixed(2).padStart(6)}% ${row.encodeMs.toFixed(1).padStart(6)} ${row.decodeMs.toFixed(1).padStart(6)} ${row.roundTrip ? "ok" : "FAILED"}${row.note ? ` (${row.note})` : ""}`, + ); + } + lines.push(` recommendation: ${a.recommendation.mode}${a.recommendation.level ? ` level ${a.recommendation.level}` : ""} — ${a.recommendation.reason}`); + lines.push(` ${a.tools.runtime}, zstd ${a.tools.zstd}, sha256 ${a.sha256.slice(0, 16)}…, ${a.at}`); + return lines.join("\n"); +} + +function describeStatus(s: ChannelStatus): string { + const c = s.configured.losslessCompression; + const e = s.effective.losslessCompression; + const lines = [ + ` ${s.channel}${s.live ? "" : " (not on the air)"}`, + ` lossless: configured ${c.mode} at ${c.boundary}, level ${c.zstdLevel}${c.tsAware ? ", ts-aware" : ""}; effective ${e.mode}${s.effective.reason ? ` — ${s.effective.reason}` : ""}`, + ` hls packaging: ${s.effective.hlsPackaging}; quality: ${s.effective.qualityProfile}; policy version ${s.configured.version}`, + ` server switch: ${s.global.enabled ? "on" : "OFF"}`, + ]; + if (s.relay) lines.push(` relaying to ${s.relay.sessions} receiver${s.relay.sessions === 1 ? "" : "s"}, generation ${s.relay.generation}`); + if (s.incoming) lines.push(` fed by ${s.incoming.from} (generation ${s.incoming.generation}, ${s.incoming.reconnects} redials${s.incoming.error ? `, last: ${s.incoming.error}` : ""})`); + const m = s.metrics; + if (m && m.blocks > 0) { + const saved = m.inputBytes - m.representationBytes; + lines.push( + ` this generation: in ${kb(m.inputBytes)}, representation ${kb(m.representationBytes)} (${saved >= 0 ? "-" : "+"}${((Math.abs(saved) * 100) / Math.max(1, m.inputBytes)).toFixed(1)}%), wire ${kb(m.wireBytes)} across listeners`, + ` blocks ${m.blocks}: ${m.compressedBlocks} ${m.activeMode === "stored" ? "compressed" : m.activeMode}, ${m.storedBlocks} stored${m.bypassed ? " (bypassed)" : ""}; latency p50 ${m.latencyMs.p50.toFixed(1)}ms p95 ${m.latencyMs.p95.toFixed(1)}ms; queue ${kb(m.queueBytes)}`, + ); + if (m.fallbackReason) lines.push(` reason: ${m.fallbackReason}`); + } + return lines.join("\n"); +} + +async function analyze(flags: Flags): Promise { + const json = wantsJson(flags); + const channel = flags.named.get("channel"); + const seconds = Number(flags.named.get("seconds") ?? 30); + if (!channel) { + const file = flags.positional[1]; + if (!file) { + console.error(USAGE); + return 2; + } + console.error(` Reading ${file}…`); + const tools = detectTools(); + const result = await analyzeFile(file, { ffprobe: tools.ffprobe }); + console.log(json ? JSON.stringify(result, null, 2) : describeAnalysis(result)); + return 0; + } + const started = await call(flags, "POST", `/api/channels/${encodeURIComponent(channel)}/compression/analyses`, { seconds }); + if (started.status !== 202 && started.status !== 200) { + console.error(`nixamp: ${started.body["error"] ?? started.status}`); + return 1; + } + let job = started.body["job"] as Job; + console.error(` Analysis ${job.id} ${started.body["existing"] ? "already running" : "started"}: up to ${seconds}s of "${channel}"…`); + while (job.status === "queued" || job.status === "running") { + await new Promise((done) => setTimeout(done, 1000)); + const poll = await call(flags, "GET", `/api/compression/analyses/${job.id}`); + if (poll.status !== 200) { + console.error(`nixamp: ${poll.body["error"] ?? poll.status}`); + return 1; + } + job = poll.body["job"] as Job; + if (job.status === "running") process.stderr.write(`\r ${kb(job.progress.bytes)} in ${(job.progress.ms / 1000).toFixed(0)}s`); + } + process.stderr.write("\n"); + if (job.status !== "done" || !job.result) { + console.error(`nixamp: analysis ${job.status}${job.error ? `: ${job.error}` : ""}`); + return 1; + } + console.log(json ? JSON.stringify(job.result, null, 2) : describeAnalysis(job.result)); + return 0; +} + +async function status(flags: Flags): Promise { + const json = wantsJson(flags); + const channel = flags.named.get("channel"); + const got = await call(flags, "GET", channel ? `/api/channels/${encodeURIComponent(channel)}/compression` : "/api/compression"); + if (got.status !== 200) { + console.error(`nixamp: ${got.body["error"] ?? got.status}`); + return 1; + } + if (json) { + console.log(JSON.stringify(got.body, null, 2)); + return 0; + } + if (channel) { + console.log(describeStatus(got.body as unknown as ChannelStatus)); + return 0; + } + const overview = got.body as { global: { enabled: boolean; hlsPackaging: string }; channels: ChannelStatus[]; pool: Record; cache: { bytes: number; entries: number } | null }; + console.log(` server switch ${overview.global.enabled ? "on" : "OFF"}; hls ${overview.global.hlsPackaging}; pool running ${overview.pool["running"]} queued ${overview.pool["queued"]}${overview.cache ? `; cache ${kb(overview.cache.bytes)} in ${overview.cache.entries} files` : ""}`); + for (const one of overview.channels) console.log(describeStatus(one)); + if (overview.channels.length === 0) console.log(" no channels with a policy or on the air"); + return 0; +} + +async function set(flags: Flags): Promise { + const channel = flags.named.get("channel"); + if (!channel) { + console.error("nixamp: say which channel: --channel ID"); + return 2; + } + const lossless: Record = {}; + const change: Record = {}; + const mode = flags.named.get("mode"); + if (mode) lossless["mode"] = mode; + const boundary = flags.named.get("boundary"); + if (boundary) lossless["boundary"] = boundary; + const level = flags.named.get("level"); + if (level) lossless["zstdLevel"] = Number(level); + const ts = flags.named.get("ts-aware"); + if (ts) lossless["tsAware"] = ts === "on" || ts === "true"; + for (const [flag, key] of [["min-savings-percent", "minSavingsPercent"], ["min-savings-bytes", "minSavingsBytes"], ["max-block-bytes", "maxBlockBytes"], ["max-hold-ms", "maxHoldMs"]] as const) { + const value = flags.named.get(flag); + if (value) lossless[key] = Number(value); + } + if (Object.keys(lossless).length > 0) change["losslessCompression"] = lossless; + const hls = flags.named.get("hls"); + if (hls) change["hlsPackaging"] = hls; + if (Object.keys(change).length === 0) { + console.error("nixamp: nothing to set; see `nixamp compression --help`"); + return 2; + } + const expect = flags.named.get("expect-version"); + const result = await call(flags, "PATCH", `/api/channels/${encodeURIComponent(channel)}/compression`, expect ? { ...change, version: Number(expect) } : change); + if (result.status !== 200) { + console.error(`nixamp: ${result.body["error"] ?? result.status}`); + return 1; + } + console.log(wantsJson(flags) ? JSON.stringify(result.body, null, 2) : describeStatus(result.body as unknown as ChannelStatus)); + return 0; +} + +async function toggle(flags: Flags, enabled: boolean): Promise { + const result = await call(flags, "PATCH", "/api/compression", { enabled }); + if (result.status !== 200) { + console.error(`nixamp: ${result.body["error"] ?? result.status}`); + return 1; + } + console.log(wantsJson(flags) ? JSON.stringify(result.body, null, 2) : ` compression is ${enabled ? "on: channels follow their own policies" : "OFF for the whole server: no new relay starts, and running ones end"}`); + return 0; +} + +async function pull(flags: Flags): Promise { + const channel = flags.named.get("channel"); + const from = flags.named.get("from"); + if (!channel || !from) { + console.error("nixamp: say which channel and where from: --channel ID --from URL"); + return 2; + } + const result = await call(flags, "POST", `/api/channels/${encodeURIComponent(channel)}/relay`, { + from, + ...(flags.named.has("from-key") ? { key: flags.named.get("from-key") } : {}), + ...(flags.named.has("name") ? { name: flags.named.get("name") } : {}), + }); + if (result.status !== 202) { + console.error(`nixamp: ${result.body["error"] ?? result.status}`); + return 1; + } + console.log(wantsJson(flags) ? JSON.stringify(result.body, null, 2) : ` "${channel}" is being brought in from ${from}; \`nixamp compression status --channel ${channel}\` says how it is going`); + return 0; +} + +/** Receive one relay (or a static representation) into a file, checking every frame. */ +async function fetchRelay(flags: Flags): Promise { + const url = flags.positional[1]; + const out = flags.named.get("out"); + if (!url || !out) { + console.error("nixamp: fetch URL --out FILE"); + return 2; + } + const file = createWriteStream(out); + let bytes = 0; + const began = Date.now(); + try { + const result = await receiveRelay({ + url, + key: flags.named.get("key") ?? null, + onStart: ({ codecs, kind }) => console.error(` Receiving${kind ? ` ${kind}` : ""} with ${codecs || "stored"}…`), + onBytes: (chunk) => + new Promise((done) => { + bytes += chunk.length; + if (bytes % (1024 * 1024) < chunk.length) process.stderr.write(`\r ${kb(bytes)}`); + if (file.write(chunk)) done(); + else file.once("drain", done); + }), + }); + await new Promise((done) => file.end(done)); + process.stderr.write("\n"); + console.log(JSON.stringify({ ok: true, out, bytes: result.bytes, frames: result.frames, generation: result.generation, ms: Date.now() - began })); + return 0; + } catch (error) { + file.destroy(); + process.stderr.write("\n"); + if (error instanceof RelayError) console.error(`nixamp: the stream broke a rule: ${error.code}: ${error.message}`); + else if (error instanceof RelayRefused) console.error(`nixamp: refused (${error.status}): ${error.message}`); + else console.error(`nixamp: ${(error as Error).message}`); + return 1; + } +} + +export async function compression(argv: string[]): Promise { + const flags = parse(argv); + const verb = flags.positional[0]; + try { + switch (verb) { + case "analyze": + case "analyse": + return await analyze(flags); + case "status": + return await status(flags); + case "set": + return await set(flags); + case "off": + return await toggle(flags, false); + case "on": + return await toggle(flags, true); + case "pull": + return await pull(flags); + case "fetch": + return await fetchRelay(flags); + default: + console.error(USAGE); + return verb === undefined || verb === "help" || flags.named.has("help") ? 0 : 2; + } + } catch (error) { + console.error(`nixamp: ${(error as Error).message}`); + return 1; + } +} diff --git a/src/compression/codec.ts b/src/compression/codec.ts new file mode 100644 index 0000000..ebb7bd9 --- /dev/null +++ b/src/compression/codec.ts @@ -0,0 +1,202 @@ +/** + * The codecs, and the pool that keeps them off the event loop. + * + * Zstandard and gzip come from node:zlib, which both Node 24 and Bun ship + * with libzstd built in: no native module to build, nothing to pin beyond + * the runtime the package already requires. The asynchronous calls run on + * the runtime's own thread pool, so a block being squeezed never holds a + * request. What this file adds is the discipline around them: a ceiling on + * how many run at once, a queue that refuses rather than grows, a deadline + * on every job, and a limit on how big a decode may get before it is called + * a bomb. + */ +import { constants, gunzip, gzip, zstdCompress, zstdDecompress } from "node:zlib"; +import { type Mode, RelayError } from "./envelope.ts"; +import { tsJoin, tsSplit } from "./ts-transform.ts"; + +export const MIN_ZSTD_LEVEL = 1; +export const MAX_ZSTD_LEVEL = 19; + +/** Encode `bytes` in `mode`. Stored is the identity, so it never goes through the pool. */ +export function encode(mode: Mode, bytes: Buffer, level = 1): Promise { + switch (mode) { + case "stored": + return Promise.resolve(bytes); + case "zstd": + return zstd(bytes, level); + case "gzip": + return new Promise((resolve, reject) => gzip(bytes, { level: Math.min(9, Math.max(1, level)) }, (error, out) => (error ? reject(error) : resolve(out)))); + case "ts-zstd": { + const split = tsSplit(bytes); + if (split === null) return Promise.reject(new RelayError("BAD_MODE", "not a transport stream: ts-zstd does not apply")); + return zstd(split, level); + } + } +} + +/** + * Decode a payload back to its original bytes. `maxOutputLength` is the + * decoded size the frame header promised; a payload that wants to be bigger + * than that is refused before it can be. + */ +export async function decode(mode: Mode, bytes: Buffer, maxOutputLength: number): Promise { + switch (mode) { + case "stored": + return bytes; + case "zstd": + return unzstd(bytes, maxOutputLength); + case "gzip": + return new Promise((resolve, reject) => + gunzip(bytes, { maxOutputLength }, (error, out) => (error ? reject(bomb(error)) : resolve(out))), + ); + case "ts-zstd": { + // The split form is a few bytes longer than the original, never shorter. + const split = await unzstd(bytes, maxOutputLength + 64); + const joined = tsJoin(split); + if (joined === null) throw new RelayError("DECODE_FAILED", "ts-zstd payload did not join back into packets"); + return joined; + } + } +} + +function zstd(bytes: Buffer, level: number): Promise { + const params = { [constants.ZSTD_c_compressionLevel]: Math.min(MAX_ZSTD_LEVEL, Math.max(MIN_ZSTD_LEVEL, level)) }; + return new Promise((resolve, reject) => zstdCompress(bytes, { params }, (error, out) => (error ? reject(error) : resolve(out)))); +} + +function unzstd(bytes: Buffer, maxOutputLength: number): Promise { + return new Promise((resolve, reject) => + zstdDecompress(bytes, { maxOutputLength }, (error, out) => (error ? reject(bomb(error)) : resolve(out))), + ); +} + +/** A decode that failed is a decode that failed; a decode that grew past its limit is named as such. */ +function bomb(error: NodeJS.ErrnoException): RelayError { + if (error.code === "ERR_BUFFER_TOO_LARGE") return new RelayError("FRAME_TOO_LARGE", "payload decodes to more than its frame promised"); + return new RelayError("DECODE_FAILED", error.message); +} + +/** What did the work, for a benchmark that has to be reproducible. */ +export function toolVersions(): { runtime: string; zstd: string; zlib: string } { + const versions = process.versions as Record; + return { + runtime: versions["bun"] ? `bun ${versions["bun"]}` : `node ${process.version}`, + zstd: versions["zstd"] ?? "bundled", + zlib: versions["zlib"] ?? "bundled", + }; +} + +export interface PoolOptions { + /** How many jobs may be in flight at once. */ + concurrency?: number; + /** How many may wait their turn before a new one is refused. */ + maxQueued?: number; + /** How long one job may take before it is abandoned. */ + timeoutMs?: number; +} + +export interface PoolStats { + running: number; + queued: number; + completed: number; + refused: number; + timedOut: number; + failed: number; +} + +/** Thrown for a job the pool would not take, or would not wait for. */ +export class PoolError extends Error { + constructor(readonly code: "BUSY" | "TIMEOUT" | "CANCELLED", message: string) { + super(message); + this.name = "PoolError"; + } +} + +/** + * A bounded queue in front of the codecs. + * + * The codecs themselves already run on other threads; what would hurt is a + * thousand blocks queued behind a slow one, each holding a quarter of a + * megabyte. So there is a ceiling on the queue, and a job past the deadline + * is dropped by whoever asked for it -- the thread finishes and its result + * is thrown away, which for a block of at most a quarter of a megabyte is a + * few milliseconds wasted, not a leak. + */ +export class Pool { + private readonly concurrency: number; + private readonly maxQueued: number; + private readonly timeoutMs: number; + private running = 0; + private readonly waiting: (() => void)[] = []; + private readonly counts = { completed: 0, refused: 0, timedOut: 0, failed: 0 }; + + constructor(options: PoolOptions = {}) { + this.concurrency = Math.max(1, options.concurrency ?? 4); + this.maxQueued = Math.max(0, options.maxQueued ?? 64); + this.timeoutMs = Math.max(1, options.timeoutMs ?? 2000); + } + + get stats(): PoolStats { + return { running: this.running, queued: this.waiting.length, ...this.counts }; + } + + async run(job: () => Promise, options: { timeoutMs?: number; signal?: AbortSignal } = {}): Promise { + if (options.signal?.aborted) throw new PoolError("CANCELLED", "cancelled before it started"); + await this.acquire(options.signal); + const deadline = options.timeoutMs ?? this.timeoutMs; + let timer: ReturnType | null = null; + let onAbort: (() => void) | null = null; + try { + const result = await Promise.race([ + job(), + new Promise((_, reject) => { + timer = setTimeout(() => reject(new PoolError("TIMEOUT", `took longer than ${deadline}ms`)), deadline); + timer.unref?.(); + if (options.signal) { + onAbort = () => reject(new PoolError("CANCELLED", "cancelled")); + options.signal.addEventListener("abort", onAbort, { once: true }); + } + }), + ]); + this.counts.completed += 1; + return result; + } catch (error) { + if (error instanceof PoolError && error.code === "TIMEOUT") this.counts.timedOut += 1; + else if (!(error instanceof PoolError)) this.counts.failed += 1; + throw error; + } finally { + if (timer) clearTimeout(timer); + if (onAbort && options.signal) options.signal.removeEventListener("abort", onAbort); + this.release(); + } + } + + /** Take a slot now, or wait for one to be handed over by `release`. */ + private async acquire(signal?: AbortSignal): Promise { + if (this.running < this.concurrency) { + this.running += 1; + return; + } + if (this.waiting.length >= this.maxQueued) { + this.counts.refused += 1; + throw new PoolError("BUSY", "the codec pool is full"); + } + await new Promise((next) => this.waiting.push(next)); + // Woken: the finishing job passed its slot to us without touching the + // count. If we no longer want it, pass it on the same way. + if (signal?.aborted) { + this.release(); + throw new PoolError("CANCELLED", "cancelled while queued"); + } + } + + /** Give the slot to the next in line, or back to the pool if nobody is waiting. */ + private release(): void { + const next = this.waiting.shift(); + if (next) { + next(); + return; + } + this.running -= 1; + } +} diff --git a/src/compression/envelope.ts b/src/compression/envelope.ts new file mode 100644 index 0000000..2899683 --- /dev/null +++ b/src/compression/envelope.ts @@ -0,0 +1,237 @@ +/** + * The wire shape of a compressed relay: what one nixamp sends another. + * + * Not a .ts file and not an HTTP body somebody's proxy might gunzip: a + * stream of framed blocks, each one independently decodable, each one + * carrying the length and the SHA-256 of the bytes it stands for, ending in + * a marker that says the stream finished rather than dropped. The layout is + * specified byte by byte in docs/stream-compression.md; this file is that + * document as code, and the test vectors there are checked against it. + * + * Everything is big-endian. Every length is checked against the negotiated + * limit before a byte is allocated for it, so a hostile header cannot ask + * for a gigabyte. + */ +import { createHash } from "node:crypto"; + +export const MAGIC = "NXS1"; +export const ENVELOPE_VERSION = 1; +export const MEDIA_TYPE = "application/vnd.nixamp.stream"; + +export const STREAM_HEADER_BYTES = 16; +export const FRAME_HEADER_BYTES = 48; + +/** Where the bytes were captured: before ffmpeg, or after the channel pipeline. */ +export type Boundary = "source" | "channel"; +const BOUNDARY_CODE: Record = { source: 0, channel: 1 }; + +/** How one block's payload was encoded. `stored` is the bytes as they were. */ +export type Mode = "stored" | "zstd" | "gzip" | "ts-zstd"; +export const MODE_CODE: Record = { stored: 0, zstd: 1, gzip: 2, "ts-zstd": 3 }; +export const MODES: readonly Mode[] = ["stored", "zstd", "gzip", "ts-zstd"]; + +export const FRAME_DATA = 1; +export const FRAME_END = 2; + +/** The decoded size one frame may claim, unless negotiated otherwise. */ +export const DEFAULT_MAX_FRAME_BYTES = 256 * 1024; +/** The most a stream header may negotiate, whatever it asks for. */ +export const CEILING_FRAME_BYTES = 16 * 1024 * 1024; +/** + * How much bigger than its original an encoded payload may be. A codec that + * cannot beat stored is stored, so anything past a small fixed allowance is + * either a bug or an attack. + */ +export const EXPANSION_ALLOWANCE = 1024; + +export interface StreamHeader { + version: number; + boundary: Boundary; + /** Which run of the source this is; changes on every restart. */ + generation: number; + /** The decoded-size limit every frame in this stream honours. */ + maxFrameBytes: number; +} + +export interface FrameHeader { + type: typeof FRAME_DATA | typeof FRAME_END; + mode: Mode; + seq: number; + originalLength: number; + encodedLength: number; + /** Of the original bytes for a data frame; of the whole generation for the end. */ + sha256: Buffer; +} + +export type RelayErrorCode = + | "BAD_MAGIC" + | "BAD_VERSION" + | "BAD_BOUNDARY" + | "BAD_LIMIT" + | "BAD_FRAME_TYPE" + | "BAD_MODE" + | "UNSUPPORTED_MODE" + | "BAD_SEQUENCE" + | "FRAME_TOO_LARGE" + | "EXPANSION_BUDGET" + | "CHECKSUM_MISMATCH" + | "LENGTH_MISMATCH" + | "DECODE_FAILED" + | "AFTER_END" + | "TRUNCATED"; + +/** A stream that broke one of the rules, and which rule. Never silently. */ +export class RelayError extends Error { + constructor(readonly code: RelayErrorCode, message: string) { + super(message); + this.name = "RelayError"; + } +} + +export function sha256(bytes: Uint8Array): Buffer { + return createHash("sha256").update(bytes).digest(); +} + +export function encodeStreamHeader(header: StreamHeader): Buffer { + if (header.version !== ENVELOPE_VERSION) throw new RelayError("BAD_VERSION", `cannot write version ${header.version}`); + if (!(header.boundary in BOUNDARY_CODE)) throw new RelayError("BAD_BOUNDARY", `no such boundary: ${header.boundary}`); + if (!Number.isInteger(header.generation) || header.generation < 0 || header.generation > 0xffff_ffff) { + throw new RelayError("BAD_LIMIT", "generation must fit in 32 bits"); + } + if (!Number.isInteger(header.maxFrameBytes) || header.maxFrameBytes < 1 || header.maxFrameBytes > CEILING_FRAME_BYTES) { + throw new RelayError("BAD_LIMIT", `maxFrameBytes must be 1..${CEILING_FRAME_BYTES}`); + } + const out = Buffer.alloc(STREAM_HEADER_BYTES); + out.write(MAGIC, 0, 4, "latin1"); + out.writeUInt8(header.version, 4); + out.writeUInt8(0, 5); // flags, none defined + out.writeUInt8(BOUNDARY_CODE[header.boundary], 6); + out.writeUInt8(0, 7); // reserved + out.writeUInt32BE(header.generation, 8); + out.writeUInt32BE(header.maxFrameBytes, 12); + return out; +} + +/** Read a stream header from the front of `bytes`. Throws on anything off. */ +export function decodeStreamHeader(bytes: Buffer): StreamHeader { + if (bytes.length < STREAM_HEADER_BYTES) throw new RelayError("TRUNCATED", "stream header is short"); + if (bytes.toString("latin1", 0, 4) !== MAGIC) throw new RelayError("BAD_MAGIC", "not a nixamp relay stream"); + const version = bytes.readUInt8(4); + if (version !== ENVELOPE_VERSION) throw new RelayError("BAD_VERSION", `envelope version ${version} is not understood`); + const boundaryCode = bytes.readUInt8(6); + const boundary = (Object.keys(BOUNDARY_CODE) as Boundary[]).find((b) => BOUNDARY_CODE[b] === boundaryCode); + if (!boundary) throw new RelayError("BAD_BOUNDARY", `boundary code ${boundaryCode} is not understood`); + const generation = bytes.readUInt32BE(8); + const maxFrameBytes = bytes.readUInt32BE(12); + if (maxFrameBytes < 1 || maxFrameBytes > CEILING_FRAME_BYTES) { + throw new RelayError("BAD_LIMIT", `maxFrameBytes ${maxFrameBytes} is outside 1..${CEILING_FRAME_BYTES}`); + } + return { version, boundary, generation, maxFrameBytes }; +} + +export function encodeFrameHeader(header: FrameHeader): Buffer { + if (header.type !== FRAME_DATA && header.type !== FRAME_END) throw new RelayError("BAD_FRAME_TYPE", `no such frame type ${header.type}`); + if (!(header.mode in MODE_CODE)) throw new RelayError("BAD_MODE", `no such mode: ${header.mode}`); + if (header.sha256.length !== 32) throw new RelayError("CHECKSUM_MISMATCH", "sha256 must be 32 bytes"); + for (const [name, value] of [["seq", header.seq], ["originalLength", header.originalLength], ["encodedLength", header.encodedLength]] as const) { + if (!Number.isInteger(value) || value < 0 || value > 0xffff_ffff) throw new RelayError("BAD_LIMIT", `${name} must fit in 32 bits`); + } + const out = Buffer.alloc(FRAME_HEADER_BYTES); + out.writeUInt8(header.type, 0); + out.writeUInt8(MODE_CODE[header.mode], 1); + out.writeUInt16BE(0, 2); // reserved + out.writeUInt32BE(header.seq, 4); + out.writeUInt32BE(header.originalLength, 8); + out.writeUInt32BE(header.encodedLength, 12); + header.sha256.copy(out, 16); + return out; +} + +/** A data frame: header then payload, as one buffer. */ +export function encodeDataFrame(seq: number, mode: Mode, original: Buffer, encoded: Buffer): Buffer { + return Buffer.concat([dataFrameHeader(seq, mode, original, encoded.length), encoded]); +} + +/** Just the header of a data frame, for a payload that is shared between listeners. */ +export function dataFrameHeader(seq: number, mode: Mode, original: Buffer, encodedLength: number, digest?: Buffer): Buffer { + return encodeFrameHeader({ + type: FRAME_DATA, + mode, + seq, + originalLength: original.length, + encodedLength, + sha256: digest ?? sha256(original), + }); +} + +/** + * The end marker: no payload, the generation's total original byte count in + * the two length fields (high word, low word) and the SHA-256 of every + * original byte in order. A stream that stops without one was cut off. + */ +export function encodeEndFrame(seq: number, totalOriginalBytes: number, digest: Buffer): Buffer { + if (!Number.isSafeInteger(totalOriginalBytes) || totalOriginalBytes < 0) throw new RelayError("BAD_LIMIT", "total must be a non-negative integer"); + return encodeFrameHeader({ + type: FRAME_END, + mode: "stored", + seq, + originalLength: Math.floor(totalOriginalBytes / 0x1_0000_0000), + encodedLength: totalOriginalBytes % 0x1_0000_0000, + sha256: digest, + }); +} + +/** The total an end frame carries, from its two halves. */ +export function endFrameTotal(header: FrameHeader): number { + return header.originalLength * 0x1_0000_0000 + header.encodedLength; +} + +export interface FrameLimits { + maxFrameBytes: number; + /** Modes this decoder can undo. A frame in any other mode is refused. */ + modes: ReadonlySet; + /** The sequence number expected next. */ + expectSeq: number; +} + +/** + * Read a frame header, checking every field against the limits before the + * caller allocates anything for the payload. + */ +export function decodeFrameHeader(bytes: Buffer, limits: FrameLimits): FrameHeader { + if (bytes.length < FRAME_HEADER_BYTES) throw new RelayError("TRUNCATED", "frame header is short"); + const type = bytes.readUInt8(0); + if (type !== FRAME_DATA && type !== FRAME_END) throw new RelayError("BAD_FRAME_TYPE", `frame type ${type} is not understood`); + const modeCode = bytes.readUInt8(1); + const mode = MODES.find((m) => MODE_CODE[m] === modeCode); + if (!mode) throw new RelayError("BAD_MODE", `mode code ${modeCode} is not understood`); + const seq = bytes.readUInt32BE(4); + if (seq !== limits.expectSeq) throw new RelayError("BAD_SEQUENCE", `expected frame ${limits.expectSeq}, got ${seq}`); + const originalLength = bytes.readUInt32BE(8); + const encodedLength = bytes.readUInt32BE(12); + const sha = Buffer.from(bytes.subarray(16, 48)); + if (type === FRAME_END) { + return { type: FRAME_END, mode: "stored", seq, originalLength, encodedLength, sha256: sha }; + } + if (!limits.modes.has(mode)) throw new RelayError("UNSUPPORTED_MODE", `this decoder cannot undo ${mode}`); + if (originalLength > limits.maxFrameBytes) { + throw new RelayError("FRAME_TOO_LARGE", `frame claims ${originalLength} original bytes, limit ${limits.maxFrameBytes}`); + } + if (encodedLength > originalLength + EXPANSION_ALLOWANCE) { + throw new RelayError("EXPANSION_BUDGET", `frame carries ${encodedLength} bytes for ${originalLength} original`); + } + if (mode === "stored" && encodedLength !== originalLength) { + throw new RelayError("LENGTH_MISMATCH", "a stored frame must carry exactly its original bytes"); + } + return { type: FRAME_DATA, mode, seq, originalLength, encodedLength, sha256: sha }; +} + +/** The codec names a peer lists in a negotiation header, kept to the ones we know. */ +export function parseModes(header: string | undefined | null): Set { + const out = new Set(); + for (const raw of (header ?? "").split(",")) { + const name = raw.trim().toLowerCase(); + if ((MODES as readonly string[]).includes(name)) out.add(name as Mode); + } + return out; +} diff --git a/src/compression/jobs.ts b/src/compression/jobs.ts new file mode 100644 index 0000000..1adfd17 --- /dev/null +++ b/src/compression/jobs.ts @@ -0,0 +1,157 @@ +/** + * Diagnostic jobs: bounded, deduplicated, cancellable, forgotten in time. + * + * An analysis reads up to thirty seconds or twenty-five megabytes of a + * channel and runs every codec over it, which is real work. So there is a + * ceiling on how many run at once, a second request for the same thing + * while the first is still going is handed the first, and a finished + * result is kept for a while and then dropped rather than for ever. + */ +import { randomBytes } from "node:crypto"; +import type { Analysis } from "./analyze.ts"; + +export type JobStatus = "queued" | "running" | "done" | "failed" | "cancelled"; + +export interface Job { + id: string; + /** What is being analysed, for the listing. */ + subject: string; + /** Who may see it: the scope that started it. */ + owner: string; + status: JobStatus; + createdAt: number; + startedAt?: number; + finishedAt?: number; + progress: { bytes: number; ms: number }; + result?: Analysis; + error?: string; +} + +export interface JobsOptions { + concurrency?: number; + /** How long a finished job is kept. */ + ttlMs?: number; + /** How many jobs, in any state, may exist at once. */ + maxJobs?: number; +} + +export type Runner = (signal: AbortSignal, progress: (bytes: number, ms: number) => void) => Promise; + +interface Slot { + job: Job; + key: string; + controller: AbortController; + run: Runner; +} + +export class AnalysisJobs { + private readonly slots = new Map(); + private readonly queue: Slot[] = []; + private running = 0; + private readonly concurrency: number; + private readonly ttlMs: number; + private readonly maxJobs: number; + + constructor(options: JobsOptions = {}) { + this.concurrency = Math.max(1, options.concurrency ?? 1); + this.ttlMs = options.ttlMs ?? 10 * 60_000; + this.maxJobs = options.maxJobs ?? 32; + } + + /** + * Start a job, or return the one already doing the same thing. `key` + * names the thing: the same key while a job is queued or running is the + * same job. Null when the server has all the jobs it will hold. + */ + start(key: string, subject: string, owner: string, run: Runner): { job: Job; existing: boolean } | null { + this.sweep(); + for (const slot of this.slots.values()) { + if (slot.key === key && (slot.job.status === "queued" || slot.job.status === "running")) return { job: slot.job, existing: true }; + } + if (this.slots.size >= this.maxJobs) return null; + const job: Job = { + id: `a${randomBytes(6).toString("hex")}`, + subject, + owner, + status: "queued", + createdAt: Date.now(), + progress: { bytes: 0, ms: 0 }, + }; + const slot: Slot = { job, key, controller: new AbortController(), run }; + this.slots.set(job.id, slot); + this.queue.push(slot); + this.pump(); + return { job, existing: false }; + } + + /** A job, if it exists and `owner` may see it. */ + get(id: string, owner: string): Job | null { + this.sweep(); + const slot = this.slots.get(id); + if (!slot || slot.job.owner !== owner) return null; + return slot.job; + } + + cancel(id: string, owner: string): boolean { + const slot = this.slots.get(id); + if (!slot || slot.job.owner !== owner) return false; + if (slot.job.status !== "queued" && slot.job.status !== "running") return false; + slot.controller.abort(); + if (slot.job.status === "queued") { + const at = this.queue.indexOf(slot); + if (at !== -1) this.queue.splice(at, 1); + slot.job.status = "cancelled"; + slot.job.finishedAt = Date.now(); + } + return true; + } + + list(owner: string): Job[] { + this.sweep(); + return [...this.slots.values()].map((slot) => slot.job).filter((job) => job.owner === owner); + } + + get counts(): { running: number; queued: number; kept: number } { + return { running: this.running, queued: this.queue.length, kept: this.slots.size }; + } + + /** Stop everything, for a server going down. */ + stopAll(): void { + for (const slot of this.slots.values()) slot.controller.abort(); + this.queue.length = 0; + } + + private pump(): void { + while (this.running < this.concurrency && this.queue.length > 0) { + const slot = this.queue.shift() as Slot; + this.running += 1; + slot.job.status = "running"; + slot.job.startedAt = Date.now(); + void slot + .run(slot.controller.signal, (bytes, ms) => { + slot.job.progress = { bytes, ms }; + }) + .then((result) => { + slot.job.result = result; + slot.job.status = slot.controller.signal.aborted ? "cancelled" : "done"; + }) + .catch((error: Error) => { + slot.job.status = slot.controller.signal.aborted ? "cancelled" : "failed"; + slot.job.error = error.message; + }) + .finally(() => { + slot.job.finishedAt = Date.now(); + this.running -= 1; + this.pump(); + }); + } + } + + /** Forget finished jobs older than the TTL. */ + private sweep(): void { + const cutoff = Date.now() - this.ttlMs; + for (const [id, slot] of this.slots) { + if (slot.job.finishedAt !== undefined && slot.job.finishedAt < cutoff) this.slots.delete(id); + } + } +} diff --git a/src/compression/metrics.ts b/src/compression/metrics.ts new file mode 100644 index 0000000..e894de7 --- /dev/null +++ b/src/compression/metrics.ts @@ -0,0 +1,109 @@ +/** + * What was measured, per channel, and only what was measured. + * + * Input bytes are what the channel produced. Wire bytes are what went to + * relay listeners, headers included, counted once per listener because + * each listener is a separate cost on the network. Neither is a saving + * until it is compared with the other, and a block that was stored saved + * nothing -- these counters say so rather than counting what compression + * might have done. + */ +export type FallbackReason = + | "compression is off" + | "already efficiently compressed" + | "receiver does not support this format" + | "processing budget exceeded" + | "original source bytes are unavailable" + | "slow listener"; + +export interface ChannelMetricsSnapshot { + generation: number; + /** Bytes in from the channel since this generation began. */ + inputBytes: number; + /** Payload bytes out, before fan-out: what one listener would receive. */ + representationBytes: number; + /** Payload plus headers, summed over every listener that was sent it. */ + wireBytes: number; + blocks: number; + storedBlocks: number; + compressedBlocks: number; + /** Which codec produced the most recent compressed block, or stored. */ + activeMode: string; + bypassed: boolean; + bypassUntil: number | null; + /** From the first byte of a block entering to its being ready, milliseconds. */ + latencyMs: { last: number; p50: number; p95: number; max: number; samples: number }; + queueBytes: number; + listeners: number; + droppedListeners: number; + codecFailures: number; + fallbackReason: FallbackReason | null; +} + +const LATENCY_WINDOW = 512; + +export class ChannelMetrics { + generation = 0; + inputBytes = 0; + representationBytes = 0; + wireBytes = 0; + blocks = 0; + storedBlocks = 0; + compressedBlocks = 0; + activeMode = "stored"; + bypassed = false; + bypassUntil: number | null = null; + queueBytes = 0; + listeners = 0; + droppedListeners = 0; + codecFailures = 0; + fallbackReason: FallbackReason | null = null; + private latencies: number[] = []; + + latency(ms: number): void { + this.latencies.push(ms); + if (this.latencies.length > LATENCY_WINDOW) this.latencies.shift(); + } + + /** A new run of the source: the per-generation counters start over. */ + reset(generation: number): void { + this.generation = generation; + this.inputBytes = 0; + this.representationBytes = 0; + this.wireBytes = 0; + this.blocks = 0; + this.storedBlocks = 0; + this.compressedBlocks = 0; + this.queueBytes = 0; + this.latencies = []; + } + + snapshot(): ChannelMetricsSnapshot { + const sorted = [...this.latencies].sort((a, b) => a - b); + const at = (q: number): number => (sorted.length === 0 ? 0 : (sorted[Math.min(sorted.length - 1, Math.floor(q * sorted.length))] as number)); + return { + generation: this.generation, + inputBytes: this.inputBytes, + representationBytes: this.representationBytes, + wireBytes: this.wireBytes, + blocks: this.blocks, + storedBlocks: this.storedBlocks, + compressedBlocks: this.compressedBlocks, + activeMode: this.activeMode, + bypassed: this.bypassed, + bypassUntil: this.bypassUntil, + latencyMs: { + last: sorted.length === 0 ? 0 : (this.latencies[this.latencies.length - 1] as number), + p50: at(0.5), + p95: at(0.95), + max: sorted.length === 0 ? 0 : (sorted[sorted.length - 1] as number), + samples: sorted.length, + }, + queueBytes: this.queueBytes, + listeners: this.listeners, + droppedListeners: this.droppedListeners, + codecFailures: this.codecFailures, + fallbackReason: this.fallbackReason, + }; + } +} diff --git a/src/compression/policy.ts b/src/compression/policy.ts new file mode 100644 index 0000000..01fa1f1 --- /dev/null +++ b/src/compression/policy.ts @@ -0,0 +1,171 @@ +/** + * What an operator can set, and what a block has to achieve to be sent + * compressed. + * + * Three settings that must never be confused: whether to squeeze the bytes + * losslessly, how to package HLS, and whether to re-encode the picture. + * Turning one on never turns another on. The first is the only one this + * module measures; the other two are carried here so that one document + * describes a channel, and so that the API can refuse a change that mixes + * them up. + */ +import { type Boundary, DEFAULT_MAX_FRAME_BYTES, CEILING_FRAME_BYTES, FRAME_HEADER_BYTES } from "./envelope.ts"; +import { MAX_ZSTD_LEVEL, MIN_ZSTD_LEVEL } from "./codec.ts"; + +export type PolicyMode = "off" | "auto" | "zstd"; +export type HlsPackaging = "mpegts" | "fmp4"; +/** `source` is the media as it came. Anything else re-encodes and says so. */ +export type QualityProfile = "source"; + +export interface LosslessPolicy { + mode: PolicyMode; + boundary: Boundary; + zstdLevel: number; + /** A block is sent compressed only if it saves at least this much, both ways. */ + minSavingsPercent: number; + minSavingsBytes: number; + maxBlockBytes: number; + /** From the first byte entering a block to its being flushed, at most. */ + maxHoldMs: number; + /** After `auto` has given up on a source, how long before it tries again. */ + resampleAfterMs: number; + /** Bytes the shared compressor may hold for a channel before it stops that relay. */ + maxChannelQueueBytes: number; + /** Unsent bytes one relay listener may fall behind by before it is cut off. */ + maxListenerQueueBytes: number; + /** The experimental transport-stream transform. Off unless asked for. */ + tsAware: boolean; +} + +export interface ChannelPolicy { + losslessCompression: LosslessPolicy; + hlsPackaging: HlsPackaging; + qualityProfile: QualityProfile; + /** Bumped on every accepted change; a conditional update names the one it saw. */ + version: number; +} + +export const DEFAULT_LOSSLESS: LosslessPolicy = { + mode: "off", + boundary: "channel", + zstdLevel: 1, + minSavingsPercent: 3, + minSavingsBytes: 512, + maxBlockBytes: DEFAULT_MAX_FRAME_BYTES, + maxHoldMs: 100, + resampleAfterMs: 60_000, + maxChannelQueueBytes: 8 * 1024 * 1024, + maxListenerQueueBytes: 1024 * 1024, + tsAware: false, +}; + +export const DEFAULT_POLICY: ChannelPolicy = { + losslessCompression: DEFAULT_LOSSLESS, + hlsPackaging: "mpegts", + qualityProfile: "source", + version: 0, +}; + +/** Fresh copies, so nobody edits the defaults in place. */ +export function defaultPolicy(): ChannelPolicy { + return { ...DEFAULT_POLICY, losslessCompression: { ...DEFAULT_LOSSLESS } }; +} + +const RANGES: Record, [number, number]> = { + zstdLevel: [MIN_ZSTD_LEVEL, MAX_ZSTD_LEVEL], + minSavingsPercent: [0, 100], + minSavingsBytes: [0, CEILING_FRAME_BYTES], + maxBlockBytes: [1024, CEILING_FRAME_BYTES], + maxHoldMs: [1, 10_000], + resampleAfterMs: [1000, 24 * 3600 * 1000], + maxChannelQueueBytes: [64 * 1024, 1024 * 1024 * 1024], + maxListenerQueueBytes: [16 * 1024, 1024 * 1024 * 1024], +}; + +export type Normalized = { ok: true; policy: ChannelPolicy } | { ok: false; errors: string[] }; + +/** + * A change, applied over what is there, checked field by field. + * + * Unknown keys are errors rather than ignored: a typo that is silently + * dropped is a setting the operator believes is on and is not. The version + * is not settable here; the store bumps it when it accepts the result. + */ +export function normalizePolicy(input: unknown, base: ChannelPolicy): Normalized { + const errors: string[] = []; + if (typeof input !== "object" || input === null || Array.isArray(input)) return { ok: false, errors: ["a policy is an object"] }; + const change = input as Record; + const policy = { ...base, losslessCompression: { ...base.losslessCompression } }; + for (const key of Object.keys(change)) { + if (!["losslessCompression", "hlsPackaging", "qualityProfile", "version"].includes(key)) errors.push(`unknown setting: ${key}`); + } + if ("hlsPackaging" in change) { + const value = change["hlsPackaging"]; + if (value === "mpegts" || value === "fmp4") policy.hlsPackaging = value; + else errors.push("hlsPackaging must be mpegts or fmp4"); + } + if ("qualityProfile" in change) { + if (change["qualityProfile"] === "source") policy.qualityProfile = "source"; + else errors.push("qualityProfile: only source is available; lower-bitrate profiles are a separate, explicit setting that this version does not ship"); + } + if ("losslessCompression" in change) { + const lc = change["losslessCompression"]; + if (typeof lc !== "object" || lc === null || Array.isArray(lc)) { + errors.push("losslessCompression is an object"); + } else { + const fields = lc as Record; + for (const [key, value] of Object.entries(fields)) { + if (key === "mode") { + if (value === "off" || value === "auto" || value === "zstd") policy.losslessCompression.mode = value; + else errors.push("losslessCompression.mode must be off, auto or zstd"); + } else if (key === "boundary") { + if (value === "channel" || value === "source") policy.losslessCompression.boundary = value; + else errors.push("losslessCompression.boundary must be channel or source"); + } else if (key === "tsAware") { + if (typeof value === "boolean") policy.losslessCompression.tsAware = value; + else errors.push("losslessCompression.tsAware must be true or false"); + } else if (key in RANGES) { + const [low, high] = RANGES[key as keyof typeof RANGES]; + if (typeof value === "number" && Number.isInteger(value) && value >= low && value <= high) { + (policy.losslessCompression as unknown as Record)[key] = value; + } else { + errors.push(`losslessCompression.${key} must be an integer from ${low} to ${high}`); + } + } else { + errors.push(`unknown setting: losslessCompression.${key}`); + } + } + } + } + if (policy.losslessCompression.minSavingsBytes >= policy.losslessCompression.maxBlockBytes) { + errors.push("losslessCompression.minSavingsBytes must be smaller than maxBlockBytes, or nothing can ever qualify"); + } + return errors.length > 0 ? { ok: false, errors } : { ok: true, policy }; +} + +/** + * Whether a compressed block earns its place. + * + * Both representations carry the same frame header, so the comparison of + * complete representations reduces to the payloads: the saving is what the + * original would have cost stored, less what the encoding costs. Both + * thresholds must hold. A tiny block can meet the percentage and still not + * be worth the decode; a huge one can save half a kilobyte and still be + * nothing. + */ +export function eligible(originalLength: number, encodedLength: number, policy: Pick): boolean { + if (originalLength <= 0) return false; + const saved = originalLength - encodedLength; + if (saved < policy.minSavingsBytes) return false; + return (saved * 100) / originalLength >= policy.minSavingsPercent; +} + +/** What a block costs on the wire in each form, header included. */ +export function wireBytes(payloadLength: number): number { + return FRAME_HEADER_BYTES + payloadLength; +} + +/** Whether two policies would produce the same bytes for the same input. */ +export function variantOf(policy: LosslessPolicy): string { + return `${policy.mode}:${policy.boundary}:${policy.zstdLevel}:${policy.tsAware ? "ts" : "plain"}:${policy.maxBlockBytes}`; +} diff --git a/src/compression/receiver.ts b/src/compression/receiver.ts new file mode 100644 index 0000000..3d85aeb --- /dev/null +++ b/src/compression/receiver.ts @@ -0,0 +1,107 @@ +/** + * The receiving end of a relay, as a client: ask another nixamp for a + * channel in the envelope, decode it, hand the original bytes on. + * + * The negotiation is two headers. `Accept` names the envelope's media type, + * which is how the server knows this is a receiver and not a browser that + * followed a link; `X-Nixamp-Stream-Codecs` lists what this decoder can + * undo, and the server compresses with nothing outside that list. A server + * that will not or cannot relay answers with JSON and an ordinary status, + * which is surfaced as an error naming the reason, never as a stream of + * something else. + */ +import { MEDIA_TYPE, type Mode, RelayError, type StreamHeader } from "./envelope.ts"; +import { RelayDecoder } from "./relay.ts"; + +export const CODECS_HEADER = "x-nixamp-stream-codecs"; +export const KIND_HEADER = "x-nixamp-kind"; +export const KEY_HEADER = "x-nixamp-key"; + +export interface ReceiveOptions { + url: string; + key: string | null; + /** What this receiver can decode. Stored is always implied. */ + modes?: Mode[]; + maxFrameBytes?: number; + /** The server said yes: what it will compress with, and what the channel carries. Before any byte. */ + onStart?: (accepted: { codecs: string; kind: "audio" | "video" | "" }) => void; + /** The stream header arrived: the generation this is. */ + onHeader?: (header: StreamHeader) => void; + onBytes: (bytes: Buffer) => void | Promise; + signal?: AbortSignal; + /** Injected by tests. */ + fetchImpl?: typeof fetch; +} + +export interface ReceiveResult { + generation: number; + frames: number; + bytes: number; +} + +/** The server said no. `status` is what it said it with. */ +export class RelayRefused extends Error { + constructor(readonly status: number, message: string) { + super(message); + this.name = "RelayRefused"; + } +} + +/** + * Pull one generation of a relay to its clean end. Resolves when the end + * frame arrives; rejects with a RelayError for a stream that broke a rule + * or stopped short, and with RelayRefused for a server that would not + * start one. + */ +export async function receiveRelay(options: ReceiveOptions): Promise { + const modes = new Set(["stored", ...(options.modes ?? ["zstd", "ts-zstd"])]); + const headers: Record = { + accept: MEDIA_TYPE, + [CODECS_HEADER]: [...modes].join(","), + }; + if (options.key) headers[KEY_HEADER] = options.key; + const send = options.fetchImpl ?? fetch; + const response = await send(options.url, { headers, ...(options.signal ? { signal: options.signal } : {}) }); + const type = response.headers.get("content-type") ?? ""; + if (response.status !== 200 || !type.startsWith(MEDIA_TYPE)) { + let reason = `${response.status}`; + try { + const body = (await response.json()) as { error?: string }; + if (typeof body.error === "string") reason = body.error; + } catch { + // Not JSON; the status is the message. + } + throw new RelayRefused(response.status, reason); + } + if (!response.body) throw new RelayRefused(response.status, "no body"); + const kindSaid = response.headers.get(KIND_HEADER); + options.onStart?.({ + codecs: response.headers.get(CODECS_HEADER) ?? "", + kind: kindSaid === "audio" || kindSaid === "video" ? kindSaid : "", + }); + const decoder = new RelayDecoder({ + modes, + ...(options.maxFrameBytes !== undefined ? { maxFrameBytes: options.maxFrameBytes } : {}), + onBytes: options.onBytes, + }); + let told = false; + const reader = response.body.getReader(); + try { + for (;;) { + const { value, done } = await reader.read(); + if (done) break; + await decoder.feed(Buffer.from(value)); + if (!told && decoder.header) { + told = true; + options.onHeader?.(decoder.header); + } + } + } catch (error) { + if (options.signal?.aborted) throw new RelayError("TRUNCATED", "cancelled"); + throw error; + } finally { + reader.releaseLock(); + } + await decoder.end(); + return { generation: decoder.header?.generation ?? 0, frames: decoder.frames, bytes: decoder.bytes }; +} diff --git a/src/compression/relay.ts b/src/compression/relay.ts new file mode 100644 index 0000000..59b426a --- /dev/null +++ b/src/compression/relay.ts @@ -0,0 +1,462 @@ +/** + * One compressor per channel, however many receivers. + * + * The encoder is a listener on the channel like any other. What it hears + * goes into blocks, each block is squeezed once on the pool, and the result + * is written to every relay session -- the payload shared, the frame header + * stamped per session because each session numbers its own frames. A + * session that joins late is first sent the channel's opening bytes (the + * header and the recent backlog, exactly what an ordinary listener gets) + * compressed for it alone, and then falls in with the shared blocks from + * the moment it joined. Nothing is sent twice and nothing is skipped: the + * encoder flushes its half-built block before it records where "the + * moment it joined" is. + * + * Bypass, overflow and slow listeners are all decided here and all bounded. + * A block that does not compress is sent stored; `auto` stops trying after + * a run of those and tries again later. A compressor that falls behind by + * more than the channel's queue limit ends its relays rather than either + * buffering for ever or stalling the channel. A session whose socket is + * not draining is cut off, on its own, and the channel never knows. + */ +import { createHash } from "node:crypto"; +import { Blocker } from "./blocks.ts"; +import { decode, encode, Pool, PoolError } from "./codec.ts"; +import { + type Boundary, + decodeFrameHeader, + decodeStreamHeader, + encodeEndFrame, + encodeFrameHeader, + encodeStreamHeader, + FRAME_DATA, + FRAME_END, + FRAME_HEADER_BYTES, + type FrameHeader, + endFrameTotal, + type Mode, + RelayError, + sha256, + STREAM_HEADER_BYTES, + type StreamHeader, +} from "./envelope.ts"; +import type { ChannelMetrics, FallbackReason } from "./metrics.ts"; +import { eligible, type LosslessPolicy } from "./policy.ts"; +import { TS_PACKET, tsLayout } from "./ts-transform.ts"; + +/** Somewhere frames go: a response, in practice. `pending` is how far behind it is. */ +export interface RelayListener { + write(chunk: Buffer): boolean; + end(): void; + pending?(): number; +} + +export interface EncodedBlock { + /** Offset of this block's first byte since the encoder attached. */ + start: number; + mode: Mode; + original: Buffer; + digest: Buffer; + payload: Buffer; +} + +/** How many blocks in a row may fail to pay before `auto` stops trying, for a while. */ +export const GIVE_UP_AFTER = 8; + +export interface RelayEncoderOptions { + generation: number; + policy: LosslessPolicy; + pool: Pool; + metrics: ChannelMetrics; + boundary: Boundary; + /** The compressor fell too far behind, or the channel ended: this relay is over. */ + onAbort: (reason: FallbackReason | "ended") => void; + /** The last session left. */ + onIdle: () => void; + now?: () => number; +} + +export class RelayEncoder { + private attached = false; + private readonly blocker: Blocker; + private chain: Promise = Promise.resolve(); + private pushed = 0; + private readonly sessions = new Set(); + private ineligibleRun = 0; + private bypassUntil: number | null = null; + private closed = false; + private readonly now: () => number; + readonly streamHeader: Buffer; + + constructor(private readonly options: RelayEncoderOptions) { + this.now = options.now ?? (() => performance.now()); + const { policy } = options; + this.blocker = new Blocker({ + maxBlockBytes: policy.maxBlockBytes, + maxHoldMs: policy.maxHoldMs, + align: policy.tsAware ? TS_PACKET : 0, + onBlock: (block) => this.onBlock(block), + }); + this.streamHeader = encodeStreamHeader({ + version: 1, + boundary: options.boundary, + generation: options.generation, + maxFrameBytes: policy.maxBlockBytes, + }); + options.metrics.reset(options.generation); + options.metrics.fallbackReason = null; + } + + get generation(): number { + return this.options.generation; + } + + get sessionCount(): number { + return this.sessions.size; + } + + /** + * Call once `Channel.listen` has returned. What it wrote before then was + * the opening bytes, which every session gets for itself as its preface; + * taking them here too would send them twice to the first session. + */ + attach(): void { + this.attached = true; + } + + /** The channel's listener face. */ + write(chunk: Buffer): boolean { + if (!this.attached || this.closed) return true; + this.options.metrics.inputBytes += chunk.length; + this.blocker.push(chunk); + return true; + } + + /** The channel ended, or started over: finish every session cleanly. */ + end(): void { + if (this.closed) return; + this.closed = true; + this.blocker.end(); + this.chain = this.chain.then(() => { + for (const session of [...this.sessions]) session.finish(); + this.options.onAbort("ended"); + }); + } + + /** + * A new receiver. `preface` is what the channel would write first to an + * ordinary listener, snapshotted by the caller in the same tick as this + * call, so that the shared blocks from here on follow it exactly. + */ + join(listener: RelayListener, preface: Buffer[]): RelaySession { + // Whatever is half-built is pre-join and is in the preface's backlog + // already: out with it before the join point is marked. + this.blocker.flush(); + const session = new RelaySession(listener, this, this.pushed, this.options.policy.maxListenerQueueBytes, (gone, reason) => { + this.sessions.delete(gone); + this.options.metrics.listeners = this.sessions.size; + if (reason !== null) { + this.options.metrics.droppedListeners += 1; + this.options.metrics.fallbackReason = reason; + } + if (this.sessions.size === 0 && !this.closed) this.options.onIdle(); + }); + this.sessions.add(session); + this.options.metrics.listeners = this.sessions.size; + // The join point is marked now; the writing starts a tick later, so + // the caller can put response headers in front of the first byte. + // No shared block can arrive in between: every block goes through the + // pool first, and the pool answers on a later tick than this one. + queueMicrotask(() => void session.start(preface)); + return session; + } + + private onBlock(block: Buffer): void { + const start = this.pushed; + this.pushed += block.length; + const metrics = this.options.metrics; + metrics.queueBytes += block.length; + if (metrics.queueBytes > this.options.policy.maxChannelQueueBytes) { + this.abort("processing budget exceeded"); + return; + } + const entered = this.now(); + this.chain = this.chain.then(async () => { + if (this.closed && this.sessions.size === 0) return; + const { mode, payload } = await this.encodeBytes(block); + metrics.queueBytes -= block.length; + metrics.latency(this.now() - entered); + metrics.blocks += 1; + metrics.representationBytes += payload.length; + if (mode === "stored") metrics.storedBlocks += 1; + else metrics.compressedBlocks += 1; + metrics.activeMode = mode; + const encoded: EncodedBlock = { start, mode, original: block, digest: sha256(block), payload }; + for (const session of [...this.sessions]) session.deliver(encoded); + }).catch(() => { + // A codec that threw something other than a pool refusal is a codec + // that cannot be trusted with the next block either. Stop the relays; + // the channel itself is untouched, and a receiver reconnects. + metrics.codecFailures += 1; + this.abort("processing budget exceeded"); + }); + } + + /** + * The bytes in the mode the policy picks: compressed when that pays, + * stored when it does not or cannot be done in time. + */ + async encodeBytes(bytes: Buffer): Promise<{ mode: Mode; payload: Buffer }> { + const policy = this.options.policy; + const metrics = this.options.metrics; + const stored = { mode: "stored" as const, payload: bytes }; + if (policy.mode === "off") return stored; + if (policy.mode === "auto" && this.bypassUntil !== null) { + if (this.now() < this.bypassUntil) return stored; + this.bypassUntil = null; + this.ineligibleRun = 0; + metrics.bypassed = false; + metrics.bypassUntil = null; + } + const candidates: Mode[] = ["zstd"]; + if (policy.tsAware && tsLayout(bytes)?.packetSize === TS_PACKET) candidates.push("ts-zstd"); + let results: Buffer[]; + try { + results = await Promise.all(candidates.map((mode) => this.options.pool.run(() => encode(mode, bytes, policy.zstdLevel)))); + } catch (error) { + metrics.codecFailures += 1; + metrics.fallbackReason = "processing budget exceeded"; + if (!(error instanceof PoolError) && !(error instanceof RelayError)) throw error; + return stored; + } + let best = 0; + for (let i = 1; i < results.length; i += 1) if ((results[i] as Buffer).length < (results[best] as Buffer).length) best = i; + const payload = results[best] as Buffer; + if (eligible(bytes.length, payload.length, policy)) { + this.ineligibleRun = 0; + return { mode: candidates[best] as Mode, payload }; + } + this.ineligibleRun += 1; + if (policy.mode === "auto" && this.ineligibleRun >= GIVE_UP_AFTER) { + this.bypassUntil = this.now() + policy.resampleAfterMs; + metrics.bypassed = true; + metrics.bypassUntil = Date.now() + policy.resampleAfterMs; + metrics.fallbackReason = "already efficiently compressed"; + } + return stored; + } + + private abort(reason: FallbackReason): void { + if (this.closed) return; + this.closed = true; + this.options.metrics.fallbackReason = reason; + for (const session of [...this.sessions]) session.close(reason); + this.options.onAbort(reason); + } +} + +/** One receiver's view of the relay: its own frame numbers, its own end marker. */ +export class RelaySession { + private seq = 0; + private total = 0; + private readonly digest = createHash("sha256"); + private queue: EncodedBlock[] = []; + private queuedBytes = 0; + private ready = false; + closed = false; + + constructor( + private readonly listener: RelayListener, + private readonly encoder: RelayEncoder, + private readonly from: number, + private readonly maxQueueBytes: number, + private readonly onClose: (session: RelaySession, reason: FallbackReason | null) => void, + ) {} + + async start(preface: Buffer[]): Promise { + if (!this.put(this.encoder.streamHeader)) return; + for (const piece of preface) { + for (let at = 0; at < piece.length && !this.closed; at += this.maxBlockBytes) { + const block = piece.subarray(at, Math.min(piece.length, at + this.maxBlockBytes)); + const { mode, payload } = await this.encoder.encodeBytes(block); + this.send(mode, block, sha256(block), payload); + } + } + if (this.closed) return; + this.ready = true; + const queued = this.queue; + this.queue = []; + this.queuedBytes = 0; + for (const block of queued) { + if (this.closed) return; + this.send(block.mode, block.original, block.digest, block.payload); + } + } + + private get maxBlockBytes(): number { + return this.encoder.streamHeader.readUInt32BE(12); + } + + deliver(block: EncodedBlock): void { + if (this.closed || block.start < this.from) return; + if (!this.ready) { + this.queue.push(block); + this.queuedBytes += block.original.length; + if (this.queuedBytes > this.maxQueueBytes) this.close("slow listener"); + return; + } + this.send(block.mode, block.original, block.digest, block.payload); + } + + private send(mode: Mode, original: Buffer, digest: Buffer, payload: Buffer): void { + if (this.closed) return; + const head = encodeFrameHeader({ + type: FRAME_DATA, + mode, + seq: this.seq, + originalLength: original.length, + encodedLength: payload.length, + sha256: digest, + }); + this.seq += 1; + this.total += original.length; + this.digest.update(original); + if (!this.put(head) || !this.put(payload)) return; + const behind = this.listener.pending?.() ?? 0; + if (behind > this.maxQueueBytes) this.close("slow listener"); + } + + /** Clean end: the end frame, then the socket. */ + finish(): void { + if (this.closed) return; + this.put(encodeEndFrame(this.seq, this.total, this.digest.digest())); + this.closed = true; + try { + this.listener.end(); + } catch { + // Gone already. + } + this.onClose(this, null); + } + + close(reason: FallbackReason): void { + if (this.closed) return; + this.closed = true; + try { + this.listener.end(); + } catch { + // Gone already. + } + this.onClose(this, reason); + } + + /** The receiver hung up. Not a fault of anybody's; nothing is counted against it. */ + leave(): void { + if (this.closed) return; + this.closed = true; + this.onClose(this, null); + } + + private put(bytes: Buffer): boolean { + try { + this.listener.write(bytes); + return true; + } catch { + this.close("slow listener"); + return false; + } + } +} + +export interface DecoderOptions { + /** What this decoder can undo. Anything else in a frame is refused. */ + modes: ReadonlySet; + /** The most a stream may negotiate; a header asking for more is refused. */ + maxFrameBytes?: number; + /** The boundary the caller expects, when it matters. */ + boundary?: Boundary; + onBytes: (bytes: Buffer) => void | Promise; +} + +/** + * The receiving end: bytes in, original bytes out, every rule checked on + * the way. Frames are validated before their payload is read, payloads are + * decoded under the size the frame promised, and the result is checked + * against the length and the digest before it is handed on. A stream that + * stops without its end frame is reported as cut off by `end()`. + */ +export class RelayDecoder { + header: StreamHeader | null = null; + frames = 0; + bytes = 0; + ended = false; + private buffer: Buffer = Buffer.alloc(0); + private seq = 0; + private pending: FrameHeader | null = null; + private readonly digest = createHash("sha256"); + private busy: Promise = Promise.resolve(); + + constructor(private readonly options: DecoderOptions) {} + + /** Feed bytes as they arrive. Serialised, so callers need not await between chunks, though they may. */ + feed(chunk: Buffer): Promise { + this.busy = this.busy.then(() => this.consume(chunk)); + return this.busy; + } + + private async consume(chunk: Buffer): Promise { + if (this.ended) throw new RelayError("AFTER_END", "bytes after the end frame"); + this.buffer = this.buffer.length === 0 ? chunk : Buffer.concat([this.buffer, chunk]); + for (;;) { + if (this.header === null) { + if (this.buffer.length < STREAM_HEADER_BYTES) return; + const header = decodeStreamHeader(this.buffer); + const cap = this.options.maxFrameBytes ?? Infinity; + if (header.maxFrameBytes > cap) throw new RelayError("BAD_LIMIT", `stream asks for ${header.maxFrameBytes}-byte frames; this receiver allows ${cap}`); + if (this.options.boundary && header.boundary !== this.options.boundary) { + throw new RelayError("BAD_BOUNDARY", `stream is at the ${header.boundary} boundary, not ${this.options.boundary}`); + } + this.header = header; + this.buffer = this.buffer.subarray(STREAM_HEADER_BYTES); + continue; + } + if (this.pending === null) { + if (this.buffer.length < FRAME_HEADER_BYTES) return; + this.pending = decodeFrameHeader(this.buffer, { + maxFrameBytes: this.header.maxFrameBytes, + modes: this.options.modes, + expectSeq: this.seq, + }); + this.buffer = this.buffer.subarray(FRAME_HEADER_BYTES); + if (this.pending.type === FRAME_END) { + const total = endFrameTotal(this.pending); + if (total !== this.bytes) throw new RelayError("LENGTH_MISMATCH", `end frame says ${total} bytes, received ${this.bytes}`); + if (!this.digest.digest().equals(this.pending.sha256)) throw new RelayError("CHECKSUM_MISMATCH", "the stream digest does not match"); + this.ended = true; + this.seq += 1; + if (this.buffer.length > 0) throw new RelayError("AFTER_END", "bytes after the end frame"); + return; + } + } + const frame = this.pending; + if (this.buffer.length < frame.encodedLength) return; + const payload = this.buffer.subarray(0, frame.encodedLength); + this.buffer = Buffer.from(this.buffer.subarray(frame.encodedLength)); + const original = await decode(frame.mode, payload, frame.originalLength); + if (original.length !== frame.originalLength) throw new RelayError("LENGTH_MISMATCH", `frame ${frame.seq} decoded to ${original.length} bytes, not ${frame.originalLength}`); + if (!sha256(original).equals(frame.sha256)) throw new RelayError("CHECKSUM_MISMATCH", `frame ${frame.seq} does not match its digest`); + this.pending = null; + this.seq += 1; + this.frames += 1; + this.bytes += original.length; + this.digest.update(original); + await this.options.onBytes(original); + } + } + + /** The connection closed. Fine after the end frame; a cut-off otherwise. */ + async end(): Promise { + await this.busy; + if (!this.ended) throw new RelayError("TRUNCATED", `the stream stopped after ${this.frames} frames without an end marker`); + } +} diff --git a/src/compression/routes.ts b/src/compression/routes.ts new file mode 100644 index 0000000..f54f2d7 --- /dev/null +++ b/src/compression/routes.ts @@ -0,0 +1,294 @@ +/** + * The HTTP face of the compression service. + * + * Under a channel: its policy (read, and change with the version you + * saw), an analysis of it (a job), and its relay -- out to another nixamp + * that asks in the envelope's media type, or in from one. Under + * /api/compression: the server-wide switch and the jobs. All of it is + * translation; the rules live in the service. The server's own helpers + * for answering (JSON, counting a stream) are handed in, so this file + * imports nothing from the server and the server one thing from it. + */ +import { createReadStream } from "node:fs"; +import type { IncomingMessage, ServerResponse } from "node:http"; +import { MEDIA_TYPE, parseModes } from "./envelope.ts"; +import { CODECS_HEADER, KIND_HEADER } from "./receiver.ts"; +import type { CompressionService } from "./service.ts"; + +export interface RouteContext { + service: CompressionService; + json: (response: ServerResponse, code: number, body: unknown) => void; + readBody: (request: IncomingMessage) => Promise; + /** Whether the caller holds the controls, as opposed to a listening link. */ + controls: () => Promise; + /** Count a streaming response in the connections view. */ + watch: (request: IncomingMessage, response: ServerResponse, track: string) => void; + cors: Record; +} + +/** Who a diagnostic job belongs to. Every control holder shares them. */ +const CONTROL = "control"; + +/** The header a conditional change names the version in, or the body's own field. */ +function expectedVersion(request: IncomingMessage, body: Record): number | undefined { + const header = request.headers["if-match"]; + const raw = typeof header === "string" ? header.replace(/^W\//, "").replace(/"/g, "") : body["version"]; + const version = Number(raw); + return raw === undefined || raw === "" || !Number.isInteger(version) ? undefined : version; +} + +async function parsed(request: IncomingMessage, ctx: RouteContext): Promise | null> { + try { + const text = await ctx.readBody(request); + const body = text === "" ? {} : (JSON.parse(text) as unknown); + return typeof body === "object" && body !== null && !Array.isArray(body) ? (body as Record) : null; + } catch { + return null; + } +} + +/** + * Routes under /api/channels/:id/. True when the request was answered. + * `action` and `file` are the path segments after the id. + */ +export async function handleChannelCompression( + request: IncomingMessage, + response: ServerResponse, + ctx: RouteContext, + id: string, + action: string | undefined, + file: string | undefined, +): Promise { + const { service, json } = ctx; + + if (action === "compression") { + if (file === undefined && request.method === "GET") { + json(response, 200, service.status(id)); + return true; + } + if (file === undefined && request.method === "PATCH") { + if (!(await ctx.controls())) { + json(response, 403, { error: "the controls are needed to change a policy" }); + return true; + } + const body = await parsed(request, ctx); + if (body === null) { + json(response, 400, { error: "send a JSON object" }); + return true; + } + const { version: _version, ...change } = body; + const result = service.set(id, change, expectedVersion(request, body)); + if (!result.ok) { + json(response, result.status, { error: result.errors.join("; "), errors: result.errors, status: service.status(id) }); + return true; + } + json(response, 200, service.status(id)); + return true; + } + if (file === "analyses" && request.method === "POST") { + if (!(await ctx.controls())) { + json(response, 403, { error: "the controls are needed to start an analysis" }); + return true; + } + const body = await parsed(request, ctx); + const seconds = Number(body?.["seconds"] ?? 30); + const started = service.analyzeChannel(id, Number.isFinite(seconds) ? seconds : 30, CONTROL); + if ("error" in started) { + json(response, started.status, { error: started.error }); + return true; + } + json(response, started.existing ? 200 : 202, { job: started.job, existing: started.existing }); + return true; + } + json(response, 405, { error: "GET or PATCH the policy; POST to analyses" }); + return true; + } + + if (action === "relay" && file === undefined) { + if (request.method === "GET") { + const accept = String(request.headers["accept"] ?? ""); + if (!accept.includes(MEDIA_TYPE)) { + json(response, 406, { + error: `this is a relay for another nixamp, sent as ${MEDIA_TYPE}; a player wants the ordinary channel URL`, + playback: `/api/channels/${id}`, + }); + return true; + } + if (request.headers["range"] !== undefined) { + json(response, 416, { error: "a relay is a live stream and cannot be asked for a range" }); + return true; + } + const offered = parseModes(String(request.headers[CODECS_HEADER] ?? "")); + const listener = { + write: (chunk: Buffer): boolean => response.write(chunk), + end: (): void => { + response.end(); + }, + pending: (): number => response.writableLength, + }; + const answer = service.relay(id, listener, offered); + if (!answer.ok) { + json(response, answer.status, { error: answer.error, code: answer.code, playback: `/api/channels/${id}` }); + return true; + } + ctx.watch(request, response, id); + response.writeHead(200, { + ...ctx.cors, + "content-type": `${MEDIA_TYPE}; version=1`, + "cache-control": "no-store", + // Never squeezed again by anything in the way: it is already framed. + "content-encoding": "identity", + [CODECS_HEADER]: answer.codecs.join(","), + [KIND_HEADER]: answer.kind, + "x-nixamp-generation": String(answer.generation), + }); + const leave = (): void => answer.session.leave(); + request.on("close", leave); + response.on("close", leave); + return true; + } + if (request.method === "POST") { + if (!(await ctx.controls())) { + json(response, 403, { error: "the controls are needed to bring a relay in" }); + return true; + } + const body = await parsed(request, ctx); + const from = typeof body?.["from"] === "string" ? body["from"] : ""; + if (from === "") { + json(response, 400, { error: "say where from: { from: \"https://host:port/api/channels//relay\" }" }); + return true; + } + const key = typeof body?.["key"] === "string" ? body["key"] : null; + const name = typeof body?.["name"] === "string" ? body["name"] : id; + const started = service.pull(id, from, key, name); + if (!started.ok) { + json(response, started.status, { error: started.error }); + return true; + } + json(response, 202, { ok: true, channel: id, from }); + return true; + } + if (request.method === "DELETE") { + if (!(await ctx.controls())) { + json(response, 403, { error: "the controls are needed to stop a relay" }); + return true; + } + const stopped = service.stopPull(id); + json(response, stopped ? 200 : 404, { ok: stopped }); + return true; + } + json(response, 405, { error: "GET to receive, POST to bring one in, DELETE to stop it" }); + return true; + } + + return false; +} + +/** Routes under /api/compression. True when the request was answered. */ +export async function handleCompressionApi(request: IncomingMessage, response: ServerResponse, ctx: RouteContext, path: string): Promise { + const { service, json } = ctx; + if (path === "/api/compression") { + if (request.method === "GET") { + json(response, 200, service.overview()); + return true; + } + if (request.method === "PATCH") { + if (!(await ctx.controls())) { + json(response, 403, { error: "the controls are needed to change the server's compression" }); + return true; + } + const body = await parsed(request, ctx); + if (body === null) { + json(response, 400, { error: "send a JSON object" }); + return true; + } + const change: { enabled?: boolean; hlsPackaging?: "mpegts" | "fmp4" } = {}; + if ("enabled" in body) { + if (typeof body["enabled"] !== "boolean") { + json(response, 400, { error: "enabled must be true or false" }); + return true; + } + change.enabled = body["enabled"]; + } + if ("hlsPackaging" in body) { + if (body["hlsPackaging"] !== "mpegts" && body["hlsPackaging"] !== "fmp4") { + json(response, 400, { error: "hlsPackaging must be mpegts or fmp4" }); + return true; + } + change.hlsPackaging = body["hlsPackaging"]; + } + json(response, 200, { global: service.setGlobal(change) }); + return true; + } + json(response, 405, { error: "GET or PATCH" }); + return true; + } + const job = /^\/api\/compression\/analyses\/([a-z0-9]+)$/.exec(path); + if (job) { + if (!(await ctx.controls())) { + json(response, 403, { error: "the controls are needed to see an analysis" }); + return true; + } + const jobId = job[1] as string; + if (request.method === "GET") { + const found = service.jobs.get(jobId, CONTROL); + if (!found) { + json(response, 404, { error: "no such analysis" }); + return true; + } + json(response, 200, { job: found }); + return true; + } + if (request.method === "DELETE") { + const cancelled = service.jobs.cancel(jobId, CONTROL); + json(response, cancelled ? 200 : 404, { ok: cancelled }); + return true; + } + json(response, 405, { error: "GET or DELETE" }); + return true; + } + return false; +} + +/** + * The static representation of a library file, at /api/media/N/relay. + * The caller has already resolved N to a path and checked that media may + * be streamed at all. + */ +export function handleStaticRelay(request: IncomingMessage, response: ServerResponse, ctx: RouteContext, file: string, title: string): void { + const { service, json } = ctx; + const accept = String(request.headers["accept"] ?? ""); + if (!accept.includes(MEDIA_TYPE)) { + json(response, 406, { error: `this is the file's relay representation, sent as ${MEDIA_TYPE}; a player wants the ordinary media URL` }); + return; + } + if (request.headers["range"] !== undefined) { + json(response, 416, { error: "the relay representation cannot be asked for a range; the ordinary media URL can" }); + return; + } + const state = service.representation(file); + if (state.state === "off") { + json(response, 409, { error: "compression is off for static files on this server", code: "COMPRESSION_OFF" }); + return; + } + if (state.state === "failed") { + json(response, 503, { error: state.reason }); + return; + } + if (state.state === "building") { + response.writeHead(202, { ...ctx.cors, "content-type": "application/json; charset=utf-8", "retry-after": "5" }); + response.end(JSON.stringify({ building: true })); + return; + } + ctx.watch(request, response, title); + response.writeHead(200, { + ...ctx.cors, + "content-type": `${MEDIA_TYPE}; version=1`, + "content-length": String(state.entry.bytes), + "content-encoding": "identity", + "cache-control": "no-store", + "x-nixamp-sha256": state.entry.sha256, + "x-nixamp-original-length": String(state.entry.size), + }); + createReadStream(state.path).pipe(response); +} diff --git a/src/compression/service.ts b/src/compression/service.ts new file mode 100644 index 0000000..809ffc8 --- /dev/null +++ b/src/compression/service.ts @@ -0,0 +1,525 @@ +/** + * The one place the server, the CLI and the API ask about compression. + * + * It owns the policies, the shared encoders, the metrics, the diagnostic + * jobs, the static cache and the incoming relays, and it is the only thing + * that touches a channel on their behalf. Every surface is a thin client + * of this: the routes translate HTTP into these calls and back, the CLI + * translates flags into the routes. There is no second copy of the rules. + */ +import { statSync } from "node:fs"; +import { codecsOf } from "../audio.ts"; +import { type Channel, type Channels, GIVE_UP, REDIAL } from "../channels.ts"; +import { type Analysis, analyzeSample, SAMPLE_MAX_BYTES, SAMPLE_MAX_SECONDS } from "./analyze.ts"; +import { Pool } from "./codec.ts"; +import { type Mode, RelayError } from "./envelope.ts"; +import { AnalysisJobs, type Job } from "./jobs.ts"; +import { ChannelMetrics, type ChannelMetricsSnapshot } from "./metrics.ts"; +import { type ChannelPolicy, type LosslessPolicy, variantOf } from "./policy.ts"; +import { receiveRelay, RelayRefused } from "./receiver.ts"; +import { RelayEncoder, type RelayListener, type RelaySession } from "./relay.ts"; +import { type GlobalSettings, PolicyStore } from "./store.ts"; +import { type Prepared, StaticCache } from "./static.ts"; + +/** The pseudo-channel whose policy governs static file representations. */ +export const STATIC_CHANNEL = "static"; + +export interface AnalyzeFileOptions { + policy?: LosslessPolicy; + ffprobe?: string[]; + signal?: AbortSignal; +} + +/** + * A file's first bytes, up to the sample limit, analysed at the source + * boundary: these are the bytes as they sit on disk, before any ffmpeg. + * Shared by the service and the CLI, which runs it with no server at all. + */ +export async function analyzeFile(path: string, options: AnalyzeFileOptions = {}): Promise { + const size = statSync(path).size; + const take = Math.min(size, SAMPLE_MAX_BYTES); + const fd = await import("node:fs/promises").then((fs) => fs.open(path, "r")); + let sample: Buffer; + try { + sample = Buffer.alloc(take); + let at = 0; + while (at < take) { + const { bytesRead } = await fd.read(sample, at, take - at, at); + if (bytesRead === 0) break; + at += bytesRead; + } + sample = sample.subarray(0, at); + } finally { + await fd.close(); + } + const ffprobe = options.ffprobe; + const codecs = ffprobe ? await codecsOf({ ffmpeg: [], ffprobe, play: null }, path) : undefined; + return analyzeSample(sample, { + boundary: "source", + source: { kind: "file", name: path }, + truncated: take < size, + tsAware: true, + ...(options.policy ? { policy: options.policy } : {}), + ...(options.signal ? { signal: options.signal } : {}), + ...(codecs && codecs.container ? { codecs } : {}), + ...(ffprobe ? { ffprobeVersion: ffprobe.join(" ") } : {}), + }); +} + +export interface CompressionServiceOptions { + channels: Channels; + /** Where settings live; null for a server that remembers nothing. */ + stateDir: string | null; + port: number; + pool?: Pool; + ffprobe?: string[]; + /** Where static representations go; null to keep none. */ + cacheDir?: string | null; + maxCacheBytes?: number; + onEvent?: (message: string) => void; +} + +export interface EffectivePolicy { + /** The policy as it applies right now, after the global switch. */ + losslessCompression: LosslessPolicy; + hlsPackaging: ChannelPolicy["hlsPackaging"]; + qualityProfile: ChannelPolicy["qualityProfile"]; + /** Why it differs from what is configured, when it does. */ + reason: string | null; +} + +export interface ChannelStatus { + channel: string; + live: boolean; + configured: ChannelPolicy; + effective: EffectivePolicy; + global: GlobalSettings; + metrics: ChannelMetricsSnapshot | null; + relay: { sessions: number; generation: number } | null; + /** An incoming relay this channel is fed by, when it is. */ + incoming: { from: string; generation: number; reconnects: number; error: string | null } | null; +} + +export type RelayAnswer = + | { ok: true; session: RelaySession; codecs: Mode[]; generation: number; kind: "audio" | "video" | "" } + | { ok: false; status: 404 | 406 | 409 | 503; code: string; error: string }; + +interface Running { + encoder: RelayEncoder; + detach: () => void; + variant: string; +} + +/** An incoming relay: this server as the receiver, dialling again when it drops. */ +class Incoming { + generation = 0; + reconnects = 0; + error: string | null = null; + private stopped = false; + private failures = 0; + private controller = new AbortController(); + private timer: ReturnType | null = null; + private channel: Channel | null = null; + + constructor( + readonly id: string, + readonly from: string, + private readonly key: string | null, + private readonly name: string, + private readonly channels: Channels, + private readonly onEvent: (message: string) => void, + private readonly onGone: (id: string) => void, + ) {} + + start(): void { + void this.dial(); + } + + private async dial(): Promise { + if (this.stopped) return; + this.controller = new AbortController(); + let received = 0; + try { + await receiveRelay({ + url: this.from, + key: this.key, + signal: this.controller.signal, + onStart: ({ kind }) => { + if (this.channel === null) { + const opened = this.channels.relayIn(this.id, this.name, kind === "video" ? "video" : "audio", this.from); + if (!opened) throw new RelayRefused(409, `channel "${this.id}" is already on`); + this.channel = opened; + } else { + // A new generation upstream: new opening boxes, so everybody + // watching here starts over too. + this.channel.rollover(); + } + }, + onHeader: (header) => { + this.generation = header.generation; + }, + onBytes: (bytes) => { + received += bytes.length; + this.channel?.receive(bytes); + }, + }); + // A clean end is the upstream's source starting over, or going off. + // Either way: dial again, and let the new generation say which. + this.error = null; + this.failures = 0; + } catch (error) { + if (this.stopped) return; + this.error = error instanceof RelayError ? `${error.code}: ${error.message}` : (error as Error).message; + this.onEvent(` relay "${this.id}" from ${this.from}: ${this.error}`); + this.failures = received > 0 ? 0 : this.failures + 1; + if (error instanceof RelayRefused && error.status === 409 && this.channel === null) { + this.stop(); + return; + } + if (this.failures >= GIVE_UP) { + this.onEvent(` relay "${this.id}" gave up: ${GIVE_UP} dials without a byte`); + this.stop(); + return; + } + } + if (this.stopped || !this.channels.has(this.id)) { + this.stop(); + return; + } + this.reconnects += 1; + this.timer = setTimeout(() => { + this.timer = null; + void this.dial(); + }, REDIAL); + this.timer.unref?.(); + } + + stop(): void { + if (this.stopped) return; + this.stopped = true; + if (this.timer) clearTimeout(this.timer); + this.timer = null; + this.controller.abort(); + this.channel?.close(); + this.channel = null; + this.onGone(this.id); + } +} + +export class CompressionService { + readonly store: PolicyStore; + readonly jobs: AnalysisJobs; + readonly pool: Pool; + readonly statics: StaticCache | null; + private readonly running = new Map(); + private readonly metrics = new Map(); + private readonly incoming = new Map(); + private generation = Math.floor(Date.now() / 1000) % 0x7fff_ffff; + + constructor(private readonly options: CompressionServiceOptions) { + this.store = new PolicyStore(options.stateDir, options.port); + this.jobs = new AnalysisJobs({ concurrency: 1 }); + this.pool = options.pool ?? new Pool({ concurrency: 4, maxQueued: 256, timeoutMs: 2000 }); + this.statics = options.cacheDir + ? new StaticCache(options.cacheDir, { pool: this.pool, ...(options.maxCacheBytes !== undefined ? { maxBytes: options.maxCacheBytes } : {}) }) + : null; + } + + private get channels(): Channels { + return this.options.channels; + } + + /** What applies to a channel right now, and why that is not what is configured, if it is not. */ + effective(id: string): EffectivePolicy { + const configured = this.store.get(id); + const global = this.store.global; + const lossless = { ...configured.losslessCompression }; + let reason: string | null = null; + if (!global.enabled && lossless.mode !== "off") { + lossless.mode = "off"; + reason = "compression is off for the whole server"; + } else if (lossless.mode !== "off" && lossless.boundary === "source") { + lossless.mode = "off"; + reason = "original source bytes are unavailable: this server's sources are read by ffmpeg, and only the channel boundary can be relayed"; + } + return { + losslessCompression: lossless, + hlsPackaging: this.store.has(id) ? configured.hlsPackaging : global.hlsPackaging, + qualityProfile: configured.qualityProfile, + reason, + }; + } + + /** For the HLS packager: how to wrap this channel. */ + packagingOf(id: string): ChannelPolicy["hlsPackaging"] { + return this.effective(id).hlsPackaging; + } + + status(id: string): ChannelStatus { + const live = this.channels.has(id); + const running = this.running.get(id); + const inbound = this.incoming.get(id); + return { + channel: id, + live, + configured: this.store.get(id), + effective: this.effective(id), + global: this.store.global, + metrics: this.metrics.get(id)?.snapshot() ?? null, + relay: running ? { sessions: running.encoder.sessionCount, generation: running.encoder.generation } : null, + incoming: inbound ? { from: inbound.from, generation: inbound.generation, reconnects: inbound.reconnects, error: inbound.error } : null, + }; + } + + /** The whole server at a glance. */ + overview(): { global: GlobalSettings; pool: Pool["stats"]; jobs: AnalysisJobs["counts"]; channels: ChannelStatus[]; cache: { bytes: number; entries: number } | null } { + const ids = new Set([...this.channels.list().map((c) => c.id), ...Object.keys(this.store.list()), ...this.running.keys(), ...this.incoming.keys()]); + return { + global: this.store.global, + pool: this.pool.stats, + jobs: this.jobs.counts, + channels: [...ids].sort().map((id) => this.status(id)), + cache: this.statics ? { bytes: this.statics.totalBytes, entries: this.statics.entries().length } : null, + }; + } + + /** Change a channel's policy. A live relay under a different variant is ended; receivers reconnect. */ + set(id: string, change: unknown, expectVersion?: number): ReturnType { + const result = this.store.set(id, change, expectVersion); + if (result.ok) this.applyPolicy(id); + return result; + } + + setGlobal(change: Partial): GlobalSettings { + const settings = this.store.setGlobal(change); + for (const id of [...this.running.keys()]) this.applyPolicy(id); + return settings; + } + + /** The kill switch, and its opposite. */ + private applyPolicy(id: string): void { + const running = this.running.get(id); + if (!running) return; + const effective = this.effective(id).losslessCompression; + if (effective.mode === "off" || variantOf(effective) !== running.variant) { + // Not a clean end: the receiver sees the stream stop without its + // marker, reports a disconnect, and dials again under the new policy. + running.encoder.end(); + } + } + + /** + * A receiver asking for a channel. Everything that can go wrong is + * answered before a byte of stream: no such channel, compression off, + * a boundary this server cannot provide, a receiver that cannot decode + * anything worth sending. + */ + relay(id: string, listener: RelayListener, offered: Set): RelayAnswer { + if (!this.channels.has(id)) return { ok: false, status: 404, code: "NO_SUCH_CHANNEL", error: "nothing is playing on that channel" }; + const configured = this.store.get(id).losslessCompression; + const effective = this.effective(id); + const policy = effective.losslessCompression; + if (policy.mode === "off") { + if (configured.mode !== "off" && configured.boundary === "source") { + return { ok: false, status: 409, code: "SOURCE_BOUNDARY_UNAVAILABLE", error: effective.reason ?? "original source bytes are unavailable" }; + } + return { ok: false, status: 409, code: "COMPRESSION_OFF", error: effective.reason ?? "compression is off for this channel; the ordinary channel URL is unaffected" }; + } + const codecs: Mode[] = ["stored"]; + if (offered.has("zstd")) codecs.push("zstd"); + if (offered.has("ts-zstd") && policy.tsAware) codecs.push("ts-zstd"); + if (codecs.length === 1) { + return { ok: false, status: 406, code: "RECEIVER_UNSUPPORTED", error: "receiver does not support this format: it must decode zstd; the ordinary channel URL is unaffected" }; + } + // A receiver that cannot undo the transform gets a variant without it. + const variantPolicy: LosslessPolicy = { ...policy, tsAware: policy.tsAware && codecs.includes("ts-zstd") }; + const variant = variantOf(variantPolicy); + let running = this.running.get(id); + if (running && running.variant !== variant) { + // One compressor per channel. A second variant would be a second + // compressor; the first receiver's policy stands until it leaves. + return { ok: false, status: 409, code: "VARIANT_IN_USE", error: "this channel is already being relayed under a different codec set; try again when that relay ends" }; + } + if (!running) { + const started = this.startEncoder(id, variantPolicy, variant); + if (!started) return { ok: false, status: 503, code: "CHANNEL_GONE", error: "the channel ended before the relay could start" }; + running = started; + } + // The opening bytes and the join point, in the same tick. + const preface = this.channels.opening(id); + const session = running.encoder.join(listener, preface); + return { ok: true, session, codecs, generation: running.encoder.generation, kind: this.channels.kindOf(id) ?? "" }; + } + + private startEncoder(id: string, policy: LosslessPolicy, variant: string): Running | null { + this.generation = (this.generation + 1) % 0xffff_ffff; + const metrics = this.metrics.get(id) ?? new ChannelMetrics(); + this.metrics.set(id, metrics); + let record: Running | null = null; + const encoder = new RelayEncoder({ + generation: this.generation, + policy, + pool: this.pool, + metrics, + boundary: "channel", + onAbort: () => { + if (record && this.running.get(id) === record) { + this.running.delete(id); + record.detach(); + } + }, + onIdle: () => { + // Nobody receiving: no compressor running. The next receiver + // starts a new generation. + if (record && this.running.get(id) === record) { + this.running.delete(id); + record.detach(); + encoder.end(); + } + }, + }); + const detach = this.channels.listen(id, { + write: (chunk) => encoder.write(chunk), + end: () => encoder.end(), + pending: () => metrics.queueBytes, + }); + if (detach === null) return null; + encoder.attach(); + record = { encoder, detach, variant }; + this.running.set(id, record); + return record; + } + + /** Start listening to another nixamp's channel as one of ours. */ + pull(id: string, from: string, key: string | null, name: string): { ok: true } | { ok: false; status: 409 | 400; error: string } { + if (!/^https?:\/\//i.test(from)) return { ok: false, status: 400, error: "the relay address must be http or https" }; + if (this.channels.has(id) || this.incoming.has(id)) return { ok: false, status: 409, error: `channel "${id}" is already on` }; + const inbound = new Incoming(id, from, key, name, this.channels, this.options.onEvent ?? (() => undefined), (gone) => this.incoming.delete(gone)); + this.incoming.set(id, inbound); + inbound.start(); + return { ok: true }; + } + + /** Stop an incoming relay, and the channel it feeds. */ + stopPull(id: string): boolean { + const inbound = this.incoming.get(id); + if (!inbound) return false; + inbound.stop(); + return true; + } + + /** + * Analyse a live channel: up to `seconds` of it, or the byte limit, + * whichever comes first, as a job. The listener is one more on the + * channel; the channel is not touched. + */ + analyzeChannel(id: string, seconds: number, owner: string): { job: Job; existing: boolean } | { error: string; status: 404 | 429 } { + if (!this.channels.has(id)) return { error: "nothing is playing on that channel", status: 404 }; + const window = Math.min(SAMPLE_MAX_SECONDS, Math.max(1, Math.floor(seconds))); + const started = this.jobs.start(`channel:${id}:${window}`, `channel ${id}`, owner, (signal, progress) => this.sampleChannel(id, window, signal, progress)); + if (started === null) return { error: "too many analyses are queued; try again later", status: 429 }; + return started; + } + + private sampleChannel(id: string, seconds: number, signal: AbortSignal, progress: (bytes: number, ms: number) => void): Promise { + return new Promise((resolve, reject) => { + const pieces: Buffer[] = []; + let bytes = 0; + const began = Date.now(); + let detach: (() => void) | null = null; + let finished = false; + const finish = (truncated: boolean): void => { + if (finished) return; + finished = true; + clearTimeout(timer); + detach?.(); + const info = this.channels.list().find((c) => c.id === id); + const sample = Buffer.concat(pieces, bytes); + analyzeSample(sample, { + boundary: "channel", + source: { kind: "channel", name: id }, + sampleMs: Date.now() - began, + truncated, + signal, + policy: this.store.get(id).losslessCompression, + tsAware: true, + ...(info?.codecs ? { codecs: info.codecs } : {}), + }).then(resolve, reject); + }; + const timer = setTimeout(() => finish(false), seconds * 1000); + timer.unref?.(); + signal.addEventListener("abort", () => { + if (finished) return; + finished = true; + clearTimeout(timer); + detach?.(); + reject(new Error("cancelled")); + }, { once: true }); + detach = this.channels.listen(id, { + write: (chunk) => { + if (finished) return true; + pieces.push(Buffer.from(chunk)); + bytes += chunk.length; + progress(bytes, Date.now() - began); + if (bytes >= SAMPLE_MAX_BYTES) finish(true); + return true; + }, + end: () => finish(false), + }); + if (detach === null) { + finished = true; + clearTimeout(timer); + reject(new Error("nothing is playing on that channel")); + } + }); + } + + /** Analyse a file on this machine: its first bytes, up to the sample limit. */ + analyzeFile(path: string, signal?: AbortSignal): Promise { + return analyzeFile(path, { + policy: this.store.get(STATIC_CHANNEL).losslessCompression, + ...(this.options.ffprobe ? { ffprobe: this.options.ffprobe } : {}), + ...(signal ? { signal } : {}), + }); + } + + /** + * The static representation of a file. Ready when it is on disk and + * still describes the file; building when this call started it (or + * found it being built); failed when the last build said why; off when + * this server keeps no cache or the static policy is off. Never a wait: + * a film takes minutes to read, and a request is not held for that. + */ + representation(file: string): { state: "off" } | { state: "building" } | { state: "failed"; reason: string } | { state: "ready"; path: string; entry: NonNullable>["entry"] } { + if (!this.statics) return { state: "off" }; + const policy = this.effective(STATIC_CHANNEL).losslessCompression; + if (policy.mode === "off") return { state: "off" }; + const variant = variantOf(policy); + const found = this.statics.lookup(file, variant); + if (found) { + this.failed.delete(file); + return { state: "ready", path: found.path, entry: found.entry }; + } + const failure = this.failed.get(file); + if (failure !== undefined) { + this.failed.delete(file); + return { state: "failed", reason: failure }; + } + void this.statics.prepare(file, policy, variant).then((result: Prepared) => { + if (!result.ok) this.failed.set(file, result.reason); + }); + return { state: "building" }; + } + + private readonly failed = new Map(); + + stopAll(): void { + for (const [id, running] of [...this.running]) { + this.running.delete(id); + running.detach(); + running.encoder.end(); + } + for (const inbound of [...this.incoming.values()]) inbound.stop(); + this.jobs.stopAll(); + } +} diff --git a/src/compression/static.ts b/src/compression/static.ts new file mode 100644 index 0000000..5df2864 --- /dev/null +++ b/src/compression/static.ts @@ -0,0 +1,264 @@ +/** + * A file's compressed representation, made once and kept. + * + * A film on disk is asked for the same way every time, so its envelope is + * built once, whole, and served as a file thereafter. The original is never + * touched: seeking, ranges and every ordinary route keep using it, and the + * representation is a second file beside it in the cache directory, named + * by what it was made from. It is published by rename, so a request never + * reads half of one; it is checked against the file's size and mtime on + * every lookup, and against the file's own digest at the end of the build, + * so a file rewritten while it was being read is not served as itself; + * and the cache as a whole has a byte budget, the least recently served + * going first. + */ +import { createHash } from "node:crypto"; +import { createReadStream, createWriteStream, mkdirSync, readdirSync, readFileSync, renameSync, rmSync, statSync, unlinkSync, writeFileSync } from "node:fs"; +import { join, resolve } from "node:path"; +import { encode, Pool } from "./codec.ts"; +import { encodeEndFrame, encodeFrameHeader, encodeStreamHeader, FRAME_DATA, FRAME_HEADER_BYTES, type Mode, sha256, STREAM_HEADER_BYTES } from "./envelope.ts"; +import { eligible, type LosslessPolicy } from "./policy.ts"; +import { TS_PACKET, tsLayout } from "./ts-transform.ts"; + +export const REPRESENTATION_SUFFIX = ".nxs"; + +export interface StaticEntry { + file: string; + size: number; + mtimeMs: number; + /** Of the original file, whole. */ + sha256: string; + /** The policy variant it was built under. Another variant is another entry. */ + variant: string; + /** How big the representation is, headers included. */ + bytes: number; + blocks: number; + compressedBlocks: number; + builtAt: number; + lastUsedAt: number; +} + +export interface StaticCacheOptions { + /** Total bytes the cache may hold. */ + maxBytes?: number; + pool?: Pool; +} + +export type Prepared = { ok: true; entry: StaticEntry; path: string } | { ok: false; reason: string }; + +export class StaticCache { + private readonly building = new Map>(); + private readonly pool: Pool; + private readonly maxBytes: number; + + constructor(readonly dir: string, options: StaticCacheOptions = {}) { + this.pool = options.pool ?? new Pool({ concurrency: 2, maxQueued: 8, timeoutMs: 10_000 }); + this.maxBytes = options.maxBytes ?? 2 * 1024 * 1024 * 1024; + mkdirSync(dir, { recursive: true }); + } + + /** The cache file name for a file under a variant: from its path, never its content, so it can be found before the content is read. */ + private nameFor(file: string, variant: string): string { + return createHash("sha256").update(`${resolve(file)}\0${variant}`).digest("hex").slice(0, 32); + } + + /** + * The representation of `file`, if one is there and still describes the + * file on disk. Null otherwise -- and a stale one is removed, not kept + * for a later lookup to be fooled by. + */ + lookup(file: string, variant: string): { entry: StaticEntry; path: string } | null { + const name = this.nameFor(file, variant); + const meta = join(this.dir, `${name}.json`); + const path = join(this.dir, `${name}${REPRESENTATION_SUFFIX}`); + let entry: StaticEntry; + try { + entry = JSON.parse(readFileSync(meta, "utf8")) as StaticEntry; + } catch { + return null; + } + let stat; + try { + stat = statSync(file); + } catch { + this.drop(name); + return null; + } + if (stat.size !== entry.size || Math.floor(stat.mtimeMs) !== Math.floor(entry.mtimeMs)) { + this.drop(name); + return null; + } + try { + if (statSync(path).size !== entry.bytes) { + this.drop(name); + return null; + } + } catch { + this.drop(name); + return null; + } + entry.lastUsedAt = Date.now(); + try { + writeFileSync(meta, JSON.stringify(entry)); + } catch { + // The timestamp is for eviction order; losing it costs nothing now. + } + return { entry, path }; + } + + /** + * Build the representation, or return the one being built. Two requests + * for the same file do one read. + */ + prepare(file: string, policy: LosslessPolicy, variant: string): Promise { + const found = this.lookup(file, variant); + if (found) return Promise.resolve({ ok: true, ...found }); + const name = this.nameFor(file, variant); + const building = this.building.get(name); + if (building) return building; + const job = this.build(file, policy, variant, name).finally(() => this.building.delete(name)); + this.building.set(name, job); + return job; + } + + private async build(file: string, policy: LosslessPolicy, variant: string, name: string): Promise { + let before; + try { + before = statSync(file); + } catch (error) { + return { ok: false, reason: (error as Error).message }; + } + if (!before.isFile()) return { ok: false, reason: "not a file" }; + const path = join(this.dir, `${name}${REPRESENTATION_SUFFIX}`); + const tmp = join(this.dir, `${name}.${process.pid}.building`); + const out = createWriteStream(tmp); + let broken: Error | null = null; + out.on("error", (error) => { + broken = error; + }); + const digest = createHash("sha256"); + let total = 0; + let bytes = 0; + let blocks = 0; + let compressedBlocks = 0; + let seq = 0; + const put = (chunk: Buffer): Promise => + new Promise((done, fail) => { + if (broken) { + fail(broken); + return; + } + bytes += chunk.length; + if (out.write(chunk)) done(); + else out.once("drain", done); + }); + try { + await put(encodeStreamHeader({ version: 1, boundary: "source", generation: 1, maxFrameBytes: policy.maxBlockBytes })); + const reader = createReadStream(file, { highWaterMark: policy.maxBlockBytes }); + for await (const piece of reader) { + const block = piece as Buffer; + digest.update(block); + total += block.length; + const { mode, payload } = await this.encodeBlock(block, policy); + blocks += 1; + if (mode !== "stored") compressedBlocks += 1; + await put(encodeFrameHeader({ type: FRAME_DATA, mode, seq, originalLength: block.length, encodedLength: payload.length, sha256: sha256(block) })); + await put(payload); + seq += 1; + } + const whole = digest.digest(); + await put(encodeEndFrame(seq, total, whole)); + await new Promise((done, fail) => out.end((error?: Error | null) => (error ? fail(error) : done()))); + // The file as it is now must be the file that was read. A file that + // changed underneath is not the file this envelope describes. + const after = statSync(file); + if (after.size !== before.size || Math.floor(after.mtimeMs) !== Math.floor(before.mtimeMs) || total !== before.size) { + unlinkSync(tmp); + return { ok: false, reason: "the file changed while it was being read" }; + } + const entry: StaticEntry = { + file: resolve(file), + size: before.size, + mtimeMs: before.mtimeMs, + sha256: whole.toString("hex"), + variant, + bytes, + blocks, + compressedBlocks, + builtAt: Date.now(), + lastUsedAt: Date.now(), + }; + this.makeRoom(bytes); + writeFileSync(join(this.dir, `${name}.json.tmp`), JSON.stringify(entry)); + renameSync(tmp, path); + renameSync(join(this.dir, `${name}.json.tmp`), join(this.dir, `${name}.json`)); + return { ok: true, entry, path }; + } catch (error) { + try { + out.destroy(); + unlinkSync(tmp); + } catch { + // Nothing to clean, or already gone. + } + return { ok: false, reason: (error as Error).message }; + } + } + + private async encodeBlock(block: Buffer, policy: LosslessPolicy): Promise<{ mode: Mode; payload: Buffer }> { + if (policy.mode === "off") return { mode: "stored", payload: block }; + const candidates: Mode[] = ["zstd"]; + if (policy.tsAware && tsLayout(block)?.packetSize === TS_PACKET) candidates.push("ts-zstd"); + const results = await Promise.all(candidates.map((mode) => this.pool.run(() => encode(mode, block, policy.zstdLevel), { timeoutMs: 10_000 }))); + let best = 0; + for (let i = 1; i < results.length; i += 1) if ((results[i] as Buffer).length < (results[best] as Buffer).length) best = i; + const payload = results[best] as Buffer; + return eligible(block.length, payload.length, policy) ? { mode: candidates[best] as Mode, payload } : { mode: "stored", payload: block }; + } + + /** Every entry on disk, oldest use first. */ + entries(): StaticEntry[] { + const list: StaticEntry[] = []; + for (const name of readdirSync(this.dir)) { + if (!name.endsWith(".json")) continue; + try { + list.push(JSON.parse(readFileSync(join(this.dir, name), "utf8")) as StaticEntry); + } catch { + // Half-written or foreign; not ours to count. + } + } + return list.sort((a, b) => a.lastUsedAt - b.lastUsedAt); + } + + get totalBytes(): number { + return this.entries().reduce((sum, entry) => sum + entry.bytes, 0); + } + + /** Drop the least recently served until `incoming` more bytes fit. */ + private makeRoom(incoming: number): void { + let total = this.totalBytes; + for (const entry of this.entries()) { + if (total + incoming <= this.maxBytes) return; + this.drop(this.nameFor(entry.file, entry.variant)); + total -= entry.bytes; + } + } + + remove(file: string, variant: string): void { + this.drop(this.nameFor(file, variant)); + } + + private drop(name: string): void { + for (const suffix of [REPRESENTATION_SUFFIX, ".json"]) { + try { + rmSync(join(this.dir, `${name}${suffix}`), { force: true }); + } catch { + // Already gone. + } + } + } + + /** What the envelope adds to a file of `size` bytes at worst: every block stored. */ + static worstCaseBytes(size: number, blockBytes: number): number { + return STREAM_HEADER_BYTES + size + FRAME_HEADER_BYTES * (Math.ceil(size / blockBytes) + 1); + } +} diff --git a/src/compression/store.ts b/src/compression/store.ts new file mode 100644 index 0000000..fdc471c --- /dev/null +++ b/src/compression/store.ts @@ -0,0 +1,126 @@ +/** + * Where a server keeps its compression settings: beside the channels it + * remembers, keyed by port for the same reason. A policy a channel does not + * have is the default, which is off. A global switch sits above every + * channel, so one command takes every relay down without editing any of + * them and without forgetting what they were set to. + */ +import { mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { type ChannelPolicy, defaultPolicy, type HlsPackaging, normalizePolicy } from "./policy.ts"; + +const FILE = "compression.json"; + +export interface GlobalSettings { + /** The kill switch. Off means no new compressed session anywhere on this server. */ + enabled: boolean; + /** How HLS is packaged when a channel does not say. */ + hlsPackaging: HlsPackaging; +} + +interface Saved { + global: GlobalSettings; + channels: Record; +} + +export class PolicyStore { + private readonly channels = new Map(); + private globalSettings: GlobalSettings = { enabled: true, hlsPackaging: "mpegts" }; + + constructor(private readonly dir: string | null, private readonly port: number) { + this.load(); + } + + get global(): GlobalSettings { + return { ...this.globalSettings }; + } + + setGlobal(change: Partial): GlobalSettings { + if (typeof change.enabled === "boolean") this.globalSettings.enabled = change.enabled; + if (change.hlsPackaging === "mpegts" || change.hlsPackaging === "fmp4") this.globalSettings.hlsPackaging = change.hlsPackaging; + this.save(); + return this.global; + } + + /** The policy for a channel: its own, or the default. Always a fresh copy. */ + get(id: string): ChannelPolicy { + const own = this.channels.get(id); + return own ? { ...own, losslessCompression: { ...own.losslessCompression } } : defaultPolicy(); + } + + has(id: string): boolean { + return this.channels.has(id); + } + + /** + * Apply a change. `expectVersion`, when given, must be the version the + * caller last saw, or the change is refused: two operators editing the + * same channel do not get to overwrite each other blind. + */ + set(id: string, change: unknown, expectVersion?: number): { ok: true; policy: ChannelPolicy } | { ok: false; status: 400 | 409 | 412; errors: string[] } { + const current = this.get(id); + if (expectVersion !== undefined && expectVersion !== current.version) { + return { ok: false, status: 412, errors: [`the policy is at version ${current.version}, not ${expectVersion}`] }; + } + const normalized = normalizePolicy(change, current); + if (!normalized.ok) return { ok: false, status: 400, errors: normalized.errors }; + const policy = { ...normalized.policy, version: current.version + 1 }; + this.channels.set(id, policy); + this.save(); + return { ok: true, policy }; + } + + /** Every channel with a policy of its own. */ + list(): Record { + return Object.fromEntries([...this.channels.entries()].map(([id, policy]) => [id, this.get(id)])); + } + + private load(): void { + if (this.dir === null) return; + let all: Record; + try { + all = JSON.parse(readFileSync(join(this.dir, FILE), "utf8")) as Record; + } catch { + return; + } + const mine = all[String(this.port)]; + if (typeof mine !== "object" || mine === null) return; + const saved = mine as Partial; + if (saved.global && typeof saved.global === "object") { + if (typeof saved.global.enabled === "boolean") this.globalSettings.enabled = saved.global.enabled; + if (saved.global.hlsPackaging === "mpegts" || saved.global.hlsPackaging === "fmp4") this.globalSettings.hlsPackaging = saved.global.hlsPackaging; + } + if (saved.channels && typeof saved.channels === "object") { + for (const [id, raw] of Object.entries(saved.channels)) { + // Checked as if it were being set now: a file edited by hand, or + // written by a version with different limits, is not trusted whole. + const { version, ...rest } = (raw ?? {}) as Partial; + const normalized = normalizePolicy(rest, defaultPolicy()); + if (!normalized.ok) continue; + this.channels.set(id, { ...normalized.policy, version: typeof version === "number" && version >= 0 ? Math.floor(version) : 0 }); + } + } + } + + private save(): void { + if (this.dir === null) return; + let all: Record = {}; + try { + all = JSON.parse(readFileSync(join(this.dir, FILE), "utf8")) as Record; + } catch { + // First time, or unreadable: start again rather than refuse to remember. + } + const saved: Saved = { global: this.globalSettings, channels: Object.fromEntries(this.channels) }; + all[String(this.port)] = saved; + try { + mkdirSync(this.dir, { recursive: true }); + // Whole, then renamed: a crash mid-write must not leave half a file + // that the next start reads as "no settings". + const tmp = join(this.dir, `${FILE}.${process.pid}.tmp`); + writeFileSync(tmp, JSON.stringify(all, null, 2)); + renameSync(tmp, join(this.dir, FILE)); + } catch { + // A state directory that cannot be written costs a memory, not a stream. + } + } +} diff --git a/src/compression/ts-transform.ts b/src/compression/ts-transform.ts new file mode 100644 index 0000000..eb5acf6 --- /dev/null +++ b/src/compression/ts-transform.ts @@ -0,0 +1,148 @@ +/** + * The experimental transport-stream transform: headers here, payloads there. + * + * An MPEG transport stream is 188-byte packets, each starting 0x47, each + * with a four-byte header whose fields (PID, continuity counter, flags) + * change a little from one packet to the next, followed by payload that is + * already-encoded video and does not compress. Interleaved, the headers are + * lost in the noise; grouped together they are a very regular sequence and + * an ordinary compressor does well on them. That is the whole idea. It is + * a reversible rearrangement in front of Zstandard, not a new compressor, + * and it is selected only when the complete output is smaller than not + * doing it. + * + * Nothing is interpreted beyond what is needed to know where a header + * ends: the sync byte, the adaptation flags and the adaptation length. A + * null packet keeps its bytes. A packet whose adaptation length overruns + * is kept whole in the header section. Bytes before the first aligned + * packet and after the last one are kept as they are. `tsJoin(tsSplit(x))` + * is `x` for every x, or `tsSplit` says no. + */ +export const TS_PACKET = 188; +/** Layouts we recognise and leave alone: timestamped and forward-error-corrected. */ +export const TS_PACKET_SIZES = [188, 192, 204] as const; +export const SYNC = 0x47; + +const VERSION = 1; +const HEAD_BYTES = 1 + 4 + 4 + 4; +/** How many packets must line up before the layout is believed. */ +const CONFIDENCE = 4; + +export interface TsLayout { + packetSize: 188 | 192 | 204; + /** Where the first full packet starts. */ + offset: number; +} + +/** + * Which packet size, if any, `bytes` is laid out in, by finding a stride at + * which the sync byte repeats. A 192-byte packet carries a four-byte + * timestamp before the sync, which is why the offset may be four. + */ +export function tsLayout(bytes: Uint8Array, minPackets = CONFIDENCE): TsLayout | null { + for (const size of TS_PACKET_SIZES) { + const skip = size === 192 ? 4 : 0; + for (let offset = 0; offset < size && offset + size * minPackets <= bytes.length; offset += 1) { + let ok = true; + for (let k = 0; k < minPackets; k += 1) { + if (bytes[offset + skip + k * size] !== SYNC) { + ok = false; + break; + } + } + if (ok) return { packetSize: size, offset }; + } + } + return null; +} + +/** How long the header part of one 188-byte packet is: 4, plus the adaptation field. */ +function headerLength(packet: Uint8Array): number { + const afc = ((packet[3] as number) >> 4) & 0x3; + if (afc === 0b10 || afc === 0b11) { + const length = packet[4] as number; + // Overrun: the whole packet is treated as header and kept intact. + if (5 + length > TS_PACKET) return TS_PACKET; + return 5 + length; + } + return 4; +} + +/** + * Split into [head][headers...][payloads...][tail]. Null when `bytes` is + * not a run of aligned 188-byte packets, which is the caller's cue to use + * the plain codec. 192- and 204-byte layouts are detected and refused here + * rather than handled: they would need their own tested join. + */ +export function tsSplit(bytes: Buffer): Buffer | null { + const layout = tsLayout(bytes); + if (layout === null || layout.packetSize !== TS_PACKET) return null; + const start = layout.offset; + // The aligned run: as many whole packets as keep syncing. The first + // packet that does not ends the run, and the rest is tail. + let count = 0; + while (start + (count + 1) * TS_PACKET <= bytes.length && bytes[start + count * TS_PACKET] === SYNC) count += 1; + const end = start + count * TS_PACKET; + const headers: Buffer[] = []; + const payloads: Buffer[] = []; + let headerBytes = 0; + for (let i = 0; i < count; i += 1) { + const packet = bytes.subarray(start + i * TS_PACKET, start + (i + 1) * TS_PACKET); + const hl = headerLength(packet); + headers.push(packet.subarray(0, hl)); + headerBytes += hl; + if (hl < TS_PACKET) payloads.push(packet.subarray(hl)); + } + const head = Buffer.alloc(HEAD_BYTES); + head.writeUInt8(VERSION, 0); + head.writeUInt32BE(start, 1); + head.writeUInt32BE(count, 5); + head.writeUInt32BE(bytes.length - end, 9); + return Buffer.concat([ + head, + bytes.subarray(0, start), + Buffer.concat(headers, headerBytes), + Buffer.concat(payloads, count * TS_PACKET - headerBytes), + bytes.subarray(end), + ]); +} + +/** The inverse. Null for anything that is not the output of `tsSplit`. */ +export function tsJoin(split: Buffer): Buffer | null { + if (split.length < HEAD_BYTES || split.readUInt8(0) !== VERSION) return null; + const headLen = split.readUInt32BE(1); + const count = split.readUInt32BE(5); + const tailLen = split.readUInt32BE(9); + const total = headLen + count * TS_PACKET + tailLen; + if (count > 0x00ff_ffff || split.length < HEAD_BYTES + headLen + tailLen) return null; + const out = Buffer.alloc(total); + split.copy(out, 0, HEAD_BYTES, HEAD_BYTES + headLen); + // Headers are variable length, so they are walked exactly as they were + // written: read one, learn its length from its own bytes, move on. The + // payloads follow all the headers, so their start is only known after + // the walk -- hence two passes. + let cursor = HEAD_BYTES + headLen; + const lengths: number[] = []; + for (let i = 0; i < count; i += 1) { + if (cursor + 4 > split.length) return null; + const hl = headerLength(split.subarray(cursor, cursor + TS_PACKET)); + if (cursor + hl > split.length) return null; + lengths.push(hl); + cursor += hl; + } + let payloadCursor = cursor; + let headerCursor = HEAD_BYTES + headLen; + for (let i = 0; i < count; i += 1) { + const hl = lengths[i] as number; + const at = headLen + i * TS_PACKET; + split.copy(out, at, headerCursor, headerCursor + hl); + headerCursor += hl; + const pl = TS_PACKET - hl; + if (payloadCursor + pl > split.length) return null; + split.copy(out, at + hl, payloadCursor, payloadCursor + pl); + payloadCursor += pl; + } + if (payloadCursor + tailLen !== split.length) return null; + split.copy(out, headLen + count * TS_PACKET, payloadCursor, payloadCursor + tailLen); + return out; +} diff --git a/src/hls.ts b/src/hls.ts index 40f5d1b..f136ee7 100644 --- a/src/hls.ts +++ b/src/hls.ts @@ -14,6 +14,7 @@ * untouched; this is one more listener on it. */ import { spawn, type ChildProcess } from "node:child_process"; +import { randomBytes } from "node:crypto"; import { mkdtempSync, readFileSync, rmSync, statSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -27,11 +28,32 @@ export const IDLE_MS = 60_000; /** How long the first playlist may take to appear before it is a failure. */ export const FIRST_PLAYLIST_MS = 20_000; -const SEGMENT = /^seg\d{5}\.ts$/; +/** + * How the segments are wrapped. MPEG-TS is what every HLS client has played + * since 2009; fragmented MP4 is the same boxes the channel already carries, + * copied into files without a second container around them, which is + * smaller and is what a modern client prefers. Neither re-encodes anything. + */ +export type Packaging = "mpegts" | "fmp4"; + +const SEGMENT = /^seg\d{5}\.(?:ts|m4s)$/; +/** + * The initialisation segment carries a token from the packager that made + * it, so a client holding the init of a previous run cannot pair it with + * the media of this one: the names do not match, and the old one is 404. + */ +const INIT = /^init-[0-9a-f]{8}\.mp4$/; /** A segment file name, or "" for anything that is not one. Never a path. */ export function segmentName(requested: string): string { - return SEGMENT.test(requested) ? requested : ""; + return SEGMENT.test(requested) || INIT.test(requested) ? requested : ""; +} + +/** What to call a segment file in a response. */ +export function segmentType(name: string): string { + if (name.endsWith(".m4s")) return "video/iso.segment"; + if (name.endsWith(".mp4")) return "video/mp4"; + return "video/mp2t"; } /** @@ -39,18 +61,24 @@ export function segmentName(requested: string): string { * * A browser resolves segment names against the playlist's URL and drops its * query, so the key that opened the playlist never reaches the segments and - * each one answers 401. The key travels on every line instead. + * each one answers 401. The key travels on every line instead -- including + * the initialisation segment named inside the EXT-X-MAP tag, which is + * fetched exactly like a segment and refused exactly like one. */ export function withKey(playlist: string, key: string): string { if (key === "") return playlist; + const query = `?k=${encodeURIComponent(key)}`; return playlist .split("\n") - .map((line) => (line !== "" && !line.startsWith("#") ? `${line}?k=${encodeURIComponent(key)}` : line)) + .map((line) => { + if (line.startsWith("#EXT-X-MAP:")) return line.replace(/URI="([^"]+)"/, (_, uri: string) => `URI="${uri}${query}"`); + return line !== "" && !line.startsWith("#") ? `${line}${query}` : line; + }) .join("\n"); } /** The ffmpeg arguments: copy what arrives on stdin into a rolling playlist. */ -export function packagerArgs(dir: string): string[] { +export function packagerArgs(dir: string, packaging: Packaging = "mpegts", token = "00000000"): string[] { return [ "-hide_banner", "-loglevel", "error", @@ -63,12 +91,53 @@ export function packagerArgs(dir: string): string[] { // keyframe so a joiner can begin anywhere; written whole then renamed so // a request never reads half a file. "-hls_flags", "delete_segments+omit_endlist+independent_segments+temp_file", - "-hls_segment_type", "mpegts", - "-hls_segment_filename", join(dir, "seg%05d.ts"), + "-hls_segment_type", packaging, + ...(packaging === "fmp4" ? ["-hls_fmp4_init_filename", `init-${token}.mp4`] : []), + "-hls_segment_filename", join(dir, packaging === "fmp4" ? "seg%05d.m4s" : "seg%05d.ts"), join(dir, "index.m3u8"), ]; } +export interface PlaylistReport { + packaging: Packaging; + targetSeconds: number; + /** Whether the playlist claims every segment starts on a keyframe. */ + independent: boolean; + /** The longest segment the playlist offers, in seconds. A copy cannot cut inside a GOP, so this can exceed the target. */ + longestSegmentSeconds: number; + segments: number; + /** Whether an initialisation segment is named. */ + initialised: boolean; +} + +/** + * What a playlist actually says, as opposed to what the packager was asked + * for: how long its segments came out, whether it names an init segment. + * A two-second target on a stream with ten-second keyframes gives + * ten-second segments, and only the playlist knows. + */ +export function playlistReport(playlist: string, packaging: Packaging): PlaylistReport { + let target = 0; + let longest = 0; + let segments = 0; + for (const line of playlist.split("\n")) { + if (line.startsWith("#EXT-X-TARGETDURATION:")) target = Number(line.slice("#EXT-X-TARGETDURATION:".length)) || 0; + const inf = /^#EXTINF:([\d.]+)/.exec(line); + if (inf) { + segments += 1; + longest = Math.max(longest, Number(inf[1]) || 0); + } + } + return { + packaging, + targetSeconds: target, + independent: playlist.includes("#EXT-X-INDEPENDENT-SEGMENTS"), + longestSegmentSeconds: longest, + segments, + initialised: playlist.includes("#EXT-X-MAP:"), + }; +} + export interface Packaged { /** Feed it the channel's bytes: the header first, then every fragment. */ write(chunk: Buffer): boolean; @@ -79,6 +148,8 @@ export interface Packaged { /** One channel's packager. */ class Packager implements Packaged { readonly dir: string; + /** Names this run's init segment, so it cannot be mixed with another run's media. */ + readonly token = randomBytes(4).toString("hex"); private child: ChildProcess | null = null; private idle: ReturnType | null = null; private stopped = false; @@ -87,6 +158,7 @@ class Packager implements Packaged { constructor( private readonly id: string, private readonly ffmpeg: string[], + readonly packaging: Packaging, private readonly onStop: (id: string) => void, private readonly onEvent: (message: string) => void, ) { @@ -96,7 +168,7 @@ class Packager implements Packaged { start(listen: (listener: Packaged) => (() => void) | null): boolean { const [command, ...prefix] = this.ffmpeg as [string, ...string[]]; try { - this.child = spawn(command, [...prefix, ...packagerArgs(this.dir)], { stdio: ["pipe", "ignore", "pipe"] }); + this.child = spawn(command, [...prefix, ...packagerArgs(this.dir, this.packaging, this.token)], { stdio: ["pipe", "ignore", "pipe"] }); } catch (error) { this.onEvent(` HLS for "${this.id}" could not start: ${(error as Error).message}`); this.stop(); @@ -217,6 +289,8 @@ export class HlsPackagers { onEvent?: (message: string) => void; /** Injected for tests: how long to wait for the first playlist. */ firstPlaylistMs?: number; + /** How to wrap a channel's segments. MPEG-TS unless something says otherwise. */ + packaging?: (id: string) => Packaging; }, ) {} @@ -228,7 +302,8 @@ export class HlsPackagers { async playlist(id: string): Promise { let packager = this.running.get(id); if (!packager) { - packager = new Packager(id, this.options.ffmpeg, (gone) => this.running.delete(gone), this.options.onEvent ?? (() => undefined)); + const packaging = this.options.packaging?.(id) ?? "mpegts"; + packager = new Packager(id, this.options.ffmpeg, packaging, (gone) => this.running.delete(gone), this.options.onEvent ?? (() => undefined)); this.running.set(id, packager); if (!packager.start((listener) => this.options.listen(id, listener))) return null; } @@ -250,6 +325,15 @@ export class HlsPackagers { return packager.segment(name); } + /** What a running packager's playlist actually says, or null when none is running. */ + report(id: string): PlaylistReport | null { + const packager = this.running.get(id); + if (!packager) return null; + const text = packager.playlist(); + if (text === "") return null; + return playlistReport(text, packager.packaging); + } + /** The channel went: stop packaging it. */ stop(id: string): void { this.running.get(id)?.stop(); diff --git a/src/main.ts b/src/main.ts index 6c4828d..8545e8e 100644 --- a/src/main.ts +++ b/src/main.ts @@ -74,6 +74,7 @@ const HELP = `nixamp — it really whips the terminal's ass. nixamp daemon start|restart|stop|status serve in the background, and let go of it nixamp attach put the player back in front of the daemon nixamp admin [--url U] [--key K] who is connected, and re-stream to them + nixamp compression analyze|status|set|pull lossless relay compression: measure, see, set nixamp login [--with github] sign in to nixamp.com, in a browser or here nixamp logout / whoami forget it, or check it nixamp token create|list|revoke tokens for a machine that cannot sign in @@ -402,6 +403,11 @@ export async function main(): Promise { await admin(rest); return; } + if (first === "compression") { + const { compression } = await import("./compression/cli.ts"); + process.exitCode = await compression(rest); + return; + } if (first === "login" || first === "signup") { const { login } = await import("./session.ts"); process.exitCode = await login(first === "signup" ? [...rest, "--signup"] : rest); diff --git a/src/server.ts b/src/server.ts index 415aa23..dfbfada 100644 --- a/src/server.ts +++ b/src/server.ts @@ -51,7 +51,9 @@ import { stateDir } from "./daemon.ts"; import { readSession } from "./session.ts"; import { Directory, ENDED_TTL_MS, parseAnnouncement, type Listing } from "./directory.ts"; import { PartyLine, telnyxSms } from "./partyline.ts"; -import { HlsPackagers, withKey } from "./hls.ts"; +import { HlsPackagers, segmentType, withKey } from "./hls.ts"; +import { CompressionService } from "./compression/service.ts"; +import { handleChannelCompression, handleCompressionApi, handleStaticRelay, type RouteContext } from "./compression/routes.ts"; import { DEFAULT_SITE as NICHEDB, Enricher, type EnrichKind, FIXTURE_TTL_MS } from "./enrich.ts"; import { contentTypeFor, downloadArgs, fileNameFor, inputArgsFor, linkChannelId, mergeDownloadArgs, playableLink, resolveLink, @@ -1275,6 +1277,8 @@ export interface HandlerOptions { ytdlp?: string[] | null; /** Channels as HLS, for Safari on a phone, which plays a live stream no other way. */ hls?: HlsPackagers; + /** Lossless relay compression, its policies, diagnostics and static representations. */ + compression?: CompressionService; /** What a name is -- a film, a channel, a fixture -- asked of nichedb.dev and remembered. */ enricher?: Enricher; /** A Netscape cookies file for sites that want a signed-in browser, when there is one. */ @@ -1473,6 +1477,24 @@ export function createHandler(engine: Engine, options: HandlerOptions) { } return channelEvent; }; + // What the compression routes need from this handler: how to answer, + // how to count a stream, and whether this caller holds the controls. + const compressionCtx = (): RouteContext | null => + options.compression + ? { + service: options.compression, + json, + readBody: (incoming) => readBody(incoming), + cors: CORS, + watch: (incoming, outgoing, track) => watch(incoming, outgoing, "stream", track), + controls: async () => { + if (key === null) return true; + if (scopeOf(keyFrom(request, url), key, null) === "control") return true; + if (options.owner) return (await options.owner.check(false, tokenFrom(request.headers))).allowed; + return false; + }, + } + : null; if (request.method === "OPTIONS") { response.writeHead(204, CORS); @@ -3097,6 +3119,12 @@ export function createHandler(engine: Engine, options: HandlerOptions) { // A channel is one publisher and everybody listening to them. Two or three // devices can publish at once, each to their own channel, and a listener // picks which to hear. + // The server's compression at a glance, its switch, and the analyses. + if (path.startsWith("/api/compression")) { + const compression = compressionCtx(); + if (compression && (await handleCompressionApi(request, response, compression, path))) return; + } + if (path === "/api/channels" && options.channels) { // Without the source. Anyone holding the listen link may ask what is // on, and the address a channel is pulled from is the one thing about @@ -3135,6 +3163,12 @@ export function createHandler(engine: Engine, options: HandlerOptions) { } } + // Its compression policy, an analysis of it, and its relay to or from + // another nixamp. The ordinary playback URL above is untouched by any + // of it: a relay is a different media type on a different path. + const compression = compressionCtx(); + if (compression && (await handleChannelCompression(request, response, compression, id, action, file))) return; + // The same channel as HLS: a playlist of short files, which is what // Safari on an iPhone plays live -- it will not take the endless MP4 // below, and spun on it a few times before giving up. Packaged on @@ -3172,7 +3206,8 @@ export function createHandler(engine: Engine, options: HandlerOptions) { watch(request, response, "stream", id); response.writeHead(200, { ...CORS, - "content-type": "video/mp2t", + // By its name: a TS segment, an fMP4 media segment, or the init. + "content-type": segmentType(file ?? ""), "cache-control": "no-store", "content-length": statSync(segment).size, }); @@ -3216,9 +3251,12 @@ export function createHandler(engine: Engine, options: HandlerOptions) { } if (action === undefined && request.method === "DELETE") { - // Asked once: the second call would answer false, having just stopped - // the thing it was asking about. - const stopped = channels.stop(id); + // A channel fed by another nixamp's relay is taken off by stopping + // the relay, which takes the channel with it; otherwise it would + // dial again the moment the channel went. Asked once either way: + // the second call would answer false, having just stopped the + // thing it was asking about. + const stopped = (options.compression?.stopPull(id) ?? false) || channels.stop(id); // Taken off on purpose is forgotten on purpose: it must not come back // at the next restart. if (stopped) { @@ -3712,6 +3750,30 @@ export function createHandler(engine: Engine, options: HandlerOptions) { return; } + // A library file in the relay envelope: the same bytes as /api/media/N, + // framed and compressed once and kept. The original stays where it is + // and keeps answering ranges; this cannot be asked for one. + const staticRelay = /^\/api\/media\/(\d+)\/relay$/.exec(path); + if (staticRelay && request.method === "GET") { + const compression = compressionCtx(); + if (!compression) { + json(response, 503, { error: "this server keeps no relay representations" }); + return; + } + if (!options.media) { + json(response, 403, { error: "media streaming is off" }); + return; + } + const index = Number(staticRelay[1]); + const file = engine.trackPath(index); + if (file === undefined) { + json(response, 404, { error: "no such track" }); + return; + } + handleStaticRelay(request, response, compression, file, engine.snapshot().tracks?.[index]?.title ?? file); + return; + } + if (path.startsWith("/api/media/")) { if (!options.media) { json(response, 403, { error: "media streaming is off" }); @@ -4298,10 +4360,22 @@ export async function serve(argv: string[], version = "0.1.0"): Promise { cacheFile: join(stateDir(), "enrich.json"), onEvent: (message) => console.log(message), }); + // Off unless a policy says otherwise: every channel plays exactly as it + // did, and a relay is a thing another nixamp asks for by name. + const compression = new CompressionService({ + channels, + stateDir: stateDir(), + port: options.port, + ffprobe: tools.ffprobe, + cacheDir: join(stateDir(), "relay-cache"), + onEvent: (message) => console.log(message), + }); const hls = new HlsPackagers({ ffmpeg: tools.ffmpeg, listen: (id, listener) => channels.listen(id, listener), onEvent: (message) => console.log(message), + // TS unless the channel's policy, or the server's, asks for fMP4. + packaging: (id) => compression.packagingOf(id), }); // The channels this server was carrying when it was last stopped, put back @@ -4698,6 +4772,7 @@ export async function serve(argv: string[], version = "0.1.0"): Promise { ytdlp: tools.ytdlp ?? null, cookies: cookiesFile(), hls, + compression, enricher, ...(tls ? { tls } : {}), // Untagged, so a directory of five thousand files answers at once; the @@ -5147,6 +5222,7 @@ export async function serve(argv: string[], version = "0.1.0"): Promise { const shutdown = (): void => { rtmp?.stop(); enricher.save(); + compression.stopAll(); hls.stopAll(); channels.stopAll(); ingest?.stopRtmp(); diff --git a/test/compression-analyze.test.ts b/test/compression-analyze.test.ts new file mode 100644 index 0000000..0cfd05d --- /dev/null +++ b/test/compression-analyze.test.ts @@ -0,0 +1,105 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { randomBytes } from "node:crypto"; +import { analyzeSample, analyzeTs, benchmark, sniffContainer } from "../src/compression/analyze.ts"; +import { TS_PACKET } from "../src/compression/ts-transform.ts"; + +/** Packets with a PCR every so often on PID 0x100, ticking at a known rate. */ +function timedTs(packets: number, kbps: number): Buffer { + const out = Buffer.alloc(packets * TS_PACKET); + const ticksPerPacket = (TS_PACKET * 8 * 27_000_000) / (kbps * 1000); + for (let i = 0; i < packets; i += 1) { + const at = i * TS_PACKET; + const pid = i % 4 === 3 ? 0x1fff : 0x100; + out[at] = 0x47; + out[at + 1] = (pid >> 8) & 0x1f; + out[at + 2] = pid & 0xff; + const pcrHere = pid === 0x100 && i % 8 === 0; + out[at + 3] = (pcrHere ? 0x30 : 0x10) | (i & 0xf); + let payloadAt = at + 4; + if (pcrHere) { + out[at + 4] = 7; + out[at + 5] = 0x10; + const ticks = Math.round(i * ticksPerPacket); + const base = Math.floor(ticks / 300); + const ext = ticks % 300; + out[at + 6] = Math.floor(base / 2 ** 25) & 0xff; + out[at + 7] = (base >> 17) & 0xff; + out[at + 8] = (base >> 9) & 0xff; + out[at + 9] = (base >> 1) & 0xff; + out[at + 10] = ((base & 1) << 7) | 0x7e | (ext >> 8); + out[at + 11] = ext & 0xff; + payloadAt = at + 12; + } + out.fill(pid === 0x1fff ? 0xff : 0x55, payloadAt, at + TS_PACKET); + } + return out; +} + +test("a container is told by its bytes, not its name", () => { + assert.equal(sniffContainer(Buffer.concat([Buffer.from([0, 0, 0, 24]), Buffer.from("ftypiso5"), Buffer.alloc(8)])), "fmp4"); + assert.equal(sniffContainer(Buffer.concat([Buffer.from([0, 0, 0, 24]), Buffer.from("ftypisom"), Buffer.alloc(8)])), "mp4"); + assert.equal(sniffContainer(Buffer.concat([Buffer.from([0, 0, 0, 8]), Buffer.from("moof"), Buffer.alloc(8)])), "fmp4"); + assert.equal(sniffContainer(timedTs(10, 2000)), "mpegts"); + assert.equal(sniffContainer(Buffer.from([0x1a, 0x45, 0xdf, 0xa3, 0, 0])), "webm"); + assert.equal(sniffContainer(Buffer.from("ID3\x04\x00")), "mp3"); + assert.equal(sniffContainer(Buffer.from([0xff, 0xfb, 0x90, 0x00])), "mp3"); + assert.equal(sniffContainer(randomBytes(4000).fill(0, 0, 12)), "unknown"); + assert.equal(sniffContainer(Buffer.alloc(0)), "unknown"); +}); + +test("the transport-stream report counts packets, padding, PIDs and the bitrate the PCR implies", () => { + const stream = timedTs(4000, 3000); + const report = analyzeTs(Buffer.concat([Buffer.from("xx"), stream, Buffer.from("tail")])); + assert.ok(report); + assert.equal(report.packetSize, 188); + assert.equal(report.offset, 2); + assert.equal(report.packets, 4000); + assert.equal(report.nullPackets, 1000); + assert.equal(report.nullShare, 0.25); + assert.equal(report.unsynced, 6); + assert.deepEqual(report.pids.map((p) => p.pid), [0x100, 0x1fff]); + assert.ok(report.pcrBitrateKbps && Math.abs(report.pcrBitrateKbps - 3000) < 30, `pcr says ${report.pcrBitrateKbps} kbps`); + assert.equal(analyzeTs(randomBytes(2000)), null); +}); + +test("the benchmark reports complete wire sizes with a checked round trip, and stored is the honest floor", async () => { + const stream = timedTs(2000, 3000); + const rows = await benchmark(stream, { blockBytes: 64 * 1024, zstdLevels: [1], tsAware: true }); + const by = Object.fromEntries(rows.map((r) => [r.mode, r])); + assert.ok(by["stored"] && by["gzip"] && by["zstd"] && by["ts-zstd"]); + for (const row of rows) assert.equal(row.roundTrip, true, row.mode); + assert.ok(by["stored"]!.wireBytes > stream.length, "the envelope costs something"); + assert.ok(by["stored"]!.savingsPercent < 0); + assert.ok(by["zstd"]!.wireBytes < by["stored"]!.wireBytes); + // On a synthetic stream this regular both codecs squeeze it to almost + // nothing, so which wins is noise; the core test shows the transform + // winning on packets with real-looking payloads. Here: it took part. + assert.ok(by["ts-zstd"]!.compressedBlocks > 0 && by["ts-zstd"]!.note === undefined, "ts-zstd was applied and round-tripped"); + assert.equal(by["zstd"]!.blocks, Math.ceil(stream.length / (64 * 1024))); + + const noise = await benchmark(randomBytes(100_000), { blockBytes: 32 * 1024, zstdLevels: [1] }); + const z = noise.find((r) => r.mode === "zstd")!; + assert.equal(z.compressedBlocks, 0, "nothing qualified"); + assert.equal(z.storedBlocks, 4); + assert.equal(z.wireBytes, noise.find((r) => r.mode === "stored")!.wireBytes, "stored blocks cost exactly what stored costs"); + const ts = await benchmark(randomBytes(10_000), { tsAware: true, zstdLevels: [] }); + assert.ok(ts.find((r) => r.mode === "ts-zstd")?.note, "ts-zstd on noise is reported as not applicable, not as a saving"); +}); + +test("an analysis names its boundary, its sample, and a recommendation that never claims a saving it did not measure", async () => { + const noise = randomBytes(50_000); + const a = await analyzeSample(noise, { boundary: "channel", source: { kind: "bytes", name: "noise" }, zstdLevels: [1], sampleMs: 2000 }); + assert.equal(a.boundary, "channel"); + assert.equal(a.container, "unknown"); + assert.equal(a.ts, null); + assert.equal(a.recommendation.mode, "stored"); + assert.equal(a.observedKbps, 200); + assert.equal(a.sha256.length, 64); + assert.ok(a.tools.runtime); + const text = Buffer.from("the same line again and again\n".repeat(3000)); + const b = await analyzeSample(text, { boundary: "source", source: { kind: "file", name: "lines.txt" }, zstdLevels: [1, 3] }); + assert.equal(b.recommendation.mode, "zstd"); + assert.ok(b.recommendation.reason.includes("saves")); + assert.equal(b.observedKbps, undefined, "no duration, no bitrate"); +}); diff --git a/test/compression-core.test.ts b/test/compression-core.test.ts new file mode 100644 index 0000000..ee52a55 --- /dev/null +++ b/test/compression-core.test.ts @@ -0,0 +1,295 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { randomBytes } from "node:crypto"; +import { + CEILING_FRAME_BYTES, + decodeFrameHeader, + decodeStreamHeader, + encodeDataFrame, + encodeEndFrame, + encodeStreamHeader, + endFrameTotal, + FRAME_END, + FRAME_HEADER_BYTES, + parseModes, + RelayError, + sha256, + STREAM_HEADER_BYTES, +} from "../src/compression/envelope.ts"; +import { decode, encode, Pool, PoolError } from "../src/compression/codec.ts"; +import { Blocker } from "../src/compression/blocks.ts"; +import { TS_PACKET, tsJoin, tsLayout, tsSplit } from "../src/compression/ts-transform.ts"; +import { eligible, normalizePolicy, defaultPolicy } from "../src/compression/policy.ts"; + +const ALL = new Set(["stored", "zstd", "gzip", "ts-zstd"] as const); + +test("the stream header is sixteen bytes, big-endian, and reads back", () => { + const bytes = encodeStreamHeader({ version: 1, boundary: "channel", generation: 7, maxFrameBytes: 262144 }); + assert.equal(bytes.length, STREAM_HEADER_BYTES); + assert.equal(bytes.toString("hex"), "4e58533101000100000000070004" + "0000"); + assert.deepEqual(decodeStreamHeader(bytes), { version: 1, boundary: "channel", generation: 7, maxFrameBytes: 262144 }); +}); + +test("a stream header from somebody else is refused for the right reason", () => { + const good = encodeStreamHeader({ version: 1, boundary: "source", generation: 1, maxFrameBytes: 1024 }); + const bad = (edit: (b: Buffer) => void): RelayError => { + const copy = Buffer.from(good); + edit(copy); + try { + decodeStreamHeader(copy); + } catch (error) { + return error as RelayError; + } + throw new Error("accepted"); + }; + assert.equal(bad((b) => b.write("NXS2", 0, "latin1")).code, "BAD_MAGIC"); + assert.equal(bad((b) => b.writeUInt8(2, 4)).code, "BAD_VERSION"); + assert.equal(bad((b) => b.writeUInt8(9, 6)).code, "BAD_BOUNDARY"); + assert.equal(bad((b) => b.writeUInt32BE(0, 12)).code, "BAD_LIMIT"); + assert.equal(bad((b) => b.writeUInt32BE(CEILING_FRAME_BYTES + 1, 12)).code, "BAD_LIMIT"); + assert.throws(() => decodeStreamHeader(good.subarray(0, 10)), (e: RelayError) => e.code === "TRUNCATED"); +}); + +test("a data frame carries its lengths and the digest of the original; the vector in the docs is this one", () => { + const original = Buffer.from("hello"); + const frame = encodeDataFrame(3, "stored", original, original); + assert.equal(frame.length, FRAME_HEADER_BYTES + 5); + assert.equal( + frame.toString("hex"), + "0100" + "0000" + "00000003" + "00000005" + "00000005" + sha256(original).toString("hex") + "68656c6c6f", + ); + const header = decodeFrameHeader(frame, { maxFrameBytes: 1024, modes: ALL, expectSeq: 3 }); + assert.equal(header.mode, "stored"); + assert.equal(header.originalLength, 5); + assert.ok(header.sha256.equals(sha256(original))); +}); + +test("the end frame vector in the docs: one frame of `hello`, then the end", () => { + const original = Buffer.from("hello"); + const frame = encodeEndFrame(1, 5, sha256(original)); + assert.equal(frame.toString("hex"), "0200" + "0000" + "00000001" + "00000000" + "00000005" + sha256(original).toString("hex")); +}); + +test("the end frame carries a 64-bit total and the stream digest", () => { + const digest = sha256(Buffer.from("everything")); + const total = 2 ** 32 + 12345; + const frame = encodeEndFrame(9, total, digest); + const header = decodeFrameHeader(frame, { maxFrameBytes: 1, modes: new Set(), expectSeq: 9 }); + assert.equal(header.type, FRAME_END); + assert.equal(endFrameTotal(header), total); + assert.ok(header.sha256.equals(digest)); +}); + +test("a frame header is checked before anything is allocated for it", () => { + const original = Buffer.alloc(100, 1); + const frame = encodeDataFrame(0, "zstd", original, Buffer.alloc(40)); + const limits = { maxFrameBytes: 100, modes: ALL, expectSeq: 0 }; + const refuse = (edit: (b: Buffer) => void, with_ = limits): string => { + const copy = Buffer.from(frame); + edit(copy); + try { + decodeFrameHeader(copy, with_); + } catch (error) { + return (error as RelayError).code; + } + return "accepted"; + }; + assert.equal(refuse(() => undefined), "accepted"); + assert.equal(refuse((b) => b.writeUInt8(7, 0)), "BAD_FRAME_TYPE"); + assert.equal(refuse((b) => b.writeUInt8(200, 1)), "BAD_MODE"); + assert.equal(refuse((b) => b.writeUInt8(2, 1), { ...limits, modes: new Set(["stored"] as const) }), "UNSUPPORTED_MODE"); + assert.equal(refuse((b) => b.writeUInt32BE(1, 4)), "BAD_SEQUENCE"); + assert.equal(refuse((b) => b.writeUInt32BE(101, 8)), "FRAME_TOO_LARGE"); + assert.equal(refuse((b) => b.writeUInt32BE(100 + 1024 + 1, 12)), "EXPANSION_BUDGET"); + assert.equal(refuse((b) => b.writeUInt8(0, 1)), "LENGTH_MISMATCH", "stored must carry exactly its bytes"); + assert.equal(refuse((b) => b.writeUInt32BE(0xffff_ffff, 8)), "FRAME_TOO_LARGE", "a gigabyte is refused on paper"); +}); + +test("codec names in a negotiation header are kept to the known ones", () => { + assert.deepEqual([...parseModes("zstd, Stored ,brotli,ts-zstd")], ["zstd", "stored", "ts-zstd"]); + assert.deepEqual([...parseModes(undefined)], []); +}); + +test("every codec round-trips, and a decode is capped at what its frame promised", async () => { + const text = Buffer.from("la ".repeat(50_000)); + for (const mode of ["stored", "zstd", "gzip"] as const) { + const encoded = await encode(mode, text, 3); + const back = await decode(mode, encoded, text.length); + assert.ok(back.equals(text), mode); + if (mode !== "stored") { + assert.ok(encoded.length < text.length / 10, `${mode} squeezed repeated text`); + await assert.rejects(decode(mode, encoded, 1000), (e: RelayError) => e.code === "FRAME_TOO_LARGE"); + } + } + await assert.rejects(decode("zstd", Buffer.from("not zstd at all"), 100), (e: RelayError) => e.code === "DECODE_FAILED"); +}); + +test("random bytes do not compress, and the eligibility rule says so", async () => { + const noise = randomBytes(64 * 1024); + const encoded = await encode("zstd", noise, 1); + assert.equal(eligible(noise.length, encoded.length, { minSavingsPercent: 3, minSavingsBytes: 512 }), false); + assert.equal(eligible(100_000, 96_000, { minSavingsPercent: 3, minSavingsBytes: 512 }), true); + assert.equal(eligible(1000, 900, { minSavingsPercent: 3, minSavingsBytes: 512 }), false, "10% but under 512 bytes"); + assert.equal(eligible(100_000, 99_000, { minSavingsPercent: 3, minSavingsBytes: 512 }), false, "1000 bytes but under 3%"); + assert.equal(eligible(0, 0, { minSavingsPercent: 0, minSavingsBytes: 0 }), false); +}); + +test("the pool bounds how many run, refuses when full, and times out a slow job", async () => { + const pool = new Pool({ concurrency: 2, maxQueued: 1, timeoutMs: 50 }); + let running = 0; + let peak = 0; + const slow = (ms: number) => async () => { + running += 1; + peak = Math.max(peak, running); + await new Promise((done) => setTimeout(done, ms)); + running -= 1; + return ms; + }; + const a = pool.run(slow(20)); + const b = pool.run(slow(20)); + const c = pool.run(slow(20)); // queued + await assert.rejects(pool.run(slow(1)), (e: PoolError) => e.code === "BUSY"); + assert.deepEqual(await Promise.all([a, b, c]), [20, 20, 20]); + assert.equal(peak, 2); + await assert.rejects(pool.run(slow(200)), (e: PoolError) => e.code === "TIMEOUT"); + await new Promise((done) => setTimeout(done, 220)); + assert.equal(pool.stats.running, 0); + assert.equal(pool.stats.refused, 1); + assert.equal(pool.stats.timedOut, 1); + const controller = new AbortController(); + controller.abort(); + await assert.rejects(pool.run(slow(1), { signal: controller.signal }), (e: PoolError) => e.code === "CANCELLED"); +}); + +test("a block is flushed when full or when its first byte has waited long enough", () => { + const blocks: Buffer[] = []; + let fire: (() => void) | null = null; + const blocker = new Blocker({ + maxBlockBytes: 10, + maxHoldMs: 100, + onBlock: (b) => blocks.push(Buffer.from(b)), + setTimer: (fn) => { + fire = fn; + return { clear: () => { fire = null; } }; + }, + }); + blocker.push(Buffer.from("abc")); + assert.equal(blocks.length, 0); + assert.ok(fire, "the clock started with the first byte"); + blocker.push(Buffer.from("defghijklmnop")); // 16 held: one full block, 6 left over + assert.deepEqual(blocks.map(String), ["abcdefghij"]); + assert.equal(blocker.pendingBytes, 6); + assert.ok(fire, "the remainder started a new clock"); + (fire as unknown as () => void)(); + assert.deepEqual(blocks.map(String), ["abcdefghij", "klmnop"]); + assert.equal(blocker.pendingBytes, 0); + fire = null; + blocker.push(Buffer.alloc(35, 0x41)); + assert.equal(blocks.length, 5, "a chunk bigger than a block is sliced"); + assert.equal(blocker.pendingBytes, 5); + blocker.end(); + assert.equal(blocks.length, 6); + blocker.push(Buffer.from("late")); + assert.equal(blocks.length, 6, "nothing after end"); +}); + +test("an aligned blocker cuts a full block at the packet boundary and keeps the rest", () => { + const blocks: Buffer[] = []; + const blocker = new Blocker({ maxBlockBytes: 1000, maxHoldMs: 100, align: 188, onBlock: (b) => blocks.push(b), setTimer: () => ({ clear: () => undefined }) }); + blocker.push(Buffer.alloc(1000, 0x47)); + assert.equal(blocks[0]?.length, 940, "five whole packets"); + assert.equal(blocker.pendingBytes, 60); +}); + +/** A plausible transport stream: N packets on a few PIDs with adaptation fields here and there. */ +function fakeTs(packets: number, seed = 1): Buffer { + const out = Buffer.alloc(packets * TS_PACKET); + let x = seed; + const rnd = (): number => { + x = (x * 1103515245 + 12345) & 0x7fffffff; + return x; + }; + for (let i = 0; i < packets; i += 1) { + const at = i * TS_PACKET; + const pid = i % 7 === 0 ? 0x1fff : [0x100, 0x101, 0x102][i % 3] as number; + out[at] = 0x47; + out[at + 1] = (pid >> 8) & 0x1f; + out[at + 2] = pid & 0xff; + const withAf = i % 5 === 0; + out[at + 3] = (withAf ? 0x30 : 0x10) | (i & 0xf); + let payloadAt = at + 4; + if (withAf) { + const len = 7 + (rnd() % 20); + out[at + 4] = len; + out[at + 5] = 0x10; // PCR flag + payloadAt = at + 5 + len; + } + for (let j = payloadAt; j < at + TS_PACKET; j += 1) out[j] = pid === 0x1fff ? 0xff : rnd() & 0xff; + } + return out; +} + +test("the transport-stream transform is exactly reversible, ragged edges and all", () => { + const stream = fakeTs(300); + for (const [head, tail] of [[0, 0], [17, 0], [0, 100], [50, 187], [187, 1]] as const) { + const sample = Buffer.concat([randomBytes(head), stream, randomBytes(tail)]); + const split = tsSplit(sample); + assert.ok(split, `head ${head} tail ${tail}`); + assert.ok(tsJoin(split).equals(sample), `head ${head} tail ${tail} joins back`); + } + assert.equal(tsSplit(randomBytes(5000)), null, "noise is not a transport stream"); + assert.equal(tsSplit(Buffer.alloc(0)), null); + assert.equal(tsJoin(Buffer.from("junk")), null); +}); + +test("a packet with an adaptation length that overruns is kept whole, and a stream that loses sync keeps every byte", () => { + const stream = fakeTs(40); + stream[5 * TS_PACKET + 3] = 0x30; + stream[5 * TS_PACKET + 4] = 250; // longer than a packet + const broken = Buffer.concat([stream.subarray(0, 20 * TS_PACKET), Buffer.from("garbage in the middle"), stream.subarray(20 * TS_PACKET)]); + const split = tsSplit(broken); + assert.ok(split); + assert.ok(tsJoin(split).equals(broken)); +}); + +test("192- and 204-byte layouts are recognised and left alone", () => { + const packets = fakeTs(20); + const timestamped = Buffer.alloc(20 * 192); + for (let i = 0; i < 20; i += 1) packets.copy(timestamped, i * 192 + 4, i * TS_PACKET, (i + 1) * TS_PACKET); + assert.deepEqual(tsLayout(timestamped), { packetSize: 192, offset: 0 }); + assert.equal(tsSplit(timestamped), null); + const fec = Buffer.alloc(20 * 204); + for (let i = 0; i < 20; i += 1) packets.copy(fec, i * 204, i * TS_PACKET, (i + 1) * TS_PACKET); + assert.deepEqual(tsLayout(fec), { packetSize: 204, offset: 0 }); + assert.equal(tsSplit(fec), null); +}); + +test("ts-zstd beats plain zstd on a padded transport stream, and both restore it", async () => { + const stream = fakeTs(1400); + const plain = await encode("zstd", stream, 1); + const aware = await encode("ts-zstd", stream, 1); + assert.ok(aware.length < plain.length, `ts-zstd ${aware.length} < zstd ${plain.length}`); + assert.ok((await decode("ts-zstd", aware, stream.length)).equals(stream)); + await assert.rejects(encode("ts-zstd", randomBytes(3000), 1), (e: RelayError) => e.code === "BAD_MODE"); +}); + +test("a policy change is checked field by field and never silently drops a typo", () => { + const base = defaultPolicy(); + const ok = normalizePolicy({ losslessCompression: { mode: "auto", zstdLevel: 3 } }, base); + assert.ok(ok.ok); + if (ok.ok) { + assert.equal(ok.policy.losslessCompression.mode, "auto"); + assert.equal(ok.policy.losslessCompression.zstdLevel, 3); + assert.equal(ok.policy.hlsPackaging, "mpegts", "untouched"); + assert.equal(base.losslessCompression.mode, "off", "the base was not edited in place"); + } + const bad = normalizePolicy({ losslessCompression: { mode: "brotli", zstdLevel: 99, typo: 1 }, qualityProfile: "low", extra: true }, base); + assert.equal(bad.ok, false); + if (!bad.ok) { + assert.equal(bad.errors.length, 5, bad.errors.join("; ")); + assert.ok(bad.errors.some((e) => e.includes("typo"))); + } + assert.equal(normalizePolicy("auto", base).ok, false); + const contradictory = normalizePolicy({ losslessCompression: { minSavingsBytes: 262144 } }, base); + assert.equal(contradictory.ok, false); +}); diff --git a/test/compression-hls-fmp4.test.ts b/test/compression-hls-fmp4.test.ts new file mode 100644 index 0000000..dc94a46 --- /dev/null +++ b/test/compression-hls-fmp4.test.ts @@ -0,0 +1,73 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { existsSync, readFileSync } from "node:fs"; +import { HlsPackagers } from "../src/hls.ts"; +import { Channels } from "../src/channels.ts"; +import { detectTools } from "../src/audio.ts"; + +/** A real ffmpeg to package with, found the way the server finds it, or none. */ +const FFMPEG = detectTools().ffmpeg; +const ffmpegHere = ((): boolean => { + const [cmd, ...rest] = FFMPEG; + if (!cmd) return false; + const r = spawnSync(cmd, [...rest, "-version"], { encoding: "utf8", timeout: 10_000 }); + return !r.error && r.status === 0; +})(); + +test("a live channel becomes fMP4 HLS: an init segment named for this run, .m4s media, the key on the map line", { skip: !ffmpegHere }, async () => { + const channels = new Channels({ ffmpeg: FFMPEG }); + const channel = channels.pull( + "test", "A test pattern", "testsrc=size=320x240:rate=25", + [ + "-c:v", "libx264", "-preset", "ultrafast", "-tune", "zerolatency", "-g", "25", "-pix_fmt", "yuv420p", "-an", + "-f", "mp4", "-movflags", "frag_keyframe+empty_moov+default_base_moof", "-frag_duration", "1000000", + ], + "video", + true, + 30_000, + ["-f", "lavfi", "-re"], + ); + assert.ok(channel); + + const said: string[] = []; + const hls = new HlsPackagers({ + ffmpeg: FFMPEG, + listen: (id, listener) => channels.listen(id, listener), + onEvent: (m) => said.push(m), + firstPlaylistMs: 40_000, + packaging: () => "fmp4", + }); + try { + const playlist = await hls.playlist("test"); + assert.ok(playlist, `no playlist: ${said.join(" | ")}`); + assert.match(playlist ?? "", /#EXT-X-MAP:URI="init-[0-9a-f]{8}\.mp4"/, "an initialisation segment, named for this run"); + const init = /URI="(init-[0-9a-f]{8}\.mp4)"/.exec(playlist ?? "")?.[1] ?? ""; + const media = (playlist ?? "").split("\n").find((line) => /^seg\d{5}\.m4s$/.test(line)) ?? ""; + assert.match(media, /^seg\d{5}\.m4s$/); + const initPath = hls.segment("test", init); + assert.ok(existsSync(initPath), "the init segment is a file on disk"); + assert.equal(readFileSync(initPath).toString("latin1", 4, 8), "ftyp", "and it begins with the file type box"); + const mediaPath = hls.segment("test", media); + assert.ok(existsSync(mediaPath)); + const box = readFileSync(mediaPath); + // A media segment is styp/moof/mdat boxes: no ftyp, no moov, those live in the init. + const types = new Set(); + for (let at = 0; at + 8 <= box.length;) { + const size = box.readUInt32BE(at); + types.add(box.toString("latin1", at + 4, at + 8)); + if (size < 8) break; + at += size; + } + assert.ok(types.has("moof") && types.has("mdat"), `segment boxes: ${[...types].join(",")}`); + assert.ok(!types.has("moov"), "the movie header is in the init, not repeated in every segment"); + assert.equal(hls.segment("test", "init-00000000.mp4"), "", "another run's init is not here"); + const report = hls.report("test"); + assert.equal(report?.packaging, "fmp4"); + assert.equal(report?.initialised, true); + assert.ok(report && report.longestSegmentSeconds > 0); + } finally { + hls.stopAll(); + channels.stopAll(); + } +}); diff --git a/test/compression-relay.test.ts b/test/compression-relay.test.ts new file mode 100644 index 0000000..fb61114 --- /dev/null +++ b/test/compression-relay.test.ts @@ -0,0 +1,220 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { randomBytes } from "node:crypto"; +import { Pool } from "../src/compression/codec.ts"; +import { encodeDataFrame, encodeEndFrame, encodeStreamHeader, RelayError, sha256 } from "../src/compression/envelope.ts"; +import { ChannelMetrics } from "../src/compression/metrics.ts"; +import { DEFAULT_LOSSLESS, type LosslessPolicy } from "../src/compression/policy.ts"; +import { GIVE_UP_AFTER, RelayDecoder, RelayEncoder, type RelayListener } from "../src/compression/relay.ts"; + +const MODES = new Set(["stored", "zstd", "ts-zstd"] as const); + +/** A listener that keeps what it was sent, and can pretend to be behind. */ +class Sink implements RelayListener { + chunks: Buffer[] = []; + ended = false; + behind = 0; + write(chunk: Buffer): boolean { + this.chunks.push(Buffer.from(chunk)); + return true; + } + end(): void { + this.ended = true; + } + pending(): number { + return this.behind; + } + get all(): Buffer { + return Buffer.concat(this.chunks); + } +} + +function policy(over: Partial = {}): LosslessPolicy { + return { ...DEFAULT_LOSSLESS, mode: "zstd", maxBlockBytes: 4096, maxHoldMs: 5, ...over }; +} + +/** Run the whole envelope through a decoder and hand back the original bytes. */ +async function decodeAll(bytes: Buffer, chunk = 1000): Promise<{ out: Buffer; decoder: RelayDecoder }> { + const pieces: Buffer[] = []; + const decoder = new RelayDecoder({ modes: MODES, onBytes: (b) => { pieces.push(b); } }); + for (let at = 0; at < bytes.length; at += chunk) await decoder.feed(bytes.subarray(at, at + chunk)); + await decoder.end(); + return { out: Buffer.concat(pieces), decoder }; +} + +const tick = (): Promise => new Promise((done) => setTimeout(done, 25)); + +test("what goes into the encoder comes out of the decoder, byte for byte, with a clean end", async () => { + const metrics = new ChannelMetrics(); + const encoder = new RelayEncoder({ generation: 3, policy: policy(), pool: new Pool(), metrics, boundary: "channel", onAbort: () => undefined, onIdle: () => undefined }); + const sink = new Sink(); + encoder.join(sink, [Buffer.from("opening bytes ".repeat(40))]); + encoder.attach(); + const live = Buffer.concat([Buffer.from("tick tock ".repeat(2000)), randomBytes(9000)]); + for (let at = 0; at < live.length; at += 1500) encoder.write(live.subarray(at, at + 1500)); + await tick(); + encoder.end(); + await tick(); + assert.ok(sink.ended); + const { out, decoder } = await decodeAll(sink.all); + assert.ok(out.equals(Buffer.concat([Buffer.from("opening bytes ".repeat(40)), live]))); + assert.ok(decoder.ended); + assert.equal(decoder.header?.generation, 3); + assert.ok(metrics.compressedBlocks > 0, "the text compressed"); + assert.ok(metrics.storedBlocks > 0, "the noise was stored"); + assert.equal(metrics.inputBytes, live.length); + assert.ok(metrics.wireBytes < metrics.inputBytes + 560, `wire ${metrics.wireBytes} against input ${metrics.inputBytes}`); + assert.ok(metrics.snapshot().latencyMs.p95 < 500); +}); + +test("one encoder serves every session: a late joiner gets its preface and then the shared blocks, nothing twice", async () => { + const metrics = new ChannelMetrics(); + const encoder = new RelayEncoder({ generation: 1, policy: policy(), pool: new Pool(), metrics, boundary: "channel", onAbort: () => undefined, onIdle: () => undefined }); + const first = new Sink(); + encoder.join(first, []); + encoder.attach(); + const part1 = Buffer.from("first part ".repeat(1000)); + encoder.write(part1); + await tick(); + // The channel has remembered part1 as its backlog; a newcomer is handed it. + const second = new Sink(); + encoder.join(second, [part1]); + const part2 = Buffer.from("second part ".repeat(1000)); + encoder.write(part2); + await tick(); + encoder.end(); + await tick(); + assert.equal(encoder.sessionCount, 0); + const a = await decodeAll(first.all); + const b = await decodeAll(second.all); + assert.ok(a.out.equals(Buffer.concat([part1, part2]))); + assert.ok(b.out.equals(Buffer.concat([part1, part2])), "the late joiner saw part1 exactly once, then part2"); + assert.equal(metrics.blocks, Math.ceil(part1.length / 4096) + Math.ceil(part2.length / 4096), "shared blocks were encoded once, not per session"); +}); + +test("auto gives up on noise after a run of stored blocks and comes back after the cooldown", async () => { + const metrics = new ChannelMetrics(); + let now = 0; + const pool = new Pool(); + const encoder = new RelayEncoder({ + generation: 1, policy: policy({ mode: "auto", resampleAfterMs: 1000 }), pool, metrics, boundary: "channel", + onAbort: () => undefined, onIdle: () => undefined, now: () => now, + }); + const sink = new Sink(); + encoder.join(sink, []); + encoder.attach(); + for (let i = 0; i < GIVE_UP_AFTER + 4; i += 1) encoder.write(randomBytes(4096)); + await tick(); + assert.equal(metrics.bypassed, true); + assert.equal(metrics.fallbackReason, "already efficiently compressed"); + assert.equal(metrics.storedBlocks, GIVE_UP_AFTER + 4); + assert.equal(pool.stats.completed, GIVE_UP_AFTER, "the blocks after giving up never went to the pool"); + const text = Buffer.from("compressible ".repeat(300)); // one block + encoder.write(text); + await tick(); + assert.equal(metrics.compressedBlocks, 0, "still bypassed: not even tried"); + assert.equal(pool.stats.completed, GIVE_UP_AFTER); + now = 2000; + encoder.write(text); + await tick(); + assert.equal(metrics.bypassed, false, "sampled again after the cooldown"); + assert.equal(metrics.compressedBlocks, 1); + assert.equal(pool.stats.completed, GIVE_UP_AFTER + 1); + encoder.end(); + await tick(); + const { out } = await decodeAll(sink.all); + assert.equal(out.length, (GIVE_UP_AFTER + 4) * 4096 + text.length * 2, "stored is still every byte"); +}); + +test("a listener that stops draining is cut off; the others carry on", async () => { + const metrics = new ChannelMetrics(); + const encoder = new RelayEncoder({ generation: 1, policy: policy({ maxListenerQueueBytes: 16 * 1024 }), pool: new Pool(), metrics, boundary: "channel", onAbort: () => undefined, onIdle: () => undefined }); + const healthy = new Sink(); + const stuck = new Sink(); + encoder.join(healthy, []); + encoder.join(stuck, []); + encoder.attach(); + stuck.behind = 20 * 1024; + encoder.write(Buffer.from("x".repeat(5000))); + await tick(); + assert.ok(stuck.ended, "cut off"); + assert.equal(healthy.ended, false); + assert.equal(metrics.droppedListeners, 1); + assert.equal(metrics.fallbackReason, "slow listener"); + encoder.write(Buffer.from("y".repeat(5000))); + await tick(); + encoder.end(); + await tick(); + const { out } = await decodeAll(healthy.all); + assert.equal(out.length, 10000); + await assert.rejects(decodeAll(stuck.all), (e: RelayError) => e.code === "TRUNCATED", "the cut-off one has no end marker"); +}); + +test("a compressor that falls too far behind ends its relays rather than growing", async () => { + const metrics = new ChannelMetrics(); + let aborted: string | null = null; + // A pool that never finishes: every block queues behind it. + const stuckPool = new Pool({ concurrency: 1, maxQueued: 1000, timeoutMs: 60_000 }); + void stuckPool.run(() => new Promise(() => undefined), { timeoutMs: 60_000 }).catch(() => undefined); + const encoder = new RelayEncoder({ + generation: 1, policy: policy({ maxChannelQueueBytes: 64 * 1024 }), pool: stuckPool, metrics, boundary: "channel", + onAbort: (reason) => { aborted = reason; }, onIdle: () => undefined, + }); + const sink = new Sink(); + encoder.join(sink, []); + encoder.attach(); + for (let i = 0; i < 20; i += 1) encoder.write(Buffer.alloc(4096, 1)); + await tick(); + assert.equal(aborted, "processing budget exceeded"); + assert.ok(sink.ended); + assert.equal(metrics.fallbackReason, "processing budget exceeded"); +}); + +test("off means stored frames: the envelope still works, nothing is compressed", async () => { + const metrics = new ChannelMetrics(); + const encoder = new RelayEncoder({ generation: 1, policy: policy({ mode: "off" }), pool: new Pool(), metrics, boundary: "channel", onAbort: () => undefined, onIdle: () => undefined }); + const sink = new Sink(); + encoder.join(sink, []); + encoder.attach(); + encoder.write(Buffer.from("a".repeat(10000))); + await tick(); + encoder.end(); + await tick(); + assert.equal(metrics.compressedBlocks, 0); + const { out } = await decodeAll(sink.all); + assert.equal(out.length, 10000); +}); + +test("the decoder refuses corruption of every kind, and says which", async () => { + const original = Buffer.from("payload ".repeat(500)); + const header = encodeStreamHeader({ version: 1, boundary: "channel", generation: 1, maxFrameBytes: 8192 }); + const frame = encodeDataFrame(0, "stored", original, original); + const end = encodeEndFrame(1, original.length, sha256(original)); + const good = Buffer.concat([header, frame, end]); + const { out } = await decodeAll(good, 7); + assert.ok(out.equals(original), "fed seven bytes at a time it still works"); + + const expect = async (bytes: Buffer, code: string, cap?: number): Promise => { + const decoder = new RelayDecoder({ modes: MODES, onBytes: () => undefined, ...(cap ? { maxFrameBytes: cap } : {}) }); + await assert.rejects(decoder.feed(bytes).then(() => decoder.end()), (e: RelayError) => e.code === code, code); + }; + await expect(good.subarray(0, good.length - 10), "TRUNCATED"); + await expect(Buffer.concat([good, Buffer.from("more")]), "AFTER_END"); + await expect(good, "BAD_LIMIT", 1024); + const flipped = Buffer.from(good); + flipped[16 + 48 + 10] ^= 0xff; + await expect(flipped, "CHECKSUM_MISMATCH"); + const wrongTotal = Buffer.concat([header, frame, encodeEndFrame(1, original.length - 1, sha256(original))]); + await expect(wrongTotal, "LENGTH_MISMATCH"); + const wrongDigest = Buffer.concat([header, frame, encodeEndFrame(1, original.length, sha256(Buffer.from("other")))]); + await expect(wrongDigest, "CHECKSUM_MISMATCH"); + const outOfOrder = Buffer.concat([header, encodeDataFrame(5, "stored", original, original)]); + await expect(outOfOrder, "BAD_SEQUENCE"); + const lying = Buffer.concat([header, encodeDataFrame(0, "zstd", Buffer.alloc(100), Buffer.from("not zstd"))]); + await expect(lying, "DECODE_FAILED"); + // A frame whose zstd payload is real but decodes to something other than it promised. + const { encode } = await import("../src/compression/codec.ts"); + const zbig = await encode("zstd", Buffer.alloc(5000, 7), 1); + const short = Buffer.concat([header, encodeDataFrame(0, "zstd", Buffer.alloc(100), zbig)]); + await expect(short, "FRAME_TOO_LARGE"); +}); diff --git a/test/compression-routes.test.ts b/test/compression-routes.test.ts new file mode 100644 index 0000000..22b8f26 --- /dev/null +++ b/test/compression-routes.test.ts @@ -0,0 +1,256 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { randomBytes } from "node:crypto"; +import type { AddressInfo } from "node:net"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { createServer, EmptyEngine } from "../src/server.ts"; +import { Channels } from "../src/channels.ts"; +import { CompressionService, STATIC_CHANNEL } from "../src/compression/service.ts"; +import { MEDIA_TYPE } from "../src/compression/envelope.ts"; +import { receiveRelay, RelayRefused } from "../src/compression/receiver.ts"; +import { RelayDecoder } from "../src/compression/relay.ts"; +import type { ChannelStatus } from "../src/compression/service.ts"; + +const CONTROL = "control-key"; +const LISTEN = "listen-key"; + +interface Started { + base: string; + channels: Channels; + compression: CompressionService; + stop: () => Promise; +} + +async function start(cacheDir: string | null = null): Promise { + const channels = new Channels({ ffmpeg: ["ffmpeg"] }); + const compression = new CompressionService({ channels, stateDir: null, port: 1, cacheDir }); + const server = createServer(new EmptyEngine(), { web: null, media: true, version: "test", key: CONTROL, listenKey: LISTEN, channels, compression }); + await new Promise((done) => server.listen(0, "127.0.0.1", done)); + const { port } = server.address() as AddressInfo; + return { + base: `http://127.0.0.1:${port}`, + channels, + compression, + stop: async () => { + compression.stopAll(); + channels.stopAll(); + await new Promise((done) => server.close(() => done())); + }, + }; +} + +const withKey = (key: string, more: Record = {}): Record => ({ "x-nixamp-key": key, ...more }); + +test("the policy routes: read with the listen key, change with the controls and the version you saw", async () => { + const s = await start(); + try { + const seen = await fetch(`${s.base}/api/channels/main/compression`, { headers: withKey(LISTEN) }); + assert.equal(seen.status, 200); + const status = (await seen.json()) as ChannelStatus; + assert.equal(status.configured.losslessCompression.mode, "off"); + assert.equal(status.live, false); + + const refused = await fetch(`${s.base}/api/channels/main/compression`, { + method: "PATCH", headers: withKey(LISTEN, { "content-type": "application/json" }), body: JSON.stringify({ losslessCompression: { mode: "auto" } }), + }); + assert.equal(refused.status, 403, "the listen key cannot change a policy"); + + const changed = await fetch(`${s.base}/api/channels/main/compression`, { + method: "PATCH", headers: withKey(CONTROL, { "content-type": "application/json", "if-match": '"0"' }), body: JSON.stringify({ losslessCompression: { mode: "auto" }, hlsPackaging: "fmp4" }), + }); + assert.equal(changed.status, 200); + const after = (await changed.json()) as ChannelStatus; + assert.equal(after.configured.losslessCompression.mode, "auto"); + assert.equal(after.configured.hlsPackaging, "fmp4"); + assert.equal(after.configured.version, 1); + assert.equal(after.configured.qualityProfile, "source", "turning compression on did not touch the quality"); + + const stale = await fetch(`${s.base}/api/channels/main/compression`, { + method: "PATCH", headers: withKey(CONTROL, { "content-type": "application/json", "if-match": '"0"' }), body: JSON.stringify({ losslessCompression: { mode: "zstd" } }), + }); + assert.equal(stale.status, 412); + const bad = await fetch(`${s.base}/api/channels/main/compression`, { + method: "PATCH", headers: withKey(CONTROL, { "content-type": "application/json" }), body: JSON.stringify({ qualityProfile: "low" }), + }); + assert.equal(bad.status, 400); + assert.ok(((await bad.json()) as { error: string }).error.includes("qualityProfile")); + + const overview = await fetch(`${s.base}/api/compression`, { headers: withKey(LISTEN) }); + assert.equal(overview.status, 200); + const all = (await overview.json()) as { global: { enabled: boolean }; channels: ChannelStatus[] }; + assert.equal(all.global.enabled, true); + assert.deepEqual(all.channels.map((c) => c.channel), ["main"]); + + const off = await fetch(`${s.base}/api/compression`, { method: "PATCH", headers: withKey(CONTROL, { "content-type": "application/json" }), body: JSON.stringify({ enabled: false }) }); + assert.equal(off.status, 200); + const now = (await (await fetch(`${s.base}/api/channels/main/compression`, { headers: withKey(LISTEN) })).json()) as ChannelStatus; + assert.equal(now.effective.losslessCompression.mode, "off"); + assert.ok(now.effective.reason?.includes("whole server")); + } finally { + await s.stop(); + } +}); + +test("a relay is negotiated by media type and codec list, and the ordinary channel URL is untouched", async () => { + const s = await start(); + try { + const channel = s.channels.attach("live", "a device", "mp3", "http")!; + const text = Buffer.from("the same frame again ".repeat(3000)); + channel.feed(text); + + const browser = await fetch(`${s.base}/api/channels/live/relay`, { headers: withKey(LISTEN) }); + assert.equal(browser.status, 406, "a player that followed the link is told where to go"); + assert.equal(((await browser.json()) as { playback: string }).playback, "/api/channels/live"); + + const off = await fetch(`${s.base}/api/channels/live/relay`, { headers: withKey(LISTEN, { accept: MEDIA_TYPE, "x-nixamp-stream-codecs": "zstd" }) }); + assert.equal(off.status, 409); + assert.equal(((await off.json()) as { code: string }).code, "COMPRESSION_OFF"); + + s.compression.set("live", { losslessCompression: { mode: "zstd", maxBlockBytes: 8192, maxHoldMs: 10 } }); + const ranged = await fetch(`${s.base}/api/channels/live/relay`, { headers: withKey(LISTEN, { accept: MEDIA_TYPE, "x-nixamp-stream-codecs": "zstd", range: "bytes=0-100" }) }); + assert.equal(ranged.status, 416, "never the wrong range, always a refusal"); + const deaf = await fetch(`${s.base}/api/channels/live/relay`, { headers: withKey(LISTEN, { accept: MEDIA_TYPE, "x-nixamp-stream-codecs": "brotli" }) }); + assert.equal(deaf.status, 406); + + // A real receiver: the backlog it is handed, then the live bytes, then the clean end. + const pieces: Buffer[] = []; + let accepted: { codecs: string; kind: string } | null = null; + const done = receiveRelay({ + url: `${s.base}/api/channels/live/relay`, + key: LISTEN, + onStart: (info) => { accepted = info; }, + onBytes: (b) => { pieces.push(b); }, + }); + await new Promise((tick) => setTimeout(tick, 100)); + const live = Buffer.concat([text, randomBytes(20_000)]); + channel.feed(live); + await new Promise((tick) => setTimeout(tick, 100)); + assert.equal(s.compression.status("live").relay?.sessions, 1); + channel.close(); + const result = await done; + assert.ok(accepted, "told what it would get before any byte"); + assert.equal(accepted!.codecs, "stored,zstd"); + assert.equal(result.bytes, live.length); + const got = Buffer.concat(pieces); + // A published channel keeps no backlog (only a pulled one does), so the + // preface is empty and the receiver gets exactly the live bytes. + assert.ok(got.equals(live), "the live bytes arrived, whole, and nothing else"); + const metrics = s.compression.status("live").metrics!; + assert.ok(metrics.compressedBlocks > 0, "the text compressed"); + assert.ok(metrics.storedBlocks > 0, "the noise was stored"); + } finally { + await s.stop(); + } +}); + +test("one nixamp pulls another's channel in, and a listener here hears the original bytes", async () => { + const upstream = await start(); + const downstream = await start(); + try { + const source = upstream.channels.attach("radio", "a device", "mp3", "http")!; + upstream.compression.set("radio", { losslessCompression: { mode: "zstd", maxBlockBytes: 8192, maxHoldMs: 10 } }); + source.feed(Buffer.from("intro ".repeat(2000))); + + const refused = await fetch(`${downstream.base}/api/channels/radio/relay`, { + method: "POST", headers: withKey(LISTEN, { "content-type": "application/json" }), body: JSON.stringify({ from: `${upstream.base}/api/channels/radio/relay`, key: LISTEN }), + }); + assert.equal(refused.status, 403); + const started = await fetch(`${downstream.base}/api/channels/radio/relay`, { + method: "POST", headers: withKey(CONTROL, { "content-type": "application/json" }), body: JSON.stringify({ from: `${upstream.base}/api/channels/radio/relay`, key: LISTEN, name: "Radio, relayed" }), + }); + assert.equal(started.status, 202); + await new Promise((tick) => setTimeout(tick, 200)); + assert.ok(downstream.channels.has("radio"), "the channel exists here now"); + const listed = (await (await fetch(`${downstream.base}/api/channels`, { headers: withKey(LISTEN) })).json()) as { channels: { id: string; via: string; source?: string }[] }; + assert.equal(listed.channels[0]?.via, "relay"); + assert.equal(listed.channels[0]?.source, undefined, "where it comes from is not for listeners"); + + // Somebody listening here, the ordinary way. + const heard: Buffer[] = []; + const listening = fetch(`${downstream.base}/api/channels/radio`, { headers: withKey(LISTEN) }).then(async (response) => { + assert.equal(response.status, 200); + assert.equal(response.headers.get("content-type"), "audio/mpeg"); + const reader = response.body!.getReader(); + for (;;) { + const { value, done } = await reader.read(); + if (done) break; + heard.push(Buffer.from(value)); + } + }); + await new Promise((tick) => setTimeout(tick, 100)); + const live = randomBytes(30_000); + source.feed(live); + await new Promise((tick) => setTimeout(tick, 200)); + const status = downstream.compression.status("radio"); + assert.equal(status.incoming?.from, `${upstream.base}/api/channels/radio/relay`); + assert.equal(status.incoming?.error, null); + + const stopped = await fetch(`${downstream.base}/api/channels/radio`, { method: "DELETE", headers: withKey(CONTROL) }); + assert.equal(stopped.status, 200); + await listening; + const all = Buffer.concat(heard); + assert.ok(all.subarray(all.length - live.length).equals(live), "the listener here got the live bytes byte for byte"); + assert.equal(downstream.channels.has("radio"), false); + assert.equal(downstream.compression.status("radio").incoming, null, "and the relay stopped dialling"); + } finally { + await downstream.stop(); + await upstream.stop(); + } +}); + +test("a library file's relay representation is built in the background, then served whole and exact", async () => { + const dir = mkdtempSync(join(tmpdir(), "nixamp-static-route-")); + const s = await start(join(dir, "cache")); + try { + const file = join(dir, "song.mp3"); + const content = Buffer.concat([Buffer.from("ID3"), Buffer.from("verse ".repeat(20_000)), randomBytes(30_000)]); + writeFileSync(file, content); + // The engine is empty; a track path comes from the library. Stand one in. + const engine = new EmptyEngine(); + engine.trackPath = (index: number) => (index === 0 ? file : undefined); + const channels = new Channels({ ffmpeg: ["ffmpeg"] }); + const compression = new CompressionService({ channels, stateDir: null, port: 2, cacheDir: join(dir, "cache2") }); + const server = createServer(engine, { web: null, media: true, version: "test", key: CONTROL, listenKey: LISTEN, channels, compression }); + await new Promise((done) => server.listen(0, "127.0.0.1", done)); + const base = `http://127.0.0.1:${(server.address() as AddressInfo).port}`; + try { + const off = await fetch(`${base}/api/media/0/relay`, { headers: withKey(LISTEN, { accept: MEDIA_TYPE }) }); + assert.equal(off.status, 409, "off until the static policy says otherwise"); + compression.set(STATIC_CHANNEL, { losslessCompression: { mode: "zstd" } }); + const first = await fetch(`${base}/api/media/0/relay`, { headers: withKey(LISTEN, { accept: MEDIA_TYPE }) }); + assert.equal(first.status, 202, "building; come back"); + let ready: Response | null = null; + for (let i = 0; i < 50 && !ready; i += 1) { + await new Promise((tick) => setTimeout(tick, 50)); + const poll = await fetch(`${base}/api/media/0/relay`, { headers: withKey(LISTEN, { accept: MEDIA_TYPE }) }); + if (poll.status === 200) ready = poll; + } + assert.ok(ready, "built"); + assert.ok(ready!.headers.get("content-type")?.startsWith(MEDIA_TYPE)); + assert.equal(ready!.headers.get("x-nixamp-original-length"), String(content.length)); + const body = Buffer.from(await ready!.arrayBuffer()); + assert.equal(body.length, Number(ready!.headers.get("content-length"))); + assert.ok(body.length < content.length, "the verse compressed"); + const pieces: Buffer[] = []; + const decoder = new RelayDecoder({ modes: new Set(["stored", "zstd", "ts-zstd"] as const), onBytes: (b) => { pieces.push(b); } }); + await decoder.feed(body); + await decoder.end(); + assert.ok(Buffer.concat(pieces).equals(content)); + const ranged = await fetch(`${base}/api/media/0/relay`, { headers: withKey(LISTEN, { accept: MEDIA_TYPE, range: "bytes=0-10" }) }); + assert.equal(ranged.status, 416); + const player = await fetch(`${base}/api/media/0/relay`, { headers: withKey(LISTEN) }); + assert.equal(player.status, 406); + const missing = await fetch(`${base}/api/media/7/relay`, { headers: withKey(LISTEN, { accept: MEDIA_TYPE }) }); + assert.equal(missing.status, 404); + await assert.rejects(receiveRelay({ url: `${base}/api/media/7/relay`, key: LISTEN, onBytes: () => undefined }), (e: RelayRefused) => e.status === 404); + } finally { + compression.stopAll(); + await new Promise((done) => server.close(() => done())); + } + } finally { + await s.stop(); + rmSync(dir, { recursive: true, force: true }); + } +}); diff --git a/test/compression-service.test.ts b/test/compression-service.test.ts new file mode 100644 index 0000000..bb01995 --- /dev/null +++ b/test/compression-service.test.ts @@ -0,0 +1,213 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { randomBytes } from "node:crypto"; +import { mkdtempSync, readFileSync, rmSync, utimesSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { PolicyStore } from "../src/compression/store.ts"; +import { AnalysisJobs } from "../src/compression/jobs.ts"; +import { StaticCache } from "../src/compression/static.ts"; +import { RelayDecoder } from "../src/compression/relay.ts"; +import { DEFAULT_LOSSLESS, variantOf } from "../src/compression/policy.ts"; +import { packagerArgs, playlistReport, segmentName, segmentType, withKey } from "../src/hls.ts"; +import { Channels } from "../src/channels.ts"; +import { CompressionService, STATIC_CHANNEL } from "../src/compression/service.ts"; +import type { Analysis } from "../src/compression/analyze.ts"; + +const scratch = (): string => mkdtempSync(join(tmpdir(), "nixamp-compression-")); + +test("a policy store keeps each channel's settings by port, checks what it reads back, and refuses a stale version", () => { + const dir = scratch(); + try { + const store = new PolicyStore(dir, 4321); + assert.equal(store.get("main").losslessCompression.mode, "off", "off until somebody says otherwise"); + const first = store.set("main", { losslessCompression: { mode: "auto", zstdLevel: 3 } }); + assert.ok(first.ok); + assert.equal(first.ok && first.policy.version, 1); + const stale = store.set("main", { losslessCompression: { mode: "zstd" } }, 0); + assert.equal(stale.ok, false); + assert.equal(!stale.ok && stale.status, 412); + const bad = store.set("main", { losslessCompression: { mode: "lzma" } }, 1); + assert.equal(!bad.ok && bad.status, 400); + assert.equal(store.get("main").losslessCompression.mode, "auto", "a refused change changes nothing"); + store.setGlobal({ enabled: false }); + + const again = new PolicyStore(dir, 4321); + assert.equal(again.get("main").losslessCompression.zstdLevel, 3); + assert.equal(again.get("main").version, 1); + assert.equal(again.global.enabled, false); + assert.equal(new PolicyStore(dir, 9999).get("main").losslessCompression.mode, "off", "another port is another line-up"); + + // Edited by hand into something impossible: dropped, not trusted. + const raw = JSON.parse(readFileSync(join(dir, "compression.json"), "utf8")) as Record }>; + raw["4321"]!.channels["main"] = { losslessCompression: { mode: "auto", zstdLevel: 900 }, version: 7 }; + writeFileSync(join(dir, "compression.json"), JSON.stringify(raw)); + assert.equal(new PolicyStore(dir, 4321).get("main").losslessCompression.mode, "off"); + assert.equal(new PolicyStore(null, 1).set("x", { hlsPackaging: "fmp4" }).ok, true, "no directory: works, remembers nothing"); + } finally { + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("jobs run one at a time, share a running one, can be cancelled, and are forgotten in time", async () => { + const jobs = new AnalysisJobs({ concurrency: 1, ttlMs: 50, maxJobs: 3 }); + const fake = { boundary: "channel" } as unknown as Analysis; + let release: (() => void) | null = null; + const slow = (signal: AbortSignal): Promise => + new Promise((done, fail) => { + release = () => done(fake); + signal.addEventListener("abort", () => fail(new Error("cancelled"))); + }); + const a = jobs.start("k1", "one", "control", slow)!; + const same = jobs.start("k1", "one", "control", slow)!; + assert.equal(same.existing, true); + assert.equal(same.job.id, a.job.id); + const b = jobs.start("k2", "two", "control", slow)!; + assert.equal(b.job.status, "queued"); + assert.equal(jobs.start("k3", "three", "control", slow)!.job.status, "queued"); + assert.equal(jobs.start("k4", "four", "control", slow), null, "full"); + assert.equal(jobs.get(a.job.id, "somebody-else"), null, "not theirs to see"); + assert.equal(jobs.cancel(b.job.id, "control"), true); + assert.equal(b.job.status, "cancelled"); + (release as unknown as () => void)(); + await new Promise((done) => setTimeout(done, 10)); + assert.equal(a.job.status, "done"); + assert.equal(jobs.counts.running, 1, "the third moved up"); + await new Promise((done) => setTimeout(done, 80)); + assert.equal(jobs.get(a.job.id, "control"), null, "forgotten after the TTL"); +}); + +/** Read a whole envelope file back through the decoder. */ +async function unpack(path: string): Promise<{ bytes: Buffer; frames: number }> { + const pieces: Buffer[] = []; + const decoder = new RelayDecoder({ modes: new Set(["stored", "zstd", "ts-zstd"] as const), onBytes: (b) => { pieces.push(b); } }); + await decoder.feed(readFileSync(path)); + await decoder.end(); + return { bytes: Buffer.concat(pieces), frames: decoder.frames }; +} + +test("a static representation is built once, published whole, restores the file exactly, and is dropped when the file changes", async () => { + const dir = scratch(); + try { + const file = join(dir, "film.ts"); + const content = Buffer.concat([Buffer.from("header ".repeat(5000)), randomBytes(50_000), Buffer.from("trailer ".repeat(5000))]); + writeFileSync(file, content); + const cache = new StaticCache(join(dir, "cache")); + const policy = { ...DEFAULT_LOSSLESS, mode: "zstd" as const, maxBlockBytes: 16 * 1024 }; + const variant = variantOf(policy); + assert.equal(cache.lookup(file, variant), null); + const [one, two] = await Promise.all([cache.prepare(file, policy, variant), cache.prepare(file, policy, variant)]); + assert.ok(one.ok && two.ok); + assert.equal(one.ok && one.entry.blocks, Math.ceil(content.length / (16 * 1024))); + assert.ok(one.ok && one.entry.compressedBlocks > 0 && one.entry.compressedBlocks < one.entry.blocks, "text squeezed, noise stored"); + assert.equal(one.ok && one.entry.bytes, one.ok ? readFileSync(one.path).length : -1, "the entry knows its own size"); + const back = await unpack(one.ok ? one.path : ""); + assert.ok(back.bytes.equals(content), "byte for byte"); + assert.ok(cache.lookup(file, variant), "found next time"); + assert.equal(cache.lookup(file, "other-variant"), null, "another variant is another entry"); + assert.ok(cache.totalBytes > 0); + + // The file changes: the entry no longer describes it and goes. + writeFileSync(file, Buffer.concat([content, Buffer.from("!")])); + assert.equal(cache.lookup(file, variant), null); + assert.equal(cache.entries().length, 0); + + // A tiny budget: the second entry pushes the first out. + const small = new StaticCache(join(dir, "small"), { maxBytes: 70_000 }); + const other = join(dir, "other.bin"); + writeFileSync(other, randomBytes(40_000)); + utimesSync(other, new Date(1_700_000_000_000), new Date(1_700_000_000_000)); + assert.ok((await small.prepare(file, policy, variant)).ok); + assert.ok((await small.prepare(other, policy, variant)).ok); + assert.equal(small.entries().length, 1, "only the newest fits"); + assert.equal(small.lookup(other, variant)?.entry.file, other); + assert.equal((await small.prepare(join(dir, "missing"), policy, variant)).ok, false); + } finally { + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("fMP4 packaging names its init segment per run, keys the EXT-X-MAP line, and types segments by name", () => { + const args = packagerArgs("/tmp/x", "fmp4", "abcd1234"); + assert.equal(args[args.indexOf("-hls_segment_type") + 1], "fmp4"); + assert.equal(args[args.indexOf("-hls_fmp4_init_filename") + 1], "init-abcd1234.mp4"); + assert.equal(args[args.indexOf("-hls_segment_filename") + 1], "/tmp/x/seg%05d.m4s"); + assert.deepEqual(args.slice(args.indexOf("-c"), args.indexOf("-c") + 2), ["-c", "copy"], "still never re-encodes"); + assert.ok(!packagerArgs("/tmp/x").includes("-hls_fmp4_init_filename"), "TS is as it was"); + + assert.equal(segmentName("seg00003.m4s"), "seg00003.m4s"); + assert.equal(segmentName("init-abcd1234.mp4"), "init-abcd1234.mp4"); + assert.equal(segmentName("init.mp4"), "", "an init without a run token is not one of ours"); + assert.equal(segmentName("init-abcd1234.mp4/../x"), ""); + assert.equal(segmentType("seg00003.m4s"), "video/iso.segment"); + assert.equal(segmentType("init-abcd1234.mp4"), "video/mp4"); + assert.equal(segmentType("seg00003.ts"), "video/mp2t"); + + const playlist = [ + "#EXTM3U", + "#EXT-X-VERSION:7", + "#EXT-X-TARGETDURATION:2", + "#EXT-X-INDEPENDENT-SEGMENTS", + '#EXT-X-MAP:URI="init-abcd1234.mp4"', + "#EXTINF:2.000000,", + "seg00012.m4s", + "#EXTINF:6.500000,", + "seg00013.m4s", + "", + ].join("\n"); + const keyed = withKey(playlist, "k/1"); + assert.ok(keyed.includes('#EXT-X-MAP:URI="init-abcd1234.mp4?k=k%2F1"'), keyed); + assert.ok(keyed.includes("seg00013.m4s?k=k%2F1")); + const report = playlistReport(playlist, "fmp4"); + assert.deepEqual(report, { packaging: "fmp4", targetSeconds: 2, independent: true, longestSegmentSeconds: 6.5, segments: 2, initialised: true }); +}); + +test("the service answers a relay only for a channel that is on, with compression on, to a receiver that can decode it", () => { + const channels = new Channels({ ffmpeg: ["ffmpeg"] }); + const service = new CompressionService({ channels, stateDir: null, port: 1 }); + const listener = { write: () => true, end: () => undefined }; + const zstd = new Set(["zstd"] as const); + let answer = service.relay("nope", listener, zstd); + assert.equal(!answer.ok && answer.status, 404); + + const channel = channels.attach("live", "a device", "mp3", "http")!; + answer = service.relay("live", listener, zstd); + assert.equal(!answer.ok && answer.code, "COMPRESSION_OFF", "off by default"); + assert.equal(service.status("live").effective.reason, null); + + service.set("live", { losslessCompression: { mode: "auto", boundary: "source" } }); + answer = service.relay("live", listener, zstd); + assert.equal(!answer.ok && answer.code, "SOURCE_BOUNDARY_UNAVAILABLE"); + assert.ok(service.status("live").effective.reason?.includes("original source bytes are unavailable")); + + service.set("live", { losslessCompression: { boundary: "channel" } }); + answer = service.relay("live", listener, new Set(["stored"] as const)); + assert.equal(!answer.ok && answer.status, 406, "a receiver that decodes nothing gets nothing"); + + service.setGlobal({ enabled: false }); + answer = service.relay("live", listener, zstd); + assert.equal(!answer.ok && answer.code, "COMPRESSION_OFF"); + assert.ok(service.status("live").effective.reason?.includes("whole server")); + service.setGlobal({ enabled: true }); + + answer = service.relay("live", listener, zstd); + assert.ok(answer.ok); + assert.deepEqual(answer.ok && answer.codecs, ["stored", "zstd"], "ts-zstd only when the policy asks for it"); + const status = service.status("live"); + assert.equal(status.relay?.sessions, 1); + assert.equal(status.metrics?.generation, status.relay?.generation); + assert.equal(channel.listeners.size, 1, "one encoder is one listener on the channel, however many receivers"); + const second = service.relay("live", { write: () => true, end: () => undefined }, zstd); + assert.ok(second.ok && second.generation === (answer.ok ? answer.generation : -1), "the second receiver joined the same encoder"); + assert.equal(channel.listeners.size, 1); + + // The static pseudo-channel governs file representations. + assert.equal(service.representation("/nonexistent").state, "off", "no cache directory"); + assert.equal(service.status(STATIC_CHANNEL).configured.losslessCompression.mode, "off"); + assert.equal(service.packagingOf("live"), "mpegts"); + service.setGlobal({ hlsPackaging: "fmp4" }); + assert.equal(service.packagingOf("never-configured"), "fmp4", "the server's default for a channel with no policy"); + assert.equal(service.packagingOf("live"), "mpegts", "a channel with its own policy keeps its own packaging"); + service.stopAll(); + channels.stopAll(); +}); diff --git a/web/package.json b/web/package.json index 52e6326..f504b15 100644 --- a/web/package.json +++ b/web/package.json @@ -1,6 +1,6 @@ { "name": "@nixamp/web", - "version": "0.15.1", + "version": "0.17.0", "private": true, "description": "The nixamp PWA: a browser player, and a remote for nixamp serve.", "license": "MIT", From a281d472f5d1d3faeca2cfcf4813b0ca9aa94fc6 Mon Sep 17 00:00:00 2001 From: Anthony Ettinger Date: Fri, 11 Sep 2026 21:15:28 +0000 Subject: [PATCH 2/2] PRD 0001: point implementation at #124 Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01MxNif5tsYq4LczgG7aE8Jp --- ...s-stream-compression-and-efficient-hls-delivery-to-nixamp.md | 2 +- prd/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/prd/0001-add-lossless-stream-compression-and-efficient-hls-delivery-to-nixamp.md b/prd/0001-add-lossless-stream-compression-and-efficient-hls-delivery-to-nixamp.md index 6a4b69e..c2adc04 100644 --- a/prd/0001-add-lossless-stream-compression-and-efficient-hls-delivery-to-nixamp.md +++ b/prd/0001-add-lossless-stream-compression-and-efficient-hls-delivery-to-nixamp.md @@ -10,7 +10,7 @@ repo: profullstack/nixamp created: "2026-09-11" updated: "2026-09-11" discussion: "https://www.reddit.com/r/compression/comments/1wdc46u/comment/p97uoz0/" -implementation: +implementation: "https://github.com/profullstack/nixamp/pull/124" tags: [nixamp, streaming, compression, lossless, mpegts, hls, performance] supersedes: superseded-by: diff --git a/prd/README.md b/prd/README.md index feee7fd..123ae62 100644 --- a/prd/README.md +++ b/prd/README.md @@ -1,6 +1,6 @@ # LogicSRC PRDs -Numbered [OpenPRD](https://github.com/profullstack/logicsrc/blob/master/docs/openprd.md) product requirements documents for this repo. One file +Numbered [OpenPRD](../docs/openprd.md) product requirements documents for this repo. One file per PRD at `prd/-.md`, four-digit ids, no gaps. `0000-template.md` is the copy-paste starting point.