0.17.0: a channel relayed to another nixamp, losslessly compressed (PRD 0001) - #124
Merged
Conversation
…RD 0001)
One nixamp can now carry another's channel with fewer bytes on the wire
and every byte restored, through a negotiated envelope on a path of its
own. Nothing about ordinary playback changes, and all of it is off until
a channel's policy says otherwise.
The envelope (application/vnd.nixamp.stream, docs/stream-compression.md):
a 16-byte stream header naming the boundary and generation, then 48-byte
frames each carrying the mode, sequence, lengths and SHA-256 of the bytes
they stand for, ending in a marker with the whole generation's total and
digest. Every field is checked against negotiated limits before a byte is
allocated; a stream that stops without its marker is reported cut off.
Compression is Zstandard from node:zlib (Bun 1.4 and Node 24 both ship
it), off the event loop on the runtime's pool behind a bounded queue with
deadlines. One encoder per channel however many receivers; a late joiner
is sent the channel's opening bytes for itself and then the shared blocks.
A block is flushed when full or after 100 ms, never waiting for a packet
boundary; it is sent compressed only when that saves 3% and 512 bytes,
stored otherwise, and `auto` stops trying after eight stored blocks in a
row until a cooldown. A compressor that falls behind ends its receivers;
a receiver that stops draining is cut off alone. Ordinary listeners now
also have a ceiling on unsent bytes, where before a stalled socket's
buffer grew until the channel ended.
Per channel: GET/PATCH /api/channels/:id/compression (conditional on the
version you saw), POST .../compression/analyses for a bounded, deduped,
cancellable analysis job, GET/POST/DELETE .../relay to serve, bring in,
or stop a relay. Server-wide: GET /api/compression and PATCH {enabled}
as the kill switch. A library file gets a representation at
/api/media/:n/relay, built once, published by rename, checked against
the file on every lookup, under a byte budget; the original keeps
answering ranges and the representation refuses them.
`nixamp compression analyze|status|set|off|on|pull|fetch` speak the same
routes; `analyze FILE` runs here with no server, reporting the container
as told by the bytes, the transport-stream layout and null share, the
bitrate the PCR implies, and every codec's complete wire size with a
checked round trip.
HLS can be packaged as fragmented MP4, opt-in: the same boxes copied
into .m4s files with an init segment named per packager run so an old
init cannot be paired with new media, the key on the EXT-X-MAP line, and
the playlist's real segment lengths reported rather than the target.
The experimental ts-zstd transform groups 188-byte packet headers apart
from payloads in front of Zstandard, reversibly, incomplete tails and
malformed adaptation fields included; it is offered only when a policy
asks and a receiver can undo it, and chosen only when smaller.
Not here yet, and answered honestly when asked for: a live source at the
source boundary (SOURCE_BOUNDARY_UNAVAILABLE), PWA and desktop controls,
MCP tools, a lower-bitrate quality profile, birnpack, and the real-feed,
client-matrix and soak measurements that gate any change of default.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MxNif5tsYq4LczgG7aE8Jp
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MxNif5tsYq4LczgG7aE8Jp
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MxNif5tsYq4LczgG7aE8Jp
Two fragmented-MP4 packagings met: the relay's, with a Packaging type and a per-run token on the init segment, and the transport-stream work's, which asks for fMP4 for an H.265 channel. The relay's shape stays and the H.265 rule folds into it: playlist(id, fmp4) picks fmp4 when asked, else the compression policy's packaging. One segmentType, the finer one. The transport tests learn the tokened init name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MEguYxw44VkpjFrxxGrNGx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the P0 baseline of OpenPRD 0001, Add lossless stream compression and efficient HLS delivery to NixAmp, and commits the PRD itself as the first entry of a new
prd/collection (logicsrc prd validate --strictpasses).What this adds
application/vnd.nixamp.stream, specified byte by byte with test vectors indocs/stream-compression.md): a 16-byte stream header naming boundary and generation, 48-byte frames each with mode, sequence, lengths and SHA-256, and an end marker carrying the generation's total and digest. Every field is validated against negotiated limits before allocation; a stream without its marker is reported as cut off, never as complete.node:zlib(Bun 1.4 and Node 24 both ship it): no native dependency, async on the runtime's thread pool, behind a bounded pool with deadlines and a decode cap that refuses bombs.autobacks off after eight stored blocks in a row and resamples after a cooldown. A compressor that falls behind ends its receivers; a slow receiver is cut off alone. Ordinary listeners now also have a ceiling on unsent bytes (a stalled socket's buffer used to grow until the channel ended).GET/PATCH /api/channels/:id/compression(conditional onIf-Match),POST …/compression/analyses(bounded, deduplicated, cancellable jobs;GET/DELETE /api/compression/analyses/:job),GET/POST/DELETE /api/channels/:id/relay(serve, bring in, stop),GET /api/compressionandPATCH {enabled}as the server-wide kill switch,GET /api/media/:n/relayfor a library file's representation (built once, published by rename, checked against the file on every lookup, byte-budgeted; the original keeps answering ranges, the representation refuses them).nixamp compression analyze|status|set|off|on|pull|fetchover the same routes;analyze FILEneeds no server and reports the container by its bytes, TS packet layout and null share, PCR bitrate, and every codec's complete wire size with a checked round trip..m4swith an init segment named per packager run, the key on theEXT-X-MAPline, and the playlist's real segment lengths reported rather than the target.ts-zstd, experimental and flagged: 188-byte packet headers grouped apart from payloads in front of Zstandard, exactly reversible (ragged edges, malformed adaptation lengths, null packets kept whole); offered only when the policy asks and the receiver can undo it, chosen only when smaller.Everything is off by default. No ordinary playback URL changes.
Not in this PR (PRD P1/P2, or gated on measurements)
sourceboundary: needs a tee in front of ffmpeg; asking for it answersSOURCE_BOUNDARY_UNAVAILABLE.qualityProfile; the API refuses anything butsourcerather than re-encoding quietly.Tests
38 new tests: envelope vectors and every refusal code, codec round trips and the bomb guard, the pool, the block builder, the TS transform, the analyzer, relay encoder/decoder end to end with corruption of every kind, policy store, jobs, static cache, fMP4 packaging with a real ffmpeg, and route integration including two servers relaying a channel and a library file served as a verified representation. Full suite: 502 pass; the one fail/error is
web/testfailing to load@profullstack/playerin this worktree, unchanged frommain.A padded testsrc transport stream (84% null packets) measures an 89% saving at the source boundary; an unpadded feed will measure close to zero, which is the honest answer the tool gives.
Note
Conflicts with #123 on the three version lines and two eight-line regions of
src/server.ts; whichever merges second rebases.🤖 Generated with Claude Code
https://claude.ai/code/session_01MxNif5tsYq4LczgG7aE8Jp